diff options
author | Alexis Lothoré <alexis.lothore@bootlin.com> | 2023-02-28 19:10:52 +0100 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2023-03-12 23:39:13 +0000 |
commit | 1eae53e277b60347753f298e1cf3376c52a981ca (patch) | |
tree | 7ece9187dd78b5041d91331bd7bf1675afe16b12 /scripts/yocto_testresults_query.py | |
parent | c31c140746c9ccf3183b8046a24ed040d5c94934 (diff) | |
download | poky-1eae53e277b60347753f298e1cf3376c52a981ca.tar.gz |
scripts/resulttool: do not count newly passing tests as regressions
resulttool regression module simply compare a base test status to a target test
result status. This approach raises many false positives since all XXX -> PASS
transitions (XXX being any status different from PASS) are flagged as
regression.
- Do not list XXX -> PASS transitions in regression report, instead count them
and print a summary of "newly passing tests"
- If an inspected pair has only "newly passing tests", do not print detailed
list and print it as "Improvement" instead of "Regression"
Updated output example looks like the following:
[...]
Improvement: oeselftest_fedora-37_qemux86-64_20230127010225
oeselftest_ubuntu-22.04_qemux86-64_20230226120516
(+1 test(s) passing)
[...]
Match: oeselftest_almalinux-8.7_qemuarm64_20230127015830
oeselftest_almalinux-8.7_qemuarm64_20230227015258
[...]
Regression: oeselftest_almalinux-9.1_qemumips_20230127000217
oeselftest_opensuseleap-15.4_qemumips_20230226130046
ptestresult.glibc-user.debug/tst-read-chk-cancel: PASS -> None
ptestresult.glibc-user.nptl/tst-mutexpi4: PASS -> FAIL
ptestresult.glibc-user.nptl/tst-mutexpi5a: PASS -> FAIL
Additionally, 44 previously failing test(s) is/are now passing
(From OE-Core rev: c335f96f687c73fde443ac330ca3e17113794d9e)
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'scripts/yocto_testresults_query.py')
0 files changed, 0 insertions, 0 deletions