linux/poky.git/scripts/lib/resulttool, branch mickledore

resulttool/report: Avoid divide by zero

2023-08-30T14:52:36+00:00

Avoid a divide by zero traceback if unfortunate test counts are encountered. (From OE-Core rev: a46782c9f54bcb0389ea1016e37b6939aae79bfb) Signed-off-by: Richard Purdie (cherry picked from commit c5aeea53dfacb53dedb8445cb3523dc3a8cb6dca) Signed-off-by: Steve Sakoman

scripts/resulttool: add mention about new detected tests

2023-08-15T16:18:49+00:00

Some regression reports show a lot of "PASSED->None" transitions. When such big lot of identical transitions are observed, it could be that tests are now failing, but it could also be that some tests has been renamed. To detect such case, add a log in regression report to report the number of new tests (i.e: tests that are present in target results but not in base result). This new log also allows to know about newly added tests bases (From OE-Core rev: 7e393ea6e46060daf3242f2f7721f7f968945122) Signed-off-by: Alexis Lothoré Signed-off-by: Alexandre Belloni (cherry picked from commit 01b5cefd07e01c7407bc663842b8a8d502358a6d) Signed-off-by: Steve Sakoman

resulttool/resultutils: allow index generation despite corrupt json

2023-08-15T16:18:49+00:00

non-release indexes will continue to generate when test output is corrupted. (From OE-Core rev: ea6c9e6a91df553e88e34a990ea15f80f469893e) Signed-off-by: Michael Halstead Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie (cherry picked from commit 1a9157684a6bff8406c9bb470cb2e16ee006bbe9) Signed-off-by: Steve Sakoman

resulttool: Improve overlapping ptest result reporting

2023-03-16T22:52:15+00:00

With the move to a ptest per image, the multiple ptest log scenario is much more likely. Tweak the handling to only warn if there are overlapping files. (From OE-Core rev: 8391f1668fcbe932ba846299b0ec22de09e06dd4) Signed-off-by: Richard Purdie

resulttool: add log --list-ptest

2023-03-14T17:13:11+00:00

Add a convenience argument to the log subcommand to list all of the ptest logs in a testresults file. (From OE-Core rev: cd2d7adf02005d46f19c93a40db1e10ce01ac261) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

lib/resulttool: fix typo breaking resulttool log --ptest

2023-03-14T17:13:11+00:00

ptestresult_get_log() looked for a key called 'ptestresuls.sections', which should be 'ptestresult.sections' (From OE-Core rev: 7c8c9f7283e54bf8b1521fbaad7dceb66a8fcdbb) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

scripts/resulttool: do not count newly passing tests as regressions

2023-03-12T23:39:13+00:00

resulttool regression module simply compare a base test status to a target test result status. This approach raises many false positives since all XXX -> PASS transitions (XXX being any status different from PASS) are flagged as regression. - Do not list XXX -> PASS transitions in regression report, instead count them and print a summary of "newly passing tests" - If an inspected pair has only "newly passing tests", do not print detailed list and print it as "Improvement" instead of "Regression" Updated output example looks like the following: [...] Improvement: oeselftest_fedora-37_qemux86-64_20230127010225 oeselftest_ubuntu-22.04_qemux86-64_20230226120516 (+1 test(s) passing) [...] Match: oeselftest_almalinux-8.7_qemuarm64_20230127015830 oeselftest_almalinux-8.7_qemuarm64_20230227015258 [...] Regression: oeselftest_almalinux-9.1_qemumips_20230127000217 oeselftest_opensuseleap-15.4_qemumips_20230226130046 ptestresult.glibc-user.debug/tst-read-chk-cancel: PASS -> None ptestresult.glibc-user.nptl/tst-mutexpi4: PASS -> FAIL ptestresult.glibc-user.nptl/tst-mutexpi5a: PASS -> FAIL Additionally, 44 previously failing test(s) is/are now passing (From OE-Core rev: c335f96f687c73fde443ac330ca3e17113794d9e) Signed-off-by: Alexis Lothoré Signed-off-by: Richard Purdie

scripts/resulttool: fix ptests results containing a non reproducible path

2023-03-12T23:39:13+00:00

Some ptests results may be wrongly sampled, resulting in some part of the error being propagated in test name. For example: "ptestresult.binutils-ld.in testcase /home/pokybuild/yocto-worker/qemumips/build/build-st-1666126/tmp/work/mips32r2-poky-linux/binutils-cross-testsuite/2.40-r0/git/ld/testsuite/ld-ctf/ctf.exp" "ptestresult.gcc.Couldn't create remote directory /tmp/runtest.455781 on ssh" "ptestresult.gcc-libstdc++-v3.Couldn't create remote directory /tmp/runtest.3814266 on target" While the root errors must be fixed in corresponding ptests packages for those tests, the test results history must still be usable even with this issue Add new filters to detect if temporary test directories (build-st-*, /tmp/runtime.*) are present in name. If so, truncate test name. As a side effect, it will aggregate multiple failing errors into one, but the regression will still be raised (From OE-Core rev: 601eecfddd26bfe2954835a73ed1116bb520235f) Signed-off-by: Alexis Lothoré Signed-off-by: Richard Purdie

scripts/resulttool: call fixup_ptest_names in regression_common

2023-03-12T23:39:12+00:00

ptests names not only need to be fixed for regression based on git testresults but also for testsresults provided "manually" Move ptests naming fixup in regression_common to share the fixup between both regression use cases (From OE-Core rev: f772ccd108dc3d618db9d479d672c0f3edd203ca) Signed-off-by: Alexis Lothoré Signed-off-by: Richard Purdie

resulttool/regression: Improve matching of poor ptest test names

2023-02-26T11:59:52+00:00

Some test case naming is poor and contains random strings, particularly lttng/babeltrace but also curl. Truncating the test names works since they contain file and line number identifiers which allows us to match them without the random components, or in the case or curl, test IDs. Going forward we may be able to improve the test names but this tweak allows historical test results to work in reports. (From OE-Core rev: 541a2e2683531355e678fd93524a0c4a8c43a8ff) Signed-off-by: Richard Purdie