summaryrefslogtreecommitdiffstats
path: root/scripts/lib/resulttool
Commit message (Collapse)AuthorAgeFilesLines
* scripts/resulttool: group all regressions in regression reportAlexis Lothoré2023-11-051-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit c304fcbe0588b1078373558c2ddf36064bcdf214 introduced a grouping when listing regressions. This grouping has been added only for ptests. It has been observed that any other kind of tests could benefit from it. For example, current regression reports can show the following: 1 regression(s) for oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_help oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED -> SKIPPED [...] This output is not so useful in its current state and should be grouped per test type too. Enable grouping for all kind of tests, to make it llok like the following in reports: 5 regression(s) for oescripts oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED -> SKIPPED (From OE-Core rev: 982798ef96e3a32bf15341bdd3bb7c4356709412) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: make additional info more compactAlexis Lothoré2023-10-231-4/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since "matched" and "improved" tests are not as important as regressions, reduce the place they take in the regression report: - merge "matched" and "improved" tests, while removing the label - add a single line of additional info per pair Those changes make the "Matches and improvements" look like the following sample: oeselftest_almalinux-9.2_qemux86-64_20230910083156 oeselftest_almalinux-8.8_qemux86-64_20231018010951 -> +7 test(s) present oeselftest_almalinux-9.2_qemux86-64_20230911010538 oeselftest_debian-11_qemux86-64_20231017150459 oeselftest_debian-11_qemux86-64_20230910012927 oeselftest_debian-11_qemux86-64_20231017151319 -> +7 test(s) present [...] (From OE-Core rev: 6de4426d9a7da67deed7d3a3918892fb56238ff3) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: rearrange regressions report orderAlexis Lothoré2023-10-231-2/+6
| | | | | | | | | | | | | | Regressions reports currently reports matching pairs and improved pairs first, then regressions. Change order to print regressions first, which is the most valuable info in the report, and then print improvements and matches at the bottom. (From OE-Core rev: 599267467430e70fa4dc8ba6b2a8b126bf6da359) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: limit the number of changes displayed per testAlexis Lothoré2023-10-231-4/+21
| | | | | | | | | | | | | | | | | | | | | | | | Most of the changes list generated in regression reports fall in one of the two following categories: - there is only a few (<10) changes listed and the info is valuable/relevant - the list is huge (> 100 ? 1000 ?) and basically tells us that the whole tests category suffers the same status (test missing, test failing, test skipped, etc) Prevent those huge, worthless lists by limiting the output for each test group: - current default limit is arbitrarily set to 50 - limit can still be overriden with a new "-l"/"--limit" flag, either with custom value, or with 0 to print the whole lists of changes - limit is applied per test family: currently it distinguishes only types of ptests, but it can be adapted to other kind of tests (From OE-Core rev: cec118406f3ad81cb4709f6e6ae1cef65799658e) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Avoid divide by zeroRichard Purdie2023-08-151-1/+4
| | | | | | | | Avoid a divide by zero traceback if unfortunate test counts are encountered. (From OE-Core rev: c5aeea53dfacb53dedb8445cb3523dc3a8cb6dca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: define custom string for "not found" test resultsAlexis Lothoré2023-08-041-0/+1
| | | | | | | | | | | | | | | | | | | Whenever a test result is present in base test result but absent from target test results, we have an entry in regression report looking like the following one: ptestresult.apr.testfile: PASSED -> None The "None" status may be a bit confusing, so replace it with a custom string which really states what "None" means in this context: ptestresult.apr.testfile: PASSED -> No matching test result (From OE-Core rev: b5f6da6e13f3484f51dfa07f6b3500aea7a21825) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: allow to replace test raw status with custom stringAlexis Lothoré2023-08-041-1/+7
| | | | | | | | | | | | | Add a STATUS_STRINGS dictionnary matching raw statuses to custom strings. Whenever a regression must be reported, raw status is searched in the custom statuses dict (key search is case insensitive). If no custom string is found, raw status is kept and used in regression report (From OE-Core rev: 9d22bfc9d0c4092dba1af0ee11a4c51b7b270786) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: add mention about new detected testsAlexis Lothoré2023-07-251-2/+14
| | | | | | | | | | | | | | | | Some regression reports show a lot of "PASSED->None" transitions. When such big lot of identical transitions are observed, it could be that tests are now failing, but it could also be that some tests has been renamed. To detect such case, add a log in regression report to report the number of new tests (i.e: tests that are present in target results but not in base result). This new log also allows to know about newly added tests bases (From OE-Core rev: 01b5cefd07e01c7407bc663842b8a8d502358a6d) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: allow index generation despite corrupt jsonMichael Halstead2023-07-191-1/+5
| | | | | | | | | | | non-release indexes will continue to generate when test output is corrupted. (From OE-Core rev: 1a9157684a6bff8406c9bb470cb2e16ee006bbe9) Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improve overlapping ptest result reportingRichard Purdie2023-03-161-5/+3
| | | | | | | | | | With the move to a ptest per image, the multiple ptest log scenario is much more likely. Tweak the handling to only warn if there are overlapping files. (From OE-Core rev: 8391f1668fcbe932ba846299b0ec22de09e06dd4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: add log --list-ptestRoss Burton2023-03-141-0/+5
| | | | | | | | | | Add a convenience argument to the log subcommand to list all of the ptest logs in a testresults file. (From OE-Core rev: cd2d7adf02005d46f19c93a40db1e10ce01ac261) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/resulttool: fix typo breaking resulttool log --ptestRoss Burton2023-03-141-1/+1
| | | | | | | | | | ptestresult_get_log() looked for a key called 'ptestresuls.sections', which should be 'ptestresult.sections' (From OE-Core rev: 7c8c9f7283e54bf8b1521fbaad7dceb66a8fcdbb) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: do not count newly passing tests as regressionsAlexis Lothoré2023-03-121-5/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | resulttool regression module simply compare a base test status to a target test result status. This approach raises many false positives since all XXX -> PASS transitions (XXX being any status different from PASS) are flagged as regression. - Do not list XXX -> PASS transitions in regression report, instead count them and print a summary of "newly passing tests" - If an inspected pair has only "newly passing tests", do not print detailed list and print it as "Improvement" instead of "Regression" Updated output example looks like the following: [...] Improvement: oeselftest_fedora-37_qemux86-64_20230127010225 oeselftest_ubuntu-22.04_qemux86-64_20230226120516 (+1 test(s) passing) [...] Match: oeselftest_almalinux-8.7_qemuarm64_20230127015830 oeselftest_almalinux-8.7_qemuarm64_20230227015258 [...] Regression: oeselftest_almalinux-9.1_qemumips_20230127000217 oeselftest_opensuseleap-15.4_qemumips_20230226130046 ptestresult.glibc-user.debug/tst-read-chk-cancel: PASS -> None ptestresult.glibc-user.nptl/tst-mutexpi4: PASS -> FAIL ptestresult.glibc-user.nptl/tst-mutexpi5a: PASS -> FAIL Additionally, 44 previously failing test(s) is/are now passing (From OE-Core rev: c335f96f687c73fde443ac330ca3e17113794d9e) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: fix ptests results containing a non reproducible pathAlexis Lothoré2023-03-121-0/+6
| | | | | | | | | | | | | | | | | | | | | | Some ptests results may be wrongly sampled, resulting in some part of the error being propagated in test name. For example: "ptestresult.binutils-ld.in testcase /home/pokybuild/yocto-worker/qemumips/build/build-st-1666126/tmp/work/mips32r2-poky-linux/binutils-cross-testsuite/2.40-r0/git/ld/testsuite/ld-ctf/ctf.exp" "ptestresult.gcc.Couldn't create remote directory /tmp/runtest.455781 on ssh" "ptestresult.gcc-libstdc++-v3.Couldn't create remote directory /tmp/runtest.3814266 on target" While the root errors must be fixed in corresponding ptests packages for those tests, the test results history must still be usable even with this issue Add new filters to detect if temporary test directories (build-st-*, /tmp/runtime.*) are present in name. If so, truncate test name. As a side effect, it will aggregate multiple failing errors into one, but the regression will still be raised (From OE-Core rev: 601eecfddd26bfe2954835a73ed1116bb520235f) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: call fixup_ptest_names in regression_commonAlexis Lothoré2023-03-121-21/+20
| | | | | | | | | | | | | ptests names not only need to be fixed for regression based on git testresults but also for testsresults provided "manually" Move ptests naming fixup in regression_common to share the fixup between both regression use cases (From OE-Core rev: f772ccd108dc3d618db9d479d672c0f3edd203ca) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression: Improve matching of poor ptest test names4.2_M3Richard Purdie2023-02-261-0/+21
| | | | | | | | | | | | | | Some test case naming is poor and contains random strings, particularly lttng/babeltrace but also curl. Truncating the test names works since they contain file and line number identifiers which allows us to match them without the random components, or in the case or curl, test IDs. Going forward we may be able to improve the test names but this tweak allows historical test results to work in reports. (From OE-Core rev: 541a2e2683531355e678fd93524a0c4a8c43a8ff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression: Ensure LTP results are only compared against other ↵Richard Purdie2023-02-261-1/+6
| | | | | | | | | | | LTP runs If a test result contains LTP test results, it should only be compared with other runs containing LTP test results. (From OE-Core rev: 4dbbf2f4a85620a08dc2fa65095dc17fe6c530f8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/regression: add metadata filtering for oeselftestAlexis Lothoré2023-02-261-0/+162
| | | | | | | | | | | | | | | | | | | | | When generating regression reports, many false positive can be observed since some tests results are compared while the corresponding tests sets are not the same, as it can be seen for example for oeselftest tests (oeselftest is run multiple time but with different parameters, resulting in different tests sets) Add a filtering mechanism in resulttool regression module to enable a better matching between tests. The METADATA_MATCH_TABLE defines that when the TEST_TYPE is "oeselftest", then resulttool should filter pairs based on OESELFTEST_METADATA appended to test configuration. If metadata is absent from test results, in order to keep compatibility with older results, add a "guessing" mechanism to generate the missing OESELFTEST_METADATA. The guessed data is tightly coupled to the autobuilder configuration, where all oe-selftest executions are described (From OE-Core rev: 94ab7c2b892bf292dd86619ca9c63ddd7bf53f3c) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/regression: remove unused importAlexis Lothoré2023-02-261-1/+0
| | | | | | | (From OE-Core rev: 4d6a74fcee1479562bf66360f20d528107c70887) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/log: Add ability to dump ltp logs as well as ptestRichard Purdie2020-05-272-15/+28
| | | | | | | | | Currently only ptest logs are accessible with the log command, this adds support so the ltp logs can be extracted too. (From OE-Core rev: 64a2121a875ce128959ee0a62e310d5f91f87b0d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Remove leftover debuggingRichard Purdie2020-05-271-1/+0
| | | | | | | | | I've long since wondered why there was some odd output in result reports, remove the leftover debug which was causing it. (From OE-Core rev: 66e96bf70753933714ff8edcc13a1f35a052656f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: Fix unicode error handlingRichard Purdie2020-02-151-4/+1
| | | | | | | | | | | | | | | | | | | | | | This error handling didn't work as expected since upon failure it would inject bytestreams back into the code leading to tracebacks. Instead, ignore the decode errors. Fixes: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 78, in <module> sys.exit(main()) File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 72, in main ret = args.func(args, logger) File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/store.py", line 70, in store resultutils.save_resultsdata(results, tempdir, ptestlogs=True) File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/resultutils.py", line 178, in save_resultsdata f.write(sectionlog) TypeError: write() argument must be str, not bytes (From OE-Core rev: 3aa0d0fd0bbf8c9c13719251a2fd1e85a22c7336) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/lib/resulttool/report: Enable report selected test case resultYeoh Ee Peng2020-02-131-5/+28
| | | | | | | | | | | | Enable reporting selected test case result given the user provided the selected test case id. If both test result id and test case id were provided, report the selected test case result from the selected test result id. (From OE-Core rev: 7161310ce32d6e0c397d0132808d556bdc80d183) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store.py: Enable add extra test environment dataYeoh Ee Peng2019-11-141-1/+4
| | | | | | | | | | | | | | | Enable the option to add extra test environment data to the configuration of each test result (as optional). Example of optional test environment data include: - custom packages included for runtime test - detail machine specification used as target - detail host environment used for bitbake (From OE-Core rev: 3cc887a72c5726024c3699e6a3deb72fad4992a9) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/report: Add total statistic to test result.Yeoh Ee Peng2019-11-142-1/+7
| | | | | | | | | Add total passed, failed, and skipped statistic to test result. (From OE-Core rev: 7ad74f99b6105f23bcf930712627aac279c6af10) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/report: Enable output raw test resultsYeoh Ee Peng2019-11-141-2/+16
| | | | | | | | | | | | | In case of debugging, report user need to acccess the raw test result. Instead of going back to source file/directory/URL to manually pull out the raw result, provide alternative way to let report showing raw test results by providing the result id (optional). (From OE-Core rev: 8b2485bc0d0fa2f72b37e8bafbfa663d456ea98c) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/report: Enable report to use regression_mapYeoh Ee Peng2019-11-142-7/+13
| | | | | | | | | | | | | | | | | | | | By default, report will use the store_map to generate the key to reference each result set. In some situation when using store_map with multiple set of tests sharing similar test configurations, the report will only showing partial result set for results that having identical result_id (use of multiconfig to run tests where it generate identical result_id). Enable report to have the option to use the regression_map (optional) instead of the default store_map, where it will take larger set of configurations to generate the key to reference each result set, this will prevent the report from only showing partial result set. (From OE-Core rev: 8577abce6feb1f529537da554d39e86a5b05ca2d) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/case.py: Encode binary data of logNathan Rossi2019-09-301-1/+5
| | | | | | | | | | | | | | | Do not decode the log content into a string only to re-encode it as binary data again. Some logs might un-intentionally contain bytes that do not decode as utf-8, as such preserve the log file content as it was on disk. Handle the decoding on the resulttool side, but also handle the failure to decode the data. (From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpersNathan Rossi2019-09-162-28/+60
| | | | | | | | | | | | | | | | Add the OEPTestResultTestCase class as a mix-in class to provide helper functions for interacting with ptestresults within the extraresults object generated by the test case. This class also provides default compression of log text and log files. Also add support to resulttool for decoding/decompressing log files embedded in the test results. (From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Handle multiple series containing ptestresultsNathan Rossi2019-09-161-12/+32
| | | | | | | | | | | | Handle multiple results series having ptestresults content. The contents are merged on a per-result basis where duplicates are ignored (with a warning message printed). The 'ptestresults.sections' collection is also merged on a per-suite basis. (From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add reproducible log extractionJoshua Watt2019-09-061-3/+30
| | | | | | | | | | | | | | | | | Adds an argument to the log subcommand to extract the raw logs from the reproducible selftest. To prevent ambiguity, the "--raw" argument has been renamed "--raw-ptest", although the old "--raw" argument is kept around for compatibility. [YOCTO #13324] (From OE-Core rev: 7a4ebb361ff1efc22e7dafadfa60c98bc8a79ed4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/lib/resulttool/report.py: Add more result typesNathan Rossi2019-08-281-3/+3
| | | | | | | | | | | Add additional result types into the dictionary to handle dejagnu style test results. These include PASS, FAIL, XPASS, XFAIL, UNSUPPORTED, UNTESTED, UNRESOLVED and ERROR. (From OE-Core rev: 684fc36402a23760b203f4761f284043031c799c) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Prevent multiple results for the same testJon Mason2019-08-211-0/+14
| | | | | | | | | | | | | Currently, if a test occurs multiple times over different series, the code will sum these. This can lead to confusion over the desired results. Change the code to report the redundant tests and skip adding an additional values. (From OE-Core rev: caeaa0648dff64c56a33f52e45e82bfab6719e3e) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/merge: Enable control TESTSERIES and extra configurationsYeoh Ee Peng2019-06-121-5/+15
| | | | | | | | | | | | | Current QA team need to merge test result files from multiple sources. Adding TESTSERIES configuration too early will have negative implication to report and regression. Enable control to add TESTSERIES when needed. Also enable adding EXECUTED_BY configuration when needed. (From OE-Core rev: 651d8d371e78e77599d56681228d5782664f7743) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store: Enable add EXECUTED_BY config to resultsYeoh Ee Peng2019-06-121-2/+7
| | | | | | | | | | | Current results stored does not have information needed to trace who executed the tests. Enable store to add EXECUTED_BY configuration to results file in order to track who executed the tests. (From OE-Core rev: dca2a57d54163a2e63b06e2f140fea3bd49cef0d) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: Enable add extra configurations to resultsYeoh Ee Peng2019-06-121-7/+12
| | | | | | | | | | | | Current resultutils library always add "TESTSERIES" configuration to results. Enhance this to allow control of adding "TESTSERIES" configuration as well as allow adding extra configurations when needed. (From OE-Core rev: 443c0acc14ef2451b10878fc83dd11b46805daf0) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Remove prints if no tests occurJon Mason2019-06-041-11/+3
| | | | | | | | | | Printing the lack of a test is not necessary (per feedback). Remove this from the template to quieten it. (From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable creation of test case configurationsangeeta jain2019-06-041-1/+30
| | | | | | | | | | | | Allow the creation of test case configuration file based on user inputs. Where this testcase configuration file will be used by the the manual execution to run selected test cases for a module rather than compulsory run all test cases in manual json file. (From OE-Core rev: 73d2a747c17779da0ca972da776b3cf02c2e1cbc) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: modify to be multi-machineJon Mason2019-06-042-67/+81
| | | | | | | | | | | | Currently, the code will sum all of the different machine results into a single report of the tests results. This can lead to confusion as to which machine may be experiencing issues. Modify the code to store the results in a per machine basis and report them accordingly. (From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib+scripts: Convert to SPDX license headersRichard Purdie2019-05-097-56/+13
| | | | | | | | | | | | | | | | | | | | | | | This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add option to dump all ptest logsJoshua Watt2019-05-082-16/+54
| | | | | | | | | | | | | | | Adds an option to dump all the ptest logs to individual files in a specified directory. If multiple test runs are present, the '--prepend-run' argument will create separate directories for each test run under the target directory and put the logs there to prevent each test run from clobbering the others. [YOCTO #13331] (From OE-Core rev: f262b9e7482e38f04dfa162f0aaf65b17e86f8b2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: add LTP compliance sectionArmin Kuster2019-04-292-1/+58
| | | | | | | (From OE-Core rev: a680d7d15fafbecf4edce9a29cc4eda16c11fd94) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: add ltp test supportArmin Kuster2019-04-292-1/+59
| | | | | | | (From OE-Core rev: 966795aa2c6960aca11a04e87f415256faf26957) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable test case configuration optionsangeeta jain2019-04-261-2/+12
| | | | | | | | | | | | | | | | | | | | | | | | Current manualexecution required user to exceute all test cases defined inside a "modulename.json" file in oeqa/manual There are cases when all test cases all not required to run for a module on specific DUT. Enable manualexecution to have the optional feature where it will use pre-defined json format test case configuration file where user will be able to select test cases from the "modulename.json" instead of running all of them. This will help in reducing testing time and reporting unneccesary skip or failures. Example pre-defined json format test case configuration file (for build-applince): { "testcases" : [ "build-appliance.build-appliance.Create_core-image-sato-sdk_using_build_appliance", "build-appliance.build-appliance.Build_a_image_without_error_(added_recipe)" ] } (From OE-Core rev: 3950c28a34f94b5907d37b579bdaee5a59794652) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resultool/resultutils: Fix module import errorRichard Purdie2019-04-261-1/+1
| | | | | | | | | Fix AttributeError: module 'urllib' has no attribute 'request' when using remote http urls. (From OE-Core rev: 9acc47236e2a9ac7f97f80040f5e508f23f01611) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add log subcommandJoshua Watt2019-04-252-1/+57
| | | | | | | | | Adds a subcommand for dumping various logs from test results (From OE-Core rev: 454b8d2cdc6f79a51e610dae92c22352850c3f7c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Load results from URLJoshua Watt2019-04-255-13/+28
| | | | | | | | | | Adds support for resulttool to load JSON files directly from a http:// or https:// URL (From OE-Core rev: 235bcf0c504e7ea253ccfb28d600898117c64c1f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Refactor and remove duplicate codeYeoh Ee Peng2019-04-121-47/+40
| | | | | | | | | | | Remove duplicate codes. Replace unnecessary class variables with local variables. Rename variables and arguments with simple and standard name. (From OE-Core rev: 6bef61d36f3328fad003c0dc9c010d3f76ba96d8) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable creation of configuration option fileYeoh Ee Peng2019-04-101-1/+53
| | | | | | | | | | | | Allow the creation of configuration option file based on user inputs. Where this configuration option file will be used by the the manual execution to display options for configuration rather than user need to inputs configuration manually. (From OE-Core rev: 23c7a037a85a825aa088bfac3a144c14087a9f49) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable configuration options selectionYeoh Ee Peng2019-04-101-9/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current manualexecution required user to input configuration manually where there were inconsistent inputs and human typo issues. Enable manualexecution to have the optional feature where it will use pre-compiled configuration options file where user will be able to select configuration from the pre-compiled list instead of manual key-in the configuration. This will eliminate human error. Expect the pre-compiled configuration options file in json format below { "bsps-hw": { "IMAGE_BASENAME": { "1": "core-image-sato-sdk" }, "MACHINE": { "1": "beaglebone-yocto", "2": "edgerouter", "3": "mpc8315e-rdb", "4": "genericx86", "5": "genericx86-64" } }, "bsps-qemu": { "IMAGE_BASENAME": { "1": "core-image-sato-sdk" }, "MACHINE": { "1": "qemuarm", "2": "qemuarm64", "3": "qemumips", "4": "qemumips64", "5": "qemuppc", "6": "qemux86", "7": "qemux86-64" } } } (From OE-Core rev: 1af7e04223b415a64e100a9f91d60f5e9b0f789a) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-03 19:52:42 +0000