Mirror of git.yoctoproject.org/poky https://git.enea.com/cgit/linux/poky.git/atom?h=yocto-4.0.8 2019-11-14T13:20:59+00:00 2019-11-14T13:20:59+00:00 Yeoh Ee Peng ee.peng.yeoh@intel.com 2019-11-07T09:50:29+00:00 urn:sha1:610b02a2fbdf66488edb91963c08daf6f928a7f7 Add total passed, failed, and skipped statistic to test result. (From OE-Core rev: 7ad74f99b6105f23bcf930712627aac279c6af10) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> 2019-06-04T22:09:24+00:00 Jon Mason jdmason@kudzu.us 2019-06-03T23:19:49+00:00 urn:sha1:b6ca6ac564cb4b8c88d811690c2b7487b58f06b4 Printing the lack of a test is not necessary (per feedback). Remove this from the template to quieten it. (From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> 2019-06-04T08:09:42+00:00 Jon Mason jdmason@kudzu.us 2019-06-02T18:29:12+00:00 urn:sha1:cb153e8da8897080e128d7e29d6a1f48e1190fdf Currently, the code will sum all of the different machine results into a single report of the tests results. This can lead to confusion as to which machine may be experiencing issues. Modify the code to store the results in a per machine basis and report them accordingly. (From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> 2019-04-29T13:16:30+00:00 Armin Kuster akuster808@gmail.com 2019-04-22T12:32:41+00:00 urn:sha1:12d8f77a88598fdf95be6936e26b2f28394bf3be (From OE-Core rev: a680d7d15fafbecf4edce9a29cc4eda16c11fd94) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> 2019-04-29T13:16:30+00:00 Armin Kuster akuster808@gmail.com 2019-04-22T12:32:38+00:00 urn:sha1:9c820850369aa0d53298291d3c77526d1817becd (From OE-Core rev: 966795aa2c6960aca11a04e87f415256faf26957) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> 2019-02-27T18:04:14+00:00 Richard Purdie richard.purdie@linuxfoundation.org 2019-02-27T17:10:19+00:00 urn:sha1:2268bf470cb3fbb2c5d12718465b53c3fbbbb67b (From OE-Core rev: 3ec9a6288793f384bf108d7194cf50147485fa28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> 2019-02-21T12:34:00+00:00 Richard Purdie richard.purdie@linuxfoundation.org 2019-02-16T18:13:00+00:00 urn:sha1:47eb3d00e9d6a66aee1283dab29af8117a006d6d This is a combined patch of the various tweaks and improvements I made to resulttool: * Avoid subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. * Avoid python keywords as variable names * Simplify dict accesses using .get() * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) * Fix oe-git-archive to use the commit/branch data from the results file * Cleaned up the command option help to match other changes * Follow the model of git branch/tag processing used by oe-build-perf-report and use that to read the data using git show to avoid branch change * Add ptest summary to the report command * Update the tests to match the above changes (From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> 2019-02-21T12:34:00+00:00 Yeoh Ee Peng ee.peng.yeoh@intel.com 2019-02-14T05:50:37+00:00 urn:sha1:1fd5ebdb06224489ad056e261962e23ece36fc87 OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file. These scripts were developed as a test result tools to manage these testresults.json file. Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files. Using the "merge" operation, user can merge multiple testresults.json files to a target file. Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository. Using the "regression-file" operation, user can perform regression analysis on testresults.json files specified. Using the "regression-dir" and "regression-git" operations, user can perform regression analysis on directory and git accordingly. These resulttool operations expect the testresults.json file to use the json format below. { "<testresult_1>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, ... "<testresult_n>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, } To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To store test result from oeqa automated tests, execute the below $ resulttool store <source_dir> <git_branch> To merge multiple testresults.json files, execute the below $ resulttool merge <base_result_file> <target_result_file> To report test report, execute the below $ resulttool report <source_dir> To perform regression file analysis, execute the below $ resulttool regression-file <base_result_file> <target_result_file> To perform regression dir analysis, execute the below $ resulttool regression-dir <base_result_dir> <target_result_dir> To perform regression git analysis, execute the below $ resulttool regression-git <source_dir> <base_branch> <target_branch> [YOCTO# 13012] [YOCTO# 12654] (From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>