diff options
author | Richard Purdie <richard.purdie@linuxfoundation.org> | 2019-02-16 18:13:00 +0000 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2019-02-21 12:34:00 +0000 |
commit | 47eb3d00e9d6a66aee1283dab29af8117a006d6d (patch) | |
tree | 9d574720d23da3edeab5648f05fb5d2ee95c9cd2 /scripts/resulttool | |
parent | beed7523b667affea71d37d88d2f5c19c935d159 (diff) | |
download | poky-47eb3d00e9d6a66aee1283dab29af8117a006d6d.tar.gz |
resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:
* Avoid subprocess.run() as its a python 3.6 feature and we
have autobuilder workers with 3.5.
* Avoid python keywords as variable names
* Simplify dict accesses using .get()
* Rename resultsutils -> resultutils to match the resultstool ->
resulttool rename
* Formalised the handling of "file_name" to "TESTSERIES" which the code
will now add into the json configuration data if its not present, based
on the directory name.
* When we don't have failed test cases, print something saying so
instead of an empty table
* Tweak the table headers in the report to be more readable (reference
"Test Series" instead if file_id and ID instead of results_id)
* Improve/simplify the max string length handling
* Merge the counts and percentage data into one table in the report
since printing two reports of the same data confuses the user
* Removed the confusing header in the regression report
* Show matches, then regressions, then unmatched runs in the regression
report, also remove chatting unneeded output
* Try harder to "pair" up matching configurations to reduce noise in
the regressions report
* Abstracted the "mapping" table concept used to pairing in the
regression code to general code in resultutils
* Created multiple mappings for results analysis, results storage and
'flattening' results data in a merge
* Simplify the merge command to take a source and a destination,
letting the destination be a directory or a file, removing the need for
an output directory parameter
* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
mappings
* Have the store command place the testresults files in a layout from
the mapping, making commits into the git repo for results storage more
useful for simple comparison purposes
* Set the oe-git-archive tag format appropriately for oeqa results
storage (and simplify the commit messages closer to their defaults)
* Fix oe-git-archive to use the commit/branch data from the results file
* Cleaned up the command option help to match other changes
* Follow the model of git branch/tag processing used by oe-build-perf-report
and use that to read the data using git show to avoid branch change
* Add ptest summary to the report command
* Update the tests to match the above changes
(From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'scripts/resulttool')
-rwxr-xr-x | scripts/resulttool | 7 |
1 files changed, 4 insertions, 3 deletions
diff --git a/scripts/resulttool b/scripts/resulttool index 13430e192a..5a89e1c9be 100755 --- a/scripts/resulttool +++ b/scripts/resulttool | |||
@@ -1,11 +1,12 @@ | |||
1 | #!/usr/bin/env python3 | 1 | #!/usr/bin/env python3 |
2 | # | 2 | # |
3 | # test results tool - tool for testresults.json (merge test results, regression analysis) | 3 | # test results tool - tool for manipulating OEQA test result json files |
4 | # (merge results, summarise results, regression analysis, generate manual test results file) | ||
4 | # | 5 | # |
5 | # To look for help information. | 6 | # To look for help information. |
6 | # $ resulttool | 7 | # $ resulttool |
7 | # | 8 | # |
8 | # To store test result from oeqa automated tests, execute the below | 9 | # To store test results from oeqa automated tests, execute the below |
9 | # $ resulttool store <source_dir> <git_branch> | 10 | # $ resulttool store <source_dir> <git_branch> |
10 | # | 11 | # |
11 | # To merge test results, execute the below | 12 | # To merge test results, execute the below |
@@ -58,7 +59,7 @@ def _validate_user_input_arguments(args): | |||
58 | return True | 59 | return True |
59 | 60 | ||
60 | def main(): | 61 | def main(): |
61 | parser = argparse_oe.ArgumentParser(description="OpenEmbedded test results tool.", | 62 | parser = argparse_oe.ArgumentParser(description="OEQA test result manipulation tool.", |
62 | epilog="Use %(prog)s <subcommand> --help to get help on a specific command") | 63 | epilog="Use %(prog)s <subcommand> --help to get help on a specific command") |
63 | parser.add_argument('-d', '--debug', help='enable debug output', action='store_true') | 64 | parser.add_argument('-d', '--debug', help='enable debug output', action='store_true') |
64 | parser.add_argument('-q', '--quiet', help='print only errors', action='store_true') | 65 | parser.add_argument('-q', '--quiet', help='print only errors', action='store_true') |