diff options
author | Yeoh Ee Peng <ee.peng.yeoh@intel.com> | 2019-02-14 13:50:37 +0800 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2019-02-21 12:34:00 +0000 |
commit | 1fd5ebdb06224489ad056e261962e23ece36fc87 (patch) | |
tree | 790b33a5498a9f97642ead84ce66dfd354bd8626 /meta/lib/oeqa/files/testresults | |
parent | 95bd530b772f97e7329749b403bf9e2dff12ff7f (diff) | |
download | poky-1fd5ebdb06224489ad056e261962e23ece36fc87.tar.gz |
resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.
These scripts were developed as a test result tools to manage
these testresults.json file.
Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json
files to a target file.
Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.
Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.
These resulttool operations expect the testresults.json file to use
the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ resulttool
To store test result from oeqa automated tests, execute the below
$ resulttool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resulttool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resulttool report <source_dir>
To perform regression file analysis, execute the below
$ resulttool regression-file <base_result_file> <target_result_file>
To perform regression dir analysis, execute the below
$ resulttool regression-dir <base_result_dir> <target_result_dir>
To perform regression git analysis, execute the below
$ resulttool regression-git <source_dir> <base_branch> <target_branch>
[YOCTO# 13012]
[YOCTO# 12654]
(From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'meta/lib/oeqa/files/testresults')
-rw-r--r-- | meta/lib/oeqa/files/testresults/testresults.json | 40 |
1 files changed, 40 insertions, 0 deletions
diff --git a/meta/lib/oeqa/files/testresults/testresults.json b/meta/lib/oeqa/files/testresults/testresults.json new file mode 100644 index 0000000000..1a62155618 --- /dev/null +++ b/meta/lib/oeqa/files/testresults/testresults.json | |||
@@ -0,0 +1,40 @@ | |||
1 | { | ||
2 | "runtime_core-image-minimal_qemuarm_20181225195701": { | ||
3 | "configuration": { | ||
4 | "DISTRO": "poky", | ||
5 | "HOST_DISTRO": "ubuntu-16.04", | ||
6 | "IMAGE_BASENAME": "core-image-minimal", | ||
7 | "IMAGE_PKGTYPE": "rpm", | ||
8 | "LAYERS": { | ||
9 | "meta": { | ||
10 | "branch": "master", | ||
11 | "commit": "801745d918e83f976c706f29669779f5b292ade3", | ||
12 | "commit_count": 52782 | ||
13 | }, | ||
14 | "meta-poky": { | ||
15 | "branch": "master", | ||
16 | "commit": "801745d918e83f976c706f29669779f5b292ade3", | ||
17 | "commit_count": 52782 | ||
18 | }, | ||
19 | "meta-yocto-bsp": { | ||
20 | "branch": "master", | ||
21 | "commit": "801745d918e83f976c706f29669779f5b292ade3", | ||
22 | "commit_count": 52782 | ||
23 | } | ||
24 | }, | ||
25 | "MACHINE": "qemuarm", | ||
26 | "STARTTIME": "20181225195701", | ||
27 | "TEST_TYPE": "runtime" | ||
28 | }, | ||
29 | "result": { | ||
30 | "apt.AptRepoTest.test_apt_install_from_repo": { | ||
31 | "log": "Test requires apt to be installed", | ||
32 | "status": "PASSED" | ||
33 | }, | ||
34 | "buildcpio.BuildCpioTest.test_cpio": { | ||
35 | "log": "Test requires autoconf to be installed", | ||
36 | "status": "ERROR" | ||
37 | } | ||
38 | } | ||
39 | } | ||
40 | } \ No newline at end of file | ||