summaryrefslogtreecommitdiffstats
path: root/scripts/lib
Commit message (Collapse)AuthorAgeFilesLines
...
* resulttool: Remove prints if no tests occurJon Mason2019-06-041-11/+3
| | | | | | | | | | Printing the lack of a test is not necessary (per feedback). Remove this from the template to quieten it. (From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable creation of test case configurationsangeeta jain2019-06-041-1/+30
| | | | | | | | | | | | Allow the creation of test case configuration file based on user inputs. Where this testcase configuration file will be used by the the manual execution to run selected test cases for a module rather than compulsory run all test cases in manual json file. (From OE-Core rev: 73d2a747c17779da0ca972da776b3cf02c2e1cbc) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: modify to be multi-machineJon Mason2019-06-042-67/+81
| | | | | | | | | | | | Currently, the code will sum all of the different machine results into a single report of the tests results. This can lead to confusion as to which machine may be experiencing issues. Modify the code to store the results in a per machine basis and report them accordingly. (From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic: bootimg-efi: add label source parameterChee Yang Lee2019-05-301-4/+15
| | | | | | | | | | | | | Add new source parameter label to allow custom boot.conf/grub.cfg label, so far it's hardcoded to "Boot". Default label to "Boot" for systemd-boot and blank for grub-efi when source parameter label are not set. (From OE-Core rev: 7a0aab1aa31e66e6bc94c04c2f6c1043b64a8967) Signed-off-by: Chee Yang Lee <chee.yang.lee at intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic/bootimg-efi: replace hardcoded volume name with labelChee Yang Lee2019-05-151-2/+4
| | | | | | | | | | | volume name should refer to --label in .wks. Replace the hardcoded volume name with label. set "ESP" as default name when no lable specified. (From OE-Core rev: 5621aceaf39ef0dc097b16c83e73b9882c987a7c) Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib+scripts: Convert to SPDX license headersRichard Purdie2019-05-0959-587/+81
| | | | | | | | | | | | | | | | | | | | | | | This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add option to dump all ptest logsJoshua Watt2019-05-082-16/+54
| | | | | | | | | | | | | | | Adds an option to dump all the ptest logs to individual files in a specified directory. If multiple test runs are present, the '--prepend-run' argument will create separate directories for each test run under the target directory and put the logs there to prevent each test run from clobbering the others. [YOCTO #13331] (From OE-Core rev: f262b9e7482e38f04dfa162f0aaf65b17e86f8b2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* recipetool: fix unbound variable when fixed SRCREV can't be foundAlex Kiernan2019-05-081-1/+1
| | | | | | | | | | | | | | | | | | | | If attempting to find a fixed SRCREV fails because the directory doesn't exit, avoid failing with: Traceback (most recent call last): File "/home/vagrant/poky/scripts/recipetool", line 121, in <module> ret = main() File "/home/vagrant/poky/scripts/recipetool", line 110, in main ret = args.func(args) File "/home/vagrant/poky/scripts/lib/recipetool/create.py", line 707, in create_recipe srcrev = stdout.rstrip() UnboundLocalError: local variable 'stdout' referenced before assignment Fixes: 000480c42797 ("recipetool / devtool: set a fixed SRCREV by default when fetching from git") (From OE-Core rev: 8e4b301573071ea04911b0df6533c25bc4596b93) Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: add LTP compliance sectionArmin Kuster2019-04-292-1/+58
| | | | | | | (From OE-Core rev: a680d7d15fafbecf4edce9a29cc4eda16c11fd94) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: add ltp test supportArmin Kuster2019-04-292-1/+59
| | | | | | | (From OE-Core rev: 966795aa2c6960aca11a04e87f415256faf26957) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable test case configuration optionsangeeta jain2019-04-261-2/+12
| | | | | | | | | | | | | | | | | | | | | | | | Current manualexecution required user to exceute all test cases defined inside a "modulename.json" file in oeqa/manual There are cases when all test cases all not required to run for a module on specific DUT. Enable manualexecution to have the optional feature where it will use pre-defined json format test case configuration file where user will be able to select test cases from the "modulename.json" instead of running all of them. This will help in reducing testing time and reporting unneccesary skip or failures. Example pre-defined json format test case configuration file (for build-applince): { "testcases" : [ "build-appliance.build-appliance.Create_core-image-sato-sdk_using_build_appliance", "build-appliance.build-appliance.Build_a_image_without_error_(added_recipe)" ] } (From OE-Core rev: 3950c28a34f94b5907d37b579bdaee5a59794652) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "wic: Set a miniumum FAT16 volume size."Angus Lees2019-04-261-1/+1
| | | | | | | | | | | | | | | | This reverts commit f7dfb4d43247d3c13a4e0a3853007d63b9512b83. FAT16 volumes do not have a minimum size of 8250 blocks (== slightly over 4MB). Exhibit A: floppy disks. The original commit message suggests this was a workaround for a bug in parted - in which case we should fix it there, or use the wic `--fixed-size` option to pass down an enforced minimum from a more context-aware point in the callstack. (From OE-Core rev: e0a1ae3863bc229512d43e4a5248551bf25950e5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resultool/resultutils: Fix module import errorRichard Purdie2019-04-261-1/+1
| | | | | | | | | Fix AttributeError: module 'urllib' has no attribute 'request' when using remote http urls. (From OE-Core rev: 9acc47236e2a9ac7f97f80040f5e508f23f01611) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add log subcommandJoshua Watt2019-04-252-1/+57
| | | | | | | | | Adds a subcommand for dumping various logs from test results (From OE-Core rev: 454b8d2cdc6f79a51e610dae92c22352850c3f7c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Load results from URLJoshua Watt2019-04-255-13/+28
| | | | | | | | | | Adds support for resulttool to load JSON files directly from a http:// or https:// URL (From OE-Core rev: 235bcf0c504e7ea253ccfb28d600898117c64c1f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Refactor and remove duplicate codeYeoh Ee Peng2019-04-121-47/+40
| | | | | | | | | | | Remove duplicate codes. Replace unnecessary class variables with local variables. Rename variables and arguments with simple and standard name. (From OE-Core rev: 6bef61d36f3328fad003c0dc9c010d3f76ba96d8) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* recipetool: add missed moduleLei Yang2019-04-121-0/+1
| | | | | | | | | | | | | | When I use recipetool to add a new recipe, it says: [snip] sys.exit(14) NameError: name 'sys' is not defined [snip] (From OE-Core rev: 051e1c27bbf6a5898f2cf13b78dd3d8d1ec31586) Signed-off-by: Lei Yang <Lei.Yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable creation of configuration option fileYeoh Ee Peng2019-04-101-1/+53
| | | | | | | | | | | | Allow the creation of configuration option file based on user inputs. Where this configuration option file will be used by the the manual execution to display options for configuration rather than user need to inputs configuration manually. (From OE-Core rev: 23c7a037a85a825aa088bfac3a144c14087a9f49) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable configuration options selectionYeoh Ee Peng2019-04-101-9/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current manualexecution required user to input configuration manually where there were inconsistent inputs and human typo issues. Enable manualexecution to have the optional feature where it will use pre-compiled configuration options file where user will be able to select configuration from the pre-compiled list instead of manual key-in the configuration. This will eliminate human error. Expect the pre-compiled configuration options file in json format below { "bsps-hw": { "IMAGE_BASENAME": { "1": "core-image-sato-sdk" }, "MACHINE": { "1": "beaglebone-yocto", "2": "edgerouter", "3": "mpc8315e-rdb", "4": "genericx86", "5": "genericx86-64" } }, "bsps-qemu": { "IMAGE_BASENAME": { "1": "core-image-sato-sdk" }, "MACHINE": { "1": "qemuarm", "2": "qemuarm64", "3": "qemumips", "4": "qemumips64", "5": "qemuppc", "6": "qemux86", "7": "qemux86-64" } } } (From OE-Core rev: 1af7e04223b415a64e100a9f91d60f5e9b0f789a) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: standard: Handle exporting generated config fragmentsNathan Rossi2019-04-091-0/+14
| | | | | | | | | | | | | | | The cml1 and ccmake bbclasses generate configuration fragment source files that must be exported from the WORKDIR as a source file to be preserved across builds. This change adds detection of the current recipes inherited classes and for cml1 and ccmake classes checks for the specific generated configuration fragment files. These files are then exported by devtool and included as SRC_URI files from within the target layer. (From OE-Core rev: 7a1fabe4a24552ce8a50c8d6009969ca52a5fd27) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Refactor and simplify codebaseYeoh Ee Peng2019-04-041-36/+20
| | | | | | | | | | Simplify and removed unnecessary codes. Refactor to allow pythonic loop. (From OE-Core rev: 84c6a992e1114685194f6e8a554bce7753c090cc) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Fixed step sorted by integerYeoh Ee Peng2019-04-041-1/+1
| | | | | | | | | | | | | Currently the manual execution display step by sorting the step as string, where steps were not being sorted correctly when there are more than 9 steps. Fixed the step sorting by sorting step as integer. (From OE-Core rev: 192c255d9f597b3526a5b94fcf3eedc4527189ed) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable display full steps without press enterYeoh Ee Peng2019-04-041-2/+3
| | | | | | | | | | | | Current manualexecution required pressing enter button to show each step information, where this was wasting execution time. Enable display full steps without needing to any press enter button. (From OE-Core rev: e44c9a018c13208fef0dcef4479ce71977628cd4) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Standardize input checkYeoh Ee Peng2019-04-041-2/+2
| | | | | | | | | | | Current input checking does not match the standard input practiced by QA team. Change the input checking to match the standard input practiced by the QA team. (From OE-Core rev: 630a93c740359a65249a198f314f15040042f8d3) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Allow extraction of ptest dataRichard Purdie2019-04-042-9/+32
| | | | | | | | | | | | | Rather than simply discarding the ptest data, change the code to discard it when writing out the new testresult files, or optionally either preserve it, or write it as seperate discrete logs. This means the autobuilder should start writing out individual ptest log files as well as allowing ueers to extract these manually. (From OE-Core rev: a1e0944bf260ef50dd7dfcb10db248fdd7f45bc9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Allow store to work on single filesRichard Purdie2019-04-041-9/+12
| | | | | | | | | | Store operations using a single file as a source weren't working as the os.walk command didn't like being given a single file. Fix the store operation to work for single files. (From OE-Core rev: eb5bbe613d8c7cbcd8b74d8ac0073c2217970410) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic/engine: Add missing newlineRichard Purdie2019-04-031-1/+1
| | | | | | | | | | | | | | | | On some hosts the wic.Wic2.test_wic_cp_ext selftest was failing as files weren't being copied into the rootfs with "wic cp". This was due to a bug added by: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=89d4a8df074598cfb3a76e41db7c45d845afd961 where there should be a second newline added at the end of the expression due to the difference in the way echo -e and printf behave. [YOCTO #13237] (From OE-Core rev: 2f3adb21cc5067458a12964d0bde235966a20a60) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: prevent starting git gc in backgroundTaras Kondratiuk2019-03-241-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Devtool creates a git repository for extracted sources in a temporary directory and then moves it to a final destination after patching is done. Unfortunately devtool is not aware that some of its git operations may have caused git garbage collector to start in background. If timing is just right a repository move fails because GC is removing git objects while they are being moved. The issue was hit on Krogoth release, but the code that moves created git repository is still the same. Disable gc.autodetach to make GC run synchronously and block git until it is done. Traceback (most recent call last): File "<snip>/openembedded-core/scripts/devtool", line 342, in <module> ret = main() File "<snip>/openembedded-core/scripts/devtool", line 329, in main ret = args.func(args, config, basepath, workspace) File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 352, in extract initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, rd) File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 644, in _extract_source shutil.move(srcsubdir, srctree) File "/usr/lib/python2.7/shutil.py", line 299, in move copytree(src, real_dst, symlinks=True) File "/usr/lib/python2.7/shutil.py", line 208, in copytree raise Error, errors Error: [('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e', '<snip>/build/grpc/grpc_src/.git/objects/5e', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e'"), ('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57', '<snip>/build/grpc/grpc_src/.git/objects/57', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57'"), many git objects ... ] (From OE-Core rev: 3da578a79195a63fb6d4bbd68d8f1548defd205f) Signed-off-by: Taras Kondratiuk <takondra@cisco.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: To output right test case idMazliana2019-03-181-9/+6
| | | | | | | | | | | | | | We found that manualexecution does not capture test suite values correctly if there are more than one test suite in test cases. After verification has made we found out we should retrieved full test cases value <test_module.test_suite.test_case> from oeqa/manual/ json file rather than split it them into new variables test_suite and test_cases. (From OE-Core rev: db08ec5f48badd5be2fa24cc0d89a4386ab44a59) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Enable roll-up report for a commitYeoh Ee Peng2019-03-181-3/+13
| | | | | | | | | | Enable roll-up all test results belong to a commit and to provide a roll-up report. (From OE-Core rev: a98a75a3db1fe9d327fd21919fb8ea09e76fb1e3) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* correct do_patch for kernel bbappend in sdkYann CARDAILLAC2019-03-121-3/+1
| | | | | | | | | | | | | do_patch rule of SDK's workspace/appends/linux-*.bbhappend may fail if script are not written in Python that was the case with Phytec's BSP, the fix was to replace the do_patch rule with : do_patch[noexec]="1" when the file was generated in scripts/lib/devtool/standard.py (From OE-Core rev: e51cb385e5573c7069de6e8308b9b68eb4a3a162) Signed-off-by: Yann CARDAILLAC <yann.cardaillac@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: Enable manual result store and regressionYeoh Ee Peng2019-03-072-19/+26
| | | | | | | | | | | | | | | To enable store for testresults.json file from manualexecution, add layers metadata to configuration and add "manual" map to resultutils.store_map. To enable regression for manual, add "manual" map to resultutils.regression_map. Also added compulsory configurations ('MACHINE', 'IMAGE_BASENAME') to manualexecution. (From OE-Core rev: 7521834f8557565d3898605257c7fd9505ea22bb) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe.scriptutils: enable color in a more flexible wayChristopher Larson2019-03-061-6/+7
| | | | | | | | | | | Rather than recreating handlers and forcing them, iterate over the handlers and enable color on ones we can handle. This makes it easier to handle color properly when we introduce the bb.msg default log filters. (From OE-Core rev: 713f44e0e0cde9b818c214002fd8b730d422fafa) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic: Ensure that sourceparams values can contain '='Jan Kiszka2019-02-281-1/+1
| | | | | | | | | This allows '--sourceparams key1=val1=val11,key2=val2'. (From OE-Core rev: e59bba5ee21e9065dce86b912996b893f8946535) Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression: Ensure regressoin results are sortedYeoh Ee Peng2019-02-281-4/+4
| | | | | | | | | Sorted regression results to provide friendly viewing of report. (From OE-Core rev: 59d827889dd8b790a7c9323c69565d6bba3041bd) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store: Fix missing variable causing testresult corruptionRichard Purdie2019-02-281-1/+1
| | | | | | (From OE-Core rev: a89846cca457de10249881ab695f0f883bfb6f40) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Ensure ptest results are sortedRichard Purdie2019-02-271-1/+1
| | | | | | (From OE-Core rev: 3ec9a6288793f384bf108d7194cf50147485fa28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Ensure test suites with no results show up on the reportRichard Purdie2019-02-271-5/+8
| | | | | | | | | ptest suites with no results don't show up on the reports even though we have a duration for them. Fix this so the fact they report no tests is visible. (From OE-Core rev: 58268151704246a81ec8dae46c26346023057554) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Handle missing metadata sections more cleanlyRichard Purdie2019-02-271-3/+6
| | | | | | | | | Currently some older results files cause the code to give tracebacks. Handle these missing sections more cleanly. (From OE-Core rev: f778c191dbd5740173b3be07f4c1655d85a07bb2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store: Handle results files for multiple revisionsRichard Purdie2019-02-271-16/+23
| | | | | | | | | Currently we cant store results if the results files span multiple different build revisons. Remove this limitation by iterating. (From OE-Core rev: 4da12c00963b02508056b87ce9b972528ce3a1be) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: deploy-target: allow use of alternate sshOlaf Mandel2019-02-251-4/+16
| | | | | | | | | | Allow the use of an alternate SSH executable for deploy-target and undeploy-target. The command can be specified with -e / --ssh-exec. (From OE-Core rev: c2684740d7fd3aeb60b3ec7888000097effe82f4) Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: Avoids tracebacks for missing logsRichard Purdie2019-02-251-1/+2
| | | | | | (From OE-Core rev: ec020699086cc56bfa37bab38065472da52b3d0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improvements to allow integration to the autobuilderRichard Purdie2019-02-218-447/+478
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a combined patch of the various tweaks and improvements I made to resulttool: * Avoid subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. * Avoid python keywords as variable names * Simplify dict accesses using .get() * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) * Fix oe-git-archive to use the commit/branch data from the results file * Cleaned up the command option help to match other changes * Follow the model of git branch/tag processing used by oe-build-perf-report and use that to read the data using git show to avoid branch change * Add ptest summary to the report command * Update the tests to match the above changes (From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: enable manual execution and result creationMazliana2019-02-211-0/+137
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Integrated “manualexecution” operation to resulttool scripts. Manual execution script is a helper script to execute all manual test cases in baseline command, which consists of user guideline steps and the expected results. The last step will ask user to provide their input to execute result. The input options are passed/failed/blocked/skipped status. The result given will be written in testresults.json including log error from the user input and configuration if there is any.The output test result for json file is created by using OEQA library. The configuration part is manually key-in by the user. The system allow user to specify how many configuration they want to add and they need to define the required configuration name and value pair. In QA perspective, "configuration" means the test environments and parameters used during QA setup before testing can be carry out. Example of configurations: image used for boot up, host machine distro used, poky configurations, etc. The purpose of adding the configuration is to standardize the output test result format between automation and manual execution. To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To execute manual test cases, execute the below $ resulttool manualexecution <manualjsonfile> By default testresults.json store in <build_dir>/tmp/log/manual/ [YOCTO #12651] (From OE-Core rev: f24dc9e87085a8fe5410feee10c7a3591fe9d816) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: enable merge, store, report and regression analysisYeoh Ee Peng2019-02-217-0/+604
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file. These scripts were developed as a test result tools to manage these testresults.json file. Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files. Using the "merge" operation, user can merge multiple testresults.json files to a target file. Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository. Using the "regression-file" operation, user can perform regression analysis on testresults.json files specified. Using the "regression-dir" and "regression-git" operations, user can perform regression analysis on directory and git accordingly. These resulttool operations expect the testresults.json file to use the json format below. { "<testresult_1>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, ... "<testresult_n>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, } To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To store test result from oeqa automated tests, execute the below $ resulttool store <source_dir> <git_branch> To merge multiple testresults.json files, execute the below $ resulttool merge <base_result_file> <target_result_file> To report test report, execute the below $ resulttool report <source_dir> To perform regression file analysis, execute the below $ resulttool regression-file <base_result_file> <target_result_file> To perform regression dir analysis, execute the below $ resulttool regression-dir <base_result_dir> <target_result_dir> To perform regression git analysis, execute the below $ resulttool regression-git <source_dir> <base_branch> <target_branch> [YOCTO# 13012] [YOCTO# 12654] (From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/buildstats: Improve error messageRichard Purdie2019-02-161-1/+1
| | | | | | | | | Just stating the recipe counts are different isn't helpful, showing the differences makes it much easier to understand what changed. (From OE-Core rev: 1d84b782e3af6f0e6922d72895c905877cc33739) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic/engine: Fix missing parted autobuilder failuresRichard Purdie2019-02-151-3/+5
| | | | | | | | | | | | OE-Core rev: a88bcbae850a2e6d182291d3f8e167aabdbe4842 broke the ability to find parted as it may be in sbin which is not in PATH for some users on some distros. Iterate on the original patch to fix this and also fix the original problem. (From OE-Core rev: af3803e5189d7814f9dbd238fb6dab200f351e1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic/engine.py: Load paths from PATH environment variableWilliam Bourque2019-02-141-2/+7
| | | | | | | | | | | | | Load self.paths from environment variable and if it fails, fall back to hardcoded list. This is required for users that would need to load different e2fsprogs binaries if their system's ones are not recent enought (From OE-Core rev: a88bcbae850a2e6d182291d3f8e167aabdbe4842) Signed-off-by: William Bourque <wbourque@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/lib/wic/engine: Fix cp's target path for ext* filesystemsLeonardo Augusto2019-02-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Python subprocess' shell=True defaults to /bin/sh[1][2], which often refers to a POSIX-compliant shell. As the -e flag is not defined in the POSIX standard[3], some shells may interpret "-e" as the first argument to the echo command, causing the rest of the command line to fail. In this particular case, "echo -e 'cd {}'" is interpreted as "-e cd {}", which causes the first line of the command to fail, and causing cp to always place the source file in the filesystem's root. Replacing "echo -e" for a printf command makes this more portable. This issue only affects "wic cp" for ext* filesystems. [1] https://docs.python.org/2/library/subprocess.html [2] https://docs.python.org/3/library/subprocess.html [3] http://pubs.opengroup.org/onlinepubs/9699919799 (From OE-Core rev: 9a5b703d4e60c3ccdf80b5289b8e6fc91133fcde) Signed-off-by: Leonardo Augusto <leobsd@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* checklayer: Avoid adding the layer if it is already presentRobert Yang2019-02-141-19/+17
| | | | | | | | | | | | | | * Rename add_layer() to add_layers() so that add_layer_dependencies() can re-use it. * Avoid adding the layer if it is already present [YOCTO #13148] (From OE-Core rev: b9cc18d83f55ff48c3d6e60c56359f6736d5a06a) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-15 13:59:42 +0000