summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* wic: make engine.py:get_partitions() resilient to parted/dmidecode stderr outputGeoff Parker2019-04-251-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Running wic commands on Debian 10 systems fail in scripts/lib/wic/engine.py:get_partitions() due to new stderr output captured when trying to parse the output from /sbin/parted as a non-root user. The parted command calls the dmidecode utility, which produces this error as a non-root user: /sys/firmware/dmi/tables/smbios_entry_point: Permission denied /dev/mem: Permission denied scripts/lib/wic/engine.py:get_partitions() calls misc.py:exec_cmd(), a subprocess wrapper which returns a combined stderr and sdtdout. These messages to stderr confuse the partition table parser in get_partitions(). This patch has the partition table parser ignore lines before the expected "BYT;" header string. Running wic in Debian 9 does not have this issue. (From OE-Core rev: 6b5681533eed4e6c00e1cc7ee442c93076f1c976) Signed-off-by: Geoff Parker <geoffhp@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/wic: Be consistent about how we call bitbakeRichard Purdie2019-04-251-11/+5
| | | | | | | | | | | | | | Whilst its tempting to call bitbake_main(), the script also directly executes bitbake. By doing both this creates all kinds of races. Therefore change to use subprocess execution in all cases. (From OE-Core rev: 0507ec4c6069490c9a503e9e2b65f0e7b1962d34) (From OE-Core rev: 2f2d52a958d3e9d511a2f5b8b14ed46466dd0d23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic: Support for changing the imager.Davis, Michael2019-04-252-1/+3
| | | | | | | | | | (From OE-Core rev: 30b33d787595b8c3120b73c755a432a0f0f30fa7) Signed-off-by: Michael Davis <michael.davis@essvote.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* pkgdata.py: avoid target-sdk-dummy-provides to mess things upChen Qi2019-04-251-0/+1
| | | | | | | | | | | | | | | | | | | | | Sometimes we meet the following failure for the test_lookup_recipe test case. AssertionError: 'zlib\nbusybox is in the RPROVIDES of target-sdk-provid[32 chars]ummy' != 'zlib\nbusybox' zlib + busybox- busybox is in the RPROVIDES of target-sdk-provides-dummy: - target-sdk-provides-dummy This is because target-sdk-provides-dummy rprovides busybox. So clean things up to avoid failure. (From OE-Core rev: 5ae49260180adc0070287db01c01a0820ce99ad3) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Refactor and simplify codebaseYeoh Ee Peng2019-04-251-36/+20
| | | | | | | | | | | | Simplify and removed unnecessary codes. Refactor to allow pythonic loop. (From OE-Core rev: 77ede7cbb7da2ed1b01d11441edcbd7dedad9b9a) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Fixed step sorted by integerYeoh Ee Peng2019-04-251-1/+1
| | | | | | | | | | | | | | | Currently the manual execution display step by sorting the step as string, where steps were not being sorted correctly when there are more than 9 steps. Fixed the step sorting by sorting step as integer. (From OE-Core rev: 1e5fba51f022a9320e72b6df753a02bfc1732af8) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable display full steps without press enterYeoh Ee Peng2019-04-251-2/+3
| | | | | | | | | | | | | | Current manualexecution required pressing enter button to show each step information, where this was wasting execution time. Enable display full steps without needing to any press enter button. (From OE-Core rev: c833aba64f26546894a0c58d8a59db75d18e2ff7) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Standardize input checkYeoh Ee Peng2019-04-251-2/+2
| | | | | | | | | | | | | Current input checking does not match the standard input practiced by QA team. Change the input checking to match the standard input practiced by the QA team. (From OE-Core rev: 6eb041fd930a3731743cc715eb8d73b045cf8932) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Allow extraction of ptest dataRichard Purdie2019-04-252-9/+32
| | | | | | | | | | | | | | | Rather than simply discarding the ptest data, change the code to discard it when writing out the new testresult files, or optionally either preserve it, or write it as seperate discrete logs. This means the autobuilder should start writing out individual ptest log files as well as allowing ueers to extract these manually. (From OE-Core rev: 08c5471c5abd95616af8df862e96072d022a6f42) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Allow store to work on single filesRichard Purdie2019-04-251-9/+12
| | | | | | | | | | | | Store operations using a single file as a source weren't working as the os.walk command didn't like being given a single file. Fix the store operation to work for single files. (From OE-Core rev: 639598aa855df523e3dd1f8df1b0eacfa7fb13d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Enable report for single result fileYeoh Ee Peng2019-04-251-10/+0
| | | | | | | | | | | | | | | Current validation check function inside resulttool disallow the report for single result file although the underlying library was able to handle both directory and file as source input to report. Removed the validation check as it was no longer needed and to enable report for single result file. (From OE-Core rev: 69ccfb9f978382f935c4d0609e09e00aab327188) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/toaster: updated test id namingYeoh Ee Peng2019-04-252-93/+93
| | | | | | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file, where test_module match the file name itself. This file was using test_module name that does not match the file name itself. Fixed test_module name as well as the test_suite name. (From OE-Core rev: 05e08a64771e40c147bdd6f02492b85edc01e946) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/toaster: transfer manual toaster test cases to oeqaYeoh Ee Peng2019-04-252-0/+3742
| | | | | | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia, manual test cases need to be migrated to OEQA. These manual test case json files will serve two use cases. Use case#1: as input to the future commandline-based manual test runner script, where this script will display actions and expected result information in user friendly text, then it will capture user input for test result and log, finally it will write test result and log into existing standardize test result json format from OEQA framework for automated tests. Use case#2: QA will open and read these json file manually for planning manual test execution. Any reader interested in understanding manual test cases will open and read these files. (From OE-Core rev: 73f5446c38a5d7766c81cda7a7bb22df9cfe0f23) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/kernel-dev.json: test id updatedsangeeta jain2019-04-251-18/+18
| | | | | | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. (From OE-Core rev: 35233e411db1a5978bfb2f4265b8e3a8fbdc4246) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/compliance-test.json: test id updated and missing test cases addedsangeeta jain2019-04-251-4/+56
| | | | | | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/compliance-test.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. Added 2 missing test cases for Beaglebone Stress Test. (From OE-Core rev: 7d95d4c6d30ae2977cd9feb5a45f96d05ff9b763) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-hw.json: test id updated and obsolete test cases removedsangeeta jain2019-04-251-220/+46
| | | | | | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/bsp-hw.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases. (From OE-Core rev: 862c96a98d2087cf05fbfeeafb57dcef1225d978) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-hw.json: add non-IA testsMazliana2019-04-251-0/+260
| | | | | | | | | | | | | | QA team found that 10 manual bsp test cases from Testopia for Beaglebone, EdgeRouter, and MPC need to be up streamed [YOCTO #12650] (From OE-Core rev: 05f785937be3d06f38cff4de238d7bd85d4a452a) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/sdk.json: Updated toolchain tarball to core-image-satosangeeta jain2019-04-251-7/+7
| | | | | | | | | | | | | | | | | Manual test step for SDK is updated. Previously toolchain was "poky-glibc-x86_64-core-image-sato-sdk<type-arch>-toolchain-<release-version>.sh" But toochain for core-image-sato-sdk is not available in releases after 2.1, hence changed it to "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh Other less significant update is to add exepcted result in intermediate test steps. (From OE-Core rev: 4e26c3a6aa647a322a8e8a3027a4114caed38f7d) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh" Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual: Add manual runtime 'compliance' testcases to jsonSudhir Sharma2019-04-251-0/+142
| | | | | | | | | | | Added json file for compliance test cases to the manual runtime tests (From OE-Core rev: 3f143119def5a84430762790f69b426fdc8ad281) Signed-off-by: Sudhir Sharma <sudhirx.sharma@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-qemu.json: Update for QEMU_USE_KVMRobert Yang2019-04-251-2/+2
| | | | | | | | | | | Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more. (From OE-Core rev: cd3c467b9419238494e8c525e86be0cf85321c4e) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/build-appliance: Drop test which is already automatedRichard Purdie2019-04-251-26/+0
| | | | | | | | | | The nightly-build-appliance target already tests this particular build. (From OE-Core rev: 90d9542900b6a941a3c3355c15dbfc2e2ac78232) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual: Remove bitbake-selftest executionRichard Purdie2019-04-251-30/+0
| | | | | | | | | | | The autobuilder runs bitbake-selftest already so we don't need to execute this manually. (From OE-Core rev: 772b2b883249c8606b2df380f749320cb683a7b2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual: transfer manual test cases from testopia to oeqaYeoh Ee Peng2019-04-259-0/+2750
| | | | | | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia, manual test cases need to be migrated to OEQA. These manual test case json files will serve two use cases. Use case#1: as input to the future commandline-based manual test runner script, where this script will display actions and expected result information in user friendly text, then it will capture user input for test result and log, finally it will write test result and log into existing standardize test result json format from OEQA framework for automated tests. Use case#2: QA will open and read these json file manually for planning manual test execution. Any reader interested in understanding manual test cases will open and read these files. (From OE-Core rev: a1675c33638288363b243e7412fe4cea63756ae6) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build-appliance-image: Update to sumo head revisionyocto-2.5.3sumo-19.0.3Richard Purdie2019-03-271-1/+1
| | | | | | (From OE-Core rev: 0a2db923fd17019d07d88204b355aa46590f0b97) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/gitarchive: Handle case where parent is only on originRichard Purdie2019-03-271-2/+2
| | | | | | | | | | | | | | | The parent code currently assumed that any parent branch is locally checked out which may not be the case. Use the local branch by default but fall back to the origin. This also means removing the later saftey check as the branch may not exist locally. This fixes the autobuilder resulttool test pushing code. (From OE-Core rev: 633e024f5d492ba38debf745b59813f6a2f4a3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build-appliance-image: Update to sumo head revisionRichard Purdie2019-03-261-1/+1
| | | | | | (From OE-Core rev: ece8242187558011940d7c6762b64c7116e38689) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* poky.conf: Bump version for 2.5.3 sumo releaseRichard Purdie2019-03-261-1/+1
| | | | | | (From meta-yocto rev: c19ec95877614314ab236fd89d9b5055f33de0c4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* yocto-uninative: Update to 2.4Richard Purdie2019-03-261-5/+4
| | | | | | | | | | | This supports glibc 2.29 which is appearing in distros like Ubuntu 19.04 (From OE-Core rev: 661ac7ed164965690a4334d372bf22f04c3bfd3a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> [Fixed up for Sumo context; no aarch64] Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-report/gitarchive: Move common useful functions to libraryRichard Purdie2019-03-262-80/+81
| | | | | | | | | | | These functions can be reused by the resulttool code so move to the common function library for this purpose. (From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385) (From OE-Core rev: 94a3ca85d30fc957f8f01a216a75342be49f9143) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-git-archive: Separate out functionality to library functionRichard Purdie2019-03-262-160/+179
| | | | | | | | | | | | | This turns the core of the script into a library function. Ultimately this will let us call that code with custom 'keywords' rather than relying on the data parsed from bitbake metadata which can't be used when archiving historical results. (From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0) (From OE-Core rev: ab5d49ba8bb02410fb9f3024f98ff19a04acb3ca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-report: Fix missing buildstats comparisionsRichard Purdie2019-03-261-2/+2
| | | | | | | | | | | Integers were being compared to strings leading to missing buildstats comparision data. Fix this. (From OE-Core rev: 2dbbf598192ae2b3aa488df042f56aa6c6634a00) (From OE-Core rev: e8651b2e01db0777b23a0359f9413c3b27bae77f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-report: Improve branch comparision handlingRichard Purdie2019-03-261-8/+23
| | | | | | | | | | | | | | | | | | | When comparing branches, correctly filter the revisions corresponding to the specific branch specified. Also use the commit numbers as a way to gauge spatially related commits for comparision meaning comparisions for out of order build revisions becomes meaninful. This should improve the reporting for autobuilder generated builds. Also improve the branch option help text. (From OE-Core rev: 9f6f4ab6eec9dca07af7f53da5f737a6167bfb38) (From OE-Core rev: d6645c4c931565d62df85db0fa71f15d51265828) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-report: Allow commits from different branchesRichard Purdie2019-03-261-0/+5
| | | | | | | | | | | | | | | The code won't currently allow comparisions of two commits on different branches even if the commits are specified by their hashes. This updates the code to search two branches for any relavent commits, hence allowing comparisions to be made. A particularly useful case is master vs. master-next for example. (From OE-Core rev: 2d835bb1570c515ae501442f3ce19fae8e249b27) (From OE-Core rev: c6351550a9bff3e0b0e3d3c0baefed4041c47b33) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-report: Allow branch without hostnameRichard Purdie2019-03-261-2/+2
| | | | | | | | | | | | Allow the branch to be set without the hostname option. Previously if hostname wasn't set, branch would be overwritten regardless of whether it was set or not. (From OE-Core rev: 3ec43bf4c6c8f7730a67f63ad4e14903f289014e) (From OE-Core rev: e95277e859adb3e3ca26c8c618954ae52ec3bff4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: To output right test case idMazliana2019-03-261-9/+6
| | | | | | | | | | | | | | We found that manualexecution does not capture test suite values correctly if there are more than one test suite in test cases. After verification has made we found out we should retrieved full test cases value <test_module.test_suite.test_case> from oeqa/manual/ json file rather than split it them into new variables test_suite and test_cases. (From OE-Core rev: d1dee26b13dd565bf06307c99b02f6d57ff0ddef) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Enable roll-up report for a commitYeoh Ee Peng2019-03-261-3/+13
| | | | | | | | | | Enable roll-up all test results belong to a commit and to provide a roll-up report. (From OE-Core rev: 8f23018bcb76c1b431f69183f993f08adb660fae) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: Enable manual result store and regressionYeoh Ee Peng2019-03-262-19/+26
| | | | | | | | | | | | | | | To enable store for testresults.json file from manualexecution, add layers metadata to configuration and add "manual" map to resultutils.store_map. To enable regression for manual, add "manual" map to resultutils.regression_map. Also added compulsory configurations ('MACHINE', 'IMAGE_BASENAME') to manualexecution. (From OE-Core rev: 514a9165790dda2afb412c519cb1444493dff1f9) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression: Ensure regressoin results are sortedYeoh Ee Peng2019-03-261-4/+4
| | | | | | | | | Sorted regression results to provide friendly viewing of report. (From OE-Core rev: de222c3c9e3aa34e42e1ad6ba6cbb9e1a18ccd80) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store: Fix missing variable causing testresult corruptionRichard Purdie2019-03-261-1/+1
| | | | | | (From OE-Core rev: 811d2be3cef340ee945335552892a2a49a46bf4b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Ensure ptest results are sortedRichard Purdie2019-03-261-1/+1
| | | | | | (From OE-Core rev: f182217970029ba0260df3c3089101352da083ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Ensure test suites with no results show up on the reportRichard Purdie2019-03-261-5/+8
| | | | | | | | | ptest suites with no results don't show up on the reports even though we have a duration for them. Fix this so the fact they report no tests is visible. (From OE-Core rev: a61ad19d5f997cb2377ca307c94ca2d53f8c0c49) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Handle missing metadata sections more cleanlyRichard Purdie2019-03-261-3/+6
| | | | | | | | | Currently some older results files cause the code to give tracebacks. Handle these missing sections more cleanly. (From OE-Core rev: 829754c380ca2b614afa23500e53a8c5b850932f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store: Handle results files for multiple revisionsRichard Purdie2019-03-261-16/+23
| | | | | | | | | Currently we cant store results if the results files span multiple different build revisons. Remove this limitation by iterating. (From OE-Core rev: 22a5982dbcb624ffcdbacef92b2a235851214663) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: Avoids tracebacks for missing logsRichard Purdie2019-03-261-1/+2
| | | | | | (From OE-Core rev: d5d62b31b192b88c6a829099b74bfb30583b066e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improvements to allow integration to the autobuilderRichard Purdie2019-03-2610-508/+530
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a combined patch of the various tweaks and improvements I made to resulttool: * Avoid subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. * Avoid python keywords as variable names * Simplify dict accesses using .get() * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) * Fix oe-git-archive to use the commit/branch data from the results file * Cleaned up the command option help to match other changes * Follow the model of git branch/tag processing used by oe-build-perf-report and use that to read the data using git show to avoid branch change * Add ptest summary to the report command * Update the tests to match the above changes (From OE-Core rev: e4195565d2a50046d4378c97f7a593c41bed51bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: enable manual execution and result creationMazliana2019-03-262-0/+145
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Integrated “manualexecution” operation to resulttool scripts. Manual execution script is a helper script to execute all manual test cases in baseline command, which consists of user guideline steps and the expected results. The last step will ask user to provide their input to execute result. The input options are passed/failed/blocked/skipped status. The result given will be written in testresults.json including log error from the user input and configuration if there is any.The output test result for json file is created by using OEQA library. The configuration part is manually key-in by the user. The system allow user to specify how many configuration they want to add and they need to define the required configuration name and value pair. In QA perspective, "configuration" means the test environments and parameters used during QA setup before testing can be carry out. Example of configurations: image used for boot up, host machine distro used, poky configurations, etc. The purpose of adding the configuration is to standardize the output test result format between automation and manual execution. To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To execute manual test cases, execute the below $ resulttool manualexecution <manualjsonfile> By default testresults.json store in <build_dir>/tmp/log/manual/ [YOCTO #12651] (From OE-Core rev: cd2ef260569a67907801299b1397db94aa871cb5) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: enable merge, store, report and regression analysisYeoh Ee Peng2019-03-2610-0/+832
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file. These scripts were developed as a test result tools to manage these testresults.json file. Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files. Using the "merge" operation, user can merge multiple testresults.json files to a target file. Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository. Using the "regression-file" operation, user can perform regression analysis on testresults.json files specified. Using the "regression-dir" and "regression-git" operations, user can perform regression analysis on directory and git accordingly. These resulttool operations expect the testresults.json file to use the json format below. { "<testresult_1>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, ... "<testresult_n>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, } To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To store test result from oeqa automated tests, execute the below $ resulttool store <source_dir> <git_branch> To merge multiple testresults.json files, execute the below $ resulttool merge <base_result_file> <target_result_file> To report test report, execute the below $ resulttool report <source_dir> To perform regression file analysis, execute the below $ resulttool regression-file <base_result_file> <target_result_file> To perform regression dir analysis, execute the below $ resulttool regression-dir <base_result_dir> <target_result_dir> To perform regression git analysis, execute the below $ resulttool regression-git <source_dir> <base_branch> <target_branch> [YOCTO# 13012] [YOCTO# 12654] (From OE-Core rev: bb0bc6368bb51ac0be77d13fe931601d493951ea) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lttng-tools: update to 2.9.11Jonathan Rajotte-Julien2019-03-251-2/+2
| | | | | | | | (From OE-Core rev: 4af5a794497746629a35726ce515556b53d40da9) Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lttng-modules: update to 2.10.9Jonathan Rajotte-Julien2019-03-251-2/+2
| | | | | | | | | | | | | | | Pertinent fix for OE-Core since 2.10.6: Fix: out of memory error handling Fix: access migrate_disable field directly Prevent allocation of buffers if exceeding available memory 2.10.9 also contains the necessary fix to support kernel up to 5.0. (From OE-Core rev: 9666e9f6058bfd8294cb4e6b03752f28159a76bc) Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lttng-ust: update to 2.10.3Jonathan Rajotte-Julien2019-03-251-2/+2
| | | | | | | | (From OE-Core rev: 9d8b3988a53449fa18398e9eaae1627923eeef97) Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-20 06:54:18 +0000