summaryrefslogtreecommitdiffstats
path: root/meta/lib
Commit message (Collapse)AuthorAgeFilesLines
* selftest/signing: Ensure build path relocation is safeRichard Purdie2020-09-161-1/+3
| | | | | | | | | | | Similarly to 04ee0e8b95cd8ed890374e0007f976684206b630, ensure only full build paths are replaced in the environment to avoid breaking buildtools. (From OE-Core rev: be07d93a4f59d4563f2d064be1997b39f05e9f0e) (From OE-Core rev: 7a46226288179df565b7c21c3316672d2e2a1ac0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/testsdk: Use original PATHRichard Purdie2020-09-161-5/+2
| | | | | | | | | | | | | | | | We want to test the SDK with PATH from the original host, not with our own tools injected via HOSTTOOLS. It even uses some tools which aren't in HOSTTOOLS. This is necessary after changing the SDK to not reset PATH to the system default which is bad for other reasons and brings the testing into sync with that change. (From OE-Core rev: 87c9602fd0dedc7bcf75b822aaf5f6ebfc17737c) (From OE-Core rev: 2cb99a44c650db7fd6fbd269f5788e4ebfd523fc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Ensure buildtools in environment variables isn't replacedRichard Purdie2020-09-161-1/+1
| | | | | | | | | | | | | This avoids the seeing broken replacements like: oe-selftest-centos/build/build-st-926tools/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt which understandably break builds. (From OE-Core rev: 04ee0e8b95cd8ed890374e0007f976684206b630) (Cherry-picked from f930e2cadb9ee69759720b6c49aeeb6dd43a7edd but adjusted for thud) (From OE-Core rev: 611d3947054dad764aeded4c6a050415f7ca4991) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* utils: fix gcc 10 version detectionCharles-Antoine Couret2020-09-161-1/+1
| | | | | | | | | | | | | | | | | Utils can not detect GCC 10 correctly due to wrong regex. It generates this error "ERROR: Can't get compiler version from gcc --version output" Sub-version numbers should be 1 or more digits instead of 1 only. (From OE-Core rev: 1d6f50a5e58f46f8af6e83c4e288d93a717187ea) (From OE-Core rev: e73228e6b039bd972d36774bfb360a638a03d821) Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 186fe4a3d390a52b87282c3e694ce3251e45ee78) Signed-off-by: Jeremy A. Puhlman <jpuhlman@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sdk: Install nativesdk locales for all TCLIBC variantsKhem Raj2019-12-161-4/+0
| | | | | | | | | | | | | | | | | | | | | | | install_locales() here is actually operating on nativesdk and only glibc is the default library for nativesdk, since thats what most of desktop/server distros use, therefore bailing out based on TCLIBC is not needed here, since nativesdk-glibc would be required for all non-glibc targetting SDKs as well. Fixes SDK install time error ERROR: OE-core's config sanity checker detected a potential misconfiguration. Either fix the cause of this error or at your own risk disable the checker (see sanity.conf). Following is the list of potential problems / advisories: Your system needs to support the en_US.UTF-8 locale. ERROR: SDK preparation failed (From OE-Core rev: 4a17afb3fe42cbc01c52b2d5357f6021bf782c01) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "OpkgPM: use --add-ignore-recommends to process BAD_RECOMMENDATIONS"Armin Kuster2019-10-172-2/+41
| | | | | | | | | | | | | This reverts commit e8cd30ba6cec854d85c7ad47edc208107858a5d7. This backport introduced an issue not seen the AB QA. Issue can be seen if BAD_RECOMMENDATIONS_append = " udev-hwdb" is used (From OE-Core rev: 5110080fbecd3f1cf43797c7eeb742951d88d1a8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/context: ensure log directory existsChen Qi2019-10-101-0/+1
| | | | | | | | | | | | | Ensure log directory exists to avoid the following error. FileNotFoundError: [Errno 2] No such file or directory: '/.../build-selftest/tmp/log/oe-selftest-results-20181207043431.log' (From OE-Core rev: c54411d0e03fe1cea8b6bb0c80dea029dd264f36) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildhistory: call a dependency parser only on actual dependency listsAlexander Kanavin2019-10-081-1/+1
| | | | | | | | | | | | Previously it was also called on filelists and possibly other items which broke the parser. (From OE-Core rev: f965ecbf558b6db1959e4ba8e599d65a5c8022b2) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* OpkgPM: use --add-ignore-recommends to process BAD_RECOMMENDATIONSAlejandro del Castillo2019-07-292-41/+2
| | | | | | | | | | | | | | | | | | Currently, BAD_RECOMMENDATIONS on the opkg backed relies on editing the opkg status file (it sets BAD_RECOMMENDATIONS pkg want state to deinstalled and pinned). This is brittle, and not consistent across the different solver backends. Use new --add-ignore-recommends flag instead. (From OE-Core rev: 0d11e813ba9b4e8de9e6e5099ff85f5d914243bc) (From OE-Core rev: bfb0acb6bc6bc11e4aa2c9527916359e1a763e85) (From OE-Core rev: 13ba66338d16cc07cb0129de932f090d0edb7760) Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/targetcontrol.py: fix qemuparams not work in runqemu with launch_cmdHongxu Jia2019-05-221-1/+1
| | | | | | | | | | | | | | | | | | | As runqemu with launch_cmd means directly run the command, don't need set rootfs or env vars. Since commit [a847dd7202 runqemu: Let qemuparams override default settings] applied in oe-core, if launch_cmd contains "qemuparams='***'", it does not work, which is overridden by latter qemuparams="-serial tcp:127.0.0.1" in QemuRunner.launch(); So we set qemuparams as a parameter in runqemu, the fix makes it work (From OE-Core rev: 7d4450d373a297f246b8c3708fd7d2cafadd3ae9) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: Replace suggogatepass with ignoring errorsRichard Purdie2019-05-221-2/+2
| | | | | | | | | | | | | We continued to see encoding problems with ssh commands run in oeqa. After much research the conclusion was we should use ignore the errors since some occasional bad locale encoding is better than the unicode decoding issues we were seeing which crashed large parts of tests. (From OE-Core rev: cf4d3230ac439118a2ad3c381bac38e11b3e14e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk: use bash to execute SDK test commandsMikko Rapeli2019-05-222-2/+3
| | | | | | | | | | | | | | | | | The commands only work with with bash. If /bin/sh is dash like in Debian, the command execution fails with errors like: Standard Output: /bin/sh: 5: export: --sysroot: bad variable name and all SDK tests fail. (From OE-Core rev: 274c22a1fd95418e4afb6633bb4b2e6debc4d7ea) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrenttest: Patch subunit module to handle classSetup failuresRichard Purdie2019-05-221-4/+23
| | | | | | | | | | | | | | | | | | | | | | | | Currently setupClass errors were not being mapped back to the failing tests and they were hence being marked as UNKNOWN and the test statistics were inaccurate. This is because whilst the errors were being encoded into the test results stream, the decoder doesn't cope with an error outside a testStart event. We patch in an addError handler to the outsideTest parser so that this does get handled in a way similar to the non-concurrent case. It would be nice if we didn't have to do this but there doesn't seem to be any other way to fix this other than forking subunit. We also make a minor change so another of our changes can cope with tests without a start time. (From OE-Core rev: 6f58c301e2d3463848df35c5b5c55d167ab34035) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Fix subunit setupClass/setupModule failure handlingRichard Purdie2019-05-221-2/+2
| | | | | | | | | | | The string format for subunit setupClass/setupModule failures is slightly different, tweak the regex to correctly handle both cases. (From OE-Core rev: f1009d6d44097a3f140e2f8679e9184031b10b44) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Handle unexpectedSucessesRichard Purdie2019-05-221-2/+8
| | | | | | | | | | | | | | Instead of showing: RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s) map unexpectedSuccesses to PASSED and improve the way they're displayed. We expect/allow ptest runner to fail but if it passes we should handle it correctly. (From OE-Core rev: 3c164b94fbb0efc513ee747cccd571a73688b541) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/ssh: Avoid unicode decode exceptionsRichard Purdie2019-05-221-2/+2
| | | | | | | | | | | | | | | | | This code really needs to be rewritten to not split potential multibyte characters, for now work around it to avoid exceptions like: File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run data = reader.read(1024, 4096) File "/usr/lib64/python3.6/codecs.py", line 503, in read newchars, decodedbytes = self.decode(data, self.errors) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte (From OE-Core rev: 53df0c93c46b6ab1366df44bd9927dc6f8699b29) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: dump stdout and stderr of each test caseMardegan, Alberto2019-05-221-4/+16
| | | | | | | | | | | | | | | | | | | | Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since `thud` we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests. We now restore this functionality by adding `stdout` and `stderr` keys to the JSON reports. This behavior is off by default; in order to enable it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake configuration. (From OE-Core rev: 68919826fe4d4db4582223569c5a939a9c2f9c72) Signed-off-by: Alberto Mardegan <amardegan@luxoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner: Fix typo in previous commitRichard Purdie2019-05-221-1/+1
| | | | | | | | (From OE-Core rev: 909baeb5901f7f212fbdd6c358c5ada6fe25e091) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemurunner: Dont mix binary and non-binary stringsRichard Purdie2019-05-221-0/+1
| | | | | | | | | | | self.msg is a str(), bootlog is b'' so this code clearly doesn't work. Add in a decode since its being used as a string. (From OE-Core rev: 5685a349eb471f4c8c9dcc2dc3b34f20369920bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* pkgdata.py: avoid target-sdk-dummy-provides to mess things upChen Qi2019-04-091-0/+1
| | | | | | | | | | | | | | | | | | | | | Sometimes we meet the following failure for the test_lookup_recipe test case. AssertionError: 'zlib\nbusybox is in the RPROVIDES of target-sdk-provid[32 chars]ummy' != 'zlib\nbusybox' zlib + busybox- busybox is in the RPROVIDES of target-sdk-provides-dummy: - target-sdk-provides-dummy This is because target-sdk-provides-dummy rprovides busybox. So clean things up to avoid failure. (From OE-Core rev: 7b5a069d4f11e36a1c407c4212aba5aafaf4ee7f) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/toaster: updated test id namingYeoh Ee Peng2019-04-092-93/+93
| | | | | | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file, where test_module match the file name itself. This file was using test_module name that does not match the file name itself. Fixed test_module name as well as the test_suite name. (From OE-Core rev: ebb3076c847c379d8c620d14927f696302fc4f26) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/toaster: transfer manual toaster test cases to oeqaYeoh Ee Peng2019-04-092-0/+3742
| | | | | | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia, manual test cases need to be migrated to OEQA. These manual test case json files will serve two use cases. Use case#1: as input to the future commandline-based manual test runner script, where this script will display actions and expected result information in user friendly text, then it will capture user input for test result and log, finally it will write test result and log into existing standardize test result json format from OEQA framework for automated tests. Use case#2: QA will open and read these json file manually for planning manual test execution. Any reader interested in understanding manual test cases will open and read these files. (From OE-Core rev: 498f84fee3c85227ffb8dba9723c2acec00e03bd) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/kernel-dev.json: test id updatedsangeeta jain2019-04-091-18/+18
| | | | | | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. (From OE-Core rev: 879f5aeac3ca4d053347e5a86a6c0fb303cf0dc3) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/compliance-test.json: test id updated and missing test cases addedsangeeta jain2019-04-091-4/+56
| | | | | | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/compliance-test.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. Added 2 missing test cases for Beaglebone Stress Test. (From OE-Core rev: 2af492e8b34e59fc58fe267da05f0868cb13ea0f) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-hw.json: test id updated and obsolete test cases removedsangeeta jain2019-04-091-220/+46
| | | | | | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/bsp-hw.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases. (From OE-Core rev: cfadaf037788f59dce84d569abaf3b1fb2d78824) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-hw.json: add non-IA testsMazliana2019-04-091-0/+260
| | | | | | | | | | | | | | QA team found that 10 manual bsp test cases from Testopia for Beaglebone, EdgeRouter, and MPC need to be up streamed [YOCTO #12650] (From OE-Core rev: f86542369fc5aabe4376e66b282981f35d437d5d) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/sdk.json: Updated toolchain tarball to core-image-satosangeeta jain2019-04-091-7/+7
| | | | | | | | | | | | | | | | | Manual test step for SDK is updated. Previously toolchain was "poky-glibc-x86_64-core-image-sato-sdk<type-arch>-toolchain-<release-version>.sh" But toochain for core-image-sato-sdk is not available in releases after 2.1, hence changed it to "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh Other less significant update is to add exepcted result in intermediate test steps. (From OE-Core rev: 88f9b2c5f86b05b42ae6429a5f74fa2f8b66e8ab) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh" Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual: Add manual runtime 'compliance' testcases to jsonSudhir Sharma2019-04-091-0/+142
| | | | | | | | | | | Added json file for compliance test cases to the manual runtime tests (From OE-Core rev: ff596eb0f0e84decdb5addadd3f1f16d62c25401) Signed-off-by: Sudhir Sharma <sudhirx.sharma@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/gitarchive: Handle case where parent is only on originRichard Purdie2019-03-261-2/+2
| | | | | | | | | | | | | | | The parent code currently assumed that any parent branch is locally checked out which may not be the case. Use the local branch by default but fall back to the origin. This also means removing the later saftey check as the branch may not exist locally. This fixes the autobuilder resulttool test pushing code. (From OE-Core rev: 36624a17d382d84647c5811134fe76251fd64dc3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improvements to allow integration to the autobuilderRichard Purdie2019-03-261-58/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a combined patch of the various tweaks and improvements I made to resulttool: * Avoid subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. * Avoid python keywords as variable names * Simplify dict accesses using .get() * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) * Fix oe-git-archive to use the commit/branch data from the results file * Cleaned up the command option help to match other changes * Follow the model of git branch/tag processing used by oe-build-perf-report and use that to read the data using git show to avoid branch change * Add ptest summary to the report command * Update the tests to match the above changes (From OE-Core rev: b4513e75f746a0989b09ee53cb85e489d41e5783) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: enable merge, store, report and regression analysisYeoh Ee Peng2019-03-262-0/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file. These scripts were developed as a test result tools to manage these testresults.json file. Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files. Using the "merge" operation, user can merge multiple testresults.json files to a target file. Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository. Using the "regression-file" operation, user can perform regression analysis on testresults.json files specified. Using the "regression-dir" and "regression-git" operations, user can perform regression analysis on directory and git accordingly. These resulttool operations expect the testresults.json file to use the json format below. { "<testresult_1>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, ... "<testresult_n>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, } To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To store test result from oeqa automated tests, execute the below $ resulttool store <source_dir> <git_branch> To merge multiple testresults.json files, execute the below $ resulttool merge <base_result_file> <target_result_file> To report test report, execute the below $ resulttool report <source_dir> To perform regression file analysis, execute the below $ resulttool regression-file <base_result_file> <target_result_file> To perform regression dir analysis, execute the below $ resulttool regression-dir <base_result_dir> <target_result_dir> To perform regression git analysis, execute the below $ resulttool regression-git <source_dir> <base_branch> <target_branch> [YOCTO# 13012] [YOCTO# 12654] (From OE-Core rev: 39cda3498ec68b91a671ead256231324d74f7d4c) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-report/gitarchive: Move common useful functions to libraryRichard Purdie2019-02-251-0/+71
| | | | | | | | | | | | | These functions can be reused by the resulttool code so move to the common function library for this purpose. (From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385) (From OE-Core rev: 4b1bd35030c5502873106782a35c4f5a9446e20c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-git-archive: Separate out functionality to library functionRichard Purdie2019-02-251-0/+173
| | | | | | | | | | | | | | | This turns the core of the script into a library function. Ultimately this will let us call that code with custom 'keywords' rather than relying on the data parsed from bitbake metadata which can't be used when archiving historical results. (From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0) (From OE-Core rev: e47c48768a01191ec36b9732288e05c9f5e80d52) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logrotate.py: restore /etc/logrotate.d/wtmpMingli Yu2019-02-251-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During the test logrotate.LogrotateTest.test_1_logrotate_setup, there is below logic: # mkdir $HOME/logrotate_dir # sed -i "s#wtmp {#wtmp {\n olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp After all logrotate.LogrotateTest finished, only cleanup $HOME/logrotate_dir as below, but don't restore the config file /etc/logrotate.d/wtmp. [snip] def tearDownClass(cls): cls.tc.target.run('rm -rf $HOME/logrotate_dir') [snip] That's to say, there is one additional line added to /etc/logrotate.d/wtmp and will make the logrotate service start failed when run systemd.SystemdBasicTests.test_systemd_failed Take an example as below when run test as root: # cat /etc/logrotate.d/wtmp # no packages own wtmp -- we'll rotate it here /var/log/wtmp { olddir /root/logrotate_dir missingok monthly create 0664 root utmp minsize 1M rotate 1 } # ls /root/logrotate_dir ls: cannot access '/root/logrotate_dir': No such file or directory # systemctl start logrotate Job for logrotate.service failed because the control process exited with error code. See "systemctl status logrotate.service" and "journalctl -xe" for details. # systemctl status logrotate logrotate.service - Rotate log files Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset> Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago Docs: man:logrotate(8) man:logrotate.conf(5) Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE) Main PID: 540 (code=exited, status=1/FAILURE) Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files... Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'. Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files. Add the logic to restore /etc/logrotate.d/wtmp to make the cleanup complete to fix the above issue. (From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f) (From OE-Core rev: 9036f78bdfde1d79af6ce57db78e2cdff9fb5677) Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/ptest: Ensure OOM errors are loggedRichard Purdie2019-02-251-1/+9
| | | | | | | | | | | | | | | Currently processed being killed by the OOM killer may not be spotted by ptest-runner. After we complete the tests, check the logs and report if there were any. This ensures the user is aware of OOM conditions affecting the ptest results. (From OE-Core rev: 20a441d53817f80e0ce1597e77f6e794422ac49a) (From OE-Core rev: f922970b119849bcdd7f7df74bd2029816ff637c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/logparser: Various misc cleanupsRichard Purdie2019-02-252-21/+13
| | | | | | | | | | | | | | | | | Get rid of further unneeded code complications: * value mappings we could just direct use * ftools when we can write files easily ourself * test result status filtering we don't use * variable overwriting module imports (From OE-Core rev: d6065f136f6d353c3054cc3f440a4e259509f876) (From OE-Core rev: ba944a72302fa088c31c7b1eee4ad9f64f9769e4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/logparser: Improve results handlingRichard Purdie2019-02-252-40/+19
| | | | | | | | | | | | | | | | | | | | | Merge the results handling into the ptest log parser as a seperate method. Drop the weird "pass.skip.fail." prefix to the results filename, its just bizarre. Drop the code turning a list into a regex then searching the regex for an item, "x in y" is perfectly capable. Use a dict, sort the keys as needed and drop the list sorting code. (From OE-Core rev: f317800e950b4a37b4034133bc52e0c47f04dc29) (From OE-Core rev: 966ffaada3e9f43a25b1361c53d4b16a521aa517) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/logparser: Add in support for duration, exitcode and logs by sectionRichard Purdie2019-02-252-7/+37
| | | | | | | | | | | | | | | Allow parsing of the ptest duration, exit code and timeout keywords from the logs, returning data on each section. Also include the logs broken out per section. (From OE-Core rev: a9a67dccaa5be0f06eedcab46dcff7cbf9202850) (From OE-Core rev: dc49021f75ed7e82713d1c9a04e045718bb9a548) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/logparser: Reform the ptest results parserRichard Purdie2019-02-252-62/+36
| | | | | | | | | | | | | | Now we have a dedicated ptest parser, merge in the remaining ptest specific pieces to further clarify and simplify the code, moving to a point where we can consider extending/enhancing it. (From OE-Core rev: 05991bb5bc8018275d03fdeecee3d5a757840c7c) (From OE-Core rev: e514c34195a7e1b2b7a1916ebd8c8ef631c60a01) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/logparser: Further simplification/clarificationRichard Purdie2019-02-252-23/+19
| | | | | | | | | | | | | Rename the paster to be ptest specific and apply some further cleanups to the code to simplify and clarify what its doing. (From OE-Core rev: 45a5886f1ec458d4c306b8d68fd31d568bc36b47) (From OE-Core rev: a4187e32f2cf90c0c7a155bef29fe558cea1edd9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/logparser: Simplify ptest log parsing codeRichard Purdie2019-02-252-58/+11
| | | | | | | | | | | | | | logparser is only used by ptest. Its slightly overcomplicated as it was intended to be reusable but wasn't. Simplify it as a dedicated parser is likely to me more readable and maintainable. (From OE-Core rev: c7478345b2b4a85cb1fec40e762633871f0e94cb) (From OE-Core rev: 1a6fcc97f3842deed3e78ededa8a21da274c0572) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/ptest: Avoid traceback for tests with no sectionRichard Purdie2019-02-251-1/+1
| | | | | | | | | | | | | Some tests end up without a section, avoid tracebacks trying to use None as a string in that case. (From OE-Core rev: 86fb5d898a29761f120c2eaa538a32cf2e078487) (From OE-Core rev: fca18914013b2f05b36e6985a7ff4a7c8dddf8a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/wic: update test case test_qemuKai Kang2019-02-251-2/+4
| | | | | | | | | | | | | | | | | | | | | This test case boots the image in qemu and checks for mounted partitions. But the outputs of mount are different between sysvinit and systemd: sysvinit: /dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt systemd: /dev/sda1 /boot\r\n/dev/sda2 /\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt So check mounted partitions by egrep rather than check output of runqemu. (From OE-Core rev: 8e3311d45ec68d74e934e966f67cf5102634981c) (From OE-Core rev: 828db9026c61d0a6ad12b8e05929eef31abfd57d) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* perl: add testdepends for sshArmin Kuster2019-02-251-0/+2
| | | | | | | | | | | | | | | | | | | fixes: DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.4 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; perl -e '$_="Uryyb, jbeyq"; tr/a-zA-Z/n-za-mN-ZA-M/;print' | DEBUG: time: 1548816904.4024463, endtime: 1548817204.397057 | DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.4 port 22: Connection refused for master/thud/sumo (From OE-Core rev: a590e7805e3bec5dd995f7ea0b9e79a21f82b48b) (From OE-Core rev: 7dbccf13493dad534c0cc604bc4c5ab16e5f0fc0) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eSDK.py: unset BBPATH and BUILDDIR to avoid eSDK failureChen Qi2019-02-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | When executing eSDK test case, the following error appears. WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead. FileExistsError: [Errno 17] File exists: '/.../tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-fcuyzsqu/tmp/sysroots/x86_64/bin/pigz' -> '/.../tmp/hosttools/pigz' So unset these two vars to avoid messing things up. (From OE-Core rev: bc07d825ce8bb3b337623c232fef61f2781c82af) (From OE-Core rev: 748f946ee74f7480200a7eb0bb0b695467b08f0a) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eSDK.py: avoid error in tearDownClass due to race condistionChen Qi2019-02-061-0/+6
| | | | | | | | | | | | | | | | | | | | When removing the temporary directory, it's possible that bitbake.lock file is removed by bitbake during the cleanup. And this leads to the following error. FileNotFoundError: [Errno 2] No such file or directory: 'bitbake.lock' So add a check to remove this file before cleaning up the temporary directory. (From OE-Core rev: 984f56b37bd0014e5bf9509fc8ed181973e61773) (From OE-Core rev: 6a5cce862f1e950439dc8ee1d84a10397a7fac4f) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/buildproject: Only clean files if we've done somethingRichard Purdie2019-02-061-0/+5
| | | | | | | | | | | | | We should only be wiping out things on target if the tests have actually run. (From OE-Core rev: d38c3eac0a5a1a9b0eb98385832e92f48145655e) (From OE-Core rev: 34efb674999781448411a04f876d6b3423ebe26c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/runqemu: Enable kvm when QEMU_USE_KVM is setRobert Yang2019-02-061-0/+5
| | | | | | | | | | | (From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da) (From OE-Core rev: ffea871575d0e80f73daf6b36c90798ce5e055e6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-qemu.json: Update for QEMU_USE_KVMRobert Yang2019-02-061-2/+2
| | | | | | | | | | | | | Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more. (From OE-Core rev: 37e5035a2ba27595eeef22062580a5b04a73ba15) (From OE-Core rev: 63f0ce0e084d2d24daeacf226d9c432e4a53fd21) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemu & runtime: qemu do not need ip input from externalYeoh Ee Peng2019-02-062-4/+3
| | | | | | | | | | | | | | | Qemu do not use the ip input from external. It will retrieve ip from QemuRunner instance and assign ip value. (From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210) (From OE-Core rev: c3d51e92f412becd3e067c637ffbdc5ad13e8894) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-12 17:08:28 +0000