summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa
Commit message (Collapse)AuthorAgeFilesLines
* oeqa/manual/toaster: updated test id namingYeoh Ee Peng2019-04-252-93/+93
| | | | | | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file, where test_module match the file name itself. This file was using test_module name that does not match the file name itself. Fixed test_module name as well as the test_suite name. (From OE-Core rev: 05e08a64771e40c147bdd6f02492b85edc01e946) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/toaster: transfer manual toaster test cases to oeqaYeoh Ee Peng2019-04-252-0/+3742
| | | | | | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia, manual test cases need to be migrated to OEQA. These manual test case json files will serve two use cases. Use case#1: as input to the future commandline-based manual test runner script, where this script will display actions and expected result information in user friendly text, then it will capture user input for test result and log, finally it will write test result and log into existing standardize test result json format from OEQA framework for automated tests. Use case#2: QA will open and read these json file manually for planning manual test execution. Any reader interested in understanding manual test cases will open and read these files. (From OE-Core rev: 73f5446c38a5d7766c81cda7a7bb22df9cfe0f23) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/kernel-dev.json: test id updatedsangeeta jain2019-04-251-18/+18
| | | | | | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. (From OE-Core rev: 35233e411db1a5978bfb2f4265b8e3a8fbdc4246) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/compliance-test.json: test id updated and missing test cases addedsangeeta jain2019-04-251-4/+56
| | | | | | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/compliance-test.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. Added 2 missing test cases for Beaglebone Stress Test. (From OE-Core rev: 7d95d4c6d30ae2977cd9feb5a45f96d05ff9b763) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-hw.json: test id updated and obsolete test cases removedsangeeta jain2019-04-251-220/+46
| | | | | | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/bsp-hw.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases. (From OE-Core rev: 862c96a98d2087cf05fbfeeafb57dcef1225d978) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-hw.json: add non-IA testsMazliana2019-04-251-0/+260
| | | | | | | | | | | | | | QA team found that 10 manual bsp test cases from Testopia for Beaglebone, EdgeRouter, and MPC need to be up streamed [YOCTO #12650] (From OE-Core rev: 05f785937be3d06f38cff4de238d7bd85d4a452a) Signed-off-by: Mazliana <mazliana.mohamad@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/sdk.json: Updated toolchain tarball to core-image-satosangeeta jain2019-04-251-7/+7
| | | | | | | | | | | | | | | | | Manual test step for SDK is updated. Previously toolchain was "poky-glibc-x86_64-core-image-sato-sdk<type-arch>-toolchain-<release-version>.sh" But toochain for core-image-sato-sdk is not available in releases after 2.1, hence changed it to "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh Other less significant update is to add exepcted result in intermediate test steps. (From OE-Core rev: 4e26c3a6aa647a322a8e8a3027a4114caed38f7d) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh" Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual: Add manual runtime 'compliance' testcases to jsonSudhir Sharma2019-04-251-0/+142
| | | | | | | | | | | Added json file for compliance test cases to the manual runtime tests (From OE-Core rev: 3f143119def5a84430762790f69b426fdc8ad281) Signed-off-by: Sudhir Sharma <sudhirx.sharma@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-qemu.json: Update for QEMU_USE_KVMRobert Yang2019-04-251-2/+2
| | | | | | | | | | | Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more. (From OE-Core rev: cd3c467b9419238494e8c525e86be0cf85321c4e) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/build-appliance: Drop test which is already automatedRichard Purdie2019-04-251-26/+0
| | | | | | | | | | The nightly-build-appliance target already tests this particular build. (From OE-Core rev: 90d9542900b6a941a3c3355c15dbfc2e2ac78232) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual: Remove bitbake-selftest executionRichard Purdie2019-04-251-30/+0
| | | | | | | | | | | The autobuilder runs bitbake-selftest already so we don't need to execute this manually. (From OE-Core rev: 772b2b883249c8606b2df380f749320cb683a7b2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual: transfer manual test cases from testopia to oeqaYeoh Ee Peng2019-04-259-0/+2750
| | | | | | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia, manual test cases need to be migrated to OEQA. These manual test case json files will serve two use cases. Use case#1: as input to the future commandline-based manual test runner script, where this script will display actions and expected result information in user friendly text, then it will capture user input for test result and log, finally it will write test result and log into existing standardize test result json format from OEQA framework for automated tests. Use case#2: QA will open and read these json file manually for planning manual test execution. Any reader interested in understanding manual test cases will open and read these files. (From OE-Core rev: a1675c33638288363b243e7412fe4cea63756ae6) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/gitarchive: Handle case where parent is only on originRichard Purdie2019-03-271-2/+2
| | | | | | | | | | | | | | | The parent code currently assumed that any parent branch is locally checked out which may not be the case. Use the local branch by default but fall back to the origin. This also means removing the later saftey check as the branch may not exist locally. This fixes the autobuilder resulttool test pushing code. (From OE-Core rev: 633e024f5d492ba38debf745b59813f6a2f4a3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-report/gitarchive: Move common useful functions to libraryRichard Purdie2019-03-261-0/+71
| | | | | | | | | | | These functions can be reused by the resulttool code so move to the common function library for this purpose. (From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385) (From OE-Core rev: 94a3ca85d30fc957f8f01a216a75342be49f9143) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-git-archive: Separate out functionality to library functionRichard Purdie2019-03-261-0/+173
| | | | | | | | | | | | | This turns the core of the script into a library function. Ultimately this will let us call that code with custom 'keywords' rather than relying on the data parsed from bitbake metadata which can't be used when archiving historical results. (From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0) (From OE-Core rev: ab5d49ba8bb02410fb9f3024f98ff19a04acb3ca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improvements to allow integration to the autobuilderRichard Purdie2019-03-261-58/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a combined patch of the various tweaks and improvements I made to resulttool: * Avoid subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. * Avoid python keywords as variable names * Simplify dict accesses using .get() * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) * Fix oe-git-archive to use the commit/branch data from the results file * Cleaned up the command option help to match other changes * Follow the model of git branch/tag processing used by oe-build-perf-report and use that to read the data using git show to avoid branch change * Add ptest summary to the report command * Update the tests to match the above changes (From OE-Core rev: e4195565d2a50046d4378c97f7a593c41bed51bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: enable merge, store, report and regression analysisYeoh Ee Peng2019-03-262-0/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file. These scripts were developed as a test result tools to manage these testresults.json file. Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files. Using the "merge" operation, user can merge multiple testresults.json files to a target file. Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository. Using the "regression-file" operation, user can perform regression analysis on testresults.json files specified. Using the "regression-dir" and "regression-git" operations, user can perform regression analysis on directory and git accordingly. These resulttool operations expect the testresults.json file to use the json format below. { "<testresult_1>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, ... "<testresult_n>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, } To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To store test result from oeqa automated tests, execute the below $ resulttool store <source_dir> <git_branch> To merge multiple testresults.json files, execute the below $ resulttool merge <base_result_file> <target_result_file> To report test report, execute the below $ resulttool report <source_dir> To perform regression file analysis, execute the below $ resulttool regression-file <base_result_file> <target_result_file> To perform regression dir analysis, execute the below $ resulttool regression-dir <base_result_dir> <target_result_dir> To perform regression git analysis, execute the below $ resulttool regression-git <source_dir> <base_branch> <target_branch> [YOCTO# 13012] [YOCTO# 12654] (From OE-Core rev: bb0bc6368bb51ac0be77d13fe931601d493951ea) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logrotate.py: restore /etc/logrotate.d/wtmpMingli Yu2019-03-241-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During the test logrotate.LogrotateTest.test_1_logrotate_setup, there is below logic: # mkdir $HOME/logrotate_dir # sed -i "s#wtmp {#wtmp {\n olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp After all logrotate.LogrotateTest finished, only cleanup $HOME/logrotate_dir as below, but don't restore the config file /etc/logrotate.d/wtmp. [snip] def tearDownClass(cls): cls.tc.target.run('rm -rf $HOME/logrotate_dir') [snip] That's to say, there is one additional line added to /etc/logrotate.d/wtmp and will make the logrotate service start failed when run systemd.SystemdBasicTests.test_systemd_failed Take an example as below when run test as root: # cat /etc/logrotate.d/wtmp # no packages own wtmp -- we'll rotate it here /var/log/wtmp { olddir /root/logrotate_dir missingok monthly create 0664 root utmp minsize 1M rotate 1 } # ls /root/logrotate_dir ls: cannot access '/root/logrotate_dir': No such file or directory # systemctl start logrotate Job for logrotate.service failed because the control process exited with error code. See "systemctl status logrotate.service" and "journalctl -xe" for details. # systemctl status logrotate logrotate.service - Rotate log files Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset> Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago Docs: man:logrotate(8) man:logrotate.conf(5) Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE) Main PID: 540 (code=exited, status=1/FAILURE) Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files... Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'. Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files. Add the logic to restore /etc/logrotate.d/wtmp to make the cleanup complete to fix the above issue. (From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f) (From OE-Core rev: a5dcf0c4bd9a0be2f2484b0b1b3e77f1f2128dfc) Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/dnf: Fix test error when static libs are enabledRichard Purdie2019-03-241-1/+1
| | | | | | | | | | | | | | | The test works by excluding curl-dev which curl-staticdev depends upon. When static libraries aren't disabled, this leads to an odd looking test failure. Simply exclude curl-staticdev as well in case its enabled to make sure the test always works. (From OE-Core rev: dfded083d9456c8d2bb168dbe8e08b2ffb3e1a26) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: devtool: Support meta being a symbolic linkPeter Kjellerstedt2019-02-251-1/+2
| | | | | | | | | | | | | | | oe-selftest's devtool tests have been broken since commit 2457cd57 (oe-selftest: devtool: avoid parallel races by using temporary copy of core) if meta is a symbolic link. (From OE-Core rev: daba6c5a991b370709d17e51305334f55a3858ec) (From OE-Core rev: 3eb59559ecd2e93fb590a330b47de1db0750fc0b) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: devtool: avoid parallel races by using temporary copy of corePaul Eggleton2019-02-251-0/+64
| | | | | | | | | | | | | | | | | | | | Some of the devtool tests make changes to files under meta/ - legitimately since we want these tests to be working with real recipes and associated files. Unfortunately with the new oe-selftest parallelisation this can break other tests if files go missing at the wrong time (among other scenarios). To avoid this issue, simply take a copy of the core repository and use that for these tests. (We copy the entire repository since changing the path of meta/ influences COREBASE and thus we need to have things like scripts/ alongside as well). (From OE-Core rev: 2457cd57b4195924ef127f497efa2f34f411e660) (From OE-Core rev: 9cb8353a4f0137823d6ed3e467db9dd7ead7b3de) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/httpserver: Rework to avoid hangs and improve loggingRichard Purdie2019-02-254-11/+32
| | | | | | | | | | | | | | | | | | | | | | testimage.bbclass installs a SIGTERM handler which conflicts with the use of multiprocessing here. This is paritcularly problematic if the http service is terminated before its started and hence before its had a chance to reset the default signal handler (as the code was written). Instead, temporarily remove testimage's handler whilst forking the http process which means the correct handler is installed and won't deadlock. Also take the opportunity to add in some log messages about the server start and shutdown so that future debugging is easier and its clearer what the code is doing. (From OE-Core rev: cc0471439aa0085ca87deccf061c5b676ef12388) (From OE-Core rev: 4eac9a5337d93b6cbd3916af97f62bb04881c9cd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/dnf: Make sure test_dnf_install does not skippedYeoh Ee Peng2019-02-251-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | During debugging dnf issue, we found that the test_dnf_install PASSED the testing even though the environment does not allow dnf install to run successfully. Further debugging had identified that current test_dnf_install will execute dnf install even when the package to be installed already exist, thus dnf install will just skipped and this test will PASSED even though it was not. To solve this, added additional logic to check if the package to be installed already exist, if yes then remove the package before actually run dnf install. This will make sure dnf install was tested as expected. (From OE-Core rev: 4f662b253f7313c4e02bfafb527cdac076b6309a) (From OE-Core rev: a80498e62898110e2ed6b01cbb9f5dd85995d13b) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* perl: add testdepends for sshArmin Kuster2019-02-251-0/+2
| | | | | | | | | | | | | | | fixes: DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.4 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; perl -e '$_="Uryyb, jbeyq"; tr/a-zA-Z/n-za-mN-ZA-M/;print' | DEBUG: time: 1548816904.4024463, endtime: 1548817204.397057 | DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.4 port 22: Connection refused for master/thud/sumo (From OE-Core rev: 76c66e061cdcdcbad73ed503668115120feb0ea3) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: rationalise Perl testsRoss Burton2019-02-253-48/+11
| | | | | | | | | | | | | | As with the Python test, this can be both better and faster. No need to copy a file, just run a one-liner. (From OE-Core rev: c6eef46747fe58bb2310be4f06d2fa9b67901d72) (From OE-Core rev: 9188ef8d1edbba8041a73d3bb8a9bfd194db0e92) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/buildproject: Only clean files if we've done somethingRichard Purdie2019-02-251-0/+5
| | | | | | | | | | | | | We should only be wiping out things on target if the tests have actually run. (From OE-Core rev: d38c3eac0a5a1a9b0eb98385832e92f48145655e) (From OE-Core rev: dab22dc58eabaeb421afa3c7de1cc08c5ec34c61) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/runqemu: Enable kvm when QEMU_USE_KVM is setRobert Yang2019-02-251-0/+5
| | | | | | | | | | | (From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da) (From OE-Core rev: 1207949c986cc9f6b3940a99e87bc4d8cfb86d5b) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemu & runtime: qemu do not need ip input from externalYeoh Ee Peng2019-02-252-4/+3
| | | | | | | | | | | | | | | Qemu do not use the ip input from external. It will retrieve ip from QemuRunner instance and assign ip value. (From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210) (From OE-Core rev: e4990ae01c9f3f486b4b745a2602795e9d496109) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/cases: Improve dependencies of kernel/gcc/build testsRichard Purdie2019-02-256-30/+31
| | | | | | | | | | | | | Mark up these tests as needing a compiler, make and kernel source code as appropriate, the image feature requirements can then be retired. (From OE-Core rev: 9f64e6c25abdf494fb511e9cd401f8dcaa08be2a) (From OE-Core rev: 544c56b6950f0598d5eaf62d9ae9d9de5fcb7eaf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/cases: Improve test dependency informationRichard Purdie2019-02-2510-25/+32
| | | | | | | | | | | | | | | | Add the OEHasPackage decorator to a variety of tests so they determine automatically if they should run against a given image. To ensure tests can do this we need to move target operations such as scp commands into the tests and out of the class startup/teardown. (From OE-Core rev: 60d6580b85714b8960a964e775d76a7f937f5e5a) (From OE-Core rev: 03b7658369bb7c1c8fbbaac7d9e281617cc16135) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/python: clean up Python testRoss Burton2019-02-251-30/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently this is three test cases: 1) test_python_exists. Fail if python3 isn't in PATH. 2) test_python_stdout. Run a Python script and check the output is as expected 3) test_python_testfile. Check that a file test_python_stdout wrote to exists. (1) should be a setup and skip the test module if it isn't present. (2) and (3) should be merged, there's no point copying over a two line Python file, and the test doesn't verify that the file doesn't exist in the first place. Rewrite the test to check that Python is present in a class setup so the entire test is skipped if it isn't and do some simple rot13 to verify that bytecode is being executed correctly. (From OE-Core rev: a35be5f32b4fe70b18ac1e2eccfd94558cecfbba) (From OE-Core rev: babdd2f6addb06c23e4882107be07034ca49bcb8) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/cases/python: use python 3 rather than python 2Alexander Kanavin2019-02-251-3/+3
| | | | | | | | | | | | | | For example, core-image-sato skipped the test alltogether, as it no longer pulls in Python 2.x at all. (From OE-Core rev: 5ad0fe9ac6b6362011a17afaa7bee8e788093915) (From OE-Core rev: bde50fc78ae75fd585f8914e458d65c328857fb2) Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/cases/rpm.py: change to use base-passwd-docChen Qi2019-02-251-13/+13
| | | | | | | | | | | | | | | | | | | | | The test cases assume that rpm-doc is built out, as it says it its comment. This is not always true. And it sometimes results in following error. | cls.tc.target.copyTo(test_file, dst) | UnboundLocalError: local variable 'test_file' referenced before assignment Change to use base-passwd-doc, as this package is more likely to be built out than rpm-doc. (From OE-Core rev: 2e23543b48921182307065c1fa9e8b9d7fbb3cdc) (From OE-Core rev: 1ce378ce07d2c49f40054893a623456c8471e177) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: replace Alex Kanavin's @intel email address with a personal oneAlexander Kanavin2019-02-254-6/+6
| | | | | | | | | | | | | | As I will be leaving Intel, this address will no longer be valid, so swap it for my personal one for now. (From OE-Core rev: ea58ff101e28dfda3410de66d775df3d8a1e5a96) (From OE-Core rev: 61bca5a7f6b3fc03cd88cbf8867708d28e0a3522) Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: make it work for multiple usersRobert Yang2019-02-253-5/+4
| | | | | | | | | | | | | | | | | | | | | | | | | There are failures when multiple users run oe-selftest on the same host: PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved- tests/201812250324_qemu' This is because /tmp/oe-saved-tests was created by user A, while user B tries to write data in it, then the error will happen. This patch can fix the problem. Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than /tmp/oe-saved-tests/ to fix the problem. (From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980) (From OE-Core rev: 872c6e5101f4f27dcfc63d141d1b6568f46d2b5f) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runtime/dnf: Add new dnf test casesJose Perez Carranza2019-02-251-0/+40
| | | | | | | | | | | | | | | | | Add test cases to test “exclude” and “installroot“ options, also modify the logic of filtering packages on the feed to have all the packages needed by the tests. [YOCTO #10744] (From OE-Core rev: 1121806603c6f621d084b692216f3f616a0768dc) (From OE-Core rev: e1b050f53ece2a31cd6866d2d737d7c67a44cea4) Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Fix for QEMU_USE_KVMRobert Yang2019-02-251-7/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: MACHINE = "qemux86" QEMU_USE_KVM = "qemux86" IMAGE_CLASSES += "testimage" $ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs [snip] File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean raise ValueError("Invalid boolean value '%s'" % value) ValueError: Invalid boolean value 'qemux86' Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm will be enabled if target_arch == build_arch or both of them are x86 archs. (From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44) (From OE-Core rev: de1b80f7f7b787f6b5b62c576ca6c62d2440031c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation ↵Richard Purdie2019-02-251-1/+2
| | | | | | | | | | | | | | warnings Use our own lsb function instead as used elsewhere by the codebase. (From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3) (From OE-Core rev: 570256a64af5a3fa994a20a5cc4c74d59ffc361f) (From OE-Core rev: d58fe9d352ae7de857e7f55b88f6e7d35b2cd706) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner: set timeout to 60s for run_serialRobert Yang2019-02-253-3/+3
| | | | | | | | | | | | | | The 5s timeout for non-kvm is too short, especially when the load is high, which leads to unexpected errors, so set timeout to 60s by default. (From OE-Core rev: 8197d0f638a760fc03062c7a9009117d083d7ead) (From OE-Core rev: 6c930a8f5c11947c3b916efe23f33af462701bee) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner: Print output when failed to loginRobert Yang2019-02-251-0/+1
| | | | | | | | | | | | | This is useful for debugging. (From OE-Core rev: da527d73f5eeae0f29b5f99aab757491d3f87ec7) (From OE-Core rev: 7fb953573635afb1b6482dbd9f51eb0f05731eae) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/esdk: Fix typo causing test failureRichard Purdie2018-12-161-1/+1
| | | | | | | | | | | | | 2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last): File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass clss.setUpClassMethod() File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass cls.tmpdir_eSDKQA = cls.tempdirobj.name AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj' (From OE-Core rev: 2c60908a2039d333a9fe2651622750ff6ed4cce1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/esdk: Ensure parent directory existsRichard Purdie2018-12-161-0/+1
| | | | | | | | | | | | | | | | | | | | INFO - ====================================================================== INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest) INFO - ---------------------------------------------------------------------- INFO - Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass clss.setUpClassMethod() File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"]) File "/usr/lib/python3.5/tempfile.py", line 929, in __init__ self.name = mkdtemp(suffix, prefix, dir) File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp _os.mkdir(file, 0o700) FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc' (From OE-Core rev: 170a601a99836d13b69e5287bee0d3e71983dd46) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* image-buildinfo,oeqa/selftest/containerimage: Ensure image-buildinfo doesn't ↵Richard Purdie2018-12-161-0/+1
| | | | | | | | | | | | | | | | | | | | | break tests Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files to fail due to the presence of an unexpected file: ['./', './etc/', - './etc/build', './etc/default/', './etc/default/postinst', Tweak the class to allow it to be disabled and disable it from the test just in case it was enabled. (From OE-Core rev: af67bf422a4df5b7e07894512ff73a5f493682ab) (From OE-Core rev: f49ab8b1610c045acaed7b964d12f07f969df856) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/metadata: Allow to function without the git moduleRichard Purdie2018-12-161-2/+15
| | | | | | | | | | | | The python git module may or may not be enabled, allow this code to function without it, falling back to the same method as metadata_scm.bbclass uses. This will be cleaned up in the next round of feature development. (From OE-Core rev: 6350586ba9f4a4107a2d457590824cd4d662d5b9) (From OE-Core rev: 32c9169b76e13e53b6a9ab4a59932cea7863d992) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Standardize json logging output directoryYeoh Ee Peng2018-12-161-1/+1
| | | | | | | | | | | | Currently sdk & sdkext will output json file to LOG_DIR, while selftest will output json file to TOPDIR/log. Standardize selftest json output file to LOG_DIR. (From OE-Core rev: 9f2e39684cbbe9f87eeef6a81961e6db783439e3) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Improvements to the json loggingRichard Purdie2018-12-161-14/+11
| | | | | | | | | | | | | | | Tweak the preceeding commit to: * Add STARTTIME to the identifier to make it unique * Use LOG_DIR * Store the layer config in a more natural json format * Drop '_' function prefixes (From OE-Core rev: 173f59acf9722e2ef27fdd49c20f7d3d664917eb) (From OE-Core rev: 3b69099edc7db99c11bfb41eab2af50bd0e3d4f2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/context: write testresult to json filesYeoh Ee Peng2018-12-161-1/+29
| | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia to store testresult, OEQA selftest need to output testresult into json files, where these json testresult files will be stored into git repository by the future test-case-management tools. By default, json testresult file will be written to "oeqa" directory under the oe-selftest log directory. To configure multiple instances of bitbake to write json testresult to a single testresult file at custom directory, user will define the variable "OEQA_JSON_RESULT_DIR" with the custom directory for json testresult. (From OE-Core rev: a95218525a4c8228fff9908ffbda85c6b85e101c) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Sort the test result output by result classRichard Purdie2018-12-161-2/+14
| | | | | | | | | | | We want to see failures/errors listed last since this is the most easily visible part of the log on consoles or autobuilder output and makes human processing easier rather than having to scroll up and scan for a single failure. (From OE-Core rev: 2cc07ab253f1ba6a1f07a66051c9ba6d98cd2357) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Always show a summary of success/fail/error/skip countsRichard Purdie2018-12-161-3/+1
| | | | | | | | | Its useful to have the counts of success/failure/error/skipped at the end of the results to allow for easier human reading of what happened. (From OE-Core rev: 5942318a261ce7a885f351e214669068ff9d8931) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/ptest: Inject results+logs into stored json results fileRichard Purdie2018-12-162-2/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the ptest results from ptest-runner, run in an image to be transferred over to the resulting json results output. Each test is given a pass/skip/fail so individual results can be monitored and the raw log output from the ptest-runner is also dumped into the results json file as this means after the fact debugging becomes much easier. Currently the log output is not split up per test but that would make a good future enhancement. I attempted to implement this as python subTests however it failed as the output was too confusing, subTests don't support any kind of log output handling, subTest successes aren't logged and it was making things far more complex than they needed to be. We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest will pass currently and we don't want that to fail the whole image test run. Its assumed there would be later analysis of the json output to determine regressions. We do have to change the test runner code so that 'unexpectedsuccess' is not a failure. Also, the test names are manipuated to remove spaces and brackets with "_" used as a replacement and any duplicate occurrences truncated. (From OE-Core rev: a13e088942e2a3c3521e98954a394e61a15234e8) (From OE-Core rev: 526ceab9d0e43f73635bb92e8dd7763ef75ad33b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-12 08:54:26 +0000