summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/core
Commit message (Collapse)AuthorAgeFilesLines
...
* oeqa/runner: Sort the test result output by result classRichard Purdie2018-11-141-1/+10
| | | | | | | | | | | We want to see failures/errors listed last since this is the most easily visible part of the log on consoles or autobuilder output and makes human processing easier rather than having to scroll up and scan for a single failure. (From OE-Core rev: 7954b19020c28a4120bc1671aa81b9e1e2b05fa2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Always show a summary of success/fail/error/skip countsRichard Purdie2018-11-141-3/+1
| | | | | | | | | Its useful to have the counts of success/failure/error/skipped at the end of the results to allow for easier human reading of what happened. (From OE-Core rev: 080d8900d470a8e7f929b0c5c2765ad461744fbb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/ptest: Inject results+logs into stored json results fileRichard Purdie2018-11-071-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the ptest results from ptest-runner, run in an image to be transferred over to the resulting json results output. Each test is given a pass/skip/fail so individual results can be monitored and the raw log output from the ptest-runner is also dumped into the results json file as this means after the fact debugging becomes much easier. Currently the log output is not split up per test but that would make a good future enhancement. I attempted to implement this as python subTests however it failed as the output was too confusing, subTests don't support any kind of log output handling, subTest successes aren't logged and it was making things far more complex than they needed to be. We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest will pass currently and we don't want that to fail the whole image test run. Its assumed there would be later analysis of the json output to determine regressions. We do have to change the test runner code so that 'unexpectedsuccess' is not a failure. Also, the test names are manipuated to remove spaces and brackets with "_" used as a replacement and any duplicate occurrences truncated. (From OE-Core rev: a13e088942e2a3c3521e98954a394e61a15234e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Don't add empty log entriesRichard Purdie2018-10-291-1/+4
| | | | | | | | | There is no point in adding empty log entries to the json result files, only add them if there is log data. (From OE-Core rev: da6aef0b6bd4655805f7f743858d4e04341f75ed) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: write testresult to json filesYeoh Ee Peng2018-10-291-1/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia to store testresult, OEQA need to output testresult into single json file, where json testresult file will be stored in git repository by the future test-case-management tools. The json testresult file will store more than one set of results, where each set of results was uniquely identified by the result_id. The result_id would be like "runtime-qemux86-core-image-sato", where it was a runtime test with target machine equal to qemux86 and running on core-image-sato image. The json testresult file will only store the latest test content for a given result_id. The json testresult file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE), result (eg. PASSED, FAILED, ERROR), test log, and result_id. Based on the destination json testresult file directory provided, it could have multiple instances of bitbake trying to write json testresult to a single testresult file, using locking a lockfile alongside the results file directory to prevent races. Also the library class inside this patch will be reused by the future test-case-management tools to write json testresult for manual test case executed. (From OE-Core rev: 00e03b5004f1eb6d59295544b3a8620504278f51) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: refactor for OEQA to write json testresultYeoh Ee Peng2018-10-181-41/+29
| | | | | | | | | | | | | | Refactor the original _getDetailsNotPassed method to return testresult details (test status and log), which will be reused by future OEQA code to write json testresult. Take the opportunity to consolidate and simplify the logic used to gather test status and log within the TestResult instance. (From OE-Core rev: 79ee7d1c371a86edeb61c99679985118da657e5d) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator: add skipIfInDataVarRichard Leitner2018-08-231-0/+14
| | | | | | | | | skipIfInDataVar will skip a test if a value is in a certain variable. (From OE-Core rev: 10b935c713748346aea6c36c2f41e0ae6c320821) Signed-off-by: Richard Leitner <richard.leitner@skidata.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Print any errors/failures earlyRichard Purdie2018-08-231-0/+8
| | | | | | | | | Its a pain to have to wait until oe-selftest finishes to see the failures for example. (From OE-Core rev: 4c499a1b10a0c2647b6a753b8f9cd934ae4ad0da) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Use the proper logger functions instead of print()Richard Purdie2018-08-231-1/+1
| | | | | | (From OE-Core rev: 5ccd2284e5dd994230e9e229b7931d049c9f46c0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/context: Only set buffer mode for non-concurrent testsRichard Purdie2018-08-231-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Periodically we'd see: NOTE: core-image-sato-1.0-r0 do_testsdk: ====================================================================== NOTE: core-image-sato-1.0-r0 do_testsdk: ERROR: broken-runner NOTE: core-image-sato-1.0-r0 do_testsdk: ---------------------------------------------------------------------- NOTE: core-image-sato-1.0-r0 do_testsdk: testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/lib/oeqa/core/utils/concurrencytest.py", line 122, in _run_test test.run(process_result) File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 1194, in run protocol = TestProtocolServer(result, self._passthrough, self._forward) File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 514, in __init__ stream = stream.buffer AttributeError: '_io.StringIO' object has no attribute 'buffer' which seems to occur if a result arrives before all the runner threads have started. The runner's result handling changes sys.stdout to a buffer temporarily which can be seen in other threads and it can sometimes fail. Since the tests are running in a separate process we don't need this buffer handling in the concurrent case so only set when not parallelising. The concurrent class handle setting buffer mode internally. (From OE-Core rev: e2c66aa0f74d19fbc7e56af1fb440ad6473d3a8e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Ensure subunit streams are flushed at exitRichard Purdie2018-08-231-0/+2
| | | | | | | | | | Without this, error output such as that in the teardown can be lost and processes may recieve signals they're not expecting causing other strange errors. (From OE-Core rev: 1e3f44737a15feb3128ba7fc0dbe896dd8782e07) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/loader: Ensure invalid test names don't trigger a tracebackRichard Purdie2018-07-261-0/+2
| | | | | | | | | | | oe-selftest -r <invalid test name> currently triggers a traceback. Ensure this doesn't happen and the user gets a sensible error message. (From OE-Core rev: f2042bf3638ed4edfb167e7f7d4be6da60997ead) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/decorator: Improve reliabilityRichard Purdie2018-07-181-7/+9
| | | | | | | | | | | | Checking if the dependency had any failure is unreliable, for example if the underlying data doesn't get transferred and the list is empty, success of the dependency is assumed. Since we now have success data available, change the code to use it. (From OE-Core rev: 4abba4c30d5a6163a968a119395a679e5e281ab4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Remove xmlrunnerRichard Purdie2018-07-181-45/+9
| | | | | | | | | | This isn't present on modern distros by default and doesn't work with testtools, needing multiple code paths in the code. Remove it in favour of finding a better replacement for results collection/analysis. (From OE-Core rev: 8001d933a8dc86004db014777f094d718086687d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Simplify codeRichard Purdie2018-07-184-16/+8
| | | | | | | | | There doesn't appear to be any reason we need this _results indirection any more so remove it. (From OE-Core rev: b618261811c48ff3b98eab1b340a8cd09ef183c6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/threaded: Remove in favour of using concurrenttestsRichard Purdie2018-07-1810-404/+12
| | | | | | | | | | | | | | | | | | | | | | | | | We have several options for parallel processing in oeqa, parallel execution of modules, threading and mulitple processes for the runners. After much experimentation is appears the most scalable and least invasive approach is multiple processes using concurrenttestsuite from testtools. This means we can drop the current threading code which is only used by the sdk test execution. oeqa/decorator/depends: Remove threading code Revert "oeqa/sdk: Enable usage of OEQA thread mode" This reverts commit adc434c0636b7dea2ef70c8d2c8e61cdb5c703b1. Revert "oeqa/core/tests: Add tests of OEQA Threaded mode" This reverts commit a4eef558c9933eb32413b61ff80a11b999951b40. Revert "oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode" This reverts commit d3d4ba902dee8b19fa1054330cffdf73f9b81fe7. (From OE-Core rev: a98ab5e560e73b6988512fbae5cefe9e42ceed53) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Add selftest parallelisation supportRichard Purdie2018-07-183-6/+282
| | | | | | | | | | | | | | | | | | | This allows oe-selftest to take a -j option which specifies how much test parallelisation to use. Currently this is "module" based with each module being split and run in a separate build directory. Further splitting could be done but this seems a good compromise between test setup and parallelism. You need python-testtools and python-subunit installed to use this but only when the -j option is specified. See notes posted to the openedmbedded-architecture list for more details about the design choices here. Some of this functionality may make more sense in the oeqa core ultimately. (From OE-Core rev: 326ababfd620ae5ea29bf486b9d68ba3d60cad30) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Default to buffer mode for testsRichard Purdie2018-07-162-6/+1
| | | | | | | | | | | | Currently some tests run in buffer mode and some don't. Those that don't can corrupt stdout/stderr. Switch to using buffer mode everywhere so we're consistent. If there is useful output on stdout/stderr, it will be displayed if the test fails. (From OE-Core rev: 978548c0abde2cb94c2782538552f39bdf2bf630) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Ensure we don't print misleading results outputRichard Purdie2018-07-161-3/+11
| | | | | | | | | | | | | | The current code assumes if something isn't a failure of some kind, it was a pass. When test case IDs weren't matching, this lead to very confusing output where things would fail, then be listed as passing. This adds code to track successes, ensuring we don't end up in this position again with unmatched entries being listed as UNKNOWN. (From OE-Core rev: 4374c296d8963e4f6a1aa7bef7983ad0a1c2fcff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Improve test case comparisionRichard Purdie2018-07-161-2/+2
| | | | | | | | | | We can directly compare the test case IDs rather than representations, then if we're using subunit to split the tests, the comparisions still work as intended. (From OE-Core rev: 72e5f46f75454ba4c445c65c1cbc616a9e72fc6e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/target/ssh.py: increase maximum read bytes from 1024 to 4096Chen Qi2018-06-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | When running testimage task for core-image-sato-sdk, the following error appeared. UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 at position 0: invalid start byte Checking the codes, I found it's caused by setting a 1024 limit for the read method of the StreamReader object. Comments from the manual: """ The chars argument indicates the number of decoded code points or bytes to return. The read() method will never return more data than requested, but it might return less, if there is not enough available. """ When running `systemctl status --full' on target, this error occurs. This patch increase the bytes limit to 4096 to fix the error. (From OE-Core rev: f1fad60ae3be4450aca6058d5665fb10a9148b44) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator/data.py: fix skipIfNotInDataVarChen Qi2018-06-041-2/+2
| | | | | | | | | | | | | | | | | | | | | | The var might not be set, resulting in unexpected error. RESULTS - multilib.MultilibTest.test_check_multilib_libc - Testcase 1593: ERROR The above error is due to MULTILIBS being not set, which is the default for OE. This patch fixes this problem. Also, the debugging message in skipIfNotInDataVar is currently confusing. Instead of DEBUG: Checking if 'MULTILIBS' value is in 'multilib:lib32' to run the test it should be DEBUG: Checking if 'MULTILIBS' value contains 'multilib:lib32' to run the test This patch also fixes it. (From OE-Core rev: 3f5c678664a2bba43d99508779dc2ce227cf52a2) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator/__init__.py: use 'cls' instead of 'obj'Chen Qi2018-06-041-3/+3
| | | | | | | | | | Use 'cls' instead of 'obj' to better reflect that registerDecorator actually serves as a class decorator. (From OE-Core rev: e06e4c859e8be5225d80806a2ebe175f0b152fe1) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator/__init__.py: set metaclass to ABCMetaChen Qi2018-06-041-2/+2
| | | | | | | | | | | | | | | | | | OETestFilter is a subclass of OETestDecorator. It wants to make use of @abstractmethod decorator. But such decorator requires metaclass to be ABCMeta to have effect. So add it now to achieve the designed behaviour. Comments from python's manual: """ Using this decorator requires that the class's metaclass is ABCMeta or is derived from it. """ (From OE-Core rev: 28c4fafb2322ea8c37bcd7710f22f46ef552a902) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader.py: support the 'auto' keywordChen Qi2018-06-041-4/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | In previous OEQA, having 'auto' in TEST_SUITES results in executing as many test cases as possible. This behaviour is broken for now. From the codes in core/loader.py, I can see that it tries to use another keyword 'all'. But in fact, it does not work. I've checked the current manual. The manual says using 'auto'. Below is the current information in manual. """ Alternatively, you can provide the "auto" option to have all applicable tests run against the image. TEST_SUITES_append = " auto" """ So we should restore this behaviour. This patch does so. Also, output warning message is some module is named as 'auto', as this is a reserved keyword. (From OE-Core rev: a65460a063a958cc887c756db5f7ab18e3f5a8c1) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* core/loader.py: fix regex to include all available test casesLeonardo Sandoval2018-01-041-1/+1
| | | | | | | | | | | | Some test cases (eSDK.oeSDK*, runtime_test/*) does not match with current regex, fix it accept all. [YOCTO #12385] (From OE-Core rev: 1ecf48fd286a77078451b67879a44f9c9dc7a894) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemu: remove elf image typeSaul Wold2018-01-021-1/+1
| | | | | | | | | | | Since we are depercating the ELF image type, we should not test it here either. (From OE-Core rev: c0c10ae9f7e206ee156a68ddbed73c8820c37824) Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Pass the value of buffer, don't force to TrueRichard Purdie2017-11-111-1/+1
| | | | | | | | The value could be False in which case we should pass that through. (From OE-Core rev: d0a3379bbcbcd8153bd59ccdb56d40fff7ad6c6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: Drop command/output logging to debug levelRichard Purdie2017-11-091-1/+1
| | | | | | | | | This ensures the console is kept clear of confusing output but that the main logs contain good debugging information. (From OE-Core rev: caeb5dcfbd3c1d71f8e0eb78b3dd45d5ce349d25) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Clean up logger handlingRichard Purdie2017-11-091-1/+1
| | | | | | | | | | | | | | | | | | The logger handling in oeqa was confused at best. This patch: a) Passes in a logger through various qemu runner pieces b) Uses that logger consistently in the code c) Creates a logger for QemuRunner outside the bitbake namespace meaning we don't conflict with the tinfoil logging changes The result of this is more consistency. For runtime tests in testimage, the logs always contain the debug info, nothing is shwon on the console. For the oe-selftests, logs are intercepted and only shown if the test fails. (From OE-Core rev: 4ff678137a55b93c9ba2cbffda34335ba859f704) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runqemu: Only show stdout/stderr upon test failureRichard Purdie2017-11-091-0/+8
| | | | | | | | | | In general we don't need to see the output of runqemu however if it fails we do. Use the buffer option that already exists in TestResult but allow us to trigger it on a per test basis. (From OE-Core rev: 86bef95bccbe9dad3dc3445ea365439861966179) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: Make _built_modules_dict() support packages correctlyPeter Kjellerstedt2017-11-081-36/+14
| | | | | | | | | | | | | | | | For test modules in a package, e.g., oelib.license, running `oe-selftest -r oelib.license` or `oe-selftest -r oelib.license.TestSimpleCombinations` would fail with a message that the specified test cases could not be found. This was due to the parsing in _built_modules_dict(), which failed to distinguish between <package>.<module>.<class> and <module>.<class>.<testcase> and treated both cases as the latter. (From OE-Core rev: 80db3d999ae26d298d9d5418a32b11a4f27af9d5) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* core/target/ssh.py: use reader to handle partial dataRobert Yang2017-08-241-2/+3
| | | | | | | | | This can avoid UnicodeDecodeError error. (From OE-Core rev: baa78420d8d8e716935852c9c7b749af0161395a) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* core/target/ssh.py: replace decode errorsRobert Yang2017-08-231-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | There might be wild strings when read from target (especially when reading ptest results), replace the errors to avoid breaking the test. Fixed: (Not always happen) $ bitbake core-image-sato -ctestimage [snip] status, output = self.target.run('ptest-runner', 0) File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 84, in run status, output = self._run(sshCmd, processTimeout, True) File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 55, in _run status, output = SSHCall(command, self.logger, timeout) File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 258, in SSHCall run() File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 236, in run output = process.communicate()[0].decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4906: invalid continuation byte [YOCTO #11547] (From OE-Core rev: d0d2f892f0bed6adb5ec6fb59d64efcc97c83e19) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* context: Include a command line argument to run all except certain testsLeonardo Sandoval2017-08-131-3/+17
| | | | | | | | | | | | | | | | | | | | | A new command line argument (-R, which is the oposite of current -r) that allows to run all test cases except the ones indicated through the command line. Some command line examples: * Run all except the distro test case: $ oe-selftest -R distrodata * Run all except the archiver test case and a single bblayers unit test $ oe-selftest -R archiver bblayers.BitbakeLayers.test_bitbakelayers_add_remove [YOCTO #11847] (From OE-Core rev: e40eeaa790b95d9c25832405c0b0d5b3a0d0292b) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: OEStreamLogger don't buffer test execution writesAníbal Limón2017-07-301-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since OEQA framework uses Python logging functionality to report test results there is a class that wraps PyUnit writes into logging commands (OEStreamLogger), so don't buffer the actual test execution to have insight of what is currently executing. This fix will change a little the test output format adding an '\n' previous the test result, for example: From: test_nonmatching_checksum (lic_checksum.LicenseTests) ... ok To: test_nonmatching_checksum (lic_checksum.LicenseTests) ... ok This is because the new line added by the PyUnit StreamLogger because currently we don't have a manner to identify when a test execution starts at report level (write msg). [YOCTO #11827] (From OE-Core rev: 4c2276469f58a88f864eb374c00dbbaace702de4) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/{core,selftest}: Add support to validate if a specified test case isn't ↵Aníbal Limón2017-07-303-2/+36
| | | | | | | | | | | | | | | | | found If some test module/case is specified to run and isn't found the OEQA framework didn't notice it, so complete the implementation using modules_required and validate for the test case prescense. Raise an exception when the test module/case required isn't found. [YOCTO #11645] (From OE-Core rev: e50b415aaaa1581473f85f0a8afa278b5f95129b) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: Generate function _make_failed_test dynamicallyAníbal Limón2017-07-211-13/+12
| | | | | | | | | | | | | | | | | | | | | | Python versions has different features from branches 3.4.x, 3.5.x and 3.6.x, i expected in wrong mode that was incremental for example changes in 3.4.4 be in 3.5.x but that's not true. The _make_failed_test internal method differs and is only available in certain versions >= 3.4.4 and in 3.5.x and 3.6.x branches but not realeses have been made including it. So to avoid futher problems inspect the _make_failed_test and generates function definition according what parameters are needed, the unique supossition is that exception argument is always passed. Related to, http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d8380d098a290510b442a7abd2dd5a50cabf5844 (From OE-Core rev: b4740d2b325a80bcecc5e56dff9add9081fcd31b) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader.py: Fix _make_failed_test for python >= 3.4.4Aníbal Limón2017-06-231-9/+13
| | | | | | | | | | | Python unittest change the signature of the _make_failed_test after python 3.4.4 don't pass the method name. (From OE-Core rev: 767b68e6ca22512ff80e6fbc42154f3f0c2206c0) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/threaded: Don't assume that results exists on logDetailsAníbal Limón2017-06-231-3/+4
| | | | | | | | (From OE-Core rev: 7beed75c97a78e945e44a55b28f0f463cd6c8dcd) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: Use full and small module name on filteringAníbal Limón2017-06-131-9/+19
| | | | | | | | | | | | | The small module name was added to support run a whole suite that has more that 3 levels in the test case name, but this broke the behaviour for use a full test case name. [YOCTO #11632] (From OE-Core rev: 9ab20ceb5801bee8dd8b218b3928720da5e1d403) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Don't log details twice if test failsAníbal Limón2017-06-131-2/+0
| | | | | | | | | | | | The details of a test failure is upper on the unittest output so don't log twice the actual failure. [YOCTO #11622] (From OE-Core rev: 0f2e81c2a4458ad0ec6bab2710952ac2c2bbf1af) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: Allow unittest.TestCase's to be executedAníbal Limón2017-06-122-16/+21
| | | | | | | | | | | | | | | Currently there was a restriction to only execute tests that's inherits from OETestCase but in some circunstancies the features from the OEQA framework isn't needed so we need to support basic unittests. [YOCTO #10828] (From OE-Core rev: baac26f1b36e89e07637b738dd31ec7356f05a02) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: Fix filtering on test modules with submodulesAníbal Limón2017-06-121-36/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Our filtering allows to specify which tests to run using, <module_name>.[test_class].[test_name] But the module name logic was restricted to only accept one level, for example: runtime_test vs oelib.types, to support multiple submodules use only the first part for filtering. This allows to run the whole tests in a module with more than tree levels. Due to the ambiguity on the test filtering options with test cases with more than tree levels the supported sytnax is, <module> or <module>.[submoduleN].[test_class].[test_name] [YOCTO #11632] (From OE-Core rev: cf2ee12b007e5570959ccfbb643159b21d90426e) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Change the order to logDetails and logSummaryAníbal Limón2017-06-091-1/+1
| | | | | | | | | | | | | Is better to log the summary at end to see in an easy way the actual result of the test run. [YOCTO #11622] (From OE-Core rev: 4e3ab36e8c90abc740cce1ba31faf6595116e1e2) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: Call parent init method in OETestLoaderAnibal Limon2017-06-061-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After Python 3.5 the init method contains initialization of _loading_packages set to avoid infinite loops in recursive loading. Fix, Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 70, in <module> ret = main() File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 57, in main results = args.func(logger, args) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 214, in run rc = self._internal_run(logger, args) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 175, in _internal_run self.tc.loadTests(self.module_paths, **self.tc_kwargs['load']) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/context.py", line 51, in loadTests self.suites = self.loader.discover() File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/loader.py", line 268, in discover pattern='*.py', top_level_dir=path) File "/usr/lib/python3.5/unittest/loader.py", line 341, in discover tests = list(self._find_tests(start_dir, pattern)) File "/usr/lib/python3.5/unittest/loader.py", line 398, in _find_tests full_path, pattern, namespace) File "/usr/lib/python3.5/unittest/loader.py", line 473, in _find_test_path self._loading_packages.add(name) AttributeError: 'OETestLoader' object has no attribute '_loading_packages' (From OE-Core rev: 17b832ba05d640570a773adbff2fa4e9ff83fce3) Signed-off-by: Anibal Limon <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-test: Move load_test_components to oeqa.utilsAníbal Limón2017-06-061-0/+1
| | | | | | | | | | | | In order to maintain compatibility with oe-selftest, the load_test_components needs to be re-used, so the script executor needs to pass to only load components supported by certain script (oe-test, oe-selftest). (From OE-Core rev: d6b78ae711b93b4059690320cb8d821aaadd1684) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core: Improve grammar in README and loader commentsRandy MacLeod2017-06-052-23/+22
| | | | | | | (From OE-Core rev: 48cd7ff7ae61ca5772a42ece9fce51976dd0d790) Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/context: Include a _pre_run methodLeonardo Sandoval2017-06-022-0/+7
| | | | | | | | | | This pre runner will serve to allow Test components executes code previously of the run a suite. (From OE-Core rev: bafb7e221d40d7a87a02cec8a97d98eec7c23438) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/context: Omit docstring output on XMLResultLeonardo Sandoval2017-06-021-1/+1
| | | | | | | | | | | By default, the xml runner class prints out the docstring for every unit test but it order to keep the same format as the standard runner, avoid docstring output setting descriptions to False. (From OE-Core rev: 05b16219698fbd30ec76697e1b3be6d31e8f8878) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-09 20:33:28 +0000