summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/core
Commit message (Collapse)AuthorAgeFilesLines
* oeqa/core/context: initialize _run_end_timeKonrad Weihmann2020-11-201-0/+1
| | | | | | | | | | | | | with _run_start_time as value. For partial results of interrupted runs, this info might be otherwise missing for at least one testcase (From OE-Core rev: 4ebf97e36d9def38fb869cbbbd5ce10f5d2669a3) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1c5e8baf57fa2a33b9ef507b11d9ea9acaa77238) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/context: expose results as variableKonrad Weihmann2020-11-201-0/+3
| | | | | | | | | | | | | | | register an unittest handler for testresults and expose it as variable result. With this even partial results from an interrupted test suite run can be made available (From OE-Core rev: 07a3ea1aece5c67d75b60417ee0978cc0e15719a) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a97ae47525157871b6c098ffc352293e365a4335) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: write @OETestTag content into json test reports for each caseAlexander Kanavin2020-08-121-0/+14
| | | | | | | | | | | | | | | | | This allows using these tags for classification and filtering of test results according to various organization-specific criteria, such as teams responsible for the test, internal test ids, feature domains and so on. Test name itself meanwhile can stay short and human-readable. (From OE-Core rev: 3801b126eb52cd46efe417111afcd27f05d8f72b) (From OE-Core rev: 9eaf11657cc8304af5cb75f98decbc64cca37801) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* ltp: make copyFrom scp command non-fatalMatthew2020-08-121-2/+5
| | | | | | | | | | | | | | | [YOCTO #13802] Make the scp failure non-fatal so the ltp tests continue to run and the rest of the logs will be available to see afterwards. (From OE-Core rev: 9390846e2abdd6837094a619ba4703a1be4a04cf) Signed-off-by: Mingde (Matthew) Zeng <matthew.zeng@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0f7d093038274f4f21f6cca39a96aac4f6c32ee3) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: fix regex to include numbersVacek, Patrick2020-07-161-1/+1
| | | | | | | | | | | | | | The previous version only included the numbers 1 and 2 in the allowed characters for the module name. In the past, this was (\w+) so all numbers were allowed. Now it explicitly includes all numbers again. (From OE-Core rev: eab35415bb2cd68872707693afcf7705675fa87e) Signed-off-by: Patrick Vacek <patrickvacek@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f241fa493536ac953c1dac303917c6e75b459e28) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: refine regex to find moduleLee Chee Yang2020-07-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | test case in format <module name>.<class name>.<test case name> this is clear when test cases is only 3 item deep. but confused when it is 4 item deep, eg, oelib.types.TestList.test_list_nosep in this case, oelib and oelib.types can both be treated as module since module name contains only lower cases and class name should contain atleast one upper case. so, always treat leading item without upper case as module also allow module name to contain dot. [YOCTO #13941] (From OE-Core rev: 838646da741cda2059ee2bd64fd1e59980fbf945) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ad81ea90a815389e45ff302a85151724c71f71c3) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Clean up separate builddir in success case when non-threadedRichard Purdie2020-07-021-19/+4
| | | | | | | | | | | | | | | | | | | | | If oe-selftest is run without -j, the separate build directory "build-st" isn't cleaned up afterwards. Mirror the behaviour of the -j option to handle this the same way, only preserve upon failure. To do this, the remove function needs to be moved to the selftest context module so that it can be accessed without requiring the testtools and subunit modules the -j option requires. A dummy wrapper class is used to wrap the tests and clean up afterwards. [YOCTO #13953] (From OE-Core rev: 20e7b1eeeb12f1cf4bd9934e0a5733c6bbe64372) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1b376ade430d40d3cfe9c18f200c764d622710e5) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: enable ovmf supportKonrad Weihmann2020-07-021-2/+4
| | | | | | | | | | | | | Add support for running wic images with EFI as testimage. Introduces a variable called QEMU_USE_OVMF for configuration. (From OE-Core rev: 458a860b45c0fb5bb3162b5a112b518c3339c1dd) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3af8aaff68ed332d812ea7dc184d392700ad7882) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: don't delete build directory for failed testsSteve Sakoman2020-06-051-2/+8
| | | | | | | | | | (From OE-Core rev: be24033acee75526a3a66c12606ef67dcb52a7ee) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3d5aa170d2e88b852bd2a4452aab9311a24badef) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testresults.json: add duration of the tests as wellAlexander Kanavin2020-05-281-1/+5
| | | | | | | | | | | | | This is printed by testimage, but isn't actually saved. It's a useful metric for tracking execution times. (From OE-Core rev: 866c652c850d9e23300218fcbe0b9e4b3ade2ebf) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8fc19639f47b959a141dae231395bbababa644e1) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: default runtime json results for testexportStefan Kral2020-03-291-4/+6
| | | | | | | | | | | | | Set the json result output dir in the oeqe runtime context to create testresults.json file by default for exported runtime test runs. Use current datetime for the json result property name (not DATETIME from build) to allow multiple result entries. (From OE-Core rev: e6c73ed8d3c5d45f387cab619ca73c21e850582f) Signed-off-by: Stefan Kral <sk@typedivision.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: enable testresults.json for testexportStefan Kral2020-03-122-4/+39
| | | | | | | | | | | | Add the option --json-result-dir to oeqa core context to enable testresults.json creation for test runs via testexport. Eg. oe-test runtime --json-result-dir . (From OE-Core rev: 9d8edf33d1f5d89b310923b0aa3cc967317c7c49) Signed-off-by: Stefan Kral <sk@typedivision.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Standardise seperate builddir for concurrent and ↵Richard Purdie2020-02-212-39/+8
| | | | | | | | | | | | | | non-concurrent selftest Currently oe-selftest reuses the current build directory and the concurrent version run with -j does not. Standardise and use a separate new build directory in both cases. This will lead to simpler code and more reliable user run tests. (From OE-Core rev: 50238b0717b04e0a1fa69d618e8c8aa8445a80b0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* concurrencytest.py: add outSideTestaddSkip for subunitTrevor Gamblin2020-02-181-0/+14
| | | | | | | | | | | | | | | | | | | | | | see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13663 When running oe-selftest in concurrency mode (e.g. with oe-selftest --run-tests oescripts.OEPybootchartguyTests -j 4), if a skip occurred during setUpClass() rather than within individual tests, the entire suite would show "UNKNOWN" as each test's result. This is because subunit doesn't know how to handle skips outside of individual tests. An example of where this occurs is when running the above call to oe-selftest in concurrency mode on a host machine that does not have python3-cairo installed. Patch subunit inside concurrencytest.py to provide a method called outSideTestaddSkip, which will allow subunit to correctly detect the skip in setUpClass(). (From OE-Core rev: 9b8734b584d6e8d9c32ff2a721b29f3f3e61cca7) Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Extend runtime testing infrastructure to allow unconventional ↵Alejandro Hernandez Samaniego2020-02-131-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | booting processes to be tested The current runtime infrastructure contains hardcoded values which Ill refer to as patterns, these patterns are either searched through or sent via the serial terminal to communicate between HOST and TARGET. These patterns are required since they allow us to check when a device has finished booting, to log in, and to check whether a command sent from our tests has returned, this way we are able to check both the status of the commands that were sent along with its output. The testing process goes somewhat as follows: 1. Launch QEMU and start booting. 2. Check when the device has booted by looking for the pattern login:. 3. Log in as the root user (default for our images). 4. Check that we were able to log in succesfully. 5. Start running the runtime test cases defined by TEST_SUITES. 6. One of such test cases could send a command to the QEMU target. 7. Check whether that command returned. 8. Check its output and status, return whether the test case passed or failed. This patch allows this set of patterns to be defined instead of being hardcoded, but it also automatically sets the defaults that we have been using in the past if they have not been manually defined, for this reason, the patch is less invasive and should not affect in any way how tests are currently being run. Cases that can be enabled with this patch: - A customized image that does not use the root user (or maybe we want to check what happens if we dont use the root user). - An image where the PS1 env variable has been modified, and the prompt pattern wouldnt match the default. - Baremetal applications, which do not follow the conventional way of booting Linux and would probably not show a prompt for a user to log in, same applies for testing bootloaders. - poky-tiny: Using DISTRO=poky-tiny and an image such as the core-image-tiny from meta-intel, which boots directly to RAM, and does not show a log in prompt since it does not contain a conventional init process. The code itself contains comments that should be self explanatory but here is an example on how these patterns can be defined in a hypothetical case where we want to run test cases as the webserver user instead: TESTIMAGE_BOOT_PATTERNS = "send_login_user search_login_succeeded" TESTIMAGE_BOOT_PATTERNS[send_login_user] = "webserver\n" TESTIMAGE_BOOT_PATTERNS[search_login_succeeded] = "webserver@[a-zA-Z0-9\-]+:~#" The variable TESTIMAGE_BOOT_PATTERNS defines which patterns to override when used to communicate with the target when booting, anyone familiar with the PACKAGECONFIG syntax should have no trouble setting these. Other patterns would still be set up as default, e.g. search_reached_prompt would still be login: The accepted flags for TESTIMAGE_BOOT_PATTERNS are the following: search_reached_prompt, send_login_user, search_login_succeeded, search_cmd_finished. They are prefixed with either search/send, to differentiate if the pattern is meant to be sent or searched to/from the target terminal. A working example of this code that falls under the baremetal case mentioned above along with a test case is present on the meta-freertos layer, which tests an RTOS image built with OpenEmbedded and automatically runs a test case on it after booting such image: As usual, INHERIT += "testimage" needs to be present on local.conf $ bitbake freertos-demo -c testimage RESULTS: RESULTS - freertos_echo.FreeRTOSTest.test_freertos_echo: PASSED (2.00s) SUMMARY: freertos-demo () - Ran 1 test in 2.006s freertos-demo - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) (From OE-Core rev: 3ab2cbfeff371e8791b031a2852eeef80101a831) Signed-off-by: Alejandro Hernandez Samaniego <aehs29@gmail.com> Signed-off-by: Alejandro Hernandez Samaniego <alejandro@enedino.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Allow testing on QEMU machines with a single serial portAlejandro Hernandez Samaniego2020-02-131-2/+3
| | | | | | | | | | | | | | | | | commmit 6cde67d0a84 enables the use of qemurunner on machines that only have a single serial port, but still sets the default value as serial_ports=2 if not provided. The testimage class does not call qemurunner with a serial_ports argument, hence always defaulting to two. Pass the serial_ports argument from the testimage class to allow tests to run on QEMU machines with a single serial port. (From OE-Core rev: a50d0163770f0b405a8de8a8a9cccd48c1de4112) Signed-off-by: Alejandro Hernandez Samaniego <alejandro@enedino.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: modernise subprocess: preexec_fn=os.setsid -> start_new_session=TrueAndré Draszik2020-02-041-1/+1
| | | | | | | | | | | | | start_new_session was added to python3 subprocess in v3.2 and is meant to take the place of the common use of preexec_fn to call os.setsid() in the child - as done here. Update to use the new equivalent. (From OE-Core rev: c3bd502f31a621394a2f6ea3ae90443ff4ac98a4) Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh oeqa/target/qemu: expose server listening port to testsAndré Draszik2019-12-162-1/+3
| | | | | | | | | | | | | Allow tests to access the listening port as just introduced. Note that when using qemu this infrastructure shouldn't be needed, but we still need to set the port to 0 so that a listening port is determined automatically (e.g. by the python http server). (From OE-Core rev: ca58559fc30842c123907fb5cc68b3356c530862) Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Optimise for autobuilder/clobberdirRichard Purdie2019-12-061-0/+9
| | | | | | | | | | | | | | | | | | | | | | We're seeing huge delays on the autobuilder during oe-selftest builddir deletion. For example there is a currently running selftest we could do with the results from and its been going 13 hours, at least 8 of which was in deletion of the builddirs. There are a variety of ways we could solve this problem however the autobuilder has a mechanism for deferred deletion, "clobberdir" which it already uses for this kind of work. Whilst in general hardcoding things like this is horrible, I believe in this case the benefits (and resulting improvements on my sanity if nothing else) mean this is a case where we should do it. If/as/when someone can come up with a better solution that is fine and this can be replaced. (From OE-Core rev: 51a84937d32a85cbdb7d5b1d1ec69a290d0c303b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core: Add qemu checksArmin Kuster2019-11-141-0/+45
| | | | | | | | | | Some test should not be run in QEMU systems so add some checks to make that easier (From OE-Core rev: 1b1e53c4ad33a71d526887d1133598f255a476ec) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core: Add a check for MACHINEArmin Kuster2019-11-141-0/+44
| | | | | | | (From OE-Core rev: e69f963e548e2f6f211a56406694c029111d7203) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator: add skipIfFeatureAndré Draszik2019-10-191-0/+18
| | | | | | | | | | skipIfFeature will skip a test if a given DIST_FEATURE or IMAGE_FEATURE is enabled. (From OE-Core rev: ff2218f7cc3992725dd35499c14ec3396120dcc5) Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/case.py: Encode binary data of logNathan Rossi2019-09-301-2/+2
| | | | | | | | | | | | | | | Do not decode the log content into a string only to re-encode it as binary data again. Some logs might un-intentionally contain bytes that do not decode as utf-8, as such preserve the log file content as it was on disk. Handle the decoding on the resulttool side, but also handle the failure to decode the data. (From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/utils/concurrencytest.py: Handle exceptions and detailsNathan Rossi2019-09-301-13/+18
| | | | | | | | | | | | | | | | | Handle the streaming of exception content with details data. The testtools package allows both 'err' and 'details' kwargs but can only pass one of them to the parent. To handle the passing of exception traceback and details data at the same time, encode the traceback into the details object and remove the 'err' arg from the add* result call. This encodes the traceback similar to how 'err' is handled without any details object. Decoding is already done by testtools when the traceback is encoded in the details object. (From OE-Core rev: 3613451825b251784b7673d89db465b9782c3a31) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrenttest: Use ionice to delete build directoriesRichard Purdie2019-09-191-1/+1
| | | | | | | | | | | | | | | | Autobuilder type infrastructure can benefit from deletion of certain files as background IO due to the way Linux filesystem priority works. We have problems where build directories as part of oe-selftest being delete starves the running tasks of IO to the point builds take much longer to compelte. Having this option of running the deletion at "idle" helps a lot with that. Use the new option added to bb.utils.prunedir(). (From OE-Core rev: d41e7018be56902b7a1be4590e468cd15e02a3b5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpersNathan Rossi2019-09-161-0/+49
| | | | | | | | | | | | | | | | Add the OEPTestResultTestCase class as a mix-in class to provide helper functions for interacting with ptestresults within the extraresults object generated by the test case. This class also provides default compression of log text and log files. Also add support to resulttool for decoding/decompressing log files embedded in the test results. (From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator: Fix super class modifying subclass tagsNathan Rossi2019-09-071-1/+2
| | | | | | | (From OE-Core rev: ba35bead1108c7d8480b785b2e59f40ea77b5549) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner.py: Fix OETestTag listingNathan Rossi2019-09-071-11/+8
| | | | | | | | | | | | Use the __oeqa_testtags attribute added by OETestTag and display no, one or more tags separated by a comma. Also change the formatting of the output so that the list of tests is formatted as "<test> (<tags>)" and remove the table header for "id" (it is no longer available). (From OE-Core rev: d62e577a4e1a5f6accbce9f7bff7317a1162d72d) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core: Rework OETestTag and remove unused OETestFilterNathan Rossi2019-09-079-138/+103
| | | | | | | | | | | | | | | | | | | | | | Rework OETestTag so that it does not rely on the existing decorator code base and instead inserts the tags into an attribute on the decorated target (e.g. class/type or method). This allows the use of OETestTag on classes and method. In order to filter tagged tests rework the loaders filtering code, removing the generic-ness (with validation and attributes/etc.) and replace it with a "tags_filter" parameter which is a function that filters a test based on the tags it has. This allows the loader user to filter on tags in more specific ways (e.g. include all untagged tests and any tests tagged with foo). Plumb all this through the context code and testing code. Update the associated tests to pass correctly with the changes. (From OE-Core rev: b8a4a4c2de68110d74607cb9807c9e741ca9441c) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core: Implement proper extra result collection and serializationNathan Rossi2019-09-072-6/+96
| | | | | | | | | | | | | | | Implement handling of extra result (e.g. ptestresult) collection with the addition of a "extraresults" extraction function in OETestResult. In order to be able to serialize and deserialize the extraresults data, allow OETestResult add* calls to take a details kwarg. The subunit module can handle cross-process transfer of binary data for the details kwarg. With a TestResult proxy class to sit inbetween to encode and decode to and from json. (From OE-Core rev: b0831d43606415807af80e2aa1d0566d0b8c209c) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/tests: Fix test_data module testsNathan Rossi2019-09-061-4/+6
| | | | | | | | | | These two tests relied on the context containing the results information. This was moved into the OETestResults class. (From OE-Core rev: ddad00bfec3b10b5a15a2a8eb57612bb5968f51f) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/tests: Skip test_fail_duplicated_moduleNathan Rossi2019-09-061-0/+1
| | | | | | | | | | | At some point the modules in tests/cases/loader/invalid/ were removed, this test relied on having tests/cases/loader/invalid/oetag.py to trigger a ImportError in the loader module. Skip this test. (From OE-Core rev: cf79aa6383e2a2773e53d5b3082154777646cda4) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: avoid class setup method to run when skipping the whole classChen Qi2019-06-212-0/+10
| | | | | | | | | | | | | | | For now, even if we have specified to skip the whole module/class via command line, e.g., `oe-selftest -R gotoolchain', the class setup method is still run. This at least results in unnecessary builds, and at worst results in ERROR, if the setup method fails. So improve the skipping mechanism to avoid class setup method to run when specified to skip. (From OE-Core rev: b0b79bf65f5e5e65958090a4a88622b42df896bf) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* context.py: avoid skipping tests by meaningless command argumentChen Qi2019-06-211-1/+1
| | | | | | | | | | | Currently `oe-selftest -R a' will skip 'archiver' tests. This is not expected. Fix it so that the '-R' should be followed by actual module/class/test names. (From OE-Core rev: de3b070fc2ddd0b63a324679ec5adbe30142fc22) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* context.py: fix skipping functionChen Qi2019-06-181-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current codes to skip test cases are logically correct, but they do not work correctly in reality. It does skip the tests as the command line argument specifies, but the related information is wrong. e.g. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:24:53,764 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:25:06,309 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:25:06,312 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:25:10,522 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:25:10,522 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,522 - oe-selftest - INFO - Skip by the command line argument "runtime_test" 2019-06-17 09:25:10,523 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:25:10,523 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,523 - oe-selftest - INFO - Skip by the command line argument "runtime_test" The archiver.Archiver.xxx tests should be skipped by 'archiver' command line argument, not 'runtime_test'. Change to use a function generator to achieve the desired effect. After the change, the effect is as follows. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:19:06,223 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:19:19,598 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:19:19,602 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:19:24,369 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,369 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,370 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,370 - oe-selftest - INFO - test_archiver_filters_by_type_and_name (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,371 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,371 - oe-selftest - INFO - test_archiver_srpm_mode (archiver.Archiver) 2019-06-17 09:19:24,371 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,372 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,372 - oe-selftest - INFO - test_bitbakelayers_add_remove (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,373 - oe-selftest - INFO - Skip by the command line argument "bblayers" 2019-06-17 09:19:24,373 - oe-selftest - INFO - test_bitbakelayers_createlayer (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,374 - oe-selftest - INFO - Skip by the command line argument "bblayers" [snip] (From OE-Core rev: 50004f431a71c71af4acf0f25403fee5a8447eab) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: Replace suggogatepass with ignoring errorsRichard Purdie2019-05-131-2/+2
| | | | | | | | | | | We continued to see encoding problems with ssh commands run in oeqa. After much research the conclusion was we should use ignore the errors since some occasional bad locale encoding is better than the unicode decoding issues we were seeing which crashed large parts of tests. (From OE-Core rev: b3bae8aba5b897c1afcc8bf4ad8929251812d2b5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrenttest: Patch subunit module to handle classSetup failuresRichard Purdie2019-05-121-4/+23
| | | | | | | | | | | | | | | | | | | | | | Currently setupClass errors were not being mapped back to the failing tests and they were hence being marked as UNKNOWN and the test statistics were inaccurate. This is because whilst the errors were being encoded into the test results stream, the decoder doesn't cope with an error outside a testStart event. We patch in an addError handler to the outsideTest parser so that this does get handled in a way similar to the non-concurrent case. It would be nice if we didn't have to do this but there doesn't seem to be any other way to fix this other than forking subunit. We also make a minor change so another of our changes can cope with tests without a start time. (From OE-Core rev: 8f7352ed9c1a3e82689b842b7f87e469ebf2e48f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Fix subunit setupClass/setupModule failure handlingRichard Purdie2019-05-121-2/+2
| | | | | | | | | The string format for subunit setupClass/setupModule failures is slightly different, tweak the regex to correctly handle both cases. (From OE-Core rev: 425fbba9348420df72de5c6292f50acaf2023d0b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Handle unexpectedSucessesRichard Purdie2019-05-091-2/+8
| | | | | | | | | | | | Instead of showing: RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s) map unexpectedSuccesses to PASSED and improve the way they're displayed. We expect/allow ptest runner to fail but if it passes we should handle it correctly. (From OE-Core rev: 7885939ca3dea9704185e93469fe515c17eb8017) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib+scripts: Convert to SPDX license headersRichard Purdie2019-05-0928-31/+108
| | | | | | | | | | | | | | | | | | | | | | | This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Drop OETestIDRichard Purdie2019-05-096-113/+8
| | | | | | | | | | These IDs refer to testopia which we're no longer using. We would now use the test names to definitively reference tests and the IDs can be dropped, along with their supporting code. (From OE-Core rev: 8e2d0575e4e7036b5f60e632f377a8ab2b96ead8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/ssh: Avoid unicode decode exceptionsRichard Purdie2019-05-011-2/+2
| | | | | | | | | | | | | | | This code really needs to be rewritten to not split potential multibyte characters, for now work around it to avoid exceptions like: File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run data = reader.read(1024, 4096) File "/usr/lib64/python3.6/codecs.py", line 503, in read newchars, decodedbytes = self.decode(data, self.errors) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte (From OE-Core rev: 17e87510378f2729208a8262695f28e1efe5eb4c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: dump stdout and stderr of each test caseMardegan, Alberto2019-04-261-4/+16
| | | | | | | | | | | | | | | | | | Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since `thud` we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests. We now restore this functionality by adding `stdout` and `stderr` keys to the JSON reports. This behavior is off by default; in order to enable it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake configuration. (From OE-Core rev: fd0048630ece5b21efb3a79e97046be0ab2a1514) Signed-off-by: Alberto Mardegan <amardegan@luxoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage.bbclass: add support for passing runqemu paramsAlexander Kanavin2019-01-261-2/+2
| | | | | | | | | This is particularly useful when setting up GL tests. (From OE-Core rev: 167a46775059b782c6f82ce8c5a47b27262e95d4) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: fix for locating meta-selftestRobert Yang2018-12-261-1/+4
| | | | | | | | | | | The previous code assumed builddir and meta-selftest are in the same dir, but this isn't always true, builddir can be anywhere, use get_test_layer() to locate meta-selftest can fix the problem. (From OE-Core rev: 56d2493a9adfcc47ae7e265439e05ff42cdbbbbf) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemu: Add support for slirpYeoh Ee Peng2018-11-231-3/+19
| | | | | | | | | | | | Enable qemu for slirp. Initialize Qemurunner with slirp. Setup ip and port attribute to enable connection with qemu running with slirp. [YOCTO#10713] (From OE-Core rev: 815e05ac4abb42238d6fdeb15d7426759e059ce1) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemu & runtime: qemu do not need ip input from externalYeoh Ee Peng2018-11-231-3/+2
| | | | | | | | | | | Qemu do not use the ip input from external. It will retrieve ip from QemuRunner instance and assign ip value. (From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Avoid unclosed file warningsRichard Purdie2018-11-161-0/+3
| | | | | | | | | Avoid an unclosed file per thread warning when running selftests concurrently by closing the result stream. (From OE-Core rev: 33a4a076e8aa72a872807332501e7f5ae1cee0e2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Correctly markup regexsRichard Purdie2018-11-141-2/+2
| | | | | | | | | Avoid the warning "DeprecationWarning: invalid escape sequence \(" by marking the regexs correctly. (From OE-Core rev: cb49980fa4a158d5529902df731dec61a8c9b3d4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/loader: Fix deprecation warningRichard Purdie2018-11-141-1/+1
| | | | | | | | | | Clean up the warning: meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec() _failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args (From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-18 18:19:39 +0000