summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/core
Commit message (Collapse)AuthorAgeFilesLines
* oeqa/core/case: add file exists assertionRoss Burton9 days1-0/+8
| | | | | | | | | | Add assertFileExists() to simply tests that want to check that a file exists. (From OE-Core rev: b62e53a0cff2522fef3b89de875c9526a626d7dd) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa decorator/data.py: add skipIfNotBuildArch decoratorMikko Rapeli2025-06-051-0/+12
| | | | | | | | | | | To limit tests to specific build host architectures. For example KVM testing will only work if target and build architectures are the same. (From OE-Core rev: c59b74b8bfd3b351a31204f33e00351ad5e5b657) Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Use single space indentation in json outputRichard Purdie2024-11-231-1/+1
| | | | | | | | | | | Using 4 space indentation in resulted in hundreds of megabytes of extra file size in general use. Reduce this to make filesizes more managable and reduce the processing cost. Some level of indentation and spacing does make the files more readable and allows use of git diff so we need to retain some of it. (From OE-Core rev: a274cdcaf852cca9497f0358f44dda99c06aacbe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/ssh: Improve performance and log sizesRichard Purdie2024-11-211-12/+32
| | | | | | | | | | | | | | | | | | The current code is not fit for purpose when handling large files via ssh. In the strace ptest case, we can end up with a 1.4GB archive being transferred for which every byte is printed into the task logfile twice over. This is then sent over bitbake IPC which compounds the problems. Make the following improvements: * when the output is large (over 64kb), don't print it * use a bytearray for better concat performance since strings are slow for this * when there is no ssh output, say that * print periodic size status output rather than the data itself since this could be binary and/or large * fix the killed process message logic which appeared broken (From OE-Core rev: e7dd009a17dc902852983a82bce41bf78bb1e242) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testexport: support for executing tests over serialAndrew Oppelt2024-09-011-0/+315
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Uses TEST_SERIALCONTROL_CMD to open a serial connection to the target and execute commands. This is a drop in replacement for the ssh target, fully supporting the same API. Supported with testexport. To use, set the following in local.conf: - TEST_TARGET to "serial" - TEST_SERIALCONTROL_CMD to a shell command or script which connects to the serial console of the target and forwards that connection to standard input/output. - TEST_SERIALCONTROL_EXTRA_ARGS (optional) any parameters that must be passed to the serial control command. - TEST_SERIALCONTROL_PS1 (optional) A regex string representing an empty prompt on the target terminal. Example: "root@target:.*# ". This is used to find an empty shell after each command is run. This field is optional and will default to "root@{MACHINE}:.*# " if no other value is given. - TEST_SERIALCONTROL_CONNECT_TIMEOUT (optional) Specifies the timeout in seconds for the initial connection to the target. Defaults to 10 if no other value is given. The serial target does have some additional limitations over the ssh target. 1. Only supports one "run" command at a time. If two threads attempt to call "run", one will block until it finishes. This is a limitation of the serial link, since two connections cannot be opened at once. 2. For file transfer, the target needs a shell and the base32 program. The file transfer implementation was chosen to be as generic as possible, so it could support as many targets as possible. 3. Transferring files is significantly slower. On a 115200 baud serial connection, the fastest observed speed was 30kbps. This is due to overhead in the implementation due to decisions documented in #2 above. (From OE-Core rev: d817b27d73d29ba2beffa2e0a4e31a14dbe0f1bf) Signed-off-by: Andrew Oppelt <andrew.j.oppelt@boeing.com> Signed-off-by: Matthew Weber <matthew.l.weber3@boeing.com> Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com> -- Tested with core-image-sato on real hardware. TEST_SERIALCONTROL_CMD was set to a bash script which connected with telnet to the target. Additionally tested with QEMU by setting TEST_SERIALCONTROL_CMD to "ssh -o StrictHostKeyChecking=no root@192.168.7.2". This imitates a serial connection to the QEMU instance. Steps: 1) Set the following in local.conf: - IMAGE_CLASSES += "testexport" - TEST_TARGET = "serial" - TEST_SERIALCONTROL_CMD="ssh -o StrictHostKeyChecking=no root@192.168.7.2" 2) Build an image - bitbake core-image-sato 3) Run the test export - bitbake -c testexport core-image-sato 4) Run the image in qemu - runqemu nographic core-image-sato 5) Navigate to the test export directory 6) Run the exported tests with target-type set to serial - ./oe-test runtime --test-data-file ./data/testdata.json --packages-manifest ./data/manifest --debug --target-type serial Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/ssh: allow to retrieve raw, unformatted ouputAlexis Lothoré2024-07-081-8/+8
| | | | | | | | | | | | | | | | | | | | The ssh target is currently well tailored to easily retrieve textual output from a command run on a remote target. It could also be used to retrieve raw data from a command run onto a remote target (for example, to feed this data directly to another program), but it currently suffers two minor issues preventing such use case: - stderr is piped to stdout, so any error log will be mixed in the program output - the final output is decoded as utf-8 and stripped Allow to return the raw, unmodified output by adding an optional "raw" parameter. Keep it to False by default to preserve the current behavior. When enabled, do not return a string but the raw output as bytes. (From OE-Core rev: 8d05dc6e2284b7ed7c32a8215b9c8bf6f7dabf00) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core: remove duplicate 'os' importEnrico Jörns2024-06-061-1/+0
| | | | | | | (From OE-Core rev: f1362fe26cc8adc09d15ea2a854af6590728c683) Signed-off-by: Enrico Jörns <ejo@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Drop target_dumper and most of monitor_dumperRichard Purdie2023-12-202-29/+1
| | | | | | | | | | | | | | | | The target_dumper code is basically broken. It has been reading binary files over the text base serial communication and runs at every command failure which makes no sense. Each run might overwrite files from the previous run and the output appears corrupted due to confusion from the binary data. For now, remove the commands and the target dumper code as the command and execution point are problematic. Also remove the same pieces of the monitor code but leave the command list since in theory this can be moved to a more useful place in the code. (From OE-Core rev: a24d787987dccc95fdd95b7e85bf525a1c55b285) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator: add skip if not qemu-usermodeAdrian Freihofer2023-12-131-0/+10
| | | | | | | | | | | Add a decorator to skip tests which require the qemu-usermode machine feature. (From OE-Core rev: 3d79615d6c9ccb1ff3766ce05389bc22cbd656e1) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/ssh: Handle SSHCall timeout error codeluca fancellu2023-11-101-1/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | The current code in ssh.py is terminating the ssh process that does not finish its computation in a given timeout (when timeout is passed), the SSHCall function is returning the process error code. The Openssl ssh before version 8.6_p1 is returning 0 when it is terminated, from commit 8a9520836e71830f4fccca066dba73fea3d16bda onwards (version >= 8.6_p1) ssh is returning 255 instead. So for version of ssh older than 8.6_p1 when the SSHCall time out, the return code will be 0, meaning success, which is wrong. Fix this issue checking if the process has timeout (hence it's been terminated) and checking if the returned code is 0, in that case set it to 255 to advertise that an error occurred. Add a test case excercising the timeout in the SSHTest, test_ssh test function. (From OE-Core rev: 948fecca1db4c7a30fcca5fcf5eef95cd12efb00) Signed-off-by: Luca Fancellu <luca.fancellu@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Remove invalid buffering optionRichard Purdie2023-10-051-2/+2
| | | | | | | | | | | | | Fix warnings from oe-selftest -j: /usr/lib/python3.10/os.py:1030: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used return io.open(fd, mode, buffering, encoding, *args, **kwargs) Remove the option since it clearly doesn't do much. (From OE-Core rev: 6b872ee72942951fd464c4c6cb9eadcb9b4749c1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Ensure class setup errors are shown to bitbake loggingRichard Purdie2023-09-261-1/+7
| | | | | | | | | | | | This took a bit of digging but failure messages from testimage are shown to bitbake's logging through stopTest. In the case of a setUpClass failure stopTest is never called and the bitbake logging never sees the error. It would still be in the task logfile. Add some code+comment to ensure logs not shown to the user mid stream are shown at the end. (From OE-Core rev: 7342c418bda2cc3c337a190089864ea731ff62e4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Streamline oe-selftest startup timeRichard Purdie2023-09-221-2/+3
| | | | | | | | | | | | | | "bitbake -e" executions from get_bb_var calls are slow and slow down oe-selftest startup. Rationalise the code to avoid them and minimise the number of "parsing" locations we use by caching key variables and passing them around more. This was particularly problematic with oe-selftest -j usage since it would have multiple bitbake -e executions per process making parallel usage particularly slow. (From OE-Core rev: 3689cadeb07d76e66f97d890e844f899f69666fe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/ssh: Further improve process exit handlingRichard Purdie2023-07-301-2/+3
| | | | | | | | | | | | | | It looks like there were further cases where orphaned processes may be left behind since the .kill() calls may be unsuccessful if the process terminated due to the terminate or through normal exit. In that situation .wait() wouldn't have been called. Further tweak the exit code paths to ensure .wait() is called to update the returncode value before returning in all cases. (From OE-Core rev: 0a0a1731e38edfa72a141e8fd8f2de52be562e94) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* target/ssh: Ensure exit code set for commandsRichard Purdie2023-07-291-0/+3
| | | | | | | | | | | | As spotted by Joshua Watt, the returncode isn't set until .poll() or .wait() is called so we need to call this after the .kill() call. This fixes return code reporting so that timeouts for example now return an exit code when they didn't before. (From OE-Core rev: 3924e94214b5135369be2551d54fb92097d35e95) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: Ensure EAGAIN doesn't truncate outputRichard Purdie2023-07-291-0/+3
| | | | | | | | | | We have a suspicion that the read() call may return EAGAIN on the non-blocking fd and this may truncate test output leading to some of our intermittent failures. Tweak the code to avoid this potential issue. (From OE-Core rev: a8920c105725431e989cceb616bd04eaa52127ec) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage/oeqa: Drop testimage_dump_host functionalityThomas Roos2023-06-291-3/+2
| | | | | | | | | | | | | | | | | The intent behind these functions was to dump the system state when issues occured but it has never really worked as we'd planned. Regular monitoring as the build runs has largely replaced this as that allows a trend to be seen rather than a spot value which was never really useful. The code is bitrotting and not functioning correctly so drop it. [YOCTO #13872] RP: Reword commit message (From OE-Core rev: dea37ba49a236029da73d5cfbfc069bffc38b508) Signed-off-by: Thomas Roos <throos@amazon.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: update options for SCPAlexis Lothoré2023-06-171-1/+4
| | | | | | | | | | By default scp expects files. Passing -r option allows to copy directories too (From OE-Core rev: f22e5af0c5f185463c6f4a7fd7f1376c7f22a4da) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: add helper to know about expected failuresAlexis Lothoré2023-06-171-0/+4
| | | | | | | | | | | | | | | | | | Testing framework currently uses the unittest.expectedFailure decorator for tests that can have intermittent failures (see PTEST_EXPECT_FAILURE = "1") in core-image-ptest.bb. While it allows upper layers to run tests without failing on "fragile" tests, it prevents those from knowing more about those failing tests since they are not accounting as failures (for example we could want to retrieve some logs about failed tests to improve them, and eventually to drop expectFailure decorator) Add a helper to allow upper layers to know about those failures which won't make global testing session (From OE-Core rev: 34595858b14f628a8282777b91c841add6ebe1ca) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/loader: Ensure module names don't contain uppercase charactersRichard Purdie2023-03-311-0/+3
| | | | | | | | | | | Python modules aren't supposed to have uppercase characters in their names according to python conventions. We have regexs in the code which work on that assumption too. Rather than showing errors under some filtering situations, make it clear and error if a problematic name is seen. (From OE-Core rev: 7964a6ee54e881a6128707a7f2a42eec2ed63881) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: loader.py: show warning when skipping selected module and abort if all ↵Martin Jansa2023-03-222-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | are skipped * skipped modules were triggering an ERROR before: poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers imagefeatures.ImageFeatures.test_image_gen_debugfs -K -B /OE/build/poky/build-eSDK 2023-03-13 15:07:53,430 - oe-selftest - ERROR - Not found eSDK.oeSDKExtSelfTest.test_install_libraries_headers in loaded test cases * but didn't show the reason why it wasn't loaded and more importantly -r was ignored when all selected modules were silently skipped * add a warning when skipping some module and abort if some modules were selected, but all ended being skipped: poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers -K -B /OE/build/poky/build-eSDK 2023-03-13 15:11:51,028 - oe-selftest - WARNING - module 'eSDK.oeSDKExtSelfTest.test_install_libraries_headers' was skipped from selected modules, because it doesn't match with module name assumptions: package and module names do not contain upper case characters, whereas class names do 2023-03-13 15:11:51,028 - oe-selftest - ERROR - All selected modules were skipped, this would trigger selftest with all tests and -r ignored. * I was hit by this in oe-selftest -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers which is skipped due to upper case characters in module name and selftest started to run all tests (archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name as first from 529) poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers -K -B /OE/build/poky/build-eSDK 2023-03-13 14:00:52,955 - oe-selftest - DEBUG - Selected tests with -r: ['eSDK.oeSDKExtSelfTest.test_install_libraries_headers'] 2023-03-13 14:00:55,531 - oe-selftest - INFO - Changing cwd to /OE/build/poky/build .. 2023-03-13 14:00:58,128 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name) I'll rename eSDK to esdk in next commit to avoid this. * also fix small typo in context I've noticed when debugging this (From OE-Core rev: 15229c9abaf4cc398c31d97d171b9fac57141873) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa ssh.py: fix hangs in run()Mikko Rapeli2023-02-231-9/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When qemu machine hangs, the ssh commands done by tests are not timing out. do_testimage() task has last logs like this: DEBUG: time: 1673531086.3155053, endtime: 1673531686.315502 The test process is stuck for hours, or for ever if the executing command or test case did not set a timeout correctly. The default 300 second timeout is not working when target hangs. Note that timeout is really a "inactive timeout" since data returned by the process will reset the timeout. Make the process stdout non-blocking so read() will always return right away using os.set_blocking() available in python 3.5 and later. Then change from python codec reader to plain read() and make the ssh subprocess stdout non-blocking. Even with select() making sure the file had input to be read, the codec reader was trying to find more stuff and blocking for ever when process hangs. While at it, add a small timeout to read data in larger chunks if possible. This avoids reading data one or few characters at a time and makes the debug logs more readable. close() the stdout file in all cases after read loop is complete. Then make sure to wait or kill the ssh subprocess in all cases. Just reading the output stream and receiving EOF there does not mean that the process exited, and wait() needs a timeout if the process is hanging. In the end kill the process and return the return value and captured output utf-8 encoded, just like before these changes. This fixes ssh run() related deadlocks when a qemu target hangs completely. (From OE-Core rev: 9c63970fce3a3d6029745252a6ec2bf9b9da862d) Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa ssh.py: add connection keep alive options to ssh clientMikko Rapeli2023-02-101-0/+2
| | | | | | | | | | | | | | | | | | Configure ssh client to test that connection with server is up. If the server does not respond within a minute then the connection, target machine or sshd daemon are stuck and it's better to exit the command execution with errors. Some tests can execute a long time without returning stdout/stderror data and it's difficult to adjust timers for those cases if connection to target machine or the target machine itself hangs and output is not expected in minutes or even hours. (From OE-Core rev: ba68ff04c5786eca7cd8dd44056705867dea8ac4) Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa ssh.py: move output prints to new lineMikko Rapeli2023-02-101-3/+3
| | | | | | | | | | | The output from is garbled otherwise and it's not easy to remove debug output form real command output on target. (From OE-Core rev: 917a70cbc43ac1c70c477b220c4115735457ef04) Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Add number of failures to summary outputRichard Purdie2022-12-221-1/+3
| | | | | | | | | | | | When running oe-selftest and seeing the end of a running log, it is extremely helpful to know if there have been failures or not to save looking at the rest of the log. Add the number of failures to the summary line so that people monitoring builds have an easier time before the end totals are printed. (From OE-Core rev: 6b23996911d91f7f99774646c6db9f3490b4cb62) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator: add decorators to skip based on HOST_ARCHRoss Burton2022-11-111-0/+24
| | | | | | | | | | | | | There are already decorators to skip on the value of MACHINE, but for flexibility it's better to skip based on the target architecture. This means, for example, the ISO image tests could skip if the architecture isn't x86. (From OE-Core rev: 0c21ff0a92906b6b4820eb8beddf8762fe70653d) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: add ignore_status argument to run()Ross Burton2022-11-021-2/+2
| | | | | | | | | | | | | | | The underlying _run() method has an ignore_status argument and can either return the exit code or raise an exception if it is non-zero. Add the same argument to the run() method so that test cases can change this value. It currently defaults to True as that is the existing behaviour, but a follow-up patch will change this to False as test cases should fail on errors unless told otherwise. (From OE-Core rev: e244228730178d15a066a1428956de328cc09671) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib: Add copyright statements to files without oneRichard Purdie2022-08-121-0/+2
| | | | | | | | | Where there isn't a copyright statement, add one to make it explicit. Also add license identifiers as MIT if there isn't one. (From OE-Core rev: bb731d1f3d2a1d50ec0aed864dbca54cf795b040) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: rationalise skipifqemu decoratorsRoss Burton2022-04-011-35/+9
| | | | | | | | (From OE-Core rev: 1a3a37cc2b16a8d5cd2258b0b35be43baa363f67) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorators/data: improve has_* logicRoss Burton2022-04-011-3/+3
| | | | | | | | | | | | | has_feature() should be splitting the feature string into substrings and then looking for membership instead of looking for simple substrings. has_machine() should be using equality instead of substrings. (From OE-Core rev: a4c63819234e252c58e040af8bbdbfb96b6feccf) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/utils/misc: remove redundant fileRoss Burton2022-04-011-47/+0
| | | | | | | | | | | This file dates back to 2016. Half of the functions have never been used, the rest are used in one place and have now been replaced. (From OE-Core rev: 5a053b2a84e7a671925fb4a78005145786c57f6a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/decorator: remove redundant codeRoss Burton2022-04-011-8/+3
| | | | | | | | | | | There's no need to wrap *tags in a potential list, as *tags will always be a tuple. (From OE-Core rev: 54210c518bcb76d80c8ec9564d1ddf344e9d8924) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: don't assume target_dumper is setRoss Burton2021-09-211-1/+3
| | | | | | | | | | The target_dumper property is only set by the QemuTarget subclass, so assign a default value and check it isn't None before calling it. (From OE-Core rev: add36cf85380bf7dc5339a9d65f951923317440b) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/target/qemu.py: display contents of dumped filesSakib Sajal2021-06-121-1/+26
| | | | | | | | | | | | | | | | During do_testimage, if the target is not started within a certain timeout, TEST_QEMUBOOT_TIMEOUT, host data is dumped to files for each command in ${TMPDIR}/log/runtime-hostdump/<datetime>_qemu/host_<seq>_<command>. Display the first 20 lines of top output and the last 20 lines of bootlog to standard output for more context for the target not being started up. (From OE-Core rev: 441390b707bf681bc308c9ebd45ea2ae20c37d7c) Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/tests/test_data.py: use weston image instead of satoAlexander Kanavin2021-04-271-1/+1
| | | | | | | (From OE-Core rev: c2ccd8c8144cdda52b858589f7d5d3a15ab28b90) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* qemurunner: Add support for qmp commandsSaul Wold2021-04-272-1/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds support for the Qemu Machine Protocol [0] extending the current dump process for Host and Target. The commands are added in the testimage.bbclass. Currently, we setup qemu to stall until qmp gets connected and sends the initialization and continue commands, this works correctly. If the UNIX Socket does not exist, we wait an timeout to ensure to socket file is created. With this version, the monitor_dumper is created in OEQemuTarget but then set in OESSHTarget as that's where we get the SSH failure happens. Python's @property is used to create a setter/getter type of setup in OESSHTarget to get overridden by OEQemuTarget. By default the data is currently dumped to files for each command in TMPDIR/log/runtime-hostdump/<date>_qmp/unknown_<seq>_qemu_monitor as this is the naming convenstion in the dump.py code. We use the qmp.py from qemu, which needs to get installed in the recipe-sysroot-native of the target image. [0] https://github.com/qemu/qemu/blob/master/docs/interop/qmp-spec.txt (From OE-Core rev: 42af4cd2df72fc8ed9deb3fde4312909842fcf91) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib/oeqa/core/tests/cases/timeout.py: add a testcase for the previous fixAlexander Kanavin2021-04-232-0/+19
| | | | | | | | | | | | | | | | | | This is the sequence that didn't properly operate: - a test case that skips and isn't executed - a second test case that is skipped via a dependency decorator, and sets a timeout - a third test case that takes longer than the timeout from the second test case Without the fix, the timeout is not cleared, and the third test case is erroneously aborted. With the fix, the timeout is cleared and the third test case is able to complete. (From OE-Core rev: 54ef07a9aa1af8f41cfb9a4802929c918efc43c8) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: tear down oeqa decorators if one of them raises an exception in setupAlexander Kanavin2021-04-232-4/+10
| | | | | | | | | | | | | | | | Some of the decorators need proper cleanup, such as OETimeout which sets a signal handler that needs to be cleared via teardown. If this is not done then the signal gets called later with unpredictable effects. This can be seen if there's a test that is skipped via a decorator and sets a timeout at the same time: the timeout isn't cleared, and is invoked later in a completely unrelated context. The test case for this is added in the next commit. (From OE-Core rev: f42a08e1aabf1ca57e0c09d69fb69cc717c7f156) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runqemu: Support RUNQEMU_TMPFS_DIR as a location to copy snapshot images toRichard Purdie2021-04-061-2/+2
| | | | | | | | | | | | | | | | | | We have a working theory that IO queues on the autobuilder are impacting runtime testing under qemu, particularly async writes which inice does not influence. We already pass the snapshot option to qemu which copies the image and runs out of the copy. Add in the ability to copy the image to a specificed location which can be a tmpfs. This means that writes to the image would no longer be blocked by other writes to disk in the system. Preliminary tests show that this does improve the qemu errors at the expense of sometimes showing qemu startup timeouts as on a loaded system with a large test image, it can take longer than 120s to copy the image to tmpfs. Having a most consistent failure mode for loaded tests is probably desireable though. (From OE-Core rev: fd1c26ab426c3699ffd8082b83d65a84c8eb8bff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Fix display of test stdout/stderrRichard Purdie2021-04-061-16/+17
| | | | | | | | | | | | | | | | If oe-selftest is run with -j, the output to stdout/stderr is being lost at present. Capture this and display it upon test failure. We have code that previously tried to enable this but it wasn't functioning correctly. This should give more usable error reports on the autobuilder. This code will mix stdout and stderr as the output is streamed from the test server without markup. This is most in keeping with subunit/testools though and the easiest way to handle the various challenges here as far as I can see. (From OE-Core rev: 6a954ce5834c8026adecff8478c3d827640bc647) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Rename variables to improve the codeRichard Purdie2021-04-061-13/+13
| | | | | | | | | | Each time I look at this code I get confused about what the different variables represent. Rename a few of them to better indicate what they represent. (From OE-Core rev: e39d97c0b191add9281bac463ca059685288c81a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/context: initialize _run_end_timeKonrad Weihmann2020-11-111-0/+1
| | | | | | | | | | with _run_start_time as value. For partial results of interrupted runs, this info might be otherwise missing for at least one testcase (From OE-Core rev: 1c5e8baf57fa2a33b9ef507b11d9ea9acaa77238) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/context: expose results as variableKonrad Weihmann2020-11-111-0/+3
| | | | | | | | | | | | register an unittest handler for testresults and expose it as variable result. With this even partial results from an interrupted test suite run can be made available (From OE-Core rev: a97ae47525157871b6c098ffc352293e365a4335) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* target/ssh.py: Add dump_target supportSaul Wold2020-09-301-0/+2
| | | | | | | | | | | | | | This adds the dump_target support when the ssh command fails with a 'No route to host'. This is will provide additional data when a Qemu target fails to respond during autobuilder testing. This does not fix 14002 [0], but may help track down why qemu looses networking [0] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14002 (From OE-Core rev: cef1a2b03b359c018911abc29db1895d0f46814c) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Add testimage_dump_target to kwargsSaul Wold2020-09-301-0/+4
| | | | | | | | | | | | | | | | | | This passes the list of commands to run on the OEQemuTarget when the TargetDumper needs to run in a test context due to a failure on the target. This is added here as a kwargs because the 'd' dictionary is not available in the staticmethod getTarget in the OERuntimeTestContextExecutor class. The OEQemuTarget is different from the QemuTarget which already uses the list of commands from testimage_dump_target from 'd'. The create_dir() is needed to initialize the TargetDumper's dump_dir variable. (From OE-Core rev: a63675fab4d9f638570912b15a07932f549cc4d1) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: write @OETestTag content into json test reports for each caseAlexander Kanavin2020-08-081-0/+14
| | | | | | | | | | | | | This allows using these tags for classification and filtering of test results according to various organization-specific criteria, such as teams responsible for the test, internal test ids, feature domains and so on. Test name itself meanwhile can stay short and human-readable. (From OE-Core rev: 3801b126eb52cd46efe417111afcd27f05d8f72b) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* ltp: make copyFrom scp command non-fatalMatthew2020-08-061-2/+5
| | | | | | | | | | | | [YOCTO #13802] Make the scp failure non-fatal so the ltp tests continue to run and the rest of the logs will be available to see afterwards. (From OE-Core rev: 0f7d093038274f4f21f6cca39a96aac4f6c32ee3) Signed-off-by: Mingde (Matthew) Zeng <matthew.zeng@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: fix regex to include numbersVacek, Patrick2020-07-081-1/+1
| | | | | | | | | | | The previous version only included the numbers 1 and 2 in the allowed characters for the module name. In the past, this was (\w+) so all numbers were allowed. Now it explicitly includes all numbers again. (From OE-Core rev: f241fa493536ac953c1dac303917c6e75b459e28) Signed-off-by: Patrick Vacek <patrickvacek@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/loader: refine regex to find moduleLee Chee Yang2020-07-031-1/+1
| | | | | | | | | | | | | | | | | | | | | test case in format <module name>.<class name>.<test case name> this is clear when test cases is only 3 item deep. but confused when it is 4 item deep, eg, oelib.types.TestList.test_list_nosep in this case, oelib and oelib.types can both be treated as module since module name contains only lower cases and class name should contain atleast one upper case. so, always treat leading item without upper case as module also allow module name to contain dot. [YOCTO #13941] (From OE-Core rev: ad81ea90a815389e45ff302a85151724c71f71c3) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Clean up separate builddir in success case when non-threadedRichard Purdie2020-06-281-19/+4
| | | | | | | | | | | | | | | | | | If oe-selftest is run without -j, the separate build directory "build-st" isn't cleaned up afterwards. Mirror the behaviour of the -j option to handle this the same way, only preserve upon failure. To do this, the remove function needs to be moved to the selftest context module so that it can be accessed without requiring the testtools and subunit modules the -j option requires. A dummy wrapper class is used to wrap the tests and clean up afterwards. [YOCTO #13953] (From OE-Core rev: 1b376ade430d40d3cfe9c18f200c764d622710e5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-02 20:46:45 +0000