linux/poky.git/meta/lib/oeqa/utils, branch uninative-2.8

oeqa/logparser: correctly set test result markers

2020-02-22T23:57:27+00:00

I spotted this by noticing ptests started to unexpectedly pass, which shouldn't happen yet. (From OE-Core rev: 1cd6e9a8fdfef927916c6cea2371fc0430b1ffdf) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

oeqa/logparser: Fix performance issues with ptest log parsing

2020-02-21T09:39:00+00:00

On the autobuilder a ptest log with 2.1 million lines took around 18 hours to process. This is clearly crazy. We can tweak the processing code to: a) Stop repeatedly joining large strings together (append to a list instead) b) Use one startswith expression instead of multiple re.search() operations With this change it takes 18 hours down to around 12s. [YOCTO #13696] (From OE-Core rev: 01c9f40b979e36a53e789a6bedd89b6d9557dce3) Signed-off-by: Richard Purdie

testimage: Extend runtime testing infrastructure to allow unconventional booting processes to be tested

2020-02-13T12:19:14+00:00

The current runtime infrastructure contains hardcoded values which Ill refer to as patterns, these patterns are either searched through or sent via the serial terminal to communicate between HOST and TARGET. These patterns are required since they allow us to check when a device has finished booting, to log in, and to check whether a command sent from our tests has returned, this way we are able to check both the status of the commands that were sent along with its output. The testing process goes somewhat as follows: 1. Launch QEMU and start booting. 2. Check when the device has booted by looking for the pattern login:. 3. Log in as the root user (default for our images). 4. Check that we were able to log in succesfully. 5. Start running the runtime test cases defined by TEST_SUITES. 6. One of such test cases could send a command to the QEMU target. 7. Check whether that command returned. 8. Check its output and status, return whether the test case passed or failed. This patch allows this set of patterns to be defined instead of being hardcoded, but it also automatically sets the defaults that we have been using in the past if they have not been manually defined, for this reason, the patch is less invasive and should not affect in any way how tests are currently being run. Cases that can be enabled with this patch: - A customized image that does not use the root user (or maybe we want to check what happens if we dont use the root user). - An image where the PS1 env variable has been modified, and the prompt pattern wouldnt match the default. - Baremetal applications, which do not follow the conventional way of booting Linux and would probably not show a prompt for a user to log in, same applies for testing bootloaders. - poky-tiny: Using DISTRO=poky-tiny and an image such as the core-image-tiny from meta-intel, which boots directly to RAM, and does not show a log in prompt since it does not contain a conventional init process. The code itself contains comments that should be self explanatory but here is an example on how these patterns can be defined in a hypothetical case where we want to run test cases as the webserver user instead: TESTIMAGE_BOOT_PATTERNS = "send_login_user search_login_succeeded" TESTIMAGE_BOOT_PATTERNS[send_login_user] = "webserver\n" TESTIMAGE_BOOT_PATTERNS[search_login_succeeded] = "webserver@[a-zA-Z0-9\-]+:~#" The variable TESTIMAGE_BOOT_PATTERNS defines which patterns to override when used to communicate with the target when booting, anyone familiar with the PACKAGECONFIG syntax should have no trouble setting these. Other patterns would still be set up as default, e.g. search_reached_prompt would still be login: The accepted flags for TESTIMAGE_BOOT_PATTERNS are the following: search_reached_prompt, send_login_user, search_login_succeeded, search_cmd_finished. They are prefixed with either search/send, to differentiate if the pattern is meant to be sent or searched to/from the target terminal. A working example of this code that falls under the baremetal case mentioned above along with a test case is present on the meta-freertos layer, which tests an RTOS image built with OpenEmbedded and automatically runs a test case on it after booting such image: As usual, INHERIT += "testimage" needs to be present on local.conf $ bitbake freertos-demo -c testimage RESULTS: RESULTS - freertos_echo.FreeRTOSTest.test_freertos_echo: PASSED (2.00s) SUMMARY: freertos-demo () - Ran 1 test in 2.006s freertos-demo - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) (From OE-Core rev: 3ab2cbfeff371e8791b031a2852eeef80101a831) Signed-off-by: Alejandro Hernandez Samaniego Signed-off-by: Alejandro Hernandez Samaniego Signed-off-by: Richard Purdie

qemurunner.py: add try/except for pid handling race

2020-02-08T13:20:02+00:00

In some instances, attempts to remove the qemu pidfile within the stop() method fail despite the os.path.exists() call immediately before implying that the file is present. Add a try/except block to log a warning if this occurs, rather than failing outright, since the process simply appears to be exiting at an inconvenient time. [YOCTO #13675] (From OE-Core rev: eadb899e23b18eb9eaff145c3bf5b20fb417c3e8) Signed-off-by: Trevor Gamblin Signed-off-by: Richard Purdie

oeqa: modernise subprocess: preexec_fn=os.setsid -> start_new_session=True

2020-02-04T15:56:29+00:00

start_new_session was added to python3 subprocess in v3.2 and is meant to take the place of the common use of preexec_fn to call os.setsid() in the child - as done here. Update to use the new equivalent. (From OE-Core rev: c3bd502f31a621394a2f6ea3ae90443ff4ac98a4) Signed-off-by: André Draszik Signed-off-by: Richard Purdie

oeqa/utils/httpserver: allow to pass in listening port

2019-12-16T23:25:49+00:00

Being able to specify the listening port is useful when running OEQA from within a docker container, e.g. crops or any other solution. In that case, a port on the outside must be mapped to a specific port inside the container. If no port is specified for the http server module in this case, the http server would choose a random port, which is unlikely to be mapped and thus won't be reachable from the outside. (From OE-Core rev: e54391ca9068661693275fbfcfded9bc0b2606f3) Signed-off-by: André Draszik Signed-off-by: Richard Purdie

oeqa: qemu: ensure we print runqemu output in the event of failure

2019-12-15T09:10:46+00:00

If we get here in the code it's because runqemu has failed, this is not a debug situation - we need to see the output, so print it as an error. Fixes [YOCTO #13681]. (From OE-Core rev: c1def38c680d64c992839166bbf9bec51eb0788a) Signed-off-by: Paul Eggleton Signed-off-by: Richard Purdie

oeqa: qemu: fix width of top output in host dump

2019-12-15T09:10:46+00:00

If runqemu fails, there is some logic to run a set of commands to dump various bits of information that might help debug the issue (particularly in a busy situation such as on the autobuilder). One of those commands is "top -bn1", however top restricts the output to the width of the calling terminal, and for whatever reason this is a little restrictive when called from inside testimage, so set COLUMNS in the environment to a high value to fix it. (Another way is to use the -w option, but that is not supported by our default busybox configuration so it will then fail when this same list of commands is used on the target). (From OE-Core rev: 07da998f5aa6c3f5ffeee5150fc579e8a1051ba4) Signed-off-by: Paul Eggleton Signed-off-by: Richard Purdie

oeqa: qemu: ensure that host dump commands can be run properly

2019-12-15T09:10:46+00:00

If runqemu fails, there is some logic to run a set of commands to dump various bits of information that might help debug the issue (particularly in a busy situation such as on the autobuilder). However, when we try to run these we are inside the normal build environment which restricts commands to be run on the host to those specified in HOSTTOOLS. Since this isn't a place where host contamination is going to be a problem, override PATH to a reasonable default so that we run the actual host tools directly to avoid the issue. (Logically we would want to use the original PATH value here, but it is not easily accessible.) (From OE-Core rev: 3f6dcf67bccb4f0fc1a9a64e02bd0e2da620ad0e) Signed-off-by: Paul Eggleton Signed-off-by: Richard Purdie

oeqa/qemurunner: Fix cmdline variable use before reference

2019-12-06T14:41:28+00:00

This avoids some tracebacks we've seen on failed autobuilder builds which would allow the real error to be seen. (From OE-Core rev: a4031935a7b8ea4f61b9020c1aa5598e186e7ad7) Signed-off-by: Richard Purdie