summaryrefslogtreecommitdiffstats
path: root/meta/classes-recipe/testimage.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* oeqa/runtime/login: Various code improvements and fixesRichard Purdie2024-03-071-0/+1
| | | | | | | | | | | | | | | | * Allow tools to be found from the host PATH so that imagemagick from a buildtools tarball/sdk can work * Reformat the code to have imports at the start of the file and have more standard formatting and whitespace * Always save copies of the images, the space imapct is negligle compared to the debug win * Write the images to ${T} * Use bb.utils.mkdirhier() instead of more complex code * Restrict the tests to images containing matchbox-desktop (From OE-Core rev: d09989b49517830297654e4d1d150aaa8723c41a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: create a list of failed test post actionsAlexis Lothoré2024-02-271-39/+2
| | | | | | | | | | | | | | | | | | | | | testimage is able to detect whenever a test run leads to some tests failing, and execute some actions in this case. The only action currently defined in such case is to retrieve artifacts from the target under test, as listed in TESTIMAGE_FAILED_QA_ARTIFACTS In order to be able to add multiple actions, define a central function to gather all "post actions" to run whenever a test has failed (run_failed_tests_post_actions). This function contains a table listing all functions to be called whenever a test fails. Any function in this table will be provided with bitbake internal data dictionary ("d") and the current runtime testing context ("tc"). Isolate all this feature in a dedicated postactions.py file inherited by testimage. This patch does not bring any functional change. (From OE-Core rev: c01aa8df0613a103859b4431d3cc5056b2fef1b8) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa: share get_json_result_dir helperAlexis Lothoré2024-02-271-9/+3
| | | | | | | | | | | | | Multiple places in oeqa need to get the log output path, and redefine a small helper to accomplish this Define this helper in lib/oeqa/utils/__init__.py and import it wherever needed to allow using it. (From OE-Core rev: 01b1a6a5a4e7cede4d23a981b5144ae9c8306274) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: log exception when failing to retrieve artifactsAlexis Lothoré2024-02-211-2/+2
| | | | | | | | | | | | | | | Despite managing to retrieve the failed ptests artifacts, testimage seems to dump some retrieval errors like the following one: WARNING: core-image-ptest-valgrind-1.0-r0 do_testimage: Can not retrieve /usr/lib/valgrind/ptest from test target Log the corresponding exception to help analyzing such issue (From OE-Core rev: 12873e5b1620414a76e4a0e87cc2c806a0513cfe) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: retrieve ptests directory when ptests failAlexis Lothoré2024-02-081-1/+4
| | | | | | | | | | | | | | | | TESTIMAGE_FAILED_QA_ARTIFACTS is set with a default, minimal list of files to retrieve whenever a runtime test fails. Add ptests directory to the list so we can get ptests artifacts (eg: logs) whenever a ptest fails. By appending the ptest directory with the multiconfig component in the path, only failing ptests will lead to corresponding ptest artifacts retrieval, instead of all ptests artifacts retrieval. While doing this addition, reinforce default value using "=" operator to make sure to get the default list in any case. (From OE-Core rev: 9357ab6c47f0a0a7000cb18358bc9775fd54e1f7) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Drop target_dumper and most of monitor_dumperRichard Purdie2023-12-201-16/+0
| | | | | | | | | | | | | | | | The target_dumper code is basically broken. It has been reading binary files over the text base serial communication and runs at every command failure which makes no sense. Each run might overwrite files from the previous run and the output appears corrupted due to confusion from the binary data. For now, remove the commands and the target dumper code as the command and execution point are problematic. Also remove the same pieces of the monitor code but leave the command list since in theory this can be moved to a more useful place in the code. (From OE-Core rev: a24d787987dccc95fdd95b7e85bf525a1c55b285) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Exclude wtmp from target-dumper commandsRichard Purdie2023-12-201-1/+1
| | | | | | | | | | | | wtmp is filled with binary data which the run_serial command can't cope with. Catting this results in confusion of the serial interface and potentially large backlogs of data in the buffers which can hang qemu. Exclude the problematic files from the command. (From OE-Core rev: 599ac08a6f6fb3f6a89a897c8e06367c63c2f979) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage.bbclass: remove QEMU_USE_SLIRP variableMikko Rapeli2023-09-141-1/+1
| | | | | | | | | | QEMU_USE_SLIRP is replaced by TEST_RUNQEMUPARAMS with "slirp" and possibly other arguments to runqemu script. (From OE-Core rev: 99fd24f0d9ff79fed389ae5a01c3031d7e7167d0) Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: respect target/server IPs when using slirpRoss Burton2023-09-141-3/+6
| | | | | | | | | | | | | | | | | | | We can't hardcode these IPs when using slirp. The target IP will need a port to be specified as this controls what port the SSH connection uses, and when slirp is used it can't bind to port 22. The qemu runner (OEQemuTarget) assumes that the first port forward is the SSH forward, but this may be wrong or a different target may be used. The server IP depends on how the virtual networking is configured. runqemu defaults to 10.0.2.x for the guests so that is a wise default, but that may be configured differently. (From OE-Core rev: 81b304e2558730de285f2773371340fc636a8ed1) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage.bbclass: detect slirp from TEST_RUNQEMUPARAMSMikko Rapeli2023-09-121-3/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also set target and server IP addresses correctly to 127.0.0.1 so that TEST_TARGET_IP and TEST_SERVER_IP don't need to be set manually with slirp. Users run qemu images with slirp networking like this: $ bitbake core-image-minimal $ ../poky/scripts/runqemu slirp slirp networking means that only one TCP port is forwarded from host system to the qemu machine so that ssh into the machine works. ping and other low level networking from host to target machine do not work, but networking from target via host does work. This is much more easy to setup than the bridge networking used by default, which I've never managed to get running on various host machines. To use slirp networking wita qemu and core-image-minimal to run testimage.bbclass tests, add these to local.conf: IMAGE_CLASSES += "testimage" IMAGE_FEATURES += "ssh-server-dropbear" TEST_RUNQEMUPARAMS += "slirp" Then image can be compiled and tested with: $ bitbake core-image-minimal $ bitbake -c testimage core-image-minimal Note that only few of the current oeqa runtime tests work against core-image-minimal, but test results look like: RESULTS: RESULTS - date.DateTest.test_date: PASSED (3.64s) RESULTS - df.DfTest.test_df: PASSED (0.55s) RESULTS - oe_syslog.SyslogTest.test_syslog_running: PASSED (0.56s) RESULTS - oe_syslog.SyslogTestConfig.test_syslog_logger: PASSED (1.88s) RESULTS - oe_syslog.SyslogTestConfig.test_syslog_restart: PASSED (0.93s) RESULTS - pam.PamBasicTest.test_pam: PASSED (2.20s) RESULTS - parselogs.ParseLogsTest.test_parselogs: PASSED (4.98s) RESULTS - ping.PingTest.test_ping: PASSED (0.05s) RESULTS - ssh.SSHTest.test_ssh: PASSED (1.28s) RESULTS - systemd.SystemdBasicTests.test_systemd_basic: PASSED (0.56s) RESULTS - systemd.SystemdBasicTests.test_systemd_failed: PASSED (1.10s) RESULTS - systemd.SystemdBasicTests.test_systemd_list: PASSED (0.92s) RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time: PASSED (0.56s) RESULTS - systemd.SystemdJournalTests.test_systemd_journal: PASSED (0.54s) RESULTS - apt.AptRepoTest.test_apt_install_from_repo: SKIPPED (0.00s) RESULTS - buildcpio.BuildCpioTest.test_cpio: SKIPPED (0.00s) RESULTS - buildgalculator.GalculatorTest.test_galculator: SKIPPED (0.00s) RESULTS - buildlzip.BuildLzipTest.test_lzip: SKIPPED (0.00s) RESULTS - connman.ConnmanTest.test_connmand_help: SKIPPED (0.00s) RESULTS - connman.ConnmanTest.test_connmand_running: SKIPPED (0.00s) RESULTS - dnf.DnfBasicTest.test_dnf_help: SKIPPED (0.00s) RESULTS - dnf.DnfBasicTest.test_dnf_history: SKIPPED (0.00s) RESULTS - dnf.DnfBasicTest.test_dnf_info: SKIPPED (0.00s) RESULTS - dnf.DnfBasicTest.test_dnf_search: SKIPPED (0.00s) RESULTS - dnf.DnfBasicTest.test_dnf_version: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_exclude: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_install: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_install_dependency: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_install_from_disk: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_install_from_http: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_installroot: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_installroot_usrmerge: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_makecache: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_reinstall: SKIPPED (0.00s) RESULTS - dnf.DnfRepoTest.test_dnf_repoinfo: SKIPPED (0.00s) RESULTS - gcc.GccCompileTest.test_gcc_compile: SKIPPED (0.00s) RESULTS - gcc.GccCompileTest.test_gpp2_compile: SKIPPED (0.00s) RESULTS - gcc.GccCompileTest.test_gpp_compile: SKIPPED (0.00s) RESULTS - gcc.GccCompileTest.test_make: SKIPPED (0.00s) RESULTS - gi.GObjectIntrospectionTest.test_python: SKIPPED (0.00s) RESULTS - go.GoHelloworldTest.test_gohelloworld: SKIPPED (0.00s) RESULTS - kernelmodule.KernelModuleTest.test_kernel_module: SKIPPED (0.00s) RESULTS - ldd.LddTest.test_ldd: SKIPPED (0.00s) RESULTS - logrotate.LogrotateTest.test_logrotate_newlog: SKIPPED (0.00s) RESULTS - logrotate.LogrotateTest.test_logrotate_wtmp: SKIPPED (0.00s) RESULTS - oe_syslog.SyslogTestConfig.test_syslog_startup_config: SKIPPED (0.00s) RESULTS - opkg.OpkgRepoTest.test_opkg_install_from_repo: SKIPPED (0.00s) RESULTS - perl.PerlTest.test_perl_works: SKIPPED (0.00s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectfail: SKIPPED (0.00s) RESULTS - ptest.PtestRunnerTest.test_ptestrunner_expectsuccess: SKIPPED (0.00s) RESULTS - python.PythonTest.test_python3: SKIPPED (0.00s) RESULTS - rpm.RpmBasicTest.test_rpm_help: SKIPPED (0.00s) RESULTS - rpm.RpmBasicTest.test_rpm_query: SKIPPED (0.00s) RESULTS - rpm.RpmBasicTest.test_rpm_query_nonroot: SKIPPED (0.00s) RESULTS - rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size: SKIPPED (0.00s) RESULTS - rpm.RpmInstallRemoveTest.test_rpm_install: SKIPPED (0.00s) RESULTS - rpm.RpmInstallRemoveTest.test_rpm_remove: SKIPPED (0.00s) RESULTS - rust.RustCompileTest.test_cargo_compile: SKIPPED (0.00s) RESULTS - rust.RustCompileTest.test_rust_compile: SKIPPED (0.00s) RESULTS - scp.ScpTest.test_scp_file: SKIPPED (0.00s) RESULTS - stap.StapTest.test_stap: SKIPPED (0.00s) RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable: SKIPPED (0.00s) RESULTS - systemd.SystemdServiceTests.test_systemd_disable_enable_ro: SKIPPED (0.00s) RESULTS - systemd.SystemdServiceTests.test_systemd_status: SKIPPED (0.00s) RESULTS - systemd.SystemdServiceTests.test_systemd_stop_start: SKIPPED (0.00s) RESULTS - weston.WestonTest.test_wayland_info: SKIPPED (0.00s) RESULTS - weston.WestonTest.test_weston_can_initialize_new_wayland_compositor: SKIPPED (0.00s) RESULTS - weston.WestonTest.test_weston_running: SKIPPED (0.00s) RESULTS - weston.WestonTest.test_weston_supports_xwayland: SKIPPED (0.00s) RESULTS - xorg.XorgTest.test_xorg_running: SKIPPED (0.00s) SUMMARY: core-image-minimal () - Ran 70 tests in 22.173s core-image-minimal - OK - All required tests passed (successes=14, skipped=56, failures=0, errors=0) NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Succeeded NOTE: Tasks Summary: Attempted 1305 tasks of which 1304 didn't need to be rerun and all succeeded. (From OE-Core rev: f4e865062cec06586d8c38c05d86a5b0d727625c) Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage/oeqa: Drop testimage_dump_host functionalityThomas Roos2023-06-291-19/+1
| | | | | | | | | | | | | | | | | The intent behind these functions was to dump the system state when issues occured but it has never really worked as we'd planned. Regular monitoring as the build runs has largely replaced this as that allows a trend to be seen rather than a spot value which was never really useful. The code is bitrotting and not functioning correctly so drop it. [YOCTO #13872] RP: Reword commit message (From OE-Core rev: dea37ba49a236029da73d5cfbfc069bffc38b508) Signed-off-by: Thomas Roos <throos@amazon.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Only note missing target directories, don't warnRichard Purdie2023-06-171-1/+1
| | | | | | | | | We don't need to see warnings for missing target debug directories. Just show a note in the logs instead. (From OE-Core rev: 52db25c58069c4f440da33daf0474255c9fa870b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: implement test artifacts retriever for failing testsAlexis Lothoré2023-06-171-0/+48
| | | | | | | | | | | | | | | | | | | | | Add a basic artifacts retrievers in testimage class which: - triggers when at least one runtime test fails but tests execution encountered no major issue - reads a list of paths to retrieve from TESTIMAGE_FAILED_QA_ARTIFACTS - checks for artifacts presence on target - retrieve those files over scp thanks to existing ssh class - store those files in an "artifacts" directory in "tmp/log/oeqa/<image>" This implementation assumes that the SSH or Qemu target has run and finished gracefully. If tests do not finish because of an exception, artifacts will not be retrieved Bring partial solution to [YOCTO #14901] (From OE-Core rev: 36ef582b8c1c99e6af1ce79ea79f5b059d2a1aad) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqemu: respect IMAGE_LINK_NAMEMartin Jansa2023-04-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | * when searching for qemuboot.conf * don't assume that IMAGE_LINK_NAME is always <rootfs>-<machine> (with <rootfs>-<machine>.qemuboot.conf) * runqemu: use IMAGE_LINK_NAME set by testimage.bbclass or query with bitbake -e * testimage.bbclass was setting DEPLOY_DIR which I don't see used anywhere else, so I assume it was supposed to be DEPLOY_DIR_IMAGE as mentioned in corresponding runqemu code, do the same with IMAGE_LINK_NAME variable * add virtual/kernel as bitbake -e target in run_bitbake_env to make sure IMAGE_LINK_NAME is defined (kernel-artifact-names.bbclass inherits image-artifact-names.bbclass as well) * improve .qemuboot.conf search 1st search for file matching the rootfs and only when not found try again with .rootfs suffix removed [YOCTO #12937] (From OE-Core rev: 716eb55bb963db7b02d985849cb025898aabc855) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Fix error message to reflect new syntaxAlejandro Hernandez Samaniego2023-02-021-1/+1
| | | | | | | | (From OE-Core rev: ec73d19d78e8f30ff9b817490c23bcdf8ea47c86) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandro@enedino.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/dnf: use dnf-test packagesRoss Burton2022-09-261-4/+1
| | | | | | | | | | | | | | | | | | Instead of installing run-postinsts with it's postinst scripts causing systemd restarts, use the new dnf-test-* packages instead. Remove from the installroot tests entirely as they're exercised enough using just busybox. Rewrite the exclude test to be simplier now these packages are not going to be part of an existing dependency chain. [ YOCTO #14787 ] (From OE-Core rev: fb1de2abc53bd742bc55cfecd384b78852c10d80) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/qemurunner: Work around possible control character contaminationRichard Purdie2022-09-131-5/+1
| | | | | | | | | | | | | | | | | Using a binary string as the login banner search expression is fraught with risks. We've seen cases on the autobuilder where "login:" is clearly shown but the code hasn't triggered. The most likely cause is hidden control characters in the output causing the search to fail. Take the opportunity to remove the horrible binary string search, at the expense of decoding the bootlog multiple times. Tweak the logging so we can know which log was printed (self.msg or bootlog) just in case this isn't the issue and we need more information in future. (From OE-Core rev: 91b9e30e08695e715ef14c3df7471e8c99f9deb5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: Update classes to match new bitbake class scope functionalityRichard Purdie2022-08-121-0/+508
Move classes to classes-global or classes-recipe as appropriate to take advantage of new bitbake functionality to check class scope/usage. (From OE-Core rev: f5c128008365e141082c129417eb72d2751e8045) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-27 07:09:59 +0000