From d6ae2c0decc0e71f1d0c1a752d9cc0602e727557 Mon Sep 17 00:00:00 2001 From: Antonin Godard Date: Thu, 26 Dec 2024 16:20:17 +0100 Subject: ref-manual: move runtime-testing section to the test-manual In the same fashion as the previous commit ("ref-manual/packages: move ptest section to the test-manual"), move the runtime testing section of the development tasks manual to the test environment manual. Add a link to it from the test-manual/intro document. (From yocto-docs rev: 6b44257874858db3aa426d3e84a79c41cb4937a3) Signed-off-by: Antonin Godard Signed-off-by: Richard Purdie --- documentation/dev-manual/index.rst | 1 - documentation/dev-manual/runtime-testing.rst | 594 --------------------------- 2 files changed, 595 deletions(-) delete mode 100644 documentation/dev-manual/runtime-testing.rst (limited to 'documentation/dev-manual') diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst index 7afd0d820e..8243c0f4cb 100644 --- a/documentation/dev-manual/index.rst +++ b/documentation/dev-manual/index.rst @@ -39,7 +39,6 @@ Yocto Project Development Tasks Manual external-scm read-only-rootfs build-quality - runtime-testing debugging licenses security-subjects diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst deleted file mode 100644 index 7a2b42f25a..0000000000 --- a/documentation/dev-manual/runtime-testing.rst +++ /dev/null @@ -1,594 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-SA-2.0-UK - -Performing Automated Runtime Testing -************************************ - -The OpenEmbedded build system makes available a series of automated -tests for images to verify runtime functionality. You can run these -tests on either QEMU or actual target hardware. Tests are written in -Python making use of the ``unittest`` module, and the majority of them -run commands on the target system over SSH. This section describes how -you set up the environment to use these tests, run available tests, and -write and add your own tests. - -For information on the test and QA infrastructure available within the -Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`" -section in the Yocto Project Reference Manual. - -Enabling Tests -============== - -Depending on whether you are planning to run tests using QEMU or on the -hardware, you have to take different steps to enable the tests. See the -following subsections for information on how to enable both types of -tests. - -Enabling Runtime Tests on QEMU ------------------------------- - -In order to run tests, you need to do the following: - -- *Set up to avoid interaction with sudo for networking:* To - accomplish this, you must do one of the following: - - - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all - commands or just for ``runqemu-ifup``. You must provide the full - path as that can change if you are using multiple clones of the - source repository. - - .. note:: - - On some distributions, you also need to comment out "Defaults - requiretty" in ``/etc/sudoers``. - - - Manually configure a tap interface for your system. - - - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which - should generate a list of tap devices. This is the option - typically chosen for Autobuilder-type environments. - - .. note:: - - - Be sure to use an absolute path when calling this script - with sudo. - - - Ensure that your host has the package ``iptables`` installed. - - - The package recipe ``qemu-helper-native`` is required to run - this script. Build the package using the following command:: - - $ bitbake qemu-helper-native - -- *Set the DISPLAY variable:* You need to set this variable so that - you have an X server available (e.g. start ``vncserver`` for a - headless machine). - -- *Be sure your host's firewall accepts incoming connections from - 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an - HTTP server on a random high number port, which is used to serve - files to the target. The DNF module serves - ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands. - That means your host's firewall must accept incoming connections from - 192.168.7.0/24, which is the default IP range used for tap devices by - ``runqemu``. - -- *Be sure your host has the correct packages installed:* Depending - your host's distribution, you need to have the following packages - installed: - - - Ubuntu and Debian: ``sysstat`` and ``iproute2`` - - - openSUSE: ``sysstat`` and ``iproute2`` - - - Fedora: ``sysstat`` and ``iproute`` - - - CentOS: ``sysstat`` and ``iproute`` - -Once you start running the tests, the following happens: - -#. A copy of the root filesystem is written to ``${WORKDIR}/testimage``. - -#. The image is booted under QEMU using the standard ``runqemu`` script. - -#. A default timeout of 500 seconds occurs to allow for the boot process - to reach the login prompt. You can change the timeout period by - setting - :term:`TEST_QEMUBOOT_TIMEOUT` - in the ``local.conf`` file. - -#. Once the boot process is reached and the login prompt appears, the - tests run. The full boot log is written to - ``${WORKDIR}/testimage/qemu_boot_log``. - -#. Each test module loads in the order found in :term:`TEST_SUITES`. You can - find the full output of the commands run over SSH in - ``${WORKDIR}/testimgage/ssh_target_log``. - -#. If no failures occur, the task running the tests ends successfully. - You can find the output from the ``unittest`` in the task log at - ``${WORKDIR}/temp/log.do_testimage``. - -Enabling Runtime Tests on Hardware ----------------------------------- - -The OpenEmbedded build system can run tests on real hardware, and for -certain devices it can also deploy the image to be tested onto the -device beforehand. - -For automated deployment, a "controller image" is installed onto the -hardware once as part of setup. Then, each time tests are to be run, the -following occurs: - -#. The controller image is booted into and used to write the image to be - tested to a second partition. - -#. The device is then rebooted using an external script that you need to - provide. - -#. The device boots into the image to be tested. - -When running tests (independent of whether the image has been deployed -automatically or not), the device is expected to be connected to a -network on a pre-determined IP address. You can either use static IP -addresses written into the image, or set the image to use DHCP and have -your DHCP server on the test network assign a known IP address based on -the MAC address of the device. - -In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an -appropriate value. For QEMU, you do not have to change anything, the -default value is "qemu". For running tests on hardware, the following -options are available: - -- *"simpleremote":* Choose "simpleremote" if you are going to run tests - on a target system that is already running the image to be tested and - is available on the network. You can use "simpleremote" in - conjunction with either real hardware or an image running within a - separately started QEMU or any other virtual machine manager. - -- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is - an EFI-based machine with ``systemd-boot`` as bootloader and - ``core-image-testmaster`` (or something similar) is installed. Also, - your hardware under test must be in a DHCP-enabled network that gives - it the same IP address for each reboot. - - If you choose "SystemdbootTarget", there are additional requirements - and considerations. See the - ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which - follows, for more information. - -- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying - images and running tests on the BeagleBone "Black" or original - "White" hardware. For information on how to use these tests, see the - comments at the top of the BeagleBoneTarget - ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file. - -- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running - tests on any generic PC that boots using GRUB. For information on how - to use these tests, see the comments at the top of the GrubTarget - ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file. - -- *"your-target":* Create your own custom target if you want to run - tests when you are deploying images and running tests on a custom - machine within your BSP layer. To do this, you need to add a Python - unit that defines the target class under ``lib/oeqa/controllers/`` - within your layer. You must also provide an empty ``__init__.py``. - For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``. - -Selecting SystemdbootTarget ---------------------------- - -If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do -not need any information in this section. You can skip down to the -":ref:`dev-manual/runtime-testing:running tests`" section. - -If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to -perform a one-time setup of your controller image by doing the following: - -#. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows:: - - EFI_PROVIDER = "systemd-boot" - -#. *Build the controller image:* Build the ``core-image-testmaster`` image. - The ``core-image-testmaster`` recipe is provided as an example for a - "controller" image and you can customize the image recipe as you would - any other recipe. - - Image recipe requirements are: - - - Inherits ``core-image`` so that kernel modules are installed. - - - Installs normal linux utilities not BusyBox ones (e.g. ``bash``, - ``coreutils``, ``tar``, ``gzip``, and ``kmod``). - - - Uses a custom :term:`Initramfs` image with a custom - installer. A normal image that you can install usually creates a - single root filesystem partition. This image uses another installer that - creates a specific partition layout. Not all Board Support - Packages (BSPs) can use an installer. For such cases, you need to - manually create the following partition layout on the target: - - - First partition mounted under ``/boot``, labeled "boot". - - - The main root filesystem partition where this image gets installed, - which is mounted under ``/``. - - - Another partition labeled "testrootfs" where test images get - deployed. - -#. *Install image:* Install the image that you just built on the target - system. - -The final thing you need to do when setting :term:`TEST_TARGET` to -"SystemdbootTarget" is to set up the test image: - -#. *Set up your local.conf file:* Make sure you have the following - statements in your ``local.conf`` file:: - - IMAGE_FSTYPES += "tar.gz" - IMAGE_CLASSES += "testimage" - TEST_TARGET = "SystemdbootTarget" - TEST_TARGET_IP = "192.168.2.3" - -#. *Build your test image:* Use BitBake to build the image:: - - $ bitbake core-image-sato - -Power Control -------------- - -For most hardware targets other than "simpleremote", you can control -power: - -- You can use :term:`TEST_POWERCONTROL_CMD` together with - :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host - and does power cycling. The test code passes one argument to that - command: off, on or cycle (off then on). Here is an example that - could appear in your ``local.conf`` file:: - - TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1" - - In this example, the expect - script does the following: - - .. code-block:: shell - - ssh test@10.11.12.1 "pyctl nuc1 arg" - - It then runs a Python script that controls power for a label called - ``nuc1``. - - .. note:: - - You need to customize :term:`TEST_POWERCONTROL_CMD` and - :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement - is that it accepts "on", "off", and "cycle" as the last argument. - -- When no command is defined, it connects to the device over SSH and - uses the classic reboot command to reboot the device. Classic reboot - is fine as long as the machine actually reboots (i.e. the SSH test - has not failed). It is useful for scenarios where you have a simple - setup, typically with a single board, and where some manual - interaction is okay from time to time. - -If you have no hardware to automatically perform power control but still -wish to experiment with automated hardware testing, you can use the -``dialog-power-control`` script that shows a dialog prompting you to perform -the required power action. This script requires either KDialog or Zenity -to be installed. To use this script, set the -:term:`TEST_POWERCONTROL_CMD` -variable as follows:: - - TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control" - -Serial Console Connection -------------------------- - -For test target classes requiring a serial console to interact with the -bootloader (e.g. BeagleBoneTarget and GrubTarget), -you need to specify a command to use to connect to the serial console of -the target machine by using the -:term:`TEST_SERIALCONTROL_CMD` -variable and optionally the -:term:`TEST_SERIALCONTROL_EXTRA_ARGS` -variable. - -These cases could be a serial terminal program if the machine is -connected to a local serial port, or a ``telnet`` or ``ssh`` command -connecting to a remote console server. Regardless of the case, the -command simply needs to connect to the serial console and forward that -connection to standard input and output as any normal terminal program -does. For example, to use the picocom terminal program on serial device -``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:: - - TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200" - -For local -devices where the serial port device disappears when the device reboots, -an additional "serdevtry" wrapper script is provided. To use this -wrapper, simply prefix the terminal command with -``${COREBASE}/scripts/contrib/serdevtry``:: - - TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0" - -Running Tests -============= - -You can start the tests automatically or manually: - -- *Automatically running tests:* To run the tests automatically after the - OpenEmbedded build system successfully creates an image, first set the - :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the - :term:`Build Directory`:: - - TESTIMAGE_AUTO = "1" - - Next, build your image. If the image successfully builds, the - tests run:: - - bitbake core-image-sato - -- *Manually running tests:* To manually run the tests, first globally - inherit the :ref:`ref-classes-testimage` class by editing your - ``local.conf`` file:: - - IMAGE_CLASSES += "testimage" - - Next, use BitBake to run the tests:: - - bitbake -c testimage image - -All test files reside in ``meta/lib/oeqa/runtime/cases`` in the -:term:`Source Directory`. A test name maps -directly to a Python module. Each test module may contain a number of -individual tests. Tests are usually grouped together by the area tested -(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``). - -You can add tests to any layer provided you place them in the proper -area and you extend :term:`BBPATH` in -the ``local.conf`` file as normal. Be sure that tests reside in -``layer/lib/oeqa/runtime/cases``. - -.. note:: - - Be sure that module names do not collide with module names used in - the default set of test modules in ``meta/lib/oeqa/runtime/cases``. - -You can change the set of tests run by appending or overriding -:term:`TEST_SUITES` variable in -``local.conf``. Each name in :term:`TEST_SUITES` represents a required test -for the image. Test modules named within :term:`TEST_SUITES` cannot be -skipped even if a test is not suitable for an image (e.g. running the -RPM tests on an image without ``rpm``). Appending "auto" to -:term:`TEST_SUITES` causes the build system to try to run all tests that are -suitable for the image (i.e. each test module may elect to skip itself). - -The order you list tests in :term:`TEST_SUITES` is important and influences -test dependencies. Consequently, tests that depend on other tests should -be added after the test on which they depend. For example, since the -``ssh`` test depends on the ``ping`` test, "ssh" needs to come after -"ping" in the list. The test class provides no re-ordering or dependency -handling. - -.. note:: - - Each module can have multiple classes with multiple test methods. - And, Python ``unittest`` rules apply. - -Here are some things to keep in mind when running tests: - -- The default tests for the image are defined as:: - - DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg" - -- Add your own test to the list of the by using the following:: - - TEST_SUITES:append = " mytest" - -- Run a specific list of tests as follows:: - - TEST_SUITES = "test1 test2 test3" - - Remember, order is important. Be sure to place a test that is - dependent on another test later in the order. - -Exporting Tests -=============== - -You can export tests so that they can run independently of the build -system. Exporting tests is required if you want to be able to hand the -test execution off to a scheduler. You can only export tests that are -defined in :term:`TEST_SUITES`. - -If your image is already built, make sure the following are set in your -``local.conf`` file:: - - INHERIT += "testexport" - TEST_TARGET_IP = "IP-address-for-the-test-target" - TEST_SERVER_IP = "IP-address-for-the-test-server" - -You can then export the tests with the -following BitBake command form:: - - $ bitbake image -c testexport - -Exporting the tests places them in the :term:`Build Directory` in -``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR` -variable. - -You can now run the tests outside of the build environment:: - - $ cd tmp/testexport/image - $ ./runexported.py testdata.json - -Here is a complete example that shows IP addresses and uses the -``core-image-sato`` image:: - - INHERIT += "testexport" - TEST_TARGET_IP = "192.168.7.2" - TEST_SERVER_IP = "192.168.7.1" - -Use BitBake to export the tests:: - - $ bitbake core-image-sato -c testexport - -Run the tests outside of -the build environment using the following:: - - $ cd tmp/testexport/core-image-sato - $ ./runexported.py testdata.json - -Writing New Tests -================= - -As mentioned previously, all new test files need to be in the proper -place for the build system to find them. New tests for additional -functionality outside of the core should be added to the layer that adds -the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as -:term:`BBPATH` is extended in the -layer's ``layer.conf`` file as normal). Just remember the following: - -- Filenames need to map directly to test (module) names. - -- Do not use module names that collide with existing core tests. - -- Minimally, an empty ``__init__.py`` file must be present in the runtime - directory. - -To create a new test, start by copying an existing module (e.g. -``oe_syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use -code from ``meta/lib/oeqa/utils``, which are helper classes. - -.. note:: - - Structure shell commands such that you rely on them and they return a - single code for success. Be aware that sometimes you will need to - parse the output. See the ``df.py`` and ``date.py`` modules for examples. - -You will notice that all test classes inherit ``oeRuntimeTest``, which -is found in ``meta/lib/oetest.py``. This base class offers some helper -attributes, which are described in the following sections: - -Class Methods -------------- - -Class methods are as follows: - -- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed - package list of the image, which is based on the manifest file that - is generated during the :ref:`ref-tasks-rootfs` task. - -- *hasFeature(feature):* Returns "True" if the feature is in - :term:`IMAGE_FEATURES` or - :term:`DISTRO_FEATURES`. - -Class Attributes ----------------- - -Class attributes are as follows: - -- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image. - Otherwise, ``pscmd`` equals "ps" (busybox). - -- *tc:* The called test context, which gives access to the - following attributes: - - - *d:* The BitBake datastore, which allows you to use stuff such - as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``. - - - *testslist and testsrequired:* Used internally. The tests - do not need these. - - - *filesdir:* The absolute path to - ``meta/lib/oeqa/runtime/files``, which contains helper files for - tests meant for copying on the target such as small files written - in C for compilation. - - - *target:* The target controller object used to deploy and - start an image on a particular target (e.g. Qemu, SimpleRemote, - and SystemdbootTarget). Tests usually use the following: - - - *ip:* The target's IP address. - - - *server_ip:* The host's IP address, which is usually used - by the DNF test suite. - - - *run(cmd, timeout=None):* The single, most used method. - This command is a wrapper for: ``ssh root@host "cmd"``. The - command returns a tuple: (status, output), which are what their - names imply - the return code of "cmd" and whatever output it - produces. The optional timeout argument represents the number - of seconds the test should wait for "cmd" to return. If the - argument is "None", the test uses the default instance's - timeout period, which is 300 seconds. If the argument is "0", - the test runs until the command returns. - - - *copy_to(localpath, remotepath):* - ``scp localpath root@ip:remotepath``. - - - *copy_from(remotepath, localpath):* - ``scp root@host:remotepath localpath``. - -Instance Attributes -------------------- - -There is a single instance attribute, which is ``target``. The ``target`` -instance attribute is identical to the class attribute of the same name, -which is described in the previous section. This attribute exists as -both an instance and class attribute so tests can use -``self.target.run(cmd)`` in instance methods instead of -``oeRuntimeTest.tc.target.run(cmd)``. - -Installing Packages in the DUT Without the Package Manager -========================================================== - -When a test requires a package built by BitBake, it is possible to -install that package. Installing the package does not require a package -manager be installed in the device under test (DUT). It does, however, -require an SSH connection and the target must be using the -``sshcontrol`` class. - -.. note:: - - This method uses ``scp`` to copy files from the host to the target, which - causes permissions and special attributes to be lost. - -A JSON file is used to define the packages needed by a test. This file -must be in the same path as the file used to define the tests. -Furthermore, the filename must map directly to the test module name with -a ``.json`` extension. - -The JSON file must include an object with the test name as keys of an -object or an array. This object (or array of objects) uses the following -data: - -- "pkg" --- a mandatory string that is the name of the package to be - installed. - -- "rm" --- an optional boolean, which defaults to "false", that specifies - to remove the package after the test. - -- "extract" --- an optional boolean, which defaults to "false", that - specifies if the package must be extracted from the package format. - When set to "true", the package is not automatically installed into - the DUT. - -Here is an example JSON file that handles test "foo" installing -package "bar" and test "foobar" installing packages "foo" and "bar". -Once the test is complete, the packages are removed from the DUT:: - - { - "foo": { - "pkg": "bar" - }, - "foobar": [ - { - "pkg": "foo", - "rm": true - }, - { - "pkg": "bar", - "rm": true - } - ] - } - -- cgit v1.2.3-54-g00ecf