From ac2d3a537751ddc3399f82fc85bfb4a6dd38cf7d Mon Sep 17 00:00:00 2001 From: Antonin Godard Date: Thu, 26 Dec 2024 16:20:17 +0100 Subject: ref-manual: move runtime-testing section to the test-manual In the same fashion as the previous commit ("ref-manual/packages: move ptest section to the test-manual"), move the runtime testing section of the development tasks manual to the test environment manual. Add a link to it from the test-manual/intro document. (From yocto-docs rev: 3128bf149f40928e6c2a3e264590a0c6c9778c6a) Signed-off-by: Antonin Godard (cherry picked from commit 6b44257874858db3aa426d3e84a79c41cb4937a3) Signed-off-by: Antonin Godard Signed-off-by: Steve Sakoman --- documentation/dev-manual/index.rst | 1 - documentation/dev-manual/runtime-testing.rst | 600 ---------------------- documentation/migration-guides/migration-1.5.rst | 4 +- documentation/ref-manual/classes.rst | 4 +- documentation/ref-manual/images.rst | 4 +- documentation/ref-manual/release-process.rst | 4 +- documentation/ref-manual/tasks.rst | 8 +- documentation/ref-manual/variables.rst | 20 +- documentation/test-manual/index.rst | 1 + documentation/test-manual/intro.rst | 4 +- documentation/test-manual/runtime-testing.rst | 601 +++++++++++++++++++++++ 11 files changed, 627 insertions(+), 624 deletions(-) delete mode 100644 documentation/dev-manual/runtime-testing.rst create mode 100644 documentation/test-manual/runtime-testing.rst diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst index 9ccf60f701..ef0512d246 100644 --- a/documentation/dev-manual/index.rst +++ b/documentation/dev-manual/index.rst @@ -39,7 +39,6 @@ Yocto Project Development Tasks Manual external-scm read-only-rootfs build-quality - runtime-testing debugging licenses security-subjects diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst deleted file mode 100644 index bd3f42ac09..0000000000 --- a/documentation/dev-manual/runtime-testing.rst +++ /dev/null @@ -1,600 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-SA-2.0-UK - -Performing Automated Runtime Testing -************************************ - -The OpenEmbedded build system makes available a series of automated -tests for images to verify runtime functionality. You can run these -tests on either QEMU or actual target hardware. Tests are written in -Python making use of the ``unittest`` module, and the majority of them -run commands on the target system over SSH. This section describes how -you set up the environment to use these tests, run available tests, and -write and add your own tests. - -For information on the test and QA infrastructure available within the -Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`" -section in the Yocto Project Reference Manual. - -Enabling Tests -============== - -Depending on whether you are planning to run tests using QEMU or on the -hardware, you have to take different steps to enable the tests. See the -following subsections for information on how to enable both types of -tests. - -Enabling Runtime Tests on QEMU ------------------------------- - -In order to run tests, you need to do the following: - -- *Set up to avoid interaction with sudo for networking:* To - accomplish this, you must do one of the following: - - - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all - commands or just for ``runqemu-ifup``. You must provide the full - path as that can change if you are using multiple clones of the - source repository. - - .. note:: - - On some distributions, you also need to comment out "Defaults - requiretty" in ``/etc/sudoers``. - - - Manually configure a tap interface for your system. - - - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which - should generate a list of tap devices. This is the option - typically chosen for Autobuilder-type environments. - - .. note:: - - - Be sure to use an absolute path when calling this script - with sudo. - - - Ensure that your host has the package ``iptables`` installed. - - - The package recipe ``qemu-helper-native`` is required to run - this script. Build the package using the following command:: - - $ bitbake qemu-helper-native - -- *Set the DISPLAY variable:* You need to set this variable so that - you have an X server available (e.g. start ``vncserver`` for a - headless machine). - -- *Be sure your host's firewall accepts incoming connections from - 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an - HTTP server on a random high number port, which is used to serve - files to the target. The DNF module serves - ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands. - That means your host's firewall must accept incoming connections from - 192.168.7.0/24, which is the default IP range used for tap devices by - ``runqemu``. - -- *Be sure your host has the correct packages installed:* Depending - your host's distribution, you need to have the following packages - installed: - - - Ubuntu and Debian: ``sysstat`` and ``iproute2`` - - - openSUSE: ``sysstat`` and ``iproute2`` - - - Fedora: ``sysstat`` and ``iproute`` - - - CentOS: ``sysstat`` and ``iproute`` - -Once you start running the tests, the following happens: - -#. A copy of the root filesystem is written to ``${WORKDIR}/testimage``. - -#. The image is booted under QEMU using the standard ``runqemu`` script. - -#. A default timeout of 500 seconds occurs to allow for the boot process - to reach the login prompt. You can change the timeout period by - setting - :term:`TEST_QEMUBOOT_TIMEOUT` - in the ``local.conf`` file. - -#. Once the boot process is reached and the login prompt appears, the - tests run. The full boot log is written to - ``${WORKDIR}/testimage/qemu_boot_log``. - -#. Each test module loads in the order found in :term:`TEST_SUITES`. You can - find the full output of the commands run over SSH in - ``${WORKDIR}/testimgage/ssh_target_log``. - -#. If no failures occur, the task running the tests ends successfully. - You can find the output from the ``unittest`` in the task log at - ``${WORKDIR}/temp/log.do_testimage``. - -Enabling Runtime Tests on Hardware ----------------------------------- - -The OpenEmbedded build system can run tests on real hardware, and for -certain devices it can also deploy the image to be tested onto the -device beforehand. - -For automated deployment, a "controller image" is installed onto the -hardware once as part of setup. Then, each time tests are to be run, the -following occurs: - -#. The controller image is booted into and used to write the image to be - tested to a second partition. - -#. The device is then rebooted using an external script that you need to - provide. - -#. The device boots into the image to be tested. - -When running tests (independent of whether the image has been deployed -automatically or not), the device is expected to be connected to a -network on a pre-determined IP address. You can either use static IP -addresses written into the image, or set the image to use DHCP and have -your DHCP server on the test network assign a known IP address based on -the MAC address of the device. - -In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an -appropriate value. For QEMU, you do not have to change anything, the -default value is "qemu". For running tests on hardware, the following -options are available: - -- *"simpleremote":* Choose "simpleremote" if you are going to run tests - on a target system that is already running the image to be tested and - is available on the network. You can use "simpleremote" in - conjunction with either real hardware or an image running within a - separately started QEMU or any other virtual machine manager. - -- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is - an EFI-based machine with ``systemd-boot`` as bootloader and - ``core-image-testmaster`` (or something similar) is installed. Also, - your hardware under test must be in a DHCP-enabled network that gives - it the same IP address for each reboot. - - If you choose "SystemdbootTarget", there are additional requirements - and considerations. See the - ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which - follows, for more information. - -- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying - images and running tests on the BeagleBone "Black" or original - "White" hardware. For information on how to use these tests, see the - comments at the top of the BeagleBoneTarget - ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file. - -- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying - images and running tests on the Ubiquiti Networks EdgeRouter Lite. - For information on how to use these tests, see the comments at the - top of the EdgeRouterTarget - ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file. - -- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running - tests on any generic PC that boots using GRUB. For information on how - to use these tests, see the comments at the top of the GrubTarget - ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file. - -- *"your-target":* Create your own custom target if you want to run - tests when you are deploying images and running tests on a custom - machine within your BSP layer. To do this, you need to add a Python - unit that defines the target class under ``lib/oeqa/controllers/`` - within your layer. You must also provide an empty ``__init__.py``. - For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``. - -Selecting SystemdbootTarget ---------------------------- - -If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do -not need any information in this section. You can skip down to the -":ref:`dev-manual/runtime-testing:running tests`" section. - -If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to -perform a one-time setup of your controller image by doing the following: - -#. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows:: - - EFI_PROVIDER = "systemd-boot" - -#. *Build the controller image:* Build the ``core-image-testmaster`` image. - The ``core-image-testmaster`` recipe is provided as an example for a - "controller" image and you can customize the image recipe as you would - any other recipe. - - Image recipe requirements are: - - - Inherits ``core-image`` so that kernel modules are installed. - - - Installs normal linux utilities not BusyBox ones (e.g. ``bash``, - ``coreutils``, ``tar``, ``gzip``, and ``kmod``). - - - Uses a custom :term:`Initramfs` image with a custom - installer. A normal image that you can install usually creates a - single root filesystem partition. This image uses another installer that - creates a specific partition layout. Not all Board Support - Packages (BSPs) can use an installer. For such cases, you need to - manually create the following partition layout on the target: - - - First partition mounted under ``/boot``, labeled "boot". - - - The main root filesystem partition where this image gets installed, - which is mounted under ``/``. - - - Another partition labeled "testrootfs" where test images get - deployed. - -#. *Install image:* Install the image that you just built on the target - system. - -The final thing you need to do when setting :term:`TEST_TARGET` to -"SystemdbootTarget" is to set up the test image: - -#. *Set up your local.conf file:* Make sure you have the following - statements in your ``local.conf`` file:: - - IMAGE_FSTYPES += "tar.gz" - IMAGE_CLASSES += "testimage" - TEST_TARGET = "SystemdbootTarget" - TEST_TARGET_IP = "192.168.2.3" - -#. *Build your test image:* Use BitBake to build the image:: - - $ bitbake core-image-sato - -Power Control -------------- - -For most hardware targets other than "simpleremote", you can control -power: - -- You can use :term:`TEST_POWERCONTROL_CMD` together with - :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host - and does power cycling. The test code passes one argument to that - command: off, on or cycle (off then on). Here is an example that - could appear in your ``local.conf`` file:: - - TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1" - - In this example, the expect - script does the following: - - .. code-block:: shell - - ssh test@10.11.12.1 "pyctl nuc1 arg" - - It then runs a Python script that controls power for a label called - ``nuc1``. - - .. note:: - - You need to customize :term:`TEST_POWERCONTROL_CMD` and - :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement - is that it accepts "on", "off", and "cycle" as the last argument. - -- When no command is defined, it connects to the device over SSH and - uses the classic reboot command to reboot the device. Classic reboot - is fine as long as the machine actually reboots (i.e. the SSH test - has not failed). It is useful for scenarios where you have a simple - setup, typically with a single board, and where some manual - interaction is okay from time to time. - -If you have no hardware to automatically perform power control but still -wish to experiment with automated hardware testing, you can use the -``dialog-power-control`` script that shows a dialog prompting you to perform -the required power action. This script requires either KDialog or Zenity -to be installed. To use this script, set the -:term:`TEST_POWERCONTROL_CMD` -variable as follows:: - - TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control" - -Serial Console Connection -------------------------- - -For test target classes requiring a serial console to interact with the -bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget), -you need to specify a command to use to connect to the serial console of -the target machine by using the -:term:`TEST_SERIALCONTROL_CMD` -variable and optionally the -:term:`TEST_SERIALCONTROL_EXTRA_ARGS` -variable. - -These cases could be a serial terminal program if the machine is -connected to a local serial port, or a ``telnet`` or ``ssh`` command -connecting to a remote console server. Regardless of the case, the -command simply needs to connect to the serial console and forward that -connection to standard input and output as any normal terminal program -does. For example, to use the picocom terminal program on serial device -``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:: - - TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200" - -For local -devices where the serial port device disappears when the device reboots, -an additional "serdevtry" wrapper script is provided. To use this -wrapper, simply prefix the terminal command with -``${COREBASE}/scripts/contrib/serdevtry``:: - - TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0" - -Running Tests -============= - -You can start the tests automatically or manually: - -- *Automatically running tests:* To run the tests automatically after the - OpenEmbedded build system successfully creates an image, first set the - :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the - :term:`Build Directory`:: - - TESTIMAGE_AUTO = "1" - - Next, build your image. If the image successfully builds, the - tests run:: - - bitbake core-image-sato - -- *Manually running tests:* To manually run the tests, first globally - inherit the :ref:`ref-classes-testimage` class by editing your - ``local.conf`` file:: - - IMAGE_CLASSES += "testimage" - - Next, use BitBake to run the tests:: - - bitbake -c testimage image - -All test files reside in ``meta/lib/oeqa/runtime/cases`` in the -:term:`Source Directory`. A test name maps -directly to a Python module. Each test module may contain a number of -individual tests. Tests are usually grouped together by the area tested -(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``). - -You can add tests to any layer provided you place them in the proper -area and you extend :term:`BBPATH` in -the ``local.conf`` file as normal. Be sure that tests reside in -``layer/lib/oeqa/runtime/cases``. - -.. note:: - - Be sure that module names do not collide with module names used in - the default set of test modules in ``meta/lib/oeqa/runtime/cases``. - -You can change the set of tests run by appending or overriding -:term:`TEST_SUITES` variable in -``local.conf``. Each name in :term:`TEST_SUITES` represents a required test -for the image. Test modules named within :term:`TEST_SUITES` cannot be -skipped even if a test is not suitable for an image (e.g. running the -RPM tests on an image without ``rpm``). Appending "auto" to -:term:`TEST_SUITES` causes the build system to try to run all tests that are -suitable for the image (i.e. each test module may elect to skip itself). - -The order you list tests in :term:`TEST_SUITES` is important and influences -test dependencies. Consequently, tests that depend on other tests should -be added after the test on which they depend. For example, since the -``ssh`` test depends on the ``ping`` test, "ssh" needs to come after -"ping" in the list. The test class provides no re-ordering or dependency -handling. - -.. note:: - - Each module can have multiple classes with multiple test methods. - And, Python ``unittest`` rules apply. - -Here are some things to keep in mind when running tests: - -- The default tests for the image are defined as:: - - DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg" - -- Add your own test to the list of the by using the following:: - - TEST_SUITES:append = " mytest" - -- Run a specific list of tests as follows:: - - TEST_SUITES = "test1 test2 test3" - - Remember, order is important. Be sure to place a test that is - dependent on another test later in the order. - -Exporting Tests -=============== - -You can export tests so that they can run independently of the build -system. Exporting tests is required if you want to be able to hand the -test execution off to a scheduler. You can only export tests that are -defined in :term:`TEST_SUITES`. - -If your image is already built, make sure the following are set in your -``local.conf`` file:: - - INHERIT += "testexport" - TEST_TARGET_IP = "IP-address-for-the-test-target" - TEST_SERVER_IP = "IP-address-for-the-test-server" - -You can then export the tests with the -following BitBake command form:: - - $ bitbake image -c testexport - -Exporting the tests places them in the :term:`Build Directory` in -``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR` -variable. - -You can now run the tests outside of the build environment:: - - $ cd tmp/testexport/image - $ ./runexported.py testdata.json - -Here is a complete example that shows IP addresses and uses the -``core-image-sato`` image:: - - INHERIT += "testexport" - TEST_TARGET_IP = "192.168.7.2" - TEST_SERVER_IP = "192.168.7.1" - -Use BitBake to export the tests:: - - $ bitbake core-image-sato -c testexport - -Run the tests outside of -the build environment using the following:: - - $ cd tmp/testexport/core-image-sato - $ ./runexported.py testdata.json - -Writing New Tests -================= - -As mentioned previously, all new test files need to be in the proper -place for the build system to find them. New tests for additional -functionality outside of the core should be added to the layer that adds -the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as -:term:`BBPATH` is extended in the -layer's ``layer.conf`` file as normal). Just remember the following: - -- Filenames need to map directly to test (module) names. - -- Do not use module names that collide with existing core tests. - -- Minimally, an empty ``__init__.py`` file must be present in the runtime - directory. - -To create a new test, start by copying an existing module (e.g. -``oe_syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use -code from ``meta/lib/oeqa/utils``, which are helper classes. - -.. note:: - - Structure shell commands such that you rely on them and they return a - single code for success. Be aware that sometimes you will need to - parse the output. See the ``df.py`` and ``date.py`` modules for examples. - -You will notice that all test classes inherit ``oeRuntimeTest``, which -is found in ``meta/lib/oetest.py``. This base class offers some helper -attributes, which are described in the following sections: - -Class Methods -------------- - -Class methods are as follows: - -- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed - package list of the image, which is based on the manifest file that - is generated during the :ref:`ref-tasks-rootfs` task. - -- *hasFeature(feature):* Returns "True" if the feature is in - :term:`IMAGE_FEATURES` or - :term:`DISTRO_FEATURES`. - -Class Attributes ----------------- - -Class attributes are as follows: - -- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image. - Otherwise, ``pscmd`` equals "ps" (busybox). - -- *tc:* The called test context, which gives access to the - following attributes: - - - *d:* The BitBake datastore, which allows you to use stuff such - as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``. - - - *testslist and testsrequired:* Used internally. The tests - do not need these. - - - *filesdir:* The absolute path to - ``meta/lib/oeqa/runtime/files``, which contains helper files for - tests meant for copying on the target such as small files written - in C for compilation. - - - *target:* The target controller object used to deploy and - start an image on a particular target (e.g. Qemu, SimpleRemote, - and SystemdbootTarget). Tests usually use the following: - - - *ip:* The target's IP address. - - - *server_ip:* The host's IP address, which is usually used - by the DNF test suite. - - - *run(cmd, timeout=None):* The single, most used method. - This command is a wrapper for: ``ssh root@host "cmd"``. The - command returns a tuple: (status, output), which are what their - names imply - the return code of "cmd" and whatever output it - produces. The optional timeout argument represents the number - of seconds the test should wait for "cmd" to return. If the - argument is "None", the test uses the default instance's - timeout period, which is 300 seconds. If the argument is "0", - the test runs until the command returns. - - - *copy_to(localpath, remotepath):* - ``scp localpath root@ip:remotepath``. - - - *copy_from(remotepath, localpath):* - ``scp root@host:remotepath localpath``. - -Instance Attributes -------------------- - -There is a single instance attribute, which is ``target``. The ``target`` -instance attribute is identical to the class attribute of the same name, -which is described in the previous section. This attribute exists as -both an instance and class attribute so tests can use -``self.target.run(cmd)`` in instance methods instead of -``oeRuntimeTest.tc.target.run(cmd)``. - -Installing Packages in the DUT Without the Package Manager -========================================================== - -When a test requires a package built by BitBake, it is possible to -install that package. Installing the package does not require a package -manager be installed in the device under test (DUT). It does, however, -require an SSH connection and the target must be using the -``sshcontrol`` class. - -.. note:: - - This method uses ``scp`` to copy files from the host to the target, which - causes permissions and special attributes to be lost. - -A JSON file is used to define the packages needed by a test. This file -must be in the same path as the file used to define the tests. -Furthermore, the filename must map directly to the test module name with -a ``.json`` extension. - -The JSON file must include an object with the test name as keys of an -object or an array. This object (or array of objects) uses the following -data: - -- "pkg" --- a mandatory string that is the name of the package to be - installed. - -- "rm" --- an optional boolean, which defaults to "false", that specifies - to remove the package after the test. - -- "extract" --- an optional boolean, which defaults to "false", that - specifies if the package must be extracted from the package format. - When set to "true", the package is not automatically installed into - the DUT. - -Here is an example JSON file that handles test "foo" installing -package "bar" and test "foobar" installing packages "foo" and "bar". -Once the test is complete, the packages are removed from the DUT:: - - { - "foo": { - "pkg": "bar" - }, - "foobar": [ - { - "pkg": "foo", - "rm": true - }, - { - "pkg": "bar", - "rm": true - } - ] - } - diff --git a/documentation/migration-guides/migration-1.5.rst b/documentation/migration-guides/migration-1.5.rst index 06334d6fc7..adfecb98aa 100644 --- a/documentation/migration-guides/migration-1.5.rst +++ b/documentation/migration-guides/migration-1.5.rst @@ -243,8 +243,8 @@ A new automated image testing framework has been added through the framework replaces the older ``imagetest-qemu`` framework. You can learn more about performing automated image tests in the -":ref:`dev-manual/runtime-testing:performing automated runtime testing`" -section in the Yocto Project Development Tasks Manual. +":ref:`test-manual/runtime-testing:performing automated runtime testing`" +section in the Yocto Project Test Environment Manual. .. _migration-1.5-build-history: diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst index b609c3af66..31bb1eeb94 100644 --- a/documentation/ref-manual/classes.rst +++ b/documentation/ref-manual/classes.rst @@ -3064,8 +3064,8 @@ after it is built, you can set :term:`TESTIMAGE_AUTO`:: TESTIMAGE_AUTO = "1" For information on how to enable, run, and create new tests, see the -":ref:`dev-manual/runtime-testing:performing automated runtime testing`" -section in the Yocto Project Development Tasks Manual. +":ref:`test-manual/runtime-testing:performing automated runtime testing`" +section in the Yocto Project Test Environment Manual. .. _ref-classes-testsdk: diff --git a/documentation/ref-manual/images.rst b/documentation/ref-manual/images.rst index f875633128..d36a4b2507 100644 --- a/documentation/ref-manual/images.rst +++ b/documentation/ref-manual/images.rst @@ -119,8 +119,8 @@ Here is a list of supported recipes: deployed to a separate partition so that you can boot into it and use it to deploy a second image to be tested. You can find more information about runtime testing in the - ":ref:`dev-manual/runtime-testing:performing automated runtime testing`" - section in the Yocto Project Development Tasks Manual. + ":ref:`test-manual/runtime-testing:performing automated runtime testing`" + section in the Yocto Project Test Environment Manual. - ``core-image-testmaster-initramfs``: A RAM-based Initial Root Filesystem (initramfs) image tailored for use with the diff --git a/documentation/ref-manual/release-process.rst b/documentation/ref-manual/release-process.rst index 89f79812e8..aed4c51aa2 100644 --- a/documentation/ref-manual/release-process.rst +++ b/documentation/ref-manual/release-process.rst @@ -148,8 +148,8 @@ Additionally, because the test strategies are visible to you as a developer, you can validate your projects. This section overviews the available test infrastructure used in the Yocto Project. For information on how to run available tests on your projects, see the -":ref:`dev-manual/runtime-testing:performing automated runtime testing`" -section in the Yocto Project Development Tasks Manual. +":ref:`test-manual/runtime-testing:performing automated runtime testing`" +section in the Yocto Project Test Environment Manual. The QA/testing infrastructure is woven into the project to the point where core developers take some of it for granted. The infrastructure diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst index 346e9491f3..c69fbaa518 100644 --- a/documentation/ref-manual/tasks.rst +++ b/documentation/ref-manual/tasks.rst @@ -629,8 +629,8 @@ information on how the root filesystem is created. Boots an image and performs runtime tests within the image. For information on automatically testing images, see the -":ref:`dev-manual/runtime-testing:performing automated runtime testing`" -section in the Yocto Project Development Tasks Manual. +":ref:`test-manual/runtime-testing:performing automated runtime testing`" +section in the Yocto Project Test Environment Manual. .. _ref-tasks-testimage_auto: @@ -642,8 +642,8 @@ after it has been built. This task is enabled when you set :term:`TESTIMAGE_AUTO` equal to "1". For information on automatically testing images, see the -":ref:`dev-manual/runtime-testing:performing automated runtime testing`" -section in the Yocto Project Development Tasks Manual. +":ref:`test-manual/runtime-testing:performing automated runtime testing`" +section in the Yocto Project Test Environment Manual. Kernel-Related Tasks ==================== diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 8ab7391f98..feb8be6be9 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -8688,8 +8688,8 @@ system and gives an overview of their function and contents. file. For more information on testing images, see the - ":ref:`dev-manual/runtime-testing:performing automated runtime testing`" - section in the Yocto Project Development Tasks Manual. + ":ref:`test-manual/runtime-testing:performing automated runtime testing`" + section in the Yocto Project Test Environment Manual. :term:`TEST_SERIALCONTROL_CMD` For automated hardware testing, specifies the command to use to @@ -8761,8 +8761,8 @@ system and gives an overview of their function and contents. TEST_SUITES = "test_A test_B" For more information on testing images, see the - ":ref:`dev-manual/runtime-testing:performing automated runtime testing`" - section in the Yocto Project Development Tasks Manual. + ":ref:`test-manual/runtime-testing:performing automated runtime testing`" + section in the Yocto Project Test Environment Manual. :term:`TEST_TARGET` Specifies the target controller to use when running tests against a @@ -8780,8 +8780,8 @@ system and gives an overview of their function and contents. You can provide the following arguments with :term:`TEST_TARGET`: - *"qemu":* Boots a QEMU image and runs the tests. See the - ":ref:`dev-manual/runtime-testing:enabling runtime tests on qemu`" section - in the Yocto Project Development Tasks Manual for more + ":ref:`test-manual/runtime-testing:enabling runtime tests on qemu`" section + in the Yocto Project Test Environment Manual for more information. - *"simpleremote":* Runs the tests on target hardware that is @@ -8796,8 +8796,8 @@ system and gives an overview of their function and contents. ``meta/lib/oeqa/controllers/simpleremote.py``. For information on running tests on hardware, see the - ":ref:`dev-manual/runtime-testing:enabling runtime tests on hardware`" - section in the Yocto Project Development Tasks Manual. + ":ref:`test-manual/runtime-testing:enabling runtime tests on hardware`" + section in the Yocto Project Test Environment Manual. :term:`TEST_TARGET_IP` The IP address of your hardware under test. The :term:`TEST_TARGET_IP` @@ -8833,8 +8833,8 @@ system and gives an overview of their function and contents. For more information on enabling, running, and writing these tests, see the - ":ref:`dev-manual/runtime-testing:performing automated runtime testing`" - section in the Yocto Project Development Tasks Manual and the + ":ref:`test-manual/runtime-testing:performing automated runtime testing`" + section in the Yocto Project Test Environment Manual and the ":ref:`ref-classes-testimage`" section. :term:`THISDIR` diff --git a/documentation/test-manual/index.rst b/documentation/test-manual/index.rst index ad71f37910..d365d337ea 100644 --- a/documentation/test-manual/index.rst +++ b/documentation/test-manual/index.rst @@ -13,6 +13,7 @@ Yocto Project Test Environment Manual intro test-process ptest + runtime-testing understand-autobuilder reproducible-builds yocto-project-compatible diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index ff9b6d3794..3e97a939e6 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst @@ -131,7 +131,9 @@ the following types of tests: $ bitbake image -c testimage The tests use the :ref:`ref-classes-testimage` - class and the :ref:`ref-tasks-testimage` task. + class and the :ref:`ref-tasks-testimage` task. See the + :ref:`test-manual/runtime-testing:Performing Automated Runtime Testing` + section of the Yocto Project Test Environment Manual for more information. - *Layer Testing:* The Autobuilder has the possibility to test whether specific layers work with the test of the system. The layers tested diff --git a/documentation/test-manual/runtime-testing.rst b/documentation/test-manual/runtime-testing.rst new file mode 100644 index 0000000000..88797c7ad8 --- /dev/null +++ b/documentation/test-manual/runtime-testing.rst @@ -0,0 +1,601 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +************************************ +Performing Automated Runtime Testing +************************************ + +The OpenEmbedded build system makes available a series of automated +tests for images to verify runtime functionality. You can run these +tests on either QEMU or actual target hardware. Tests are written in +Python making use of the ``unittest`` module, and the majority of them +run commands on the target system over SSH. This section describes how +you set up the environment to use these tests, run available tests, and +write and add your own tests. + +For information on the test and QA infrastructure available within the +Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`" +section in the Yocto Project Reference Manual. + +Enabling Tests +============== + +Depending on whether you are planning to run tests using QEMU or on the +hardware, you have to take different steps to enable the tests. See the +following subsections for information on how to enable both types of +tests. + +Enabling Runtime Tests on QEMU +------------------------------ + +In order to run tests, you need to do the following: + +- *Set up to avoid interaction with sudo for networking:* To + accomplish this, you must do one of the following: + + - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all + commands or just for ``runqemu-ifup``. You must provide the full + path as that can change if you are using multiple clones of the + source repository. + + .. note:: + + On some distributions, you also need to comment out "Defaults + requiretty" in ``/etc/sudoers``. + + - Manually configure a tap interface for your system. + + - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which + should generate a list of tap devices. This is the option + typically chosen for Autobuilder-type environments. + + .. note:: + + - Be sure to use an absolute path when calling this script + with sudo. + + - Ensure that your host has the package ``iptables`` installed. + + - The package recipe ``qemu-helper-native`` is required to run + this script. Build the package using the following command:: + + $ bitbake qemu-helper-native + +- *Set the DISPLAY variable:* You need to set this variable so that + you have an X server available (e.g. start ``vncserver`` for a + headless machine). + +- *Be sure your host's firewall accepts incoming connections from + 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an + HTTP server on a random high number port, which is used to serve + files to the target. The DNF module serves + ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands. + That means your host's firewall must accept incoming connections from + 192.168.7.0/24, which is the default IP range used for tap devices by + ``runqemu``. + +- *Be sure your host has the correct packages installed:* Depending + your host's distribution, you need to have the following packages + installed: + + - Ubuntu and Debian: ``sysstat`` and ``iproute2`` + + - openSUSE: ``sysstat`` and ``iproute2`` + + - Fedora: ``sysstat`` and ``iproute`` + + - CentOS: ``sysstat`` and ``iproute`` + +Once you start running the tests, the following happens: + +#. A copy of the root filesystem is written to ``${WORKDIR}/testimage``. + +#. The image is booted under QEMU using the standard ``runqemu`` script. + +#. A default timeout of 500 seconds occurs to allow for the boot process + to reach the login prompt. You can change the timeout period by + setting + :term:`TEST_QEMUBOOT_TIMEOUT` + in the ``local.conf`` file. + +#. Once the boot process is reached and the login prompt appears, the + tests run. The full boot log is written to + ``${WORKDIR}/testimage/qemu_boot_log``. + +#. Each test module loads in the order found in :term:`TEST_SUITES`. You can + find the full output of the commands run over SSH in + ``${WORKDIR}/testimgage/ssh_target_log``. + +#. If no failures occur, the task running the tests ends successfully. + You can find the output from the ``unittest`` in the task log at + ``${WORKDIR}/temp/log.do_testimage``. + +Enabling Runtime Tests on Hardware +---------------------------------- + +The OpenEmbedded build system can run tests on real hardware, and for +certain devices it can also deploy the image to be tested onto the +device beforehand. + +For automated deployment, a "controller image" is installed onto the +hardware once as part of setup. Then, each time tests are to be run, the +following occurs: + +#. The controller image is booted into and used to write the image to be + tested to a second partition. + +#. The device is then rebooted using an external script that you need to + provide. + +#. The device boots into the image to be tested. + +When running tests (independent of whether the image has been deployed +automatically or not), the device is expected to be connected to a +network on a pre-determined IP address. You can either use static IP +addresses written into the image, or set the image to use DHCP and have +your DHCP server on the test network assign a known IP address based on +the MAC address of the device. + +In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an +appropriate value. For QEMU, you do not have to change anything, the +default value is "qemu". For running tests on hardware, the following +options are available: + +- *"simpleremote":* Choose "simpleremote" if you are going to run tests + on a target system that is already running the image to be tested and + is available on the network. You can use "simpleremote" in + conjunction with either real hardware or an image running within a + separately started QEMU or any other virtual machine manager. + +- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is + an EFI-based machine with ``systemd-boot`` as bootloader and + ``core-image-testmaster`` (or something similar) is installed. Also, + your hardware under test must be in a DHCP-enabled network that gives + it the same IP address for each reboot. + + If you choose "SystemdbootTarget", there are additional requirements + and considerations. See the + ":ref:`test-manual/runtime-testing:selecting systemdboottarget`" section, which + follows, for more information. + +- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying + images and running tests on the BeagleBone "Black" or original + "White" hardware. For information on how to use these tests, see the + comments at the top of the BeagleBoneTarget + ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file. + +- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying + images and running tests on the Ubiquiti Networks EdgeRouter Lite. + For information on how to use these tests, see the comments at the + top of the EdgeRouterTarget + ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file. + +- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running + tests on any generic PC that boots using GRUB. For information on how + to use these tests, see the comments at the top of the GrubTarget + ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file. + +- *"your-target":* Create your own custom target if you want to run + tests when you are deploying images and running tests on a custom + machine within your BSP layer. To do this, you need to add a Python + unit that defines the target class under ``lib/oeqa/controllers/`` + within your layer. You must also provide an empty ``__init__.py``. + For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``. + +Selecting SystemdbootTarget +--------------------------- + +If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do +not need any information in this section. You can skip down to the +":ref:`test-manual/runtime-testing:running tests`" section. + +If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to +perform a one-time setup of your controller image by doing the following: + +#. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows:: + + EFI_PROVIDER = "systemd-boot" + +#. *Build the controller image:* Build the ``core-image-testmaster`` image. + The ``core-image-testmaster`` recipe is provided as an example for a + "controller" image and you can customize the image recipe as you would + any other recipe. + + Image recipe requirements are: + + - Inherits ``core-image`` so that kernel modules are installed. + + - Installs normal linux utilities not BusyBox ones (e.g. ``bash``, + ``coreutils``, ``tar``, ``gzip``, and ``kmod``). + + - Uses a custom :term:`Initramfs` image with a custom + installer. A normal image that you can install usually creates a + single root filesystem partition. This image uses another installer that + creates a specific partition layout. Not all Board Support + Packages (BSPs) can use an installer. For such cases, you need to + manually create the following partition layout on the target: + + - First partition mounted under ``/boot``, labeled "boot". + + - The main root filesystem partition where this image gets installed, + which is mounted under ``/``. + + - Another partition labeled "testrootfs" where test images get + deployed. + +#. *Install image:* Install the image that you just built on the target + system. + +The final thing you need to do when setting :term:`TEST_TARGET` to +"SystemdbootTarget" is to set up the test image: + +#. *Set up your local.conf file:* Make sure you have the following + statements in your ``local.conf`` file:: + + IMAGE_FSTYPES += "tar.gz" + IMAGE_CLASSES += "testimage" + TEST_TARGET = "SystemdbootTarget" + TEST_TARGET_IP = "192.168.2.3" + +#. *Build your test image:* Use BitBake to build the image:: + + $ bitbake core-image-sato + +Power Control +------------- + +For most hardware targets other than "simpleremote", you can control +power: + +- You can use :term:`TEST_POWERCONTROL_CMD` together with + :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host + and does power cycling. The test code passes one argument to that + command: off, on or cycle (off then on). Here is an example that + could appear in your ``local.conf`` file:: + + TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1" + + In this example, the expect + script does the following: + + .. code-block:: shell + + ssh test@10.11.12.1 "pyctl nuc1 arg" + + It then runs a Python script that controls power for a label called + ``nuc1``. + + .. note:: + + You need to customize :term:`TEST_POWERCONTROL_CMD` and + :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement + is that it accepts "on", "off", and "cycle" as the last argument. + +- When no command is defined, it connects to the device over SSH and + uses the classic reboot command to reboot the device. Classic reboot + is fine as long as the machine actually reboots (i.e. the SSH test + has not failed). It is useful for scenarios where you have a simple + setup, typically with a single board, and where some manual + interaction is okay from time to time. + +If you have no hardware to automatically perform power control but still +wish to experiment with automated hardware testing, you can use the +``dialog-power-control`` script that shows a dialog prompting you to perform +the required power action. This script requires either KDialog or Zenity +to be installed. To use this script, set the +:term:`TEST_POWERCONTROL_CMD` +variable as follows:: + + TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control" + +Serial Console Connection +------------------------- + +For test target classes requiring a serial console to interact with the +bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget), +you need to specify a command to use to connect to the serial console of +the target machine by using the +:term:`TEST_SERIALCONTROL_CMD` +variable and optionally the +:term:`TEST_SERIALCONTROL_EXTRA_ARGS` +variable. + +These cases could be a serial terminal program if the machine is +connected to a local serial port, or a ``telnet`` or ``ssh`` command +connecting to a remote console server. Regardless of the case, the +command simply needs to connect to the serial console and forward that +connection to standard input and output as any normal terminal program +does. For example, to use the picocom terminal program on serial device +``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:: + + TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200" + +For local +devices where the serial port device disappears when the device reboots, +an additional "serdevtry" wrapper script is provided. To use this +wrapper, simply prefix the terminal command with +``${COREBASE}/scripts/contrib/serdevtry``:: + + TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0" + +Running Tests +============= + +You can start the tests automatically or manually: + +- *Automatically running tests:* To run the tests automatically after the + OpenEmbedded build system successfully creates an image, first set the + :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the + :term:`Build Directory`:: + + TESTIMAGE_AUTO = "1" + + Next, build your image. If the image successfully builds, the + tests run:: + + bitbake core-image-sato + +- *Manually running tests:* To manually run the tests, first globally + inherit the :ref:`ref-classes-testimage` class by editing your + ``local.conf`` file:: + + IMAGE_CLASSES += "testimage" + + Next, use BitBake to run the tests:: + + bitbake -c testimage image + +All test files reside in ``meta/lib/oeqa/runtime/cases`` in the +:term:`Source Directory`. A test name maps +directly to a Python module. Each test module may contain a number of +individual tests. Tests are usually grouped together by the area tested +(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``). + +You can add tests to any layer provided you place them in the proper +area and you extend :term:`BBPATH` in +the ``local.conf`` file as normal. Be sure that tests reside in +``layer/lib/oeqa/runtime/cases``. + +.. note:: + + Be sure that module names do not collide with module names used in + the default set of test modules in ``meta/lib/oeqa/runtime/cases``. + +You can change the set of tests run by appending or overriding +:term:`TEST_SUITES` variable in +``local.conf``. Each name in :term:`TEST_SUITES` represents a required test +for the image. Test modules named within :term:`TEST_SUITES` cannot be +skipped even if a test is not suitable for an image (e.g. running the +RPM tests on an image without ``rpm``). Appending "auto" to +:term:`TEST_SUITES` causes the build system to try to run all tests that are +suitable for the image (i.e. each test module may elect to skip itself). + +The order you list tests in :term:`TEST_SUITES` is important and influences +test dependencies. Consequently, tests that depend on other tests should +be added after the test on which they depend. For example, since the +``ssh`` test depends on the ``ping`` test, "ssh" needs to come after +"ping" in the list. The test class provides no re-ordering or dependency +handling. + +.. note:: + + Each module can have multiple classes with multiple test methods. + And, Python ``unittest`` rules apply. + +Here are some things to keep in mind when running tests: + +- The default tests for the image are defined as:: + + DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg" + +- Add your own test to the list of the by using the following:: + + TEST_SUITES:append = " mytest" + +- Run a specific list of tests as follows:: + + TEST_SUITES = "test1 test2 test3" + + Remember, order is important. Be sure to place a test that is + dependent on another test later in the order. + +Exporting Tests +=============== + +You can export tests so that they can run independently of the build +system. Exporting tests is required if you want to be able to hand the +test execution off to a scheduler. You can only export tests that are +defined in :term:`TEST_SUITES`. + +If your image is already built, make sure the following are set in your +``local.conf`` file:: + + INHERIT += "testexport" + TEST_TARGET_IP = "IP-address-for-the-test-target" + TEST_SERVER_IP = "IP-address-for-the-test-server" + +You can then export the tests with the +following BitBake command form:: + + $ bitbake image -c testexport + +Exporting the tests places them in the :term:`Build Directory` in +``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR` +variable. + +You can now run the tests outside of the build environment:: + + $ cd tmp/testexport/image + $ ./runexported.py testdata.json + +Here is a complete example that shows IP addresses and uses the +``core-image-sato`` image:: + + INHERIT += "testexport" + TEST_TARGET_IP = "192.168.7.2" + TEST_SERVER_IP = "192.168.7.1" + +Use BitBake to export the tests:: + + $ bitbake core-image-sato -c testexport + +Run the tests outside of +the build environment using the following:: + + $ cd tmp/testexport/core-image-sato + $ ./runexported.py testdata.json + +Writing New Tests +================= + +As mentioned previously, all new test files need to be in the proper +place for the build system to find them. New tests for additional +functionality outside of the core should be added to the layer that adds +the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as +:term:`BBPATH` is extended in the +layer's ``layer.conf`` file as normal). Just remember the following: + +- Filenames need to map directly to test (module) names. + +- Do not use module names that collide with existing core tests. + +- Minimally, an empty ``__init__.py`` file must be present in the runtime + directory. + +To create a new test, start by copying an existing module (e.g. +``oe_syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use +code from ``meta/lib/oeqa/utils``, which are helper classes. + +.. note:: + + Structure shell commands such that you rely on them and they return a + single code for success. Be aware that sometimes you will need to + parse the output. See the ``df.py`` and ``date.py`` modules for examples. + +You will notice that all test classes inherit ``oeRuntimeTest``, which +is found in ``meta/lib/oetest.py``. This base class offers some helper +attributes, which are described in the following sections: + +Class Methods +------------- + +Class methods are as follows: + +- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed + package list of the image, which is based on the manifest file that + is generated during the :ref:`ref-tasks-rootfs` task. + +- *hasFeature(feature):* Returns "True" if the feature is in + :term:`IMAGE_FEATURES` or + :term:`DISTRO_FEATURES`. + +Class Attributes +---------------- + +Class attributes are as follows: + +- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image. + Otherwise, ``pscmd`` equals "ps" (busybox). + +- *tc:* The called test context, which gives access to the + following attributes: + + - *d:* The BitBake datastore, which allows you to use stuff such + as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``. + + - *testslist and testsrequired:* Used internally. The tests + do not need these. + + - *filesdir:* The absolute path to + ``meta/lib/oeqa/runtime/files``, which contains helper files for + tests meant for copying on the target such as small files written + in C for compilation. + + - *target:* The target controller object used to deploy and + start an image on a particular target (e.g. Qemu, SimpleRemote, + and SystemdbootTarget). Tests usually use the following: + + - *ip:* The target's IP address. + + - *server_ip:* The host's IP address, which is usually used + by the DNF test suite. + + - *run(cmd, timeout=None):* The single, most used method. + This command is a wrapper for: ``ssh root@host "cmd"``. The + command returns a tuple: (status, output), which are what their + names imply - the return code of "cmd" and whatever output it + produces. The optional timeout argument represents the number + of seconds the test should wait for "cmd" to return. If the + argument is "None", the test uses the default instance's + timeout period, which is 300 seconds. If the argument is "0", + the test runs until the command returns. + + - *copy_to(localpath, remotepath):* + ``scp localpath root@ip:remotepath``. + + - *copy_from(remotepath, localpath):* + ``scp root@host:remotepath localpath``. + +Instance Attributes +------------------- + +There is a single instance attribute, which is ``target``. The ``target`` +instance attribute is identical to the class attribute of the same name, +which is described in the previous section. This attribute exists as +both an instance and class attribute so tests can use +``self.target.run(cmd)`` in instance methods instead of +``oeRuntimeTest.tc.target.run(cmd)``. + +Installing Packages in the DUT Without the Package Manager +========================================================== + +When a test requires a package built by BitBake, it is possible to +install that package. Installing the package does not require a package +manager be installed in the device under test (DUT). It does, however, +require an SSH connection and the target must be using the +``sshcontrol`` class. + +.. note:: + + This method uses ``scp`` to copy files from the host to the target, which + causes permissions and special attributes to be lost. + +A JSON file is used to define the packages needed by a test. This file +must be in the same path as the file used to define the tests. +Furthermore, the filename must map directly to the test module name with +a ``.json`` extension. + +The JSON file must include an object with the test name as keys of an +object or an array. This object (or array of objects) uses the following +data: + +- "pkg" --- a mandatory string that is the name of the package to be + installed. + +- "rm" --- an optional boolean, which defaults to "false", that specifies + to remove the package after the test. + +- "extract" --- an optional boolean, which defaults to "false", that + specifies if the package must be extracted from the package format. + When set to "true", the package is not automatically installed into + the DUT. + +Here is an example JSON file that handles test "foo" installing +package "bar" and test "foobar" installing packages "foo" and "bar". +Once the test is complete, the packages are removed from the DUT:: + + { + "foo": { + "pkg": "bar" + }, + "foobar": [ + { + "pkg": "foo", + "rm": true + }, + { + "pkg": "bar", + "rm": true + } + ] + } + -- cgit v1.2.3-54-g00ecf