1 files changed, 594 insertions, 0 deletions

diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst
new file mode 100644
index 0000000000..7a2b42f25a
--- /dev/null
+++ b/documentation/dev-manual/runtime-testing.rst
@@ -0,0 +1,594 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+Performing Automated Runtime Testing
+************************************
+
+The OpenEmbedded build system makes available a series of automated
+tests for images to verify runtime functionality. You can run these
+tests on either QEMU or actual target hardware. Tests are written in
+Python making use of the ``unittest`` module, and the majority of them
+run commands on the target system over SSH. This section describes how
+you set up the environment to use these tests, run available tests, and
+write and add your own tests.
+
+For information on the test and QA infrastructure available within the
+Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
+section in the Yocto Project Reference Manual.
+
+Enabling Tests
+==============
+
+Depending on whether you are planning to run tests using QEMU or on the
+hardware, you have to take different steps to enable the tests. See the
+following subsections for information on how to enable both types of
+tests.
+
+Enabling Runtime Tests on QEMU
+------------------------------
+
+In order to run tests, you need to do the following:
+
+-  *Set up to avoid interaction with sudo for networking:* To
+   accomplish this, you must do one of the following:
+
+   -  Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
+      commands or just for ``runqemu-ifup``. You must provide the full
+      path as that can change if you are using multiple clones of the
+      source repository.
+
+      .. note::
+
+         On some distributions, you also need to comment out "Defaults
+         requiretty" in ``/etc/sudoers``.
+
+   -  Manually configure a tap interface for your system.
+
+   -  Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
+      should generate a list of tap devices. This is the option
+      typically chosen for Autobuilder-type environments.
+
+      .. note::
+
+         -  Be sure to use an absolute path when calling this script
+            with sudo.
+
+         -  Ensure that your host has the package ``iptables`` installed.
+
+         -  The package recipe ``qemu-helper-native`` is required to run
+            this script. Build the package using the following command::
+
+               $ bitbake qemu-helper-native
+
+-  *Set the DISPLAY variable:* You need to set this variable so that
+   you have an X server available (e.g. start ``vncserver`` for a
+   headless machine).
+
+-  *Be sure your host's firewall accepts incoming connections from
+   192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
+   HTTP server on a random high number port, which is used to serve
+   files to the target. The DNF module serves
+   ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
+   That means your host's firewall must accept incoming connections from
+   192.168.7.0/24, which is the default IP range used for tap devices by
+   ``runqemu``.
+
+-  *Be sure your host has the correct packages installed:* Depending
+   your host's distribution, you need to have the following packages
+   installed:
+
+   -  Ubuntu and Debian: ``sysstat`` and ``iproute2``
+
+   -  openSUSE: ``sysstat`` and ``iproute2``
+
+   -  Fedora: ``sysstat`` and ``iproute``
+
+   -  CentOS: ``sysstat`` and ``iproute``
+
+Once you start running the tests, the following happens:
+
+#. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
+
+#. The image is booted under QEMU using the standard ``runqemu`` script.
+
+#. A default timeout of 500 seconds occurs to allow for the boot process
+   to reach the login prompt. You can change the timeout period by
+   setting
+   :term:`TEST_QEMUBOOT_TIMEOUT`
+   in the ``local.conf`` file.
+
+#. Once the boot process is reached and the login prompt appears, the
+   tests run. The full boot log is written to
+   ``${WORKDIR}/testimage/qemu_boot_log``.
+
+#. Each test module loads in the order found in :term:`TEST_SUITES`. You can
+   find the full output of the commands run over SSH in
+   ``${WORKDIR}/testimgage/ssh_target_log``.
+
+#. If no failures occur, the task running the tests ends successfully.
+   You can find the output from the ``unittest`` in the task log at
+   ``${WORKDIR}/temp/log.do_testimage``.
+
+Enabling Runtime Tests on Hardware
+----------------------------------
+
+The OpenEmbedded build system can run tests on real hardware, and for
+certain devices it can also deploy the image to be tested onto the
+device beforehand.
+
+For automated deployment, a "controller image" is installed onto the
+hardware once as part of setup. Then, each time tests are to be run, the
+following occurs:
+
+#. The controller image is booted into and used to write the image to be
+   tested to a second partition.
+
+#. The device is then rebooted using an external script that you need to
+   provide.
+
+#. The device boots into the image to be tested.
+
+When running tests (independent of whether the image has been deployed
+automatically or not), the device is expected to be connected to a
+network on a pre-determined IP address. You can either use static IP
+addresses written into the image, or set the image to use DHCP and have
+your DHCP server on the test network assign a known IP address based on
+the MAC address of the device.
+
+In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
+appropriate value. For QEMU, you do not have to change anything, the
+default value is "qemu". For running tests on hardware, the following
+options are available:
+
+-  *"simpleremote":* Choose "simpleremote" if you are going to run tests
+   on a target system that is already running the image to be tested and
+   is available on the network. You can use "simpleremote" in
+   conjunction with either real hardware or an image running within a
+   separately started QEMU or any other virtual machine manager.
+
+-  *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
+   an EFI-based machine with ``systemd-boot`` as bootloader and
+   ``core-image-testmaster`` (or something similar) is installed. Also,
+   your hardware under test must be in a DHCP-enabled network that gives
+   it the same IP address for each reboot.
+
+   If you choose "SystemdbootTarget", there are additional requirements
+   and considerations. See the
+   ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which
+   follows, for more information.
+
+-  *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
+   images and running tests on the BeagleBone "Black" or original
+   "White" hardware. For information on how to use these tests, see the
+   comments at the top of the BeagleBoneTarget
+   ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
+
+-  *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
+   tests on any generic PC that boots using GRUB. For information on how
+   to use these tests, see the comments at the top of the GrubTarget
+   ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
+
+-  *"your-target":* Create your own custom target if you want to run
+   tests when you are deploying images and running tests on a custom
+   machine within your BSP layer. To do this, you need to add a Python
+   unit that defines the target class under ``lib/oeqa/controllers/``
+   within your layer. You must also provide an empty ``__init__.py``.
+   For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
+
+Selecting SystemdbootTarget
+---------------------------
+
+If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
+not need any information in this section. You can skip down to the
+":ref:`dev-manual/runtime-testing:running tests`" section.
+
+If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
+perform a one-time setup of your controller image by doing the following:
+
+#. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
+
+      EFI_PROVIDER = "systemd-boot"
+
+#. *Build the controller image:* Build the ``core-image-testmaster`` image.
+   The ``core-image-testmaster`` recipe is provided as an example for a
+   "controller" image and you can customize the image recipe as you would
+   any other recipe.
+
+   Image recipe requirements are:
+
+   -  Inherits ``core-image`` so that kernel modules are installed.
+
+   -  Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
+      ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
+
+   -  Uses a custom :term:`Initramfs` image with a custom
+      installer. A normal image that you can install usually creates a
+      single root filesystem partition. This image uses another installer that
+      creates a specific partition layout. Not all Board Support
+      Packages (BSPs) can use an installer. For such cases, you need to
+      manually create the following partition layout on the target:
+
+      -  First partition mounted under ``/boot``, labeled "boot".
+
+      -  The main root filesystem partition where this image gets installed,
+         which is mounted under ``/``.
+
+      -  Another partition labeled "testrootfs" where test images get
+         deployed.
+
+#. *Install image:* Install the image that you just built on the target
+   system.
+
+The final thing you need to do when setting :term:`TEST_TARGET` to
+"SystemdbootTarget" is to set up the test image:
+
+#. *Set up your local.conf file:* Make sure you have the following
+   statements in your ``local.conf`` file::
+
+      IMAGE_FSTYPES += "tar.gz"
+      IMAGE_CLASSES += "testimage"
+      TEST_TARGET = "SystemdbootTarget"
+      TEST_TARGET_IP = "192.168.2.3"
+
+#. *Build your test image:* Use BitBake to build the image::
+
+      $ bitbake core-image-sato
+
+Power Control
+-------------
+
+For most hardware targets other than "simpleremote", you can control
+power:
+
+-  You can use :term:`TEST_POWERCONTROL_CMD` together with
+   :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
+   and does power cycling. The test code passes one argument to that
+   command: off, on or cycle (off then on). Here is an example that
+   could appear in your ``local.conf`` file::
+
+      TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
+
+   In this example, the expect
+   script does the following:
+
+   .. code-block:: shell
+
+      ssh test@10.11.12.1 "pyctl nuc1 arg"
+
+   It then runs a Python script that controls power for a label called
+   ``nuc1``.
+
+   .. note::
+
+      You need to customize :term:`TEST_POWERCONTROL_CMD` and
+      :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
+      is that it accepts "on", "off", and "cycle" as the last argument.
+
+-  When no command is defined, it connects to the device over SSH and
+   uses the classic reboot command to reboot the device. Classic reboot
+   is fine as long as the machine actually reboots (i.e. the SSH test
+   has not failed). It is useful for scenarios where you have a simple
+   setup, typically with a single board, and where some manual
+   interaction is okay from time to time.
+
+If you have no hardware to automatically perform power control but still
+wish to experiment with automated hardware testing, you can use the
+``dialog-power-control`` script that shows a dialog prompting you to perform
+the required power action. This script requires either KDialog or Zenity
+to be installed. To use this script, set the
+:term:`TEST_POWERCONTROL_CMD`
+variable as follows::
+
+   TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
+
+Serial Console Connection
+-------------------------
+
+For test target classes requiring a serial console to interact with the
+bootloader (e.g. BeagleBoneTarget and GrubTarget),
+you need to specify a command to use to connect to the serial console of
+the target machine by using the
+:term:`TEST_SERIALCONTROL_CMD`
+variable and optionally the
+:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
+variable.
+
+These cases could be a serial terminal program if the machine is
+connected to a local serial port, or a ``telnet`` or ``ssh`` command
+connecting to a remote console server. Regardless of the case, the
+command simply needs to connect to the serial console and forward that
+connection to standard input and output as any normal terminal program
+does. For example, to use the picocom terminal program on serial device
+``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
+
+   TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
+
+For local
+devices where the serial port device disappears when the device reboots,
+an additional "serdevtry" wrapper script is provided. To use this
+wrapper, simply prefix the terminal command with
+``${COREBASE}/scripts/contrib/serdevtry``::
+
+   TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
+
+Running Tests
+=============
+
+You can start the tests automatically or manually:
+
+-  *Automatically running tests:* To run the tests automatically after the
+   OpenEmbedded build system successfully creates an image, first set the
+   :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the
+   :term:`Build Directory`::
+
+      TESTIMAGE_AUTO = "1"
+
+   Next, build your image. If the image successfully builds, the
+   tests run::
+
+      bitbake core-image-sato
+
+-  *Manually running tests:* To manually run the tests, first globally
+   inherit the :ref:`ref-classes-testimage` class by editing your
+   ``local.conf`` file::
+
+      IMAGE_CLASSES += "testimage"
+
+   Next, use BitBake to run the tests::
+
+      bitbake -c testimage image
+
+All test files reside in ``meta/lib/oeqa/runtime/cases`` in the
+:term:`Source Directory`. A test name maps
+directly to a Python module. Each test module may contain a number of
+individual tests. Tests are usually grouped together by the area tested
+(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``).
+
+You can add tests to any layer provided you place them in the proper
+area and you extend :term:`BBPATH` in
+the ``local.conf`` file as normal. Be sure that tests reside in
+``layer/lib/oeqa/runtime/cases``.
+
+.. note::
+
+   Be sure that module names do not collide with module names used in
+   the default set of test modules in ``meta/lib/oeqa/runtime/cases``.
+
+You can change the set of tests run by appending or overriding
+:term:`TEST_SUITES` variable in
+``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
+for the image. Test modules named within :term:`TEST_SUITES` cannot be
+skipped even if a test is not suitable for an image (e.g. running the
+RPM tests on an image without ``rpm``). Appending "auto" to
+:term:`TEST_SUITES` causes the build system to try to run all tests that are
+suitable for the image (i.e. each test module may elect to skip itself).
+
+The order you list tests in :term:`TEST_SUITES` is important and influences
+test dependencies. Consequently, tests that depend on other tests should
+be added after the test on which they depend. For example, since the
+``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
+"ping" in the list. The test class provides no re-ordering or dependency
+handling.
+
+.. note::
+
+   Each module can have multiple classes with multiple test methods.
+   And, Python ``unittest`` rules apply.
+
+Here are some things to keep in mind when running tests:
+
+-  The default tests for the image are defined as::
+
+      DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
+
+-  Add your own test to the list of the by using the following::
+
+      TEST_SUITES:append = " mytest"
+
+-  Run a specific list of tests as follows::
+
+     TEST_SUITES = "test1 test2 test3"
+
+   Remember, order is important. Be sure to place a test that is
+   dependent on another test later in the order.
+
+Exporting Tests
+===============
+
+You can export tests so that they can run independently of the build
+system. Exporting tests is required if you want to be able to hand the
+test execution off to a scheduler. You can only export tests that are
+defined in :term:`TEST_SUITES`.
+
+If your image is already built, make sure the following are set in your
+``local.conf`` file::
+
+   INHERIT += "testexport"
+   TEST_TARGET_IP = "IP-address-for-the-test-target"
+   TEST_SERVER_IP = "IP-address-for-the-test-server"
+
+You can then export the tests with the
+following BitBake command form::
+
+   $ bitbake image -c testexport
+
+Exporting the tests places them in the :term:`Build Directory` in
+``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
+variable.
+
+You can now run the tests outside of the build environment::
+
+   $ cd tmp/testexport/image
+   $ ./runexported.py testdata.json
+
+Here is a complete example that shows IP addresses and uses the
+``core-image-sato`` image::
+
+   INHERIT += "testexport"
+   TEST_TARGET_IP = "192.168.7.2"
+   TEST_SERVER_IP = "192.168.7.1"
+
+Use BitBake to export the tests::
+
+   $ bitbake core-image-sato -c testexport
+
+Run the tests outside of
+the build environment using the following::
+
+   $ cd tmp/testexport/core-image-sato
+   $ ./runexported.py testdata.json
+
+Writing New Tests
+=================
+
+As mentioned previously, all new test files need to be in the proper
+place for the build system to find them. New tests for additional
+functionality outside of the core should be added to the layer that adds
+the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as
+:term:`BBPATH` is extended in the
+layer's ``layer.conf`` file as normal). Just remember the following:
+
+-  Filenames need to map directly to test (module) names.
+
+-  Do not use module names that collide with existing core tests.
+
+-  Minimally, an empty ``__init__.py`` file must be present in the runtime
+   directory.
+
+To create a new test, start by copying an existing module (e.g.
+``oe_syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
+code from ``meta/lib/oeqa/utils``, which are helper classes.
+
+.. note::
+
+   Structure shell commands such that you rely on them and they return a
+   single code for success. Be aware that sometimes you will need to
+   parse the output. See the ``df.py`` and ``date.py`` modules for examples.
+
+You will notice that all test classes inherit ``oeRuntimeTest``, which
+is found in ``meta/lib/oetest.py``. This base class offers some helper
+attributes, which are described in the following sections:
+
+Class Methods
+-------------
+
+Class methods are as follows:
+
+-  *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
+   package list of the image, which is based on the manifest file that
+   is generated during the :ref:`ref-tasks-rootfs` task.
+
+-  *hasFeature(feature):* Returns "True" if the feature is in
+   :term:`IMAGE_FEATURES` or
+   :term:`DISTRO_FEATURES`.
+
+Class Attributes
+----------------
+
+Class attributes are as follows:
+
+-  *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
+   Otherwise, ``pscmd`` equals "ps" (busybox).
+
+-  *tc:* The called test context, which gives access to the
+   following attributes:
+
+   -  *d:* The BitBake datastore, which allows you to use stuff such
+      as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
+
+   -  *testslist and testsrequired:* Used internally. The tests
+      do not need these.
+
+   -  *filesdir:* The absolute path to
+      ``meta/lib/oeqa/runtime/files``, which contains helper files for
+      tests meant for copying on the target such as small files written
+      in C for compilation.
+
+   -  *target:* The target controller object used to deploy and
+      start an image on a particular target (e.g. Qemu, SimpleRemote,
+      and SystemdbootTarget). Tests usually use the following:
+
+      -  *ip:* The target's IP address.
+
+      -  *server_ip:* The host's IP address, which is usually used
+         by the DNF test suite.
+
+      -  *run(cmd, timeout=None):* The single, most used method.
+         This command is a wrapper for: ``ssh root@host "cmd"``. The
+         command returns a tuple: (status, output), which are what their
+         names imply - the return code of "cmd" and whatever output it
+         produces. The optional timeout argument represents the number
+         of seconds the test should wait for "cmd" to return. If the
+         argument is "None", the test uses the default instance's
+         timeout period, which is 300 seconds. If the argument is "0",
+         the test runs until the command returns.
+
+      -  *copy_to(localpath, remotepath):*
+         ``scp localpath root@ip:remotepath``.
+
+      -  *copy_from(remotepath, localpath):*
+         ``scp root@host:remotepath localpath``.
+
+Instance Attributes
+-------------------
+
+There is a single instance attribute, which is ``target``. The ``target``
+instance attribute is identical to the class attribute of the same name,
+which is described in the previous section. This attribute exists as
+both an instance and class attribute so tests can use
+``self.target.run(cmd)`` in instance methods instead of
+``oeRuntimeTest.tc.target.run(cmd)``.
+
+Installing Packages in the DUT Without the Package Manager
+==========================================================
+
+When a test requires a package built by BitBake, it is possible to
+install that package. Installing the package does not require a package
+manager be installed in the device under test (DUT). It does, however,
+require an SSH connection and the target must be using the
+``sshcontrol`` class.
+
+.. note::
+
+   This method uses ``scp`` to copy files from the host to the target, which
+   causes permissions and special attributes to be lost.
+
+A JSON file is used to define the packages needed by a test. This file
+must be in the same path as the file used to define the tests.
+Furthermore, the filename must map directly to the test module name with
+a ``.json`` extension.
+
+The JSON file must include an object with the test name as keys of an
+object or an array. This object (or array of objects) uses the following
+data:
+
+-  "pkg" --- a mandatory string that is the name of the package to be
+   installed.
+
+-  "rm" --- an optional boolean, which defaults to "false", that specifies
+   to remove the package after the test.
+
+-  "extract" --- an optional boolean, which defaults to "false", that
+   specifies if the package must be extracted from the package format.
+   When set to "true", the package is not automatically installed into
+   the DUT.
+
+Here is an example JSON file that handles test "foo" installing
+package "bar" and test "foobar" installing packages "foo" and "bar".
+Once the test is complete, the packages are removed from the DUT::
+
+     {
+         "foo": {
+             "pkg": "bar"
+         },
+         "foobar": [
+             {
+                 "pkg": "foo",
+                 "rm": true
+             },
+             {
+                 "pkg": "bar",
+                 "rm": true
+             }
+         ]
+     }
+