ref-manual: move runtime-testing section to the test-manual

In the same fashion as the previous commit ("ref-manual/packages: move
ptest section to the test-manual"), move the runtime testing section of
the development tasks manual to the test environment manual.

Add a link to it from the test-manual/intro document.

(From yocto-docs rev: 3128bf149f40928e6c2a3e264590a0c6c9778c6a)

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
(cherry picked from commit 6b44257874858db3aa426d3e84a79c41cb4937a3)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>

2 files changed, 0 insertions, 601 deletions

diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index 9ccf60f701..ef0512d246 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -39,7 +39,6 @@ Yocto Project Development Tasks Manual
    external-scm
    read-only-rootfs
    build-quality
-   runtime-testing
    debugging
    licenses
    security-subjects
diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst
deleted file mode 100644
index bd3f42ac09..0000000000
--- a/documentation/dev-manual/runtime-testing.rst
+++ /dev/null
@@ -1,600 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
-
-Performing Automated Runtime Testing
-************************************
-
-The OpenEmbedded build system makes available a series of automated
-tests for images to verify runtime functionality. You can run these
-tests on either QEMU or actual target hardware. Tests are written in
-Python making use of the ``unittest`` module, and the majority of them
-run commands on the target system over SSH. This section describes how
-you set up the environment to use these tests, run available tests, and
-write and add your own tests.
-
-For information on the test and QA infrastructure available within the
-Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
-section in the Yocto Project Reference Manual.
-
-Enabling Tests
-==============
-
-Depending on whether you are planning to run tests using QEMU or on the
-hardware, you have to take different steps to enable the tests. See the
-following subsections for information on how to enable both types of
-tests.
-
-Enabling Runtime Tests on QEMU
-------------------------------
-
-In order to run tests, you need to do the following:
-
--  *Set up to avoid interaction with sudo for networking:* To
-   accomplish this, you must do one of the following:
-
-   -  Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
-      commands or just for ``runqemu-ifup``. You must provide the full
-      path as that can change if you are using multiple clones of the
-      source repository.
-
-      .. note::
-
-         On some distributions, you also need to comment out "Defaults
-         requiretty" in ``/etc/sudoers``.
-
-   -  Manually configure a tap interface for your system.
-
-   -  Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
-      should generate a list of tap devices. This is the option
-      typically chosen for Autobuilder-type environments.
-
-      .. note::
-
-         -  Be sure to use an absolute path when calling this script
-            with sudo.
-
-         -  Ensure that your host has the package ``iptables`` installed.
-
-         -  The package recipe ``qemu-helper-native`` is required to run
-            this script. Build the package using the following command::
-
-               $ bitbake qemu-helper-native
-
--  *Set the DISPLAY variable:* You need to set this variable so that
-   you have an X server available (e.g. start ``vncserver`` for a
-   headless machine).
-
--  *Be sure your host's firewall accepts incoming connections from
-   192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
-   HTTP server on a random high number port, which is used to serve
-   files to the target. The DNF module serves
-   ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
-   That means your host's firewall must accept incoming connections from
-   192.168.7.0/24, which is the default IP range used for tap devices by
-   ``runqemu``.
-
--  *Be sure your host has the correct packages installed:* Depending
-   your host's distribution, you need to have the following packages
-   installed:
-
-   -  Ubuntu and Debian: ``sysstat`` and ``iproute2``
-
-   -  openSUSE: ``sysstat`` and ``iproute2``
-
-   -  Fedora: ``sysstat`` and ``iproute``
-
-   -  CentOS: ``sysstat`` and ``iproute``
-
-Once you start running the tests, the following happens:
-
-#. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
-
-#. The image is booted under QEMU using the standard ``runqemu`` script.
-
-#. A default timeout of 500 seconds occurs to allow for the boot process
-   to reach the login prompt. You can change the timeout period by
-   setting
-   :term:`TEST_QEMUBOOT_TIMEOUT`
-   in the ``local.conf`` file.
-
-#. Once the boot process is reached and the login prompt appears, the
-   tests run. The full boot log is written to
-   ``${WORKDIR}/testimage/qemu_boot_log``.
-
-#. Each test module loads in the order found in :term:`TEST_SUITES`. You can
-   find the full output of the commands run over SSH in
-   ``${WORKDIR}/testimgage/ssh_target_log``.
-
-#. If no failures occur, the task running the tests ends successfully.
-   You can find the output from the ``unittest`` in the task log at
-   ``${WORKDIR}/temp/log.do_testimage``.
-
-Enabling Runtime Tests on Hardware
-----------------------------------
-
-The OpenEmbedded build system can run tests on real hardware, and for
-certain devices it can also deploy the image to be tested onto the
-device beforehand.
-
-For automated deployment, a "controller image" is installed onto the
-hardware once as part of setup. Then, each time tests are to be run, the
-following occurs:
-
-#. The controller image is booted into and used to write the image to be
-   tested to a second partition.
-
-#. The device is then rebooted using an external script that you need to
-   provide.
-
-#. The device boots into the image to be tested.
-
-When running tests (independent of whether the image has been deployed
-automatically or not), the device is expected to be connected to a
-network on a pre-determined IP address. You can either use static IP
-addresses written into the image, or set the image to use DHCP and have
-your DHCP server on the test network assign a known IP address based on
-the MAC address of the device.
-
-In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
-appropriate value. For QEMU, you do not have to change anything, the
-default value is "qemu". For running tests on hardware, the following
-options are available:
-
--  *"simpleremote":* Choose "simpleremote" if you are going to run tests
-   on a target system that is already running the image to be tested and
-   is available on the network. You can use "simpleremote" in
-   conjunction with either real hardware or an image running within a
-   separately started QEMU or any other virtual machine manager.
-
--  *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
-   an EFI-based machine with ``systemd-boot`` as bootloader and
-   ``core-image-testmaster`` (or something similar) is installed. Also,
-   your hardware under test must be in a DHCP-enabled network that gives
-   it the same IP address for each reboot.
-
-   If you choose "SystemdbootTarget", there are additional requirements
-   and considerations. See the
-   ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which
-   follows, for more information.
-
--  *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
-   images and running tests on the BeagleBone "Black" or original
-   "White" hardware. For information on how to use these tests, see the
-   comments at the top of the BeagleBoneTarget
-   ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
-
--  *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
-   images and running tests on the Ubiquiti Networks EdgeRouter Lite.
-   For information on how to use these tests, see the comments at the
-   top of the EdgeRouterTarget
-   ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
-
--  *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
-   tests on any generic PC that boots using GRUB. For information on how
-   to use these tests, see the comments at the top of the GrubTarget
-   ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
-
--  *"your-target":* Create your own custom target if you want to run
-   tests when you are deploying images and running tests on a custom
-   machine within your BSP layer. To do this, you need to add a Python
-   unit that defines the target class under ``lib/oeqa/controllers/``
-   within your layer. You must also provide an empty ``__init__.py``.
-   For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
-
-Selecting SystemdbootTarget
----------------------------
-
-If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
-not need any information in this section. You can skip down to the
-":ref:`dev-manual/runtime-testing:running tests`" section.
-
-If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
-perform a one-time setup of your controller image by doing the following:
-
-#. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
-
-      EFI_PROVIDER = "systemd-boot"
-
-#. *Build the controller image:* Build the ``core-image-testmaster`` image.
-   The ``core-image-testmaster`` recipe is provided as an example for a
-   "controller" image and you can customize the image recipe as you would
-   any other recipe.
-
-   Image recipe requirements are:
-
-   -  Inherits ``core-image`` so that kernel modules are installed.
-
-   -  Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
-      ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
-
-   -  Uses a custom :term:`Initramfs` image with a custom
-      installer. A normal image that you can install usually creates a
-      single root filesystem partition. This image uses another installer that
-      creates a specific partition layout. Not all Board Support
-      Packages (BSPs) can use an installer. For such cases, you need to
-      manually create the following partition layout on the target:
-
-      -  First partition mounted under ``/boot``, labeled "boot".
-
-      -  The main root filesystem partition where this image gets installed,
-         which is mounted under ``/``.
-
-      -  Another partition labeled "testrootfs" where test images get
-         deployed.
-
-#. *Install image:* Install the image that you just built on the target
-   system.
-
-The final thing you need to do when setting :term:`TEST_TARGET` to
-"SystemdbootTarget" is to set up the test image:
-
-#. *Set up your local.conf file:* Make sure you have the following
-   statements in your ``local.conf`` file::
-
-      IMAGE_FSTYPES += "tar.gz"
-      IMAGE_CLASSES += "testimage"
-      TEST_TARGET = "SystemdbootTarget"
-      TEST_TARGET_IP = "192.168.2.3"
-
-#. *Build your test image:* Use BitBake to build the image::
-
-      $ bitbake core-image-sato
-
-Power Control
--------------
-
-For most hardware targets other than "simpleremote", you can control
-power:
-
--  You can use :term:`TEST_POWERCONTROL_CMD` together with
-   :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
-   and does power cycling. The test code passes one argument to that
-   command: off, on or cycle (off then on). Here is an example that
-   could appear in your ``local.conf`` file::
-
-      TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
-
-   In this example, the expect
-   script does the following:
-
-   .. code-block:: shell
-
-      ssh test@10.11.12.1 "pyctl nuc1 arg"
-
-   It then runs a Python script that controls power for a label called
-   ``nuc1``.
-
-   .. note::
-
-      You need to customize :term:`TEST_POWERCONTROL_CMD` and
-      :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
-      is that it accepts "on", "off", and "cycle" as the last argument.
-
--  When no command is defined, it connects to the device over SSH and
-   uses the classic reboot command to reboot the device. Classic reboot
-   is fine as long as the machine actually reboots (i.e. the SSH test
-   has not failed). It is useful for scenarios where you have a simple
-   setup, typically with a single board, and where some manual
-   interaction is okay from time to time.
-
-If you have no hardware to automatically perform power control but still
-wish to experiment with automated hardware testing, you can use the
-``dialog-power-control`` script that shows a dialog prompting you to perform
-the required power action. This script requires either KDialog or Zenity
-to be installed. To use this script, set the
-:term:`TEST_POWERCONTROL_CMD`
-variable as follows::
-
-   TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
-
-Serial Console Connection
--------------------------
-
-For test target classes requiring a serial console to interact with the
-bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
-you need to specify a command to use to connect to the serial console of
-the target machine by using the
-:term:`TEST_SERIALCONTROL_CMD`
-variable and optionally the
-:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
-variable.
-
-These cases could be a serial terminal program if the machine is
-connected to a local serial port, or a ``telnet`` or ``ssh`` command
-connecting to a remote console server. Regardless of the case, the
-command simply needs to connect to the serial console and forward that
-connection to standard input and output as any normal terminal program
-does. For example, to use the picocom terminal program on serial device
-``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
-
-   TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
-
-For local
-devices where the serial port device disappears when the device reboots,
-an additional "serdevtry" wrapper script is provided. To use this
-wrapper, simply prefix the terminal command with
-``${COREBASE}/scripts/contrib/serdevtry``::
-
-   TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
-
-Running Tests
-=============
-
-You can start the tests automatically or manually:
-
--  *Automatically running tests:* To run the tests automatically after the
-   OpenEmbedded build system successfully creates an image, first set the
-   :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the
-   :term:`Build Directory`::
-
-      TESTIMAGE_AUTO = "1"
-
-   Next, build your image. If the image successfully builds, the
-   tests run::
-
-      bitbake core-image-sato
-
--  *Manually running tests:* To manually run the tests, first globally
-   inherit the :ref:`ref-classes-testimage` class by editing your
-   ``local.conf`` file::
-
-      IMAGE_CLASSES += "testimage"
-
-   Next, use BitBake to run the tests::
-
-      bitbake -c testimage image
-
-All test files reside in ``meta/lib/oeqa/runtime/cases`` in the
-:term:`Source Directory`. A test name maps
-directly to a Python module. Each test module may contain a number of
-individual tests. Tests are usually grouped together by the area tested
-(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``).
-
-You can add tests to any layer provided you place them in the proper
-area and you extend :term:`BBPATH` in
-the ``local.conf`` file as normal. Be sure that tests reside in
-``layer/lib/oeqa/runtime/cases``.
-
-.. note::
-
-   Be sure that module names do not collide with module names used in
-   the default set of test modules in ``meta/lib/oeqa/runtime/cases``.
-
-You can change the set of tests run by appending or overriding
-:term:`TEST_SUITES` variable in
-``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
-for the image. Test modules named within :term:`TEST_SUITES` cannot be
-skipped even if a test is not suitable for an image (e.g. running the
-RPM tests on an image without ``rpm``). Appending "auto" to
-:term:`TEST_SUITES` causes the build system to try to run all tests that are
-suitable for the image (i.e. each test module may elect to skip itself).
-
-The order you list tests in :term:`TEST_SUITES` is important and influences
-test dependencies. Consequently, tests that depend on other tests should
-be added after the test on which they depend. For example, since the
-``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
-"ping" in the list. The test class provides no re-ordering or dependency
-handling.
-
-.. note::
-
-   Each module can have multiple classes with multiple test methods.
-   And, Python ``unittest`` rules apply.
-
-Here are some things to keep in mind when running tests:
-
--  The default tests for the image are defined as::
-
-      DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
-
--  Add your own test to the list of the by using the following::
-
-      TEST_SUITES:append = " mytest"
-
--  Run a specific list of tests as follows::
-
-     TEST_SUITES = "test1 test2 test3"
-
-   Remember, order is important. Be sure to place a test that is
-   dependent on another test later in the order.
-
-Exporting Tests
-===============
-
-You can export tests so that they can run independently of the build
-system. Exporting tests is required if you want to be able to hand the
-test execution off to a scheduler. You can only export tests that are
-defined in :term:`TEST_SUITES`.
-
-If your image is already built, make sure the following are set in your
-``local.conf`` file::
-
-   INHERIT += "testexport"
-   TEST_TARGET_IP = "IP-address-for-the-test-target"
-   TEST_SERVER_IP = "IP-address-for-the-test-server"
-
-You can then export the tests with the
-following BitBake command form::
-
-   $ bitbake image -c testexport
-
-Exporting the tests places them in the :term:`Build Directory` in
-``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
-variable.
-
-You can now run the tests outside of the build environment::
-
-   $ cd tmp/testexport/image
-   $ ./runexported.py testdata.json
-
-Here is a complete example that shows IP addresses and uses the
-``core-image-sato`` image::
-
-   INHERIT += "testexport"
-   TEST_TARGET_IP = "192.168.7.2"
-   TEST_SERVER_IP = "192.168.7.1"
-
-Use BitBake to export the tests::
-
-   $ bitbake core-image-sato -c testexport
-
-Run the tests outside of
-the build environment using the following::
-
-   $ cd tmp/testexport/core-image-sato
-   $ ./runexported.py testdata.json
-
-Writing New Tests
-=================
-
-As mentioned previously, all new test files need to be in the proper
-place for the build system to find them. New tests for additional
-functionality outside of the core should be added to the layer that adds
-the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as
-:term:`BBPATH` is extended in the
-layer's ``layer.conf`` file as normal). Just remember the following:
-
--  Filenames need to map directly to test (module) names.
-
--  Do not use module names that collide with existing core tests.
-
--  Minimally, an empty ``__init__.py`` file must be present in the runtime
-   directory.
-
-To create a new test, start by copying an existing module (e.g.
-``oe_syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
-code from ``meta/lib/oeqa/utils``, which are helper classes.
-
-.. note::
-
-   Structure shell commands such that you rely on them and they return a
-   single code for success. Be aware that sometimes you will need to
-   parse the output. See the ``df.py`` and ``date.py`` modules for examples.
-
-You will notice that all test classes inherit ``oeRuntimeTest``, which
-is found in ``meta/lib/oetest.py``. This base class offers some helper
-attributes, which are described in the following sections:
-
-Class Methods
--------------
-
-Class methods are as follows:
-
--  *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
-   package list of the image, which is based on the manifest file that
-   is generated during the :ref:`ref-tasks-rootfs` task.
-
--  *hasFeature(feature):* Returns "True" if the feature is in
-   :term:`IMAGE_FEATURES` or
-   :term:`DISTRO_FEATURES`.
-
-Class Attributes
-----------------
-
-Class attributes are as follows:
-
--  *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
-   Otherwise, ``pscmd`` equals "ps" (busybox).
-
--  *tc:* The called test context, which gives access to the
-   following attributes:
-
-   -  *d:* The BitBake datastore, which allows you to use stuff such
-      as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
-
-   -  *testslist and testsrequired:* Used internally. The tests
-      do not need these.
-
-   -  *filesdir:* The absolute path to
-      ``meta/lib/oeqa/runtime/files``, which contains helper files for
-      tests meant for copying on the target such as small files written
-      in C for compilation.
-
-   -  *target:* The target controller object used to deploy and
-      start an image on a particular target (e.g. Qemu, SimpleRemote,
-      and SystemdbootTarget). Tests usually use the following:
-
-      -  *ip:* The target's IP address.
-
-      -  *server_ip:* The host's IP address, which is usually used
-         by the DNF test suite.
-
-      -  *run(cmd, timeout=None):* The single, most used method.
-         This command is a wrapper for: ``ssh root@host "cmd"``. The
-         command returns a tuple: (status, output), which are what their
-         names imply - the return code of "cmd" and whatever output it
-         produces. The optional timeout argument represents the number
-         of seconds the test should wait for "cmd" to return. If the
-         argument is "None", the test uses the default instance's
-         timeout period, which is 300 seconds. If the argument is "0",
-         the test runs until the command returns.
-
-      -  *copy_to(localpath, remotepath):*
-         ``scp localpath root@ip:remotepath``.
-
-      -  *copy_from(remotepath, localpath):*
-         ``scp root@host:remotepath localpath``.
-
-Instance Attributes
--------------------
-
-There is a single instance attribute, which is ``target``. The ``target``
-instance attribute is identical to the class attribute of the same name,
-which is described in the previous section. This attribute exists as
-both an instance and class attribute so tests can use
-``self.target.run(cmd)`` in instance methods instead of
-``oeRuntimeTest.tc.target.run(cmd)``.
-
-Installing Packages in the DUT Without the Package Manager
-==========================================================
-
-When a test requires a package built by BitBake, it is possible to
-install that package. Installing the package does not require a package
-manager be installed in the device under test (DUT). It does, however,
-require an SSH connection and the target must be using the
-``sshcontrol`` class.
-
-.. note::
-
-   This method uses ``scp`` to copy files from the host to the target, which
-   causes permissions and special attributes to be lost.
-
-A JSON file is used to define the packages needed by a test. This file
-must be in the same path as the file used to define the tests.
-Furthermore, the filename must map directly to the test module name with
-a ``.json`` extension.
-
-The JSON file must include an object with the test name as keys of an
-object or an array. This object (or array of objects) uses the following
-data:
-
--  "pkg" --- a mandatory string that is the name of the package to be
-   installed.
-
--  "rm" --- an optional boolean, which defaults to "false", that specifies
-   to remove the package after the test.
-
--  "extract" --- an optional boolean, which defaults to "false", that
-   specifies if the package must be extracted from the package format.
-   When set to "true", the package is not automatically installed into
-   the DUT.
-
-Here is an example JSON file that handles test "foo" installing
-package "bar" and test "foobar" installing packages "foo" and "bar".
-Once the test is complete, the packages are removed from the DUT::
-
-     {
-         "foo": {
-             "pkg": "bar"
-         },
-         "foobar": [
-             {
-                 "pkg": "foo",
-                 "rm": true
-             },
-             {
-                 "pkg": "bar",
-                 "rm": true
-             }
-         ]
-     }
-