ref-manual, dev-manual: Applied review comments for test section

The section and variables for performing automated runtime tests
on images was reviewed by Paul Eggleton.  The suggested changes
were made.

(From yocto-docs rev: c2f84ea3c162892e4da3df30fb833f88bab3d3cc)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

3 files changed, 109 insertions, 81 deletions

diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml
index 2dbc3d14b8..4d1b731fa3 100644
--- a/documentation/dev-manual/dev-manual-common-tasks.xml
+++ b/documentation/dev-manual/dev-manual-common-tasks.xml
@@ -4252,24 +4252,22 @@
         </section>
     </section>
 
-    <section id="running-and-writing-tests-for-a-qemu-image">
-        <title>Running and Writing Tests for a QEMU Image</title>
+    <section id="performing-automated-runtime-testing">
+        <title>Performing Automated Runtime Testing</title>
 
         <para>
             The OpenEmbedded build system makes available a series of automated
-            tests for QEMU images.
-            These tests are commands that run on the target system over
-            <filename>ssh</filename> and are written in Python, thus making
-            use of the <filename>unittest</filename> module.
-        </para>
-
-        <para>
+            tests for images.
+            <note>
+                The current release of Yocto Project supports these tests
+                for QEMU images only.
+            </note>
+            These tests are written in Python making use of the
+            <filename>unittest</filename> module, and the majority of them
+            run commands on the target system over
+            <filename>ssh</filename>.
             This section describes how you set up the environment to use these
             tests, run available tests, and write and add your own tests.
-            You can the same information in a different form and with example
-            log files of test runs on the
-            <ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
-            Wiki page.
         </para>
 
         <section id="qemu-image-enabling-tests">
@@ -4278,9 +4276,8 @@
             <para>
                 In order to run tests, you need to do the following:
                 <itemizedlist>
-                    <listitem><para><emphasis>Ensure you run the test
-                        automatically with no interaction and under
-                        <filename>sudo</filename>:</emphasis>
+                    <listitem><para><emphasis>Set up to avoid interaction
+                        with <filename>sudo</filename> for networking:</emphasis>
                         To accomplish this, you must do one of the
                         following:
                         <itemizedlist>
@@ -4300,12 +4297,13 @@
                             <listitem><para>Manually configure a tap interface
                                 for your system.</para></listitem>
                             <listitem><para>Run as root the script in
-                                <filename>scripts/runqemu-gen-tapdev</filename>,
+                                <filename>scripts/runqemu-gen-tapdevs</filename>,
                                 which should generate a list of tap devices.
-                                List generation is usually done in environments
-                                similar to AutoBuilder.</para></listitem>
+                                This is the option typically chosen for
+                                Autobuilder-type environments.
+                                </para></listitem>
                         </itemizedlist></para></listitem>
-                    <listitem><para><emphasis>Set up the
+                    <listitem><para><emphasis>Set the
                         <filename>DISPLAY</filename> variable:</emphasis>
                         You need to set this variable so that you have an X
                         server available (e.g. start
@@ -4321,11 +4319,10 @@
                         <filename>${DEPLOY_DIR}/rpm</filename> so it can run
                         smart channel commands. That means your host's firewall
                         must accept incoming connections from 192.168.7.0/24,
-                        which is the default class used for
-                        <filename>tap0</filename> devices by
-                        <filename>runqemu</filename>.</para></listitem>
-                    <listitem><para><emphasis>Have your QEMU image built:</emphasis>
-                        The QEMU image on which you want to run tests needs to
+                        which is the default IP range used for tap devices
+                        by <filename>runqemu</filename>.</para></listitem>
+                    <listitem><para><emphasis>Have your image built:</emphasis>
+                        The image on which you want to run tests needs to
                         be built.
                         For example, the following command would build the
                         <filename>core-image-sato</filename> image when
@@ -4333,18 +4330,34 @@
                         <literallayout class='monospaced'>
     bitbake core-image-sato
                         </literallayout></para></listitem>
-                    <listitem><para><emphasis>Globally inherit
-                        <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink></emphasis>
-                        Edit your <filename>local.conf</filename> file as
+                    <listitem><para><emphasis>Run the tests:</emphasis>
+                        You can start the tests automatically or manually.
+                        To have the tests start automatically after the
+                        OpenEmbedded build system successfully creates a QEMU
+                        image, set the
+                        <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_IMAGE'><filename>TEST_IMAGE</filename></ulink>
+                        variable to "1" in your
+                        <filename>local.conf</filename> file in the
+                        <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
+                        To manually run the tests, globally inherit
+                        <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink>
+                        by editing your <filename>local.conf</filename> file as
                         follows:
                         <literallayout class='monospaced'>
     INHERIT += "testimage"
+                        </literallayout>
+                        After editing your
+                        <filename>local.conf</filename> file, use BitBake to
+                        run the tests:
+                        <literallayout class='monospaced'>
+     bitbake -c testimage &lt;qemu-image&gt;
                         </literallayout></para></listitem>
                 </itemizedlist>
             </para>
 
             <note>
-                If you built your QEMU image using <filename>rm_work</filename>,
+                Regardless of how you initiate the tests, if you built your
+                image using <filename>rm_work</filename>,
                 most of the tests will fail with errors because they rely on
                 <filename>${WORKDIR}/installed_pkgs.txt</filename>.
             </note>
@@ -4355,14 +4368,16 @@
 
             <para>
                 After setting up to run the tests, use BitBake to start
-                the standard suite of tests:
+                the standard suite of tests manually or, if you are running
+                them automatically, build your image:
                 <literallayout class='monospaced'>
      bitbake core-image-sato -c testimage
+     bitbake core-image-sato
                 </literallayout>
             </para>
 
             <para>
-                Once you start the tests, the following happens:
+                Once the tests start, the following happens:
                 <itemizedlist>
                     <listitem><para>A copy of the root filesystem is written
                         to <filename>${WORKDIR}/testimage</filename>.
@@ -4400,10 +4415,10 @@
                 All test files reside in
                 <filename>meta/lib/oeqa/runtime</filename> in the
                 <link linkend='source-directory'>Source Directory</link>.
-                Tests (also referred to as modules) have a one-to-one
-                relationship with the filenames in <filename>runtime</filename>.
-                Modules can have multiple classes and test methods and are
-                usually grouped together by the area tested (e.g tests for
+                A test name (sometimes referred to as a module) maps to a
+                method name within a class within a module.
+                Tests can be used within multiple classes and the tests
+                are usually grouped together by the area tested (e.g tests for
                 <filename>systemd</filename> reside in
                 <filename>meta/lib/oeqa/runtime/systemd.py</filename>).
             </para>
@@ -4412,12 +4427,13 @@
                 You can add tests to any layer provided you place them in the
                 proper area and you extend
                 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
-                in the <filename>local.conf</filename> file.
+                in the <filename>local.conf</filename> file as normal.
                 Be sure that tests reside in
                 <filename>&lt;layer&gt;/lib/oeqa/runtime</filename>.
                 <note>
                     Be sure that module names do not collide with module names
-                    provided by the Yocto Project.
+                    used in the default set of test modules in
+                    <filename>meta/lib/oeqa/runtime</filename>.
                 </note>
             </para>
 
@@ -4458,12 +4474,12 @@
                     <listitem><para>Run the default set of tests simply by
                         running the following:
                         <literallayout class='monospaced'>
-     bitbake &lt;qemu_image&gt; -c testimage
+     bitbake &lt;image&gt; -c testimage
                         </literallayout></para></listitem>
                     <listitem><para>The default tests for the image are defined
                         as:
                         <literallayout class='monospaced'>
-     DEFAULT_TEST_SUITES_pn-&lt;qemu_image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"
+     DEFAULT_TEST_SUITES_pn-&lt;image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"
                         </literallayout></para></listitem>
                     <listitem><para>Add your own test to the list of the
                         by using the following:
@@ -4487,15 +4503,16 @@
             <para>
                 As mentioned previously, all new test files need to be in the
                 proper place for the build system to find them.
-                New tests can go in the
-                <link linkend='source-directory'>Source Directory</link> at
-                <filename>meta/lib/oeqa/runtime</filename>.
-                Alternatively, a layer can add its own tests in
-                <filename>&lt;layer&gt;/lib/oeqa/runtime</filename> as long
-                as you extend <filename>BBPATH</filename>.
-                Just remember that filenames need to map directory to test
-                (module) names and you not use module names that collide with
-                existing core tests.
+                New tests for additional functionality outside of the core
+                should be added to the layer that adds the functionality, in
+                <filename>&lt;layer&gt;/lib/oeqa/runtime</filename> (as
+                long as
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
+                is extended in the layer's
+                <filename>layer.conf</filename> file as normal.
+                Just remember that filenames need to map directly to test
+                (module) names and that you do not use module names that
+                collide with existing core tests.
             </para>
 
             <para>
diff --git a/documentation/ref-manual/ref-classes.xml b/documentation/ref-manual/ref-classes.xml
index 5870d0f44a..b1a8e64ce0 100644
--- a/documentation/ref-manual/ref-classes.xml
+++ b/documentation/ref-manual/ref-classes.xml
@@ -940,8 +940,12 @@
 
     <para>
         You can use this class to enable running a series of automated tests
-        for QEMU images.
-        The class handles loading the tests and starting the QEMU image.
+        for images.
+        The class handles loading the tests and starting the image.
+        <note>
+            The current release of Yocto Project supports these tests
+            for QEMU images only.
+        </note>
     </para>
 
     <para>
@@ -950,15 +954,13 @@
         The tests are commands that run on the target system over
         <filename>ssh</filename>.
         they are written in Python and make use of the
-        <filename>unittest modulebuild</filename> software.
+        <filename>unittest</filename> module.
     </para>
 
     <para>
         For information on how to enable, run, and create new tests, see the
-        "<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
-        section and the
-        <ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
-        Wiki page.
+        "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
+        section.
     </para>
 </section>
 
diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index 662a230650..d71378d271 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -5220,21 +5220,27 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
         <glossentry id='var-TEST_IMAGE'><glossterm>TEST_IMAGE</glossterm>
             <glossdef>
                 <para>
-                    Automatically runs the series of automated tests for QEMU
-                    images when a QEMU image is successfully built.
-                    These tests are commands that run on the target system over
-                    <filename>ssh</filename> and are written in Python.
+                    Automatically runs the series of automated tests for
+                    images when an image is successfully built.
+                    <note>
+                        The current release of Yocto Project supports these tests
+                        for QEMU images only.
+                    </note>
+                    These tests are written in Python making use of the
+                    <filename>unittest</filename> module, and the majority of
+                    them run commands on the target system over
+                    <filename>ssh</filename>.
                     You can set this variable to "1" in your
                     <filename>local.conf</filename> file in the
                     <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
                     to have the OpenEmbedded build system automatically run
-                    these tests after a QEMU image successfully builds:
+                    these tests after an image successfully builds:
                     <literallayout class='monospaced'>
      TEST_IMAGE = "1"
                     </literallayout>
                     For more information on enabling, running, and writing
                     these tests, see the
-                    "<ulink url='&YOCTO_DOCS_DEV_URL;#running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
+                    "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
                     section in the Yocto Project Development Manual and the
                     "<link linkend='ref-classes-testimage'><filename>testimage.bbclass</filename></link>"
                     section.
@@ -5245,24 +5251,18 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
         <glossentry id='var-TEST_QEMUBOOT_TIMEOUT'><glossterm>TEST_QEMUBOOT_TIMEOUT</glossterm>
             <glossdef>
                 <para>
-                    The time in seconds allowed for a QEMU image to boot before
-                    tests begin to run against the image using the following
-                    command:
-                    <literallayout class='monospaced'>
-     bitbake &lt;qemu_image&gt; -c testimage
-                    </literallayout>
-                </para>
-
-                <para>
+                    The time in seconds allowed for an image to boot before
+                    automated runtime tests begin to run against an
+                    image.
                     The default timeout period to allow the boot process to
                     reach the login prompt is 500 seconds.
-                    You can edit this default value in the
+                    You can specify a different value in the
                     <filename>local.conf</filename> file.
                 </para>
 
                 <para>
-                    For more information on testing QEMU images, see the
-                    "<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
+                    For more information on testing images, see the
+                    "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
                     section in the Yocto Project Development Manual.
                 </para>
             </glossdef>
@@ -5272,15 +5272,16 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
             <glossdef>
                 <para>
                     An ordered list of tests (modules) to run against
-                    a QEMU image using the following command:
-                    <literallayout class='monospaced'>
-     bitbake &lt;qemu_image&gt; -c testimage
-                    </literallayout>
+                    an image when performing automated runtime testing.
                 </para>
 
                 <para>
-                    The Yocto Project provides a core set of tests that can
-                    be used against QEMU images.
+                    The OpenEmbedded build system provides a core set of tests
+                    that can be used against images.
+                    <note>
+                        The current release of Yocto Project supports these
+                        tests for QEMU images only.
+                    </note>
                     Tests include <filename>ping</filename>,
                     <filename>ssh</filename>, <filename>df</filename> among
                     others.
@@ -5289,6 +5290,14 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     <literallayout class='monospaced'>
      TEST_SUITES_append = " mytest"
                     </literallayout>
+                    Alternatively, you can provide the "auto" option to
+                    have all applicable tests run against the image.
+                    <literallayout class='monospaced'>
+     TEST_SUITES_append = " auto"
+                    </literallayout>
+                    Using this option causes the build system to automatically
+                    run tests that are applicable to the image.
+                    Tests that are not applicable are skipped.
                 </para>
 
                 <para>
@@ -5307,8 +5316,8 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                 </para>
 
                 <para>
-                    For more information on testing QEMU images, see the
-                    "<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
+                    For more information on testing images, see the
+                    "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
                     section in the Yocto Project Development Manual.
                 </para>
             </glossdef>