1 files changed, 173 insertions, 114 deletions

diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst
index 2f11e2beb3..b0bc24c9b0 100644
--- a/documentation/test-manual/test-manual-intro.rst
+++ b/documentation/test-manual/test-manual-intro.rst
@@ -23,26 +23,25 @@ project core.
 
 Currently, the Yocto Project Test Environment Manual has no projected
 release date. This manual is a work-in-progress and is being initially
-loaded with information from the `README <>`__ files and notes from key
+loaded with information from the README files and notes from key
 engineers:
 
--  *``yocto-autobuilder2``:* This
-   ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md
+-  *yocto-autobuilder2:* This
+   :yocto_git:`README.md </cgit.cgi/yocto-autobuilder2/tree/README.md>`
    is the main README which detials how to set up the Yocto Project
    Autobuilder. The ``yocto-autobuilder2`` repository represents the
    Yocto Project's console UI plugin to Buildbot and the configuration
    necessary to configure Buildbot to perform the testing the project
    requires.
 
--  *``yocto-autobuilder-helper``:* This
-   ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README
+-  *yocto-autobuilder-helper:* This :yocto_git:`README </cgit.cgi/yocto-autobuilder-helper/tree/README/>`
    and repository contains Yocto Project Autobuilder Helper scripts and
    configuration. The ``yocto-autobuilder-helper`` repository contains
    the "glue" logic that defines which tests to run and how to run them.
    As a result, it can be used by any Continuous Improvement (CI) system
    to run builds, support getting the correct code revisions, configure
    builds and layers, run builds, and collect results. The code is
-   independent of any CI system, which means the code can work Buildbot,
+   independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
    Jenkins, or others. This repository has a branch per release of the
    project defining the tests to run on a per release basis.
 
@@ -124,21 +123,22 @@ thefollowing types of tests:
    ensure we have no single point of failure and can ensure the
    different distros work effectively.
 
--  *eSDK Testing:* Image tests initiated through the following command:
-   $ bitbake image -c testsdkext The tests utilize the ``testsdkext``
-   class and the ``do_testsdkext`` task.
+-  *eSDK Testing:* Image tests initiated through the following command::
+
+      $ bitbake image -c testsdkext
+
+   The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
 
 -  *Feature Testing:* Various scenario-based tests are run through the
-   `OpenEmbedded
-   Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__
-   (oe-selftest). We test oe-selftest on each of the main distrubutions
+   :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
    we support.
 
--  *Image Testing:* Image tests initiated through the following command:
-   $ bitbake image -c testimage The tests utilize the
-   :ref:`testimage* <ref-classes-testimage*>`
-   classes and the
-   :ref:`ref-tasks-testimage` task.
+-  *Image Testing:* Image tests initiated through the following command::
+
+      $ bitbake image -c testimage
+
+   The tests utilize the :ref:`testimage* <ref-classes-testimage*>`
+   classes and the :ref:`ref-tasks-testimage` task.
 
 -  *Layer Testing:* The Autobuilder has the possibility to test whether
    specific layers work with the test of the system. The layers tested
@@ -147,20 +147,22 @@ thefollowing types of tests:
 
 -  *Package Testing:* A Package Test (ptest) runs tests against packages
    built by the OpenEmbedded build system on the target machine. See the
-   "`Testing Packages With
-   ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section
+   :ref:`Testing Packages With
+   ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section
    in the Yocto Project Development Tasks Manual and the
    ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
    information on Ptest.
 
--  *SDK Testing:* Image tests initiated through the following command: $
-   bitbake image -c testsdk The tests utilize the
-   :ref:`testsdk <ref-classes-testsdk>` class and
+-  *SDK Testing:* Image tests initiated through the following command::
+
+      $ bitbake image -c testsdk
+
+   The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and
    the ``do_testsdk`` task.
 
 -  *Unit Testing:* Unit tests on various components of the system run
-   through ``oe-selftest`` and
-   ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__.
+   through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and
+   :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`.
 
 -  *Automatic Upgrade Helper:* This target tests whether new versions of
    software are available and whether we can automatically upgrade to
@@ -174,36 +176,43 @@ How Tests Map to Areas of Code
 
 Tests map into the codebase as follows:
 
--  *bitbake-selftest*:
+-  *bitbake-selftest:*
 
    These tests are self-contained and test BitBake as well as its APIs,
    which include the fetchers. The tests are located in
    ``bitbake/lib/*/tests``.
 
-   From within the BitBake repository, run the following: $
-   bitbake-selftest
+   From within the BitBake repository, run the following::
+
+      $ bitbake-selftest
 
    To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
-   variable when running "bitbake-selftest" as follows: $
-   BB_SKIP_NETTEST=yes bitbake-selftest
+   variable when running "bitbake-selftest" as follows::
+
+      $ BB_SKIP_NETTEST=yes bitbake-selftest
 
    The default output is quiet and just prints a summary of what was
-   run. To see more information, there is a verbose option:$
-   bitbake-selftest -v
+   run. To see more information, there is a verbose option::
+
+      $ bitbake-selftest -v
 
    Use this option when you wish to skip tests that access the network,
    which are mostly necessary to test the fetcher modules. To specify
    individual test modules to run, append the test module name to the
    "bitbake-selftest" command. For example, to specify the tests for the
-   bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can
-   also specify individual tests by defining the full name and module
-   plus the class path of the test, for example: $ bitbake-selftest
-   bb.tests.data.TestOverrides.test_one_override
+   bb.data.module, run::
+
+      $ bitbake-selftest bb.test.data.module
+
+   You can also specify individual tests by defining the full name and module
+   plus the class path of the test, for example::
+
+      $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
 
    The tests are based on `Python
    unittest <https://docs.python.org/3/library/unittest.html>`__.
 
--  *oe-selftest*:
+-  *oe-selftest:*
 
    -  These tests use OE to test the workflows, which include testing
       specific features, behaviors of tasks, and API unit tests.
@@ -219,31 +228,40 @@ Tests map into the codebase as follows:
    -  The code for the tests resides in
       ``meta/lib/oeqa/selftest/cases/``.
 
-   -  To run all the tests, enter the following command: $ oe-selftest
-      -a
+   -  To run all the tests, enter the following command::
+
+         $ oe-selftest -a
 
    -  To run a specific test, use the following command form where
-      testname is the name of the specific test: $ oe-selftest -r
-      testname For example, the following command would run the tinfoil
-      getVar API test:$ oe-selftest -r
-      tinfoil.TinfoilTests.test_getvarIt is also possible to run a set
+      testname is the name of the specific test::
+
+         $ oe-selftest -r <testname>
+
+      For example, the following command would run the tinfoil
+      getVar API test::
+
+         $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
+
+      It is also possible to run a set
       of tests. For example the following command will run all of the
-      tinfoil tests:$ oe-selftest -r tinfoil
+      tinfoil tests::
+
+         $ oe-selftest -r tinfoil
 
 -  *testimage:*
 
    -  These tests build an image, boot it, and run tests against the
       image's content.
 
-   -  The code for these tests resides in
-      ``meta/lib/oeqa/runtime/cases/``.
+   -  The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``.
+
+   -  You need to set the :term:`IMAGE_CLASSES` variable as follows::
 
-   -  You need to set the
-      :term:`IMAGE_CLASSES`
-      variable as follows: IMAGE_CLASSES += "testimage"
+         IMAGE_CLASSES += "testimage"
 
-   -  Run the tests using the following command form: $ bitbake image -c
-      testimage
+   -  Run the tests using the following command form::
+
+         $ bitbake image -c testimage
 
 -  *testsdk:*
 
@@ -252,8 +270,9 @@ Tests map into the codebase as follows:
 
    -  The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
 
-   -  Run the test using the following command form: $ bitbake image -c
-      testsdk
+   -  Run the test using the following command form::
+
+         $ bitbake image -c testsdk
 
 -  *testsdk_ext:*
 
@@ -262,8 +281,9 @@ Tests map into the codebase as follows:
 
    -  The code for these tests resides in ``meta/lib/oeqa/esdk``.
 
-   -  To run the tests, use the following command form: $ bitbake image
-      -c testsdkext
+   -  To run the tests, use the following command form::
+
+         $ bitbake image -c testsdkext
 
 -  *oe-build-perf-test:*
 
@@ -272,8 +292,11 @@ Tests map into the codebase as follows:
 
    -  The code for these tests resides in ``meta/lib/oeqa/buildperf``.
 
-   -  To run the tests, use the following command form: $
-      oe-build-perf-test optionsThe command takes a number of options,
+   -  To run the tests, use the following command form::
+
+         $ oe-build-perf-test <options>
+
+      The command takes a number of options,
       such as where to place the test results. The Autobuilder Helper
       Scripts include the ``build-perf-test-wrapper`` script with
       examples of how to use the oe-build-perf-test from the command
@@ -285,9 +308,9 @@ Tests map into the codebase as follows:
       Use the ``oe-build-perf-report`` command to generate text reports
       and HTML reports with graphs of the performance data. For
       examples, see
-      `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html <#>`__
+      :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html`
       and
-      `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt <#>`__.
+      :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`.
 
    -  The tests are contained in ``lib/oeqa/buildperf/test_basic.py``.
 
@@ -295,7 +318,7 @@ Test Examples
 =============
 
 This section provides example tests for each of the tests listed in the
-`How Tests Map to Areas of Code <#test-test-mapping>`__ section.
+:ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section.
 
 For oeqa tests, testcases for each area reside in the main test
 directory at ``meta/lib/oeqa/selftest/cases`` directory.
@@ -308,14 +331,20 @@ directory.
 ``bitbake-selftest``
 --------------------
 
-A simple test example from ``lib/bb/tests/data.py`` is: class
-DataExpansions(unittest.TestCase): def setUp(self): self.d =
-bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] =
-"value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def
-test_one_var(self): val = self.d.expand("${foo}")
-self.assertEqual(str(val), "value_of_foo")
+A simple test example from ``lib/bb/tests/data.py`` is::
+
+   class DataExpansions(unittest.TestCase):
+      def setUp(self):
+            self.d = bb.data.init()
+            self.d["foo"] = "value_of_foo"
+            self.d["bar"] = "value_of_bar"
+            self.d["value_of_foo"] = "value_of_'value_of_foo'"
 
-In this example, a ```DataExpansions`` <>`__ class of tests is created,
+      def test_one_var(self):
+            val = self.d.expand("${foo}")
+            self.assertEqual(str(val), "value_of_foo")
+
+In this example, a ``DataExpansions`` class of tests is created,
 derived from standard python unittest. The class has a common ``setUp``
 function which is shared by all the tests in the class. A simple test is
 then added to test that when a variable is expanded, the correct value
@@ -323,7 +352,7 @@ is found.
 
 Bitbake selftests are straightforward python unittest. Refer to the
 Python unittest documentation for additional information on writing
-these tests at: `https://docs.python.org/3/library/unittest.html <#>`__.
+these tests at: https://docs.python.org/3/library/unittest.html.
 
 .. _oe-selftest-example:
 
@@ -334,14 +363,15 @@ These tests are more complex due to the setup required behind the scenes
 for full builds. Rather than directly using Python's unittest, the code
 wraps most of the standard objects. The tests can be simple, such as
 testing a command from within the OE build environment using the
-following example:class BitbakeLayers(OESelftestTestCase): def
-test_bitbakelayers_showcrossdepends(self): result =
-runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in
-result.output, msg = "No dependencies were shown. bitbake-layers
-show-cross-depends output: %s"% result.output)
+following example::
+
+   class BitbakeLayers(OESelftestTestCase):
+      def test_bitbakelayers_showcrossdepends(self):
+            result = runCmd('bitbake-layers show-cross-depends')
+            self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
 
 This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
-creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived
+creates a testcase from the ``OESelftestTestCase`` class, derived
 from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
 and checks the output to ensure it contains something we know should be
 here.
@@ -367,7 +397,7 @@ so tests within a given test class should always run in the same build,
 while tests in different classes or modules may be split into different
 builds. There is no data store available for these tests since the tests
 launch the ``bitbake`` command and exist outside of its context. As a
-result, common bitbake library functions (bb.*) are also unavailable.
+result, common bitbake library functions (bb.\*) are also unavailable.
 
 .. _testimage-example:
 
@@ -378,14 +408,18 @@ These tests are run once an image is up and running, either on target
 hardware or under QEMU. As a result, they are assumed to be running in a
 target image environment, as opposed to a host build environment. A
 simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
-the following:class PythonTest(OERuntimeTestCase):
-@OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core'])
-def test_python3(self): cmd = "python3 -c \\"import codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output =
-self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output
-self.assertEqual(status, 0, msg=msg)
-
-In this example, the ```OERuntimeTestCase`` <>`__ class wraps
+the following::
+
+   class PythonTest(OERuntimeTestCase):
+      @OETestDepends(['ssh.SSHTest.test_ssh'])
+      @OEHasPackage(['python3-core'])
+      def test_python3(self):
+         cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+         status, output = self.target.run(cmd)
+         msg = 'Exit status was not 0. Output: %s' % output
+         self.assertEqual(status, 0, msg=msg)
+
+In this example, the ``OERuntimeTestCase`` class wraps
 ``unittest.TestCase``. Within the test, ``self.target`` represents the
 target system, where commands can be run on it using the ``run()``
 method.
@@ -402,19 +436,30 @@ the image.
 
 These tests are run against built extensible SDKs (eSDKs). The tests can
 assume that the eSDK environment has already been setup. An example from
-``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class
-DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls):
-myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst =
-os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src,
-cls.myapp_dst) subprocess.check_output(['git', 'init', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'],
-cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m',
-"'test commit'"], cwd=cls.myapp_dst) @classmethod def
-tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def
-\_test_devtool_build(self, directory): self._run('devtool add myapp %s'
-% directory) try: self._run('devtool build myapp') finally:
-self._run('devtool reset myapp') def test_devtool_build_make(self):
-self._test_devtool_build(self.myapp_dst)In this example, the ``devtool``
+``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
+
+   class DevtoolTest(OESDKExtTestCase):
+      @classmethod def setUpClass(cls):
+         myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
+         cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
+         shutil.copytree(myapp_src, cls.myapp_dst)
+         subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
+         subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
+         subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
+
+      @classmethod
+      def tearDownClass(cls):
+         shutil.rmtree(cls.myapp_dst)
+      def _test_devtool_build(self, directory):
+         self._run('devtool add myapp %s' % directory)
+         try:
+         self._run('devtool build myapp')
+         finally:
+         self._run('devtool reset myapp')
+      def test_devtool_build_make(self):
+         self._test_devtool_build(self.myapp_dst)
+
+In this example, the ``devtool``
 command is tested to see whether a sample application can be built with
 the ``devtool build`` command within the eSDK.
 
@@ -426,14 +471,20 @@ the ``devtool build`` command within the eSDK.
 These tests are run against built SDKs. The tests can assume that an SDK
 has already been extracted and its environment file has been sourced. A
 simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
-following:class Python3Test(OESDKTestCase): def setUp(self): if not
-(self.tc.hasHostPackage("nativesdk-python3-core") or
-self.tc.hasHostPackage("python3-core-native")): raise
-unittest.SkipTest("No python3 package in the SDK") def
-test_python3(self): cmd = "python3 -c \\"import codecs;
-print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd)
-self.assertEqual(output, "Hello, world\n")In this example, if
-nativesdk-python3-core has been installed into the SDK, the code runs
+following::
+
+   class Python3Test(OESDKTestCase):
+      def setUp(self):
+            if not (self.tc.hasHostPackage("nativesdk-python3-core") or
+                  self.tc.hasHostPackage("python3-core-native")):
+               raise unittest.SkipTest("No python3 package in the SDK")
+
+      def test_python3(self):
+            cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+            output = self._run(cmd)
+            self.assertEqual(output, "Hello, world\n")
+
+In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
 the python3 interpreter with a basic command to check it is working
 correctly. The test would only run if python3 is installed in the SDK.
 
@@ -444,17 +495,25 @@ correctly. The test would only run if python3 is installed in the SDK.
 
 The performance tests usually measure how long operations take and the
 resource utilisation as that happens. An example from
-``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class
-Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake
--p)""" # Drop all caches and parse self.rm_cache()
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no
-caches)') # Drop tmp/cache
-oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no
-tmp/cache)') # Parse with fully cached data
-self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p
-(cached)')This example shows how three specific parsing timings are
+``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
+
+   class Test3(BuildPerfTestCase):
+      def test3(self):
+            """Bitbake parsing (bitbake -p)"""
+            # Drop all caches and parse
+            self.rm_cache()
+            oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+            self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
+                     'bitbake -p (no caches)')
+            # Drop tmp/cache
+            oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+            self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
+                     'bitbake -p (no tmp/cache)')
+            # Parse with fully cached data
+            self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
+                     'bitbake -p (cached)')
+
+This example shows how three specific parsing timings are
 measured, with and without various caches, to show how BitBake’s parsing
 performance trends over time.