ref-manual: Added section on test and QA infrastructure

Fixes [YOCTO #10887]

There needed to be an overview of the test and QA infrastructure
used by the YP development team for releases and that is visible
to the developer.  I added a new section for this information into
the existing section on YP release processes.

(From yocto-docs rev: 17b3d6898b37bdbfbd23a333a4d659a9746696d0)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

1 files changed, 131 insertions, 0 deletions

diff --git a/documentation/ref-manual/ref-release-process.xml b/documentation/ref-manual/ref-release-process.xml
index 1b337f8bd7..fe3ba0933e 100644
--- a/documentation/ref-manual/ref-release-process.xml
+++ b/documentation/ref-manual/ref-release-process.xml
@@ -117,6 +117,137 @@
     </para>
 </section>
 
+<section id='testing-and-quality-assurance'>
+    <title>Testing and Quality Assurance</title>
+
+    <para>
+        Part of the Yocto Project development and release process is quality
+        assurance through the execution of test strategies.
+        Test strategies provide the Yocto Project team a way to ensure a
+        release is validated.
+        Additionally, because the test strategies are visible to you as a
+        developer, you can validate your projects.
+        This section overviews the available test infrastructure used in the
+        Yocto Project.
+        For information on how to run available tests on your projects, see the
+        "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
+        section in the Yocto Project Development Manual.
+    </para>
+
+    <para>
+        The QA/testing infrastructure is woven into the project to the point
+        where core developers take some of it for granted.
+        The infrastructure consists of the following pieces:
+        <itemizedlist>
+            <listitem><para>
+                <filename>bitbake-selftest</filename>:
+                A standalone command that runs unit tests on key pieces of
+                BitBake and its fetchers.
+                </para></listitem>
+            <listitem><para>
+                <link linkend='ref-classes-sanity'><filename>sanity.bbclass</filename></link>:
+                This automatically included class checks the build environment
+                for missing tools (e.g. <filename>gcc</filename>) or common
+                misconfigurations such as
+                <link linkend='var-MACHINE'><filename>MACHINE</filename></link>
+                set incorrectly.
+                </para></listitem>
+            <listitem><para>
+                <link linkend='ref-classes-insane'><filename>insane.bbclass</filename></link>:
+                This class checks the generated output from builds for sanity.
+                For example, if building for an ARM target, did the build
+                produce ARM binaries.
+                If, for example, the build produced PPC binaries then there
+                is a problem.
+                </para></listitem>
+            <listitem><para>
+                <link linkend='ref-classes-testimage*'><filename>testimage.bbclass</filename></link>:
+                This class performs runtime testing of images after they are
+                built.
+                The tests are usually used with
+                <ulink url='&YOCTO_DOCS_DEV_URL;#dev-manual-qemu'>QEMU</ulink>
+                to boot the images and check the combined runtime result
+                boot operation and functions.
+                However, the test can also use the IP address of a machine to
+                test.
+                </para></listitem>
+            <listitem><para>
+                <ulink url='&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest'><filename>ptest</filename></ulink>:
+                Runs tests against packages produced during the build for a
+                given piece of software.
+                The test allows the packages to be be run within a target
+                image.
+                </para></listitem>
+            <listitem><para>
+                <filename>oe-selftest</filename>:
+                Tests combination BitBake invocations.
+                These tests operate outside the OpenEmbedded build system
+                itself.
+                The <filename>oe-selftest</filename> can run all tests by
+                default or can run selected tests or test suites.
+                <note>
+                    Running <filename>oe-selftest</filename> requires
+                    host packages beyond the "Essential" grouping.
+                    See the
+                    "<link linkend='required-packages-for-the-host-development-system'>Required Packages for the Host Development System</link>"
+                    section for more information.
+                </note>
+                </para></listitem>
+        </itemizedlist>
+    </para>
+
+    <para>
+        Originally, much of this testing was done manually.
+        However, significant effort has been made to automate the tests so
+        that more people can use them and the Yocto Project development team
+        can run them faster and more efficiently.
+    </para>
+
+    <para>
+        The Yocto Project's main Autobuilder
+        (<filename>autobuilder.yoctoproject.org</filename>) publicly tests
+        each Yocto Project release's code in the OE-Core, Poky, and BitBake
+        repositories.
+        The testing occurs for both the current state of the
+        "master" branch and also for submitted patches.
+        Testing for submitted patches usually occurs in the
+        "ross/mut" branch in the <filename>poky-contrib</filename> repository
+        (i.e. the master-under-test branch) or in the "master-next" branch
+        in the <filename>poky</filename> repository.
+        <note>
+            You can find all these branches in the Yocto Project
+            <ulink url='&YOCTO_DOCS_DEV_URL;#source-repositories'>Source Repositories</ulink>.
+        </note>
+        Testing within these public branches ensures in a publicly visible way
+        that all of the main supposed architectures and recipes in OE-Core
+        successfully build and behave properly.
+    </para>
+
+    <para>
+        Various features such as <filename>multilib</filename>, sub
+        architectures (e.g. <filename>x32</filename>,
+        <filename>poky-tiny</filename>, <filename>musl</filename>,
+        <filename>no-x11</filename> and and so forth),
+        <filename>bitbake-selftest</filename>, and
+        <filename>oe-selftest</filename> are tested as part of
+        the QA process of a release.
+        Complete testing and validation for a release takes the Autobuilder
+        workers several hours.
+        <note>
+            The Autobuilder workers are non-homogeneous, which means regular
+            testing across a variety of Linux distributions occurs.
+            The Autobuilder is limited to only testing QEMU-based setups and
+            not real hardware.
+        </note>
+    </para>
+
+    <para>
+        Finally, in addition to the Autobuilder's tests, the Yocto Project
+        QA team also performs testing on a variety of platforms, which includes
+        actual hardware, to ensure expected results.
+    </para>
+</section>
+
 </chapter>
 <!--
 vim: expandtab tw=80 ts=4