path: root/documentation/test-manual/test-manual-intro.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
<!--SPDX-License-Identifier: CC-BY-2.0-UK-->

<chapter id='test-manual-intro'>

<title>The Yocto Project Test Environment Manual</title>
    <section id='test-welcome'>
        <title>Welcome</title>

        <para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in
            progress. The manual contains information about the testing environment used by the
            Yocto Project to make sure each major and minor release works as intended. All the
            project’s testing infrastructure and processes are publicly visible and available so
            that the community can see what testing is being performed, how it’s being done and the
            current status of the tests and the project at any given time. It is intended that Other
            organizations can leverage off the process and testing environment used by the Yocto
            Project to create their own automated, production test environment, building upon the
            foundations from the project core. </para>

        <para> Currently, the Yocto Project Test Environment Manual has no projected release date.
            This manual is a work-in-progress and is being initially loaded with information from
            the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
                <listitem>
                    <para>
                        <emphasis><filename>yocto-autobuilder2</filename>:</emphasis> This <ulink
                            url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md"
                                ><filename>README.md</filename></ulink> is the main README which
                        detials how to set up the Yocto Project Autobuilder. The
                            <filename>yocto-autobuilder2</filename> repository represents the Yocto
                        Project's console UI plugin to Buildbot and the configuration necessary to
                        configure Buildbot to perform the testing the project requires. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis><filename>yocto-autobuilder-helper</filename>:</emphasis> This
                            <ulink
                            url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README"
                                ><filename>README</filename></ulink> and repository contains Yocto
                        Project Autobuilder Helper scripts and configuration. The
                            <filename>yocto-autobuilder-helper</filename> repository contains the
                        "glue" logic that defines which tests to run and how to run them. As a
                        result, it can be used by any Continuous Improvement (CI) system to run
                        builds, support getting the correct code revisions, configure builds and
                        layers, run builds, and collect results. The code is independent of any CI
                        system, which means the code can work Buildbot, Jenkins, or others. This
                        repository has a branch per release of the project defining the tests to run
                        on a per release basis.</para>
                </listitem>
            </itemizedlist>
        </para>
    </section>

    <section id='test-yocto-project-autobuilder-overview'>
        <title>Yocto Project Autobuilder Overview</title>

        <para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and
            procedures used by the Yocto Project to test released software across supported hardware
            in an automated and regular fashion. Basically, during the development of a Yocto
            Project release, the Autobuilder tests if things work. The Autobuilder builds all test
            targets and runs all the tests. </para>

        <para>The Yocto Project uses now uses standard upstream  <ulink
                url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive
            its integration and testing. Buildbot Nine has a plug-in interface that the Yocto
            Project customizes using code from the <filename>yocto-autobuilder2</filename>
            repository, adding its own console UI plugin. The resulting UI plug-in allows you to
            visualize builds in a way suited to the project's needs.</para>

        <para>A <filename>helper</filename> layer provides configuration and job management through
            scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The
                <filename>helper</filename> layer contains the bulk of the build configuration
            information and is release-specific, which makes it highly customizable on a per-project
            basis. The layer is CI system-agnostic and contains a number of Helper scripts that can
            generate build configurations from simple JSON files. <note>
                <para>The project uses Buildbot for historical reasons but also because many of the
                    project developers have knowledge of python. It is possible to use the outer
                    layers from another Continuous Integration (CI) system such as <ulink
                        url="https://en.wikipedia.org/wiki/Jenkins_(software)">Jenkins</ulink>
                    instead of Buildbot. </para>
            </note>
        </para>

        <para> The following figure shows the Yocto Project Autobuilder stack with a topology that
            includes a controller and a cluster of workers: <imagedata
                fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center"
                scalefit="1"/>
        </para>
    </section>

    <section id='test-project-tests'>
        <title>Yocto Project Tests - Types of Testing Overview</title>

        <para>The Autobuilder tests different elements of the project by using thefollowing types of
            tests: <itemizedlist>
                <listitem>
                    <para>
                        <emphasis>Build Testing:</emphasis> Tests whether specific configurations
                        build by varying <ulink url="&YOCTO_DOCS_REF_URL;#var-MACHINE"
                                ><filename>MACHINE</filename></ulink>, <ulink
                            url="&YOCTO_DOCS_REF_URL;#var-DISTRO"
                            ><filename>DISTRO</filename></ulink>, other configuration options, and
                        the specific target images being built (or world). Used to trigger builds of
                        all the different test configurations on the Autobuilder. Builds usually
                        cover many different targets for different architectures, machines, and
                        distributions, as well as different configurations, such as different init
                        systems. The Autobuilder tests literally hundreds of configurations and
                        targets. <itemizedlist>
                            <listitem>
                                <para>
                                    <emphasis>Sanity Checks During the Build Process:</emphasis>
                                    Tests initiated through the <ulink
                                        url="&YOCTO_DOCS_REF_URL;#ref-classes-insane"
                                            ><filename>insane</filename></ulink> class. These checks
                                    ensure the output of the builds are correct. For example, does
                                    the ELF architecture in the generated binaries match the target
                                    system? ARM binaries would not work in a MIPS system! </para>
                            </listitem>
                        </itemizedlist></para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>Build Performance Testing:</emphasis> Tests whether or not
                        commonly used steps during builds work efficiently and avoid regressions.
                        Tests to time commonly used usage scenarios are run through
                            <filename>oe-build-perf-test</filename>. These tests are run on isolated
                        machines so that the time measurements of the tests are accurate and no
                        other processes interfere with the timing results. The project currently
                        tests performance on two different distributions, Fedora and Ubuntu, to
                        ensure we have no single point of failure and can ensure the different
                        distros work effectively. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>eSDK Testing:</emphasis> Image tests initiated through the
                        following command:
                        <literallayout class="monospaced">
     $ bitbake <replaceable>image</replaceable> -c testsdkext
                    </literallayout>
                        The tests utilize the <filename>testsdkext</filename> class and the
                            <filename>do_testsdkext</filename> task. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>Feature Testing:</emphasis> Various scenario-based tests are run
                        through the <ulink url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
                            >OpenEmbedded Self-Test</ulink> (oe-selftest). We test oe-selftest on
                        each of the main distrubutions we support. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>Image Testing:</emphasis> Image tests initiated through the
                        following command:
                        <literallayout class="monospaced">
     $ bitbake <replaceable>image</replaceable> -c testimage
                    </literallayout>
                        The tests utilize the <ulink
                            url="&YOCTO_DOCS_REF_URL;#ref-classes-testimage*"
                                ><filename>testimage*</filename></ulink> classes and the <ulink
                            url="&YOCTO_DOCS_REF_URL;#ref-tasks-testimage"
                                ><filename>do_testimage</filename></ulink> task. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>Layer Testing:</emphasis> The Autobuilder has the possibility to
                        test whether specific layers work with the test of the system. The layers
                        tested may be selected by members of the project. Some key community layers
                        are also tested periodically.</para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>Package Testing:</emphasis> A Package Test (ptest) runs tests
                        against packages built by the OpenEmbedded build system on the target
                        machine. See the "<ulink
                            url="&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest">Testing Packages
                            With ptest</ulink>" section in the Yocto Project Development Tasks
                        Manual and the "<ulink url="&YOCTO_WIKI_URL;/wiki/Ptest">Ptest</ulink>" Wiki
                        page for more information on Ptest. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>SDK Testing:</emphasis> Image tests initiated through the
                        following command:
                        <literallayout class="monospaced">
     $ bitbake <replaceable>image</replaceable> -c testsdk
                    </literallayout>
                        The tests utilize the <ulink url="&YOCTO_DOCS_REF_URL;#ref-classes-testsdk"
                                ><filename>testsdk</filename></ulink> class and the
                            <filename>do_testsdk</filename> task. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>Unit Testing:</emphasis> Unit tests on various components of the
                        system run through <filename>oe-selftest</filename> and <ulink
                            url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
                                ><filename>bitbake-selftest</filename></ulink>. </para>
                </listitem>
                <listitem>
                    <para>
                        <emphasis>Automatic Upgrade Helper:</emphasis> This target tests whether new
                        versions of software are available and whether we can automatically upgrade
                        to those new versions. If so, this target emails the maintainers with a
                        patch to let them know this is possible.</para>
                </listitem>
            </itemizedlist>
        </para>
    </section>

    <section id='test-test-mapping'>
        <title>How Tests Map to Areas of Code</title>

        <para>
            Tests map into the codebase as follows:
            <itemizedlist>
                <listitem><para>
                        <emphasis>bitbake-selftest</emphasis>: </para>
                    <para>These tests are self-contained and test BitBake as well as its APIs, which
                        include the fetchers. The tests are located in
                            <filename>bitbake/lib/*/tests</filename>. </para>
                    <para>From within the BitBake repository, run the following:
                        <literallayout class="monospaced">
     $ bitbake-selftest
                            </literallayout>
                    </para>
                    <para>To skip tests that access the Internet, use the
                            <filename>BB_SKIP_NETTEST</filename> variable when running
                        "bitbake-selftest" as follows:
                        <literallayout class="monospaced">
     $ BB_SKIP_NETTEST=yes bitbake-selftest
                            </literallayout></para>
                    <para>The default output is quiet and just prints a summary of what was run. To
                        see more information, there is a verbose
                        option:<literallayout class="monospaced">
     $ bitbake-selftest -v
                             </literallayout></para>
                    <para>Use this option when you wish to skip tests that access the network, which
                        are mostly necessary to test the fetcher modules. To specify individual test
                        modules to run, append the test module name to the "bitbake-selftest"
                        command. For example, to specify the tests for the bb.data.module, run:
                        <literallayout class="monospaced">
     $ bitbake-selftest bb.test.data.module
                            </literallayout>You
                        can also specify individual tests by defining the full name and module plus
                        the class path of the test, for example:
                        <literallayout class="monospaced">
     $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
                             </literallayout></para>
                    <para>The tests are based on <ulink
                            url="https://docs.python.org/3/library/unittest.html">Python
                            unittest</ulink>. </para></listitem>
                <listitem><para>
                        <emphasis>oe-selftest</emphasis>: <itemizedlist>
                            <listitem>
                                <para>These tests use OE to test the workflows, which include
                                    testing specific features, behaviors of tasks, and API unit
                                    tests. </para>
                            </listitem>
                            <listitem>
                                <para>The tests can take advantage of parallelism through the "-j"
                                    option, which can specify a number of threads to spread the
                                    tests across. Note that all tests from a given class of tests
                                    will run in the same thread. To parallelize large numbers of
                                    tests you can split the class into multiple units.</para>
                            </listitem>
                            <listitem>
                                <para>The tests are based on Python unittest. </para>
                            </listitem>
                            <listitem>
                                <para>The code for the tests resides in
                                        <filename>meta/lib/oeqa/selftest/cases/</filename>. </para>
                            </listitem>
                            <listitem>
                                <para>To run all the tests, enter the following command:
                                    <literallayout class="monospaced">
     $ oe-selftest -a
                            </literallayout>
                                </para>
                            </listitem>
                            <listitem>
                                <para>To run a specific test, use the following command form where
                                        <replaceable>testname</replaceable> is the name of the
                                    specific test:
                                    <literallayout class="monospaced">
     $ oe-selftest -r <replaceable>testname</replaceable>
                            </literallayout>
                                    For example, the following command would run the tinfoil getVar
                                    API
                                    test:<literallayout class="monospaced">
     $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
                            </literallayout>It
                                    is also possible to run a set of tests. For example the
                                    following command will run all of the tinfoil
                                    tests:<literallayout class="monospaced">
     $ oe-selftest -r  tinfoil
                            </literallayout></para>
                            </listitem>
                        </itemizedlist>
                    </para></listitem>
                <listitem><para>
                    <emphasis>testimage:</emphasis>
                    <itemizedlist>
                        <listitem><para>
                            These tests build an image, boot it, and run tests
                            against the image's content.
                            </para></listitem>
                        <listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem>
                        <listitem><para>
                            You need to set the
                            <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink>
                            variable as follows:
                            <literallayout class='monospaced'>
     IMAGE_CLASSES += "testimage"
                            </literallayout>
                            </para></listitem>
                        <listitem><para>
                            Run the tests using the following command form:
                            <literallayout class='monospaced'>
     $ bitbake <replaceable>image</replaceable> -c testimage
                            </literallayout>
                            </para></listitem>
                    </itemizedlist>
                    </para></listitem>
                <listitem><para>
                    <emphasis>testsdk:</emphasis>
                    <itemizedlist>
                        <listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem>
                        <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem>
                        <listitem><para>Run the test using the following command form:
                                    <literallayout class="monospaced">
     $ bitbake <replaceable>image</replaceable> -c testsdk
                            </literallayout>
                                </para></listitem>
                    </itemizedlist>
                    </para></listitem>
                <listitem><para>
                    <emphasis>testsdk_ext:</emphasis>
                    <itemizedlist>
                        <listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem>
                        <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem>
                        <listitem><para>To run the tests, use the following command form:
                                    <literallayout class="monospaced">
     $ bitbake <replaceable>image</replaceable> -c testsdkext
                            </literallayout>
                                </para></listitem>
                    </itemizedlist>
                    </para></listitem>


                <listitem><para>
                    <emphasis>oe-build-perf-test:</emphasis>
                    <itemizedlist>
                        <listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem>
                        <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem>
                        <listitem><para>To run the tests, use the following command form:
                                    <literallayout class="monospaced">
    $ oe-build-perf-test <replaceable>options</replaceable>
                            </literallayout>The
                                    command takes a number of options, such as where to place the
                                    test results. The Autobuilder Helper Scripts include the
                                        <filename>build-perf-test-wrapper</filename> script with
                                    examples of how to use the oe-build-perf-test from the command
                                    line.</para>
                                <para>Use the <filename>oe-git-archive</filename> command to store
                                    test results into a Git repository. </para>
                                <para>Use the <filename>oe-build-perf-report</filename> command to
                                    generate text reports and HTML reports with graphs of the
                                    performance data. For examples, see <link linkend=""
                                        >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>
                                    and <link linkend=""
                                        >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>
                            <listitem>
                                <para>The tests are contained in
                                        <filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>
                            </listitem>
                    </itemizedlist>
                    </para></listitem>




            </itemizedlist>
        </para>
    </section>

    <section id='test-examples'>
        <title>Test Examples</title>

        <para>This section provides example tests for each of the tests listed in the <link
                linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>
        <para>For oeqa tests, testcases for each area reside in the main test directory at
                <filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>
        <para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>
            directory. </para>

        <section id='bitbake-selftest-example'>
            <title><filename>bitbake-selftest</filename></title>

            <para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:
                <literallayout class="monospaced">
     class DataExpansions(unittest.TestCase):
         def setUp(self):
             self.d = bb.data.init()
             self.d["foo"] = "value_of_foo"
             self.d["bar"] = "value_of_bar"       
             self.d["value_of_foo"] = "value_of_'value_of_foo'"
           
          def test_one_var(self):
             val =  self.d.expand("${foo}")
             self.assertEqual(str(val), "value_of_foo")
                     </literallayout>
            </para>
            <para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class
                of tests is created, derived from standard python unittest. The class has a common
                    <filename>setUp</filename> function which is shared by all the tests in the
                class. A simple test is then added to test that when a variable is expanded, the
                correct value is found.</para>
            <para>Bitbake selftests are straightforward python unittest. Refer to the Python
                unittest documentation for additional information on writing these tests at: <link
                    linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>
        </section>

        <section id='oe-selftest-example'>
            <title><filename>oe-selftest</filename></title>

            <para>These tests are more complex due to the setup required behind the scenes for full
                builds. Rather than directly using Python's unittest, the code wraps most of the
                standard objects. The tests can be simple, such as testing a command from within the
                OE build environment using the following
                example:<literallayout class="monospaced">
     class BitbakeLayers(OESelftestTestCase):
          def test_bitbakelayers_showcrossdepends(self):
             result = runCmd('bitbake-layers show-cross-depends')       
             self.assertTrue('aspell' in result.output, msg = "No dependencies 
     were shown. bitbake-layers show-cross-depends output:
           %s"% result.output)
                   </literallayout></para>
            <para>This example, taken from
                    <filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
                testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>
                class, derived from <filename>unittest.TestCase</filename>, which runs the
                    <filename>bitbake-layers</filename> command and checks the output to ensure it
                contains something we know should be here.</para>
            <para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can
                assist with common tasks, including:<itemizedlist>
                    <listitem>
                        <para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use
                                <filename>oeqa.utils.commands.get_bb_var()</filename> or use
                                <filename>oeqa.utils.commands.get_bb_vars()</filename> for more than
                            one variable</para>
                    </listitem>
                    <listitem>
                        <para><emphasis>Running a bitbake invocation for a build:</emphasis> Use
                                <filename>oeqa.utils.commands.bitbake()</filename></para>
                    </listitem>
                    <listitem>
                        <para><emphasis>Running a command:</emphasis> Use
                                <filename>oeqa.utils.commandsrunCmd()</filename></para>
                    </listitem>
                </itemizedlist></para>
            <para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for
                launching the <filename>runqemu</filename> command for testing things within a
                running, virtualized image.</para>
            <para>You can run these tests in parallel. Parallelism works per test class, so tests
                within a given test class should always run in the same build, while  tests in
                different classes or modules may be split into different builds. There is no data
                store available for these tests since the tests launch the
                    <filename>bitbake</filename> command and exist outside of its context. As a
                result, common bitbake library functions (bb.*) are also unavailable.</para>
        </section>

        <section id='testimage-example'>
            <title><filename>testimage</filename></title>

            <para>These tests are run once an image is up and running, either on target hardware or
                under QEMU. As a result, they are assumed to be running in a target image
                environment, as opposed to a host build environment. A simple example from
                    <filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the
                following:<literallayout class="monospaced">
     class PythonTest(OERuntimeTestCase):
         @OETestDepends(['ssh.SSHTest.test_ssh'])
         @OEHasPackage(['python3-core'])
         def test_python3(self):
             cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,
     jbeyq', 'rot13'))\""
             status, output = self.target.run(cmd)
             msg = 'Exit status was not 0. Output: %s' % output
             self.assertEqual(status, 0, msg=msg)
                    </literallayout></para>
            <para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>
                class wraps <filename>unittest.TestCase</filename>. Within the test,
                    <filename>self.target</filename> represents the target system, where commands
                can be run on it using the <filename>run()</filename> method. </para>
            <para>To ensure certain test or package dependencies are met, you can use the
                    <filename>OETestDepends</filename> and <filename>OEHasPackage</filename>
                decorators. For example, the test in this example would only make sense if
                python3-core is installed in the image.</para>
        </section>

        <section id='testsdk_ext-example'>
            <title><filename>testsdk_ext</filename></title>

            <para>These tests are run against built extensible SDKs (eSDKs). The tests can assume
                that the eSDK environment has already been setup. An example from
                    <filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
                following:<literallayout class="monospaced">
     class DevtoolTest(OESDKExtTestCase):
          @classmethod
          def setUpClass(cls):
          myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
          cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
          shutil.copytree(myapp_src, cls.myapp_dst)
          subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
          subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
          subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)

         @classmethod
         def tearDownClass(cls):
         shutil.rmtree(cls.myapp_dst)
         def _test_devtool_build(self, directory):
         self._run('devtool add myapp %s' % directory)
         try:
         self._run('devtool build myapp')
         finally:
         self._run('devtool reset myapp')
         def test_devtool_build_make(self):
         self._test_devtool_build(self.myapp_dst)
                    </literallayout>In
                this example, the <filename>devtool</filename> command is tested to see whether a
                sample application can be built with the <filename>devtool build</filename> command
                within the eSDK.</para>
        </section>

        <section id='testsdk-example'>
            <title><filename>testsdk</filename></title>

            <para>These tests are run against built SDKs. The tests can assume that an SDK has
                already been extracted and its environment file has been sourced. A simple example
                from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
                following:<literallayout class="monospaced">
     class Python3Test(OESDKTestCase):
         def setUp(self):
             if not (self.tc.hasHostPackage("nativesdk-python3-core") or
                     self.tc.hasHostPackage("python3-core-native")):
                 raise unittest.SkipTest("No python3 package in the SDK")

         def test_python3(self):
             cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
             output = self._run(cmd)
             self.assertEqual(output, "Hello, world\n")
                    </literallayout>In
                this example, if nativesdk-python3-core has been installed into the SDK, the code
                runs the python3 interpreter with a basic command to check it is working correctly.
                The test would only run if python3 is installed in the SDK.</para>
        </section>

        <section id='oe-build-perf-test-example'>
            <title><filename>oe-build-perf-test</filename></title>

            <para>The performance tests usually measure how long operations take and the resource
                utilisation as that happens. An example from
                    <filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
                following:<literallayout class="monospaced">
     class Test3(BuildPerfTestCase):

       def test3(self):
           """Bitbake parsing (bitbake -p)"""
           # Drop all caches and parse
           self.rm_cache()
           oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
           self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
                                      'bitbake -p (no caches)')
           # Drop tmp/cache
           oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
           self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
                                      'bitbake -p (no tmp/cache)')
           # Parse with fully cached data
           self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
                                      'bitbake -p (cached)')
                    </literallayout>This
                example shows how three specific parsing timings are measured, with and without
                various caches, to show how BitBake’s parsing performance trends over time.</para>
        </section>
    </section>
    <section id='test-writing-considerations'>
        <title>Considerations When Writing Tests</title>
        <para>When writing good tests, there are several things to keep in mind. Since things
            running on the Autobuilder are accessed concurrently by multiple workers, consider the
            following:</para>
        <formalpara>
            <title>Running "cleanall" is not permitted</title>
            <para>This can delete files from DL_DIR which would potentially break other builds
                running in parallel. If this is required, DL_DIR must be set to an isolated
                directory.</para>
        </formalpara>
        <formalpara>
            <title>Running "cleansstate" is not permitted</title>
            <para>This can delete files from SSTATE_DIR which would potentially break other builds
                running in parallel. If this is required, SSTATE_DIR must be set to an isolated
                directory. Alternatively, you can use the "-f" option with the
                    <filename>bitbake</filename> command to "taint" tasks by changing the sstate
                checksums to ensure sstate cache items will not be reused.</para>
        </formalpara>
        <formalpara>
            <title>Tests should not change the metadata</title>
            <para>This is particularly true for oe-selftests since these can run in parallel and
                changing metadata leads to changing checksums, which confuses BitBake while running
                in parallel. If this is necessary, copy layers to a temporary location and modify
                them. Some tests need to change metadata, such as the devtool tests. To prevent the
                metadate from changes, set up temporary copies of that data first.</para>
        </formalpara>
    </section>








</chapter>
<!--
vim: expandtab tw=80 ts=4
-->