From 6f248db9a26e963ae8379e40371ed478ed8e3d3f Mon Sep 17 00:00:00 2001 From: Nicolas Dechesne Date: Fri, 13 Nov 2020 01:25:13 +0100 Subject: sphinx: remove test-manual This is a new manual in YP 3.2, let's remove it from the 3.1 Sphinx backport. (From yocto-docs rev: baf4410abad44b7f1fc6e4f1c445060b87c21c32) Signed-off-by: Nicolas Dechesne Signed-off-by: Richard Purdie --- documentation/index.rst | 1 - .../overview-manual/overview-manual-yp-intro.rst | 2 +- .../test-manual/figures/ab-test-cluster.png | Bin 18684 -> 0 bytes .../test-manual/figures/test-manual-title.png | Bin 15382 -> 0 bytes documentation/test-manual/history.rst | 16 - .../test-manual/test-manual-customization.xsl | 29 - documentation/test-manual/test-manual-intro.rst | 550 ------------ documentation/test-manual/test-manual-intro.xml | 624 ------------- documentation/test-manual/test-manual-style.css | 991 --------------------- .../test-manual/test-manual-test-process.rst | 103 --- .../test-manual/test-manual-test-process.xml | 110 --- .../test-manual-understand-autobuilder.rst | 305 ------- .../test-manual-understand-autobuilder.xml | 314 ------- documentation/test-manual/test-manual.rst | 18 - documentation/test-manual/test-manual.xml | 104 --- 15 files changed, 1 insertion(+), 3166 deletions(-) delete mode 100644 documentation/test-manual/figures/ab-test-cluster.png delete mode 100644 documentation/test-manual/figures/test-manual-title.png delete mode 100644 documentation/test-manual/history.rst delete mode 100644 documentation/test-manual/test-manual-customization.xsl delete mode 100644 documentation/test-manual/test-manual-intro.rst delete mode 100644 documentation/test-manual/test-manual-intro.xml delete mode 100644 documentation/test-manual/test-manual-style.css delete mode 100644 documentation/test-manual/test-manual-test-process.rst delete mode 100644 documentation/test-manual/test-manual-test-process.xml delete mode 100644 documentation/test-manual/test-manual-understand-autobuilder.rst delete mode 100644 documentation/test-manual/test-manual-understand-autobuilder.xml delete mode 100644 documentation/test-manual/test-manual.rst delete mode 100755 documentation/test-manual/test-manual.xml (limited to 'documentation') diff --git a/documentation/index.rst b/documentation/index.rst index 258ecb81a7..a1605ade91 100644 --- a/documentation/index.rst +++ b/documentation/index.rst @@ -33,7 +33,6 @@ Welcome to The Yocto Project's documentation! Profile and Tracing Manual Application Development and the Extensible SDK (eSDK) Toaster Manual - Test Environment Manual Bitbake User Manual .. toctree:: diff --git a/documentation/overview-manual/overview-manual-yp-intro.rst b/documentation/overview-manual/overview-manual-yp-intro.rst index 5cdab7ca4a..2b3c3b07c7 100644 --- a/documentation/overview-manual/overview-manual-yp-intro.rst +++ b/documentation/overview-manual/overview-manual-yp-intro.rst @@ -401,7 +401,7 @@ activities using the Yocto Project: benefit of the development community. You can learn more about the AutoBuilder used by the Yocto Project - Autobuilder :doc:`here <../test-manual/test-manual-understand-autobuilder>`. + Autobuilder `here <&YOCTO_AB_URL;>`__. - *Cross-Prelink:* Prelinking is the process of pre-computing the load addresses and link tables generated by the dynamic linker as compared diff --git a/documentation/test-manual/figures/ab-test-cluster.png b/documentation/test-manual/figures/ab-test-cluster.png deleted file mode 100644 index 6a6a7882b4..0000000000 Binary files a/documentation/test-manual/figures/ab-test-cluster.png and /dev/null differ diff --git a/documentation/test-manual/figures/test-manual-title.png b/documentation/test-manual/figures/test-manual-title.png deleted file mode 100644 index c709cb9d09..0000000000 Binary files a/documentation/test-manual/figures/test-manual-title.png and /dev/null differ diff --git a/documentation/test-manual/history.rst b/documentation/test-manual/history.rst deleted file mode 100644 index 76d43091a5..0000000000 --- a/documentation/test-manual/history.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.0-UK - -*********************** -Manual Revision History -*********************** - -.. list-table:: - :widths: 10 15 40 - :header-rows: 1 - - * - Revision - - Date - - Note - * - 3.2 - - October 2020 - - The initial document released with the Yocto Project 3.2 Release diff --git a/documentation/test-manual/test-manual-customization.xsl b/documentation/test-manual/test-manual-customization.xsl deleted file mode 100644 index 17bf57c2d1..0000000000 --- a/documentation/test-manual/test-manual-customization.xsl +++ /dev/null @@ -1,29 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst deleted file mode 100644 index 53ad650b35..0000000000 --- a/documentation/test-manual/test-manual-intro.rst +++ /dev/null @@ -1,550 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.0-UK - -***************************************** -The Yocto Project Test Environment Manual -***************************************** - -.. _test-welcome: - -Welcome -======= - -Welcome to the Yocto Project Test Environment Manual! This manual is a -work in progress. The manual contains information about the testing -environment used by the Yocto Project to make sure each major and minor -release works as intended. All the project's testing infrastructure and -processes are publicly visible and available so that the community can -see what testing is being performed, how it's being done and the current -status of the tests and the project at any given time. It is intended -that Other organizations can leverage off the process and testing -environment used by the Yocto Project to create their own automated, -production test environment, building upon the foundations from the -project core. - -Currently, the Yocto Project Test Environment Manual has no projected -release date. This manual is a work-in-progress and is being initially -loaded with information from the README files and notes from key -engineers: - -- *yocto-autobuilder2:* This - :yocto_git:`README.md ` - is the main README which detials how to set up the Yocto Project - Autobuilder. The ``yocto-autobuilder2`` repository represents the - Yocto Project's console UI plugin to Buildbot and the configuration - necessary to configure Buildbot to perform the testing the project - requires. - -- *yocto-autobuilder-helper:* This :yocto_git:`README ` - and repository contains Yocto Project Autobuilder Helper scripts and - configuration. The ``yocto-autobuilder-helper`` repository contains - the "glue" logic that defines which tests to run and how to run them. - As a result, it can be used by any Continuous Improvement (CI) system - to run builds, support getting the correct code revisions, configure - builds and layers, run builds, and collect results. The code is - independent of any CI system, which means the code can work `Buildbot `__, - Jenkins, or others. This repository has a branch per release of the - project defining the tests to run on a per release basis. - -.. _test-yocto-project-autobuilder-overview: - -Yocto Project Autobuilder Overview -================================== - -The Yocto Project Autobuilder collectively refers to the software, -tools, scripts, and procedures used by the Yocto Project to test -released software across supported hardware in an automated and regular -fashion. Basically, during the development of a Yocto Project release, -the Autobuilder tests if things work. The Autobuilder builds all test -targets and runs all the tests. - -The Yocto Project uses now uses standard upstream -`Buildbot `__ (version 9) to -drive its integration and testing. Buildbot Nine has a plug-in interface -that the Yocto Project customizes using code from the -``yocto-autobuilder2`` repository, adding its own console UI plugin. The -resulting UI plug-in allows you to visualize builds in a way suited to -the project's needs. - -A ``helper`` layer provides configuration and job management through -scripts found in the ``yocto-autobuilder-helper`` repository. The -``helper`` layer contains the bulk of the build configuration -information and is release-specific, which makes it highly customizable -on a per-project basis. The layer is CI system-agnostic and contains a -number of Helper scripts that can generate build configurations from -simple JSON files. - -.. note:: - - The project uses Buildbot for historical reasons but also because - many of the project developers have knowledge of python. It is - possible to use the outer layers from another Continuous Integration - (CI) system such as - `Jenkins `__ - instead of Buildbot. - -The following figure shows the Yocto Project Autobuilder stack with a -topology that includes a controller and a cluster of workers: - -.. image:: figures/ab-test-cluster.png - :align: center - -.. _test-project-tests: - -Yocto Project Tests - Types of Testing Overview -=============================================== - -The Autobuilder tests different elements of the project by using -thefollowing types of tests: - -- *Build Testing:* Tests whether specific configurations build by - varying :term:`MACHINE`, - :term:`DISTRO`, other configuration - options, and the specific target images being built (or world). Used - to trigger builds of all the different test configurations on the - Autobuilder. Builds usually cover many different targets for - different architectures, machines, and distributions, as well as - different configurations, such as different init systems. The - Autobuilder tests literally hundreds of configurations and targets. - - - *Sanity Checks During the Build Process:* Tests initiated through - the :ref:`insane ` - class. These checks ensure the output of the builds are correct. - For example, does the ELF architecture in the generated binaries - match the target system? ARM binaries would not work in a MIPS - system! - -- *Build Performance Testing:* Tests whether or not commonly used steps - during builds work efficiently and avoid regressions. Tests to time - commonly used usage scenarios are run through ``oe-build-perf-test``. - These tests are run on isolated machines so that the time - measurements of the tests are accurate and no other processes - interfere with the timing results. The project currently tests - performance on two different distributions, Fedora and Ubuntu, to - ensure we have no single point of failure and can ensure the - different distros work effectively. - -- *eSDK Testing:* Image tests initiated through the following command:: - - $ bitbake image -c testsdkext - - The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task. - -- *Feature Testing:* Various scenario-based tests are run through the - :ref:`OpenEmbedded Self test (oe-selftest) `. We test oe-selftest on each of the main distrubutions - we support. - -- *Image Testing:* Image tests initiated through the following command:: - - $ bitbake image -c testimage - - The tests utilize the :ref:`testimage* ` - classes and the :ref:`ref-tasks-testimage` task. - -- *Layer Testing:* The Autobuilder has the possibility to test whether - specific layers work with the test of the system. The layers tested - may be selected by members of the project. Some key community layers - are also tested periodically. - -- *Package Testing:* A Package Test (ptest) runs tests against packages - built by the OpenEmbedded build system on the target machine. See the - :ref:`Testing Packages With - ptest ` section - in the Yocto Project Development Tasks Manual and the - ":yocto_wiki:`Ptest `" Wiki page for more - information on Ptest. - -- *SDK Testing:* Image tests initiated through the following command:: - - $ bitbake image -c testsdk - - The tests utilize the :ref:`testsdk ` class and - the ``do_testsdk`` task. - -- *Unit Testing:* Unit tests on various components of the system run - through :ref:`bitbake-selftest ` and - :ref:`oe-selftest `. - -- *Automatic Upgrade Helper:* This target tests whether new versions of - software are available and whether we can automatically upgrade to - those new versions. If so, this target emails the maintainers with a - patch to let them know this is possible. - -.. _test-test-mapping: - -How Tests Map to Areas of Code -============================== - -Tests map into the codebase as follows: - -- *bitbake-selftest:* - - These tests are self-contained and test BitBake as well as its APIs, - which include the fetchers. The tests are located in - ``bitbake/lib/*/tests``. - - From within the BitBake repository, run the following:: - - $ bitbake-selftest - - To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` - variable when running "bitbake-selftest" as follows:: - - $ BB_SKIP_NETTEST=yes bitbake-selftest - - The default output is quiet and just prints a summary of what was - run. To see more information, there is a verbose option:: - - $ bitbake-selftest -v - - Use this option when you wish to skip tests that access the network, - which are mostly necessary to test the fetcher modules. To specify - individual test modules to run, append the test module name to the - "bitbake-selftest" command. For example, to specify the tests for the - bb.data.module, run:: - - $ bitbake-selftest bb.test.data.module - - You can also specify individual tests by defining the full name and module - plus the class path of the test, for example:: - - $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override - - The tests are based on `Python - unittest `__. - -- *oe-selftest:* - - - These tests use OE to test the workflows, which include testing - specific features, behaviors of tasks, and API unit tests. - - - The tests can take advantage of parallelism through the "-j" - option, which can specify a number of threads to spread the tests - across. Note that all tests from a given class of tests will run - in the same thread. To parallelize large numbers of tests you can - split the class into multiple units. - - - The tests are based on Python unittest. - - - The code for the tests resides in - ``meta/lib/oeqa/selftest/cases/``. - - - To run all the tests, enter the following command:: - - $ oe-selftest -a - - - To run a specific test, use the following command form where - testname is the name of the specific test:: - - $ oe-selftest -r - - For example, the following command would run the tinfoil - getVar API test:: - - $ oe-selftest -r tinfoil.TinfoilTests.test_getvar - - It is also possible to run a set - of tests. For example the following command will run all of the - tinfoil tests:: - - $ oe-selftest -r tinfoil - -- *testimage:* - - - These tests build an image, boot it, and run tests against the - image's content. - - - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``. - - - You need to set the :term:`IMAGE_CLASSES` variable as follows:: - - IMAGE_CLASSES += "testimage" - - - Run the tests using the following command form:: - - $ bitbake image -c testimage - -- *testsdk:* - - - These tests build an SDK, install it, and then run tests against - that SDK. - - - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. - - - Run the test using the following command form:: - - $ bitbake image -c testsdk - -- *testsdk_ext:* - - - These tests build an extended SDK (eSDK), install that eSDK, and - run tests against the eSDK. - - - The code for these tests resides in ``meta/lib/oeqa/esdk``. - - - To run the tests, use the following command form:: - - $ bitbake image -c testsdkext - -- *oe-build-perf-test:* - - - These tests run through commonly used usage scenarios and measure - the performance times. - - - The code for these tests resides in ``meta/lib/oeqa/buildperf``. - - - To run the tests, use the following command form:: - - $ oe-build-perf-test - - The command takes a number of options, - such as where to place the test results. The Autobuilder Helper - Scripts include the ``build-perf-test-wrapper`` script with - examples of how to use the oe-build-perf-test from the command - line. - - Use the ``oe-git-archive`` command to store test results into a - Git repository. - - Use the ``oe-build-perf-report`` command to generate text reports - and HTML reports with graphs of the performance data. For - examples, see - :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` - and - :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. - - - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. - -Test Examples -============= - -This section provides example tests for each of the tests listed in the -:ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section. - -For oeqa tests, testcases for each area reside in the main test -directory at ``meta/lib/oeqa/selftest/cases`` directory. - -For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/`` -directory. - -.. _bitbake-selftest-example: - -``bitbake-selftest`` --------------------- - -A simple test example from ``lib/bb/tests/data.py`` is:: - - class DataExpansions(unittest.TestCase): - def setUp(self): - self.d = bb.data.init() - self.d["foo"] = "value_of_foo" - self.d["bar"] = "value_of_bar" - self.d["value_of_foo"] = "value_of_'value_of_foo'" - - def test_one_var(self): - val = self.d.expand("${foo}") - self.assertEqual(str(val), "value_of_foo") - -In this example, a ``DataExpansions`` class of tests is created, -derived from standard python unittest. The class has a common ``setUp`` -function which is shared by all the tests in the class. A simple test is -then added to test that when a variable is expanded, the correct value -is found. - -Bitbake selftests are straightforward python unittest. Refer to the -Python unittest documentation for additional information on writing -these tests at: https://docs.python.org/3/library/unittest.html. - -.. _oe-selftest-example: - -``oe-selftest`` ---------------- - -These tests are more complex due to the setup required behind the scenes -for full builds. Rather than directly using Python's unittest, the code -wraps most of the standard objects. The tests can be simple, such as -testing a command from within the OE build environment using the -following example:: - - class BitbakeLayers(OESelftestTestCase): - def test_bitbakelayers_showcrossdepends(self): - result = runCmd('bitbake-layers show-cross-depends') - self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output) - -This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, -creates a testcase from the ``OESelftestTestCase`` class, derived -from ``unittest.TestCase``, which runs the ``bitbake-layers`` command -and checks the output to ensure it contains something we know should be -here. - -The ``oeqa.utils.commands`` module contains Helpers which can assist -with common tasks, including: - -- *Obtaining the value of a bitbake variable:* Use - ``oeqa.utils.commands.get_bb_var()`` or use - ``oeqa.utils.commands.get_bb_vars()`` for more than one variable - -- *Running a bitbake invocation for a build:* Use - ``oeqa.utils.commands.bitbake()`` - -- *Running a command:* Use ``oeqa.utils.commandsrunCmd()`` - -There is also a ``oeqa.utils.commands.runqemu()`` function for launching -the ``runqemu`` command for testing things within a running, virtualized -image. - -You can run these tests in parallel. Parallelism works per test class, -so tests within a given test class should always run in the same build, -while tests in different classes or modules may be split into different -builds. There is no data store available for these tests since the tests -launch the ``bitbake`` command and exist outside of its context. As a -result, common bitbake library functions (bb.\*) are also unavailable. - -.. _testimage-example: - -``testimage`` -------------- - -These tests are run once an image is up and running, either on target -hardware or under QEMU. As a result, they are assumed to be running in a -target image environment, as opposed to a host build environment. A -simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains -the following:: - - class PythonTest(OERuntimeTestCase): - @OETestDepends(['ssh.SSHTest.test_ssh']) - @OEHasPackage(['python3-core']) - def test_python3(self): - cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" - status, output = self.target.run(cmd) - msg = 'Exit status was not 0. Output: %s' % output - self.assertEqual(status, 0, msg=msg) - -In this example, the ``OERuntimeTestCase`` class wraps -``unittest.TestCase``. Within the test, ``self.target`` represents the -target system, where commands can be run on it using the ``run()`` -method. - -To ensure certain test or package dependencies are met, you can use the -``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test -in this example would only make sense if python3-core is installed in -the image. - -.. _testsdk_ext-example: - -``testsdk_ext`` ---------------- - -These tests are run against built extensible SDKs (eSDKs). The tests can -assume that the eSDK environment has already been setup. An example from -``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: - - class DevtoolTest(OESDKExtTestCase): - @classmethod def setUpClass(cls): - myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") - cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") - shutil.copytree(myapp_src, cls.myapp_dst) - subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) - - @classmethod - def tearDownClass(cls): - shutil.rmtree(cls.myapp_dst) - def _test_devtool_build(self, directory): - self._run('devtool add myapp %s' % directory) - try: - self._run('devtool build myapp') - finally: - self._run('devtool reset myapp') - def test_devtool_build_make(self): - self._test_devtool_build(self.myapp_dst) - -In this example, the ``devtool`` -command is tested to see whether a sample application can be built with -the ``devtool build`` command within the eSDK. - -.. _testsdk-example: - -``testsdk`` ------------ - -These tests are run against built SDKs. The tests can assume that an SDK -has already been extracted and its environment file has been sourced. A -simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the -following:: - - class Python3Test(OESDKTestCase): - def setUp(self): - if not (self.tc.hasHostPackage("nativesdk-python3-core") or - self.tc.hasHostPackage("python3-core-native")): - raise unittest.SkipTest("No python3 package in the SDK") - - def test_python3(self): - cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" - output = self._run(cmd) - self.assertEqual(output, "Hello, world\n") - -In this example, if nativesdk-python3-core has been installed into the SDK, the code runs -the python3 interpreter with a basic command to check it is working -correctly. The test would only run if python3 is installed in the SDK. - -.. _oe-build-perf-test-example: - -``oe-build-perf-test`` ----------------------- - -The performance tests usually measure how long operations take and the -resource utilisation as that happens. An example from -``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: - - class Test3(BuildPerfTestCase): - def test3(self): - """Bitbake parsing (bitbake -p)""" - # Drop all caches and parse - self.rm_cache() - oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) - self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', - 'bitbake -p (no caches)') - # Drop tmp/cache - oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) - self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', - 'bitbake -p (no tmp/cache)') - # Parse with fully cached data - self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', - 'bitbake -p (cached)') - -This example shows how three specific parsing timings are -measured, with and without various caches, to show how BitBake's parsing -performance trends over time. - -.. _test-writing-considerations: - -Considerations When Writing Tests -================================= - -When writing good tests, there are several things to keep in mind. Since -things running on the Autobuilder are accessed concurrently by multiple -workers, consider the following: - -**Running "cleanall" is not permitted.** - -This can delete files from DL_DIR which would potentially break other -builds running in parallel. If this is required, DL_DIR must be set to -an isolated directory. - -**Running "cleansstate" is not permitted.** - -This can delete files from SSTATE_DIR which would potentially break -other builds running in parallel. If this is required, SSTATE_DIR must -be set to an isolated directory. Alternatively, you can use the "-f" -option with the ``bitbake`` command to "taint" tasks by changing the -sstate checksums to ensure sstate cache items will not be reused. - -**Tests should not change the metadata.** - -This is particularly true for oe-selftests since these can run in -parallel and changing metadata leads to changing checksums, which -confuses BitBake while running in parallel. If this is necessary, copy -layers to a temporary location and modify them. Some tests need to -change metadata, such as the devtool tests. To prevent the metadate from -changes, set up temporary copies of that data first. diff --git a/documentation/test-manual/test-manual-intro.xml b/documentation/test-manual/test-manual-intro.xml deleted file mode 100644 index 0cdbee4d8e..0000000000 --- a/documentation/test-manual/test-manual-intro.xml +++ /dev/null @@ -1,624 +0,0 @@ - %poky; ] > - - - - -The Yocto Project Test Environment Manual -
- Welcome - - Welcome to the Yocto Project Test Environment Manual! This manual is a work in - progress. The manual contains information about the testing environment used by the - Yocto Project to make sure each major and minor release works as intended. All the - project's testing infrastructure and processes are publicly visible and available so - that the community can see what testing is being performed, how it's being done and the - current status of the tests and the project at any given time. It is intended that Other - organizations can leverage off the process and testing environment used by the Yocto - Project to create their own automated, production test environment, building upon the - foundations from the project core. - - Currently, the Yocto Project Test Environment Manual has no projected release date. - This manual is a work-in-progress and is being initially loaded with information from - the README files and notes from key engineers: - - - yocto-autobuilder2: This README.md is the main README which - detials how to set up the Yocto Project Autobuilder. The - yocto-autobuilder2 repository represents the Yocto - Project's console UI plugin to Buildbot and the configuration necessary to - configure Buildbot to perform the testing the project requires. - - - - yocto-autobuilder-helper: This - README and repository contains Yocto - Project Autobuilder Helper scripts and configuration. The - yocto-autobuilder-helper repository contains the - "glue" logic that defines which tests to run and how to run them. As a - result, it can be used by any Continuous Improvement (CI) system to run - builds, support getting the correct code revisions, configure builds and - layers, run builds, and collect results. The code is independent of any CI - system, which means the code can work Buildbot, Jenkins, or others. This - repository has a branch per release of the project defining the tests to run - on a per release basis. - - - -
- -
- Yocto Project Autobuilder Overview - - The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and - procedures used by the Yocto Project to test released software across supported hardware - in an automated and regular fashion. Basically, during the development of a Yocto - Project release, the Autobuilder tests if things work. The Autobuilder builds all test - targets and runs all the tests. - - The Yocto Project uses now uses standard upstream Buildbot (version 9) to drive - its integration and testing. Buildbot Nine has a plug-in interface that the Yocto - Project customizes using code from the yocto-autobuilder2 - repository, adding its own console UI plugin. The resulting UI plug-in allows you to - visualize builds in a way suited to the project's needs. - - A helper layer provides configuration and job management through - scripts found in the yocto-autobuilder-helper repository. The - helper layer contains the bulk of the build configuration - information and is release-specific, which makes it highly customizable on a per-project - basis. The layer is CI system-agnostic and contains a number of Helper scripts that can - generate build configurations from simple JSON files. - The project uses Buildbot for historical reasons but also because many of the - project developers have knowledge of python. It is possible to use the outer - layers from another Continuous Integration (CI) system such as Jenkins - instead of Buildbot. - - - - The following figure shows the Yocto Project Autobuilder stack with a topology that - includes a controller and a cluster of workers: - -
- -
- Yocto Project Tests - Types of Testing Overview - - The Autobuilder tests different elements of the project by using thefollowing types of - tests: - - - Build Testing: Tests whether specific configurations - build by varying MACHINE, DISTRO, other configuration options, and - the specific target images being built (or world). Used to trigger builds of - all the different test configurations on the Autobuilder. Builds usually - cover many different targets for different architectures, machines, and - distributions, as well as different configurations, such as different init - systems. The Autobuilder tests literally hundreds of configurations and - targets. - - - Sanity Checks During the Build Process: - Tests initiated through the insane class. These checks - ensure the output of the builds are correct. For example, does - the ELF architecture in the generated binaries match the target - system? ARM binaries would not work in a MIPS system! - - - - - - Build Performance Testing: Tests whether or not - commonly used steps during builds work efficiently and avoid regressions. - Tests to time commonly used usage scenarios are run through - oe-build-perf-test. These tests are run on isolated - machines so that the time measurements of the tests are accurate and no - other processes interfere with the timing results. The project currently - tests performance on two different distributions, Fedora and Ubuntu, to - ensure we have no single point of failure and can ensure the different - distros work effectively. - - - - eSDK Testing: Image tests initiated through the - following command: - - $ bitbake image -c testsdkext - - The tests utilize the testsdkext class and the - do_testsdkext task. - - - - Feature Testing: Various scenario-based tests are run - through the OpenEmbedded Self-Test (oe-selftest). We test oe-selftest on - each of the main distrubutions we support. - - - - Image Testing: Image tests initiated through the - following command: - - $ bitbake image -c testimage - - The tests utilize the testimage* classes and the do_testimage task. - - - - Layer Testing: The Autobuilder has the possibility to - test whether specific layers work with the test of the system. The layers - tested may be selected by members of the project. Some key community layers - are also tested periodically. - - - - Package Testing: A Package Test (ptest) runs tests - against packages built by the OpenEmbedded build system on the target - machine. See the "Testing Packages - With ptest" section in the Yocto Project Development Tasks - Manual and the "Ptest" Wiki - page for more information on Ptest. - - - - SDK Testing: Image tests initiated through the - following command: - - $ bitbake image -c testsdk - - The tests utilize the testsdk class and the - do_testsdk task. - - - - Unit Testing: Unit tests on various components of the - system run through oe-selftest and bitbake-selftest. - - - - Automatic Upgrade Helper: This target tests whether new - versions of software are available and whether we can automatically upgrade - to those new versions. If so, this target emails the maintainers with a - patch to let them know this is possible. - - - -
- -
- How Tests Map to Areas of Code - - - Tests map into the codebase as follows: - - - bitbake-selftest: - These tests are self-contained and test BitBake as well as its APIs, which - include the fetchers. The tests are located in - bitbake/lib/*/tests. - From within the BitBake repository, run the following: - - $ bitbake-selftest - - - To skip tests that access the Internet, use the - BB_SKIP_NETTEST variable when running - "bitbake-selftest" as follows: - - $ BB_SKIP_NETTEST=yes bitbake-selftest - - The default output is quiet and just prints a summary of what was run. To - see more information, there is a verbose - option: - $ bitbake-selftest -v - - Use this option when you wish to skip tests that access the network, which - are mostly necessary to test the fetcher modules. To specify individual test - modules to run, append the test module name to the "bitbake-selftest" - command. For example, to specify the tests for the bb.data.module, run: - - $ bitbake-selftest bb.test.data.module - You - can also specify individual tests by defining the full name and module plus - the class path of the test, for example: - - $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override - - The tests are based on Python - unittest. - - oe-selftest: - - These tests use OE to test the workflows, which include - testing specific features, behaviors of tasks, and API unit - tests. - - - The tests can take advantage of parallelism through the "-j" - option, which can specify a number of threads to spread the - tests across. Note that all tests from a given class of tests - will run in the same thread. To parallelize large numbers of - tests you can split the class into multiple units. - - - The tests are based on Python unittest. - - - The code for the tests resides in - meta/lib/oeqa/selftest/cases/. - - - To run all the tests, enter the following command: - - $ oe-selftest -a - - - - - To run a specific test, use the following command form where - testname is the name of the - specific test: - - $ oe-selftest -r testname - - For example, the following command would run the tinfoil getVar - API - test: - $ oe-selftest -r tinfoil.TinfoilTests.test_getvar - It - is also possible to run a set of tests. For example the - following command will run all of the tinfoil - tests: - $ oe-selftest -r tinfoil - - - - - - testimage: - - - These tests build an image, boot it, and run tests - against the image's content. - - The code for these tests resides in meta/lib/oeqa/runtime/cases/. - - You need to set the - IMAGE_CLASSES - variable as follows: - - IMAGE_CLASSES += "testimage" - - - - Run the tests using the following command form: - - $ bitbake image -c testimage - - - - - - testsdk: - - These tests build an SDK, install it, and then run tests against that SDK. - The code for these tests resides in meta/lib/oeqa/sdk/cases/. - Run the test using the following command form: - - $ bitbake image -c testsdk - - - - - - testsdk_ext: - - These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. - The code for these tests resides in meta/lib/oeqa/esdk. - To run the tests, use the following command form: - - $ bitbake image -c testsdkext - - - - - - - - oe-build-perf-test: - - These tests run through commonly used usage scenarios and measure the performance times. - The code for these tests resides in meta/lib/oeqa/buildperf. - To run the tests, use the following command form: - - $ oe-build-perf-test options - The - command takes a number of options, such as where to place the - test results. The Autobuilder Helper Scripts include the - build-perf-test-wrapper script with - examples of how to use the oe-build-perf-test from the command - line. - Use the oe-git-archive command to store - test results into a Git repository. - Use the oe-build-perf-report command to - generate text reports and HTML reports with graphs of the - performance data. For examples, see http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html - and http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt. - - The tests are contained in - lib/oeqa/buildperf/test_basic.py. - - - - - - - - - -
- -
- Test Examples - - This section provides example tests for each of the tests listed in the How Tests Map to Areas of Code section. - For oeqa tests, testcases for each area reside in the main test directory at - meta/lib/oeqa/selftest/cases directory. - For oe-selftest. bitbake testcases reside in the lib/bb/tests/ - directory. - -
- <filename>bitbake-selftest</filename> - - A simple test example from lib/bb/tests/data.py is: - - class DataExpansions(unittest.TestCase): - def setUp(self): - self.d = bb.data.init() - self.d["foo"] = "value_of_foo" - self.d["bar"] = "value_of_bar" - self.d["value_of_foo"] = "value_of_'value_of_foo'" - - def test_one_var(self): - val = self.d.expand("${foo}") - self.assertEqual(str(val), "value_of_foo") - - - In this example, a DataExpansions class - of tests is created, derived from standard python unittest. The class has a common - setUp function which is shared by all the tests in the - class. A simple test is then added to test that when a variable is expanded, the - correct value is found. - Bitbake selftests are straightforward python unittest. Refer to the Python - unittest documentation for additional information on writing these tests at: https://docs.python.org/3/library/unittest.html. -
- -
- <filename>oe-selftest</filename> - - These tests are more complex due to the setup required behind the scenes for full - builds. Rather than directly using Python's unittest, the code wraps most of the - standard objects. The tests can be simple, such as testing a command from within the - OE build environment using the following - example: - class BitbakeLayers(OESelftestTestCase): - def test_bitbakelayers_showcrossdepends(self): - result = runCmd('bitbake-layers show-cross-depends') - self.assertTrue('aspell' in result.output, msg = "No dependencies - were shown. bitbake-layers show-cross-depends output: - %s"% result.output) - - This example, taken from - meta/lib/oeqa/selftest/cases/bblayers.py, creates a - testcase from the OESelftestTestCase - class, derived from unittest.TestCase, which runs the - bitbake-layers command and checks the output to ensure it - contains something we know should be here. - The oeqa.utils.commands module contains Helpers which can - assist with common tasks, including: - - Obtaining the value of a bitbake variable: Use - oeqa.utils.commands.get_bb_var() or use - oeqa.utils.commands.get_bb_vars() for more than - one variable - - - Running a bitbake invocation for a build: Use - oeqa.utils.commands.bitbake() - - - Running a command: Use - oeqa.utils.commandsrunCmd() - - - There is also a oeqa.utils.commands.runqemu() function for - launching the runqemu command for testing things within a - running, virtualized image. - You can run these tests in parallel. Parallelism works per test class, so tests - within a given test class should always run in the same build, while tests in - different classes or modules may be split into different builds. There is no data - store available for these tests since the tests launch the - bitbake command and exist outside of its context. As a - result, common bitbake library functions (bb.*) are also unavailable. -
- -
- <filename>testimage</filename> - - These tests are run once an image is up and running, either on target hardware or - under QEMU. As a result, they are assumed to be running in a target image - environment, as opposed to a host build environment. A simple example from - meta/lib/oeqa/runtime/cases/python.py contains the - following: - class PythonTest(OERuntimeTestCase): - @OETestDepends(['ssh.SSHTest.test_ssh']) - @OEHasPackage(['python3-core']) - def test_python3(self): - cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, - jbeyq', 'rot13'))\"" - status, output = self.target.run(cmd) - msg = 'Exit status was not 0. Output: %s' % output - self.assertEqual(status, 0, msg=msg) - - In this example, the OERuntimeTestCase - class wraps unittest.TestCase. Within the test, - self.target represents the target system, where commands - can be run on it using the run() method. - To ensure certain test or package dependencies are met, you can use the - OETestDepends and OEHasPackage - decorators. For example, the test in this example would only make sense if - python3-core is installed in the image. -
- -
- <filename>testsdk_ext</filename> - - These tests are run against built extensible SDKs (eSDKs). The tests can assume - that the eSDK environment has already been setup. An example from - meta/lib/oeqa/sdk/cases/devtool.py contains the - following: - class DevtoolTest(OESDKExtTestCase): - @classmethod - def setUpClass(cls): - myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") - cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") - shutil.copytree(myapp_src, cls.myapp_dst) - subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) - - @classmethod - def tearDownClass(cls): - shutil.rmtree(cls.myapp_dst) - def _test_devtool_build(self, directory): - self._run('devtool add myapp %s' % directory) - try: - self._run('devtool build myapp') - finally: - self._run('devtool reset myapp') - def test_devtool_build_make(self): - self._test_devtool_build(self.myapp_dst) - In - this example, the devtool command is tested to see whether a - sample application can be built with the devtool build command - within the eSDK. -
- -
- <filename>testsdk</filename> - - These tests are run against built SDKs. The tests can assume that an SDK has - already been extracted and its environment file has been sourced. A simple example - from meta/lib/oeqa/sdk/cases/python2.py contains the - following: - class Python3Test(OESDKTestCase): - def setUp(self): - if not (self.tc.hasHostPackage("nativesdk-python3-core") or - self.tc.hasHostPackage("python3-core-native")): - raise unittest.SkipTest("No python3 package in the SDK") - - def test_python3(self): - cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" - output = self._run(cmd) - self.assertEqual(output, "Hello, world\n") - In - this example, if nativesdk-python3-core has been installed into the SDK, the code - runs the python3 interpreter with a basic command to check it is working correctly. - The test would only run if python3 is installed in the SDK. -
- -
- <filename>oe-build-perf-test</filename> - - The performance tests usually measure how long operations take and the resource - utilisation as that happens. An example from - meta/lib/oeqa/buildperf/test_basic.py contains the - following: - class Test3(BuildPerfTestCase): - - def test3(self): - """Bitbake parsing (bitbake -p)""" - # Drop all caches and parse - self.rm_cache() - oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) - self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', - 'bitbake -p (no caches)') - # Drop tmp/cache - oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) - self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', - 'bitbake -p (no tmp/cache)') - # Parse with fully cached data - self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', - 'bitbake -p (cached)') - This - example shows how three specific parsing timings are measured, with and without - various caches, to show how BitBake's parsing performance trends over time. -
-
-
- Considerations When Writing Tests - When writing good tests, there are several things to keep in mind. Since things - running on the Autobuilder are accessed concurrently by multiple workers, consider the - following: - - Running "cleanall" is not permitted - This can delete files from DL_DIR which would potentially break other builds - running in parallel. If this is required, DL_DIR must be set to an isolated - directory. - - - Running "cleansstate" is not permitted - This can delete files from SSTATE_DIR which would potentially break other builds - running in parallel. If this is required, SSTATE_DIR must be set to an isolated - directory. Alternatively, you can use the "-f" option with the - bitbake command to "taint" tasks by changing the sstate - checksums to ensure sstate cache items will not be reused. - - - Tests should not change the metadata - This is particularly true for oe-selftests since these can run in parallel and - changing metadata leads to changing checksums, which confuses BitBake while running - in parallel. If this is necessary, copy layers to a temporary location and modify - them. Some tests need to change metadata, such as the devtool tests. To prevent the - metadate from changes, set up temporary copies of that data first. - -
- - - - - - - - - - diff --git a/documentation/test-manual/test-manual-style.css b/documentation/test-manual/test-manual-style.css deleted file mode 100644 index 10ee4c79cf..0000000000 --- a/documentation/test-manual/test-manual-style.css +++ /dev/null @@ -1,991 +0,0 @@ -/* - Generic XHTML / DocBook XHTML CSS Stylesheet. - - SPDX-License-Identifier: CC-BY-2.0-UK - - Browser wrangling and typographic design by - Oyvind Kolas / pippin@gimp.org - - Customised for Poky by - Matthew Allum / mallum@o-hand.com - - Thanks to: - Liam R. E. Quin - William Skaggs - Jakub Steiner - - Structure - --------- - - The stylesheet is divided into the following sections: - - Positioning - Margins, paddings, width, font-size, clearing. - Decorations - Borders, style - Colors - Colors - Graphics - Graphical backgrounds - Nasty IE tweaks - Workarounds needed to make it work in internet explorer, - currently makes the stylesheet non validating, but up until - this point it is validating. - Mozilla extensions - Transparency for footer - Rounded corners on boxes - -*/ - - - /*************** / - / Positioning / -/ ***************/ - -body { - font-family: Verdana, Sans, sans-serif; - - min-width: 640px; - width: 80%; - margin: 0em auto; - padding: 2em 5em 5em 5em; - color: #333; -} - -h1,h2,h3,h4,h5,h6,h7 { - font-family: Arial, Sans; - color: #00557D; - clear: both; -} - -h1 { - font-size: 2em; - text-align: left; - padding: 0em 0em 0em 0em; - margin: 2em 0em 0em 0em; -} - -h2.subtitle { - margin: 0.10em 0em 3.0em 0em; - padding: 0em 0em 0em 0em; - font-size: 1.8em; - padding-left: 20%; - font-weight: normal; - font-style: italic; -} - -h2 { - margin: 2em 0em 0.66em 0em; - padding: 0.5em 0em 0em 0em; - font-size: 1.5em; - font-weight: bold; -} - -h3.subtitle { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; - font-size: 142.14%; - text-align: right; -} - -h3 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 140%; - font-weight: bold; -} - -h4 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 120%; - font-weight: bold; -} - -h5 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 110%; - font-weight: bold; -} - -h6 { - margin: 1em 0em 0em 0em; - padding: 1em 0em 0em 0em; - font-size: 110%; - font-weight: bold; -} - -.authorgroup { - background-color: transparent; - background-repeat: no-repeat; - padding-top: 256px; - background-image: url("figures/test-manual-title.png"); - background-position: left top; - margin-top: -256px; - padding-right: 50px; - margin-left: 0px; - text-align: right; - width: 740px; -} - -h3.author { - margin: 0em 0me 0em 0em; - padding: 0em 0em 0em 0em; - font-weight: normal; - font-size: 100%; - color: #333; - clear: both; -} - -.author tt.email { - font-size: 66%; -} - -.titlepage hr { - width: 0em; - clear: both; -} - -.revhistory { - padding-top: 2em; - clear: both; -} - -.toc, -.list-of-tables, -.list-of-examples, -.list-of-figures { - padding: 1.33em 0em 2.5em 0em; - color: #00557D; -} - -.toc p, -.list-of-tables p, -.list-of-figures p, -.list-of-examples p { - padding: 0em 0em 0em 0em; - padding: 0em 0em 0.3em; - margin: 1.5em 0em 0em 0em; -} - -.toc p b, -.list-of-tables p b, -.list-of-figures p b, -.list-of-examples p b{ - font-size: 100.0%; - font-weight: bold; -} - -.toc dl, -.list-of-tables dl, -.list-of-figures dl, -.list-of-examples dl { - margin: 0em 0em 0.5em 0em; - padding: 0em 0em 0em 0em; -} - -.toc dt { - margin: 0em 0em 0em 0em; - padding: 0em 0em 0em 0em; -} - -.toc dd { - margin: 0em 0em 0em 2.6em; - padding: 0em 0em 0em 0em; -} - -div.glossary dl, -div.variablelist dl { -} - -.glossary dl dt, -.variablelist dl dt, -.variablelist dl dt span.term { - font-weight: normal; - width: 20em; - text-align: right; -} - -.variablelist dl dt { - margin-top: 0.5em; -} - -.glossary dl dd, -.variablelist dl dd { - margin-top: 0em; - margin-left: 25.5em; -} - -.glossary dd p, -.variablelist dd p { - margin-top: 0em; - margin-bottom: 1em; -} - - -div.calloutlist table td { - padding: 0em 0em 0em 0em; - margin: 0em 0em 0em 0em; -} - -div.calloutlist table td p { - margin-top: 0em; - margin-bottom: 1em; -} - -div p.copyright { - text-align: left; -} - -div.legalnotice p.legalnotice-title { - margin-bottom: 0em; -} - -p { - line-height: 1.5em; - margin-top: 0em; - -} - -dl { - padding-top: 0em; -} - -hr { - border: solid 1px; -} - - -.mediaobject, -.mediaobjectco { - text-align: center; -} - -img { - border: none; -} - -ul { - padding: 0em 0em 0em 1.5em; -} - -ul li { - padding: 0em 0em 0em 0em; -} - -ul li p { - text-align: left; -} - -table { - width :100%; -} - -th { - padding: 0.25em; - text-align: left; - font-weight: normal; - vertical-align: top; -} - -td { - padding: 0.25em; - vertical-align: top; -} - -p a[id] { - margin: 0px; - padding: 0px; - display: inline; - background-image: none; -} - -a { - text-decoration: underline; - color: #444; -} - -pre { - overflow: auto; -} - -a:hover { - text-decoration: underline; - /*font-weight: bold;*/ -} - -/* This style defines how the permalink character - appears by itself and when hovered over with - the mouse. */ - -[alt='Permalink'] { color: #eee; } -[alt='Permalink']:hover { color: black; } - - -div.informalfigure, -div.informalexample, -div.informaltable, -div.figure, -div.table, -div.example { - margin: 1em 0em; - padding: 1em; - page-break-inside: avoid; -} - - -div.informalfigure p.title b, -div.informalexample p.title b, -div.informaltable p.title b, -div.figure p.title b, -div.example p.title b, -div.table p.title b{ - padding-top: 0em; - margin-top: 0em; - font-size: 100%; - font-weight: normal; -} - -.mediaobject .caption, -.mediaobject .caption p { - text-align: center; - font-size: 80%; - padding-top: 0.5em; - padding-bottom: 0.5em; -} - -.epigraph { - padding-left: 55%; - margin-bottom: 1em; -} - -.epigraph p { - text-align: left; -} - -.epigraph .quote { - font-style: italic; -} -.epigraph .attribution { - font-style: normal; - text-align: right; -} - -span.application { - font-style: italic; -} - -.programlisting { - font-family: monospace; - font-size: 80%; - white-space: pre; - margin: 1.33em 0em; - padding: 1.33em; -} - -.tip, -.warning, -.caution, -.note { - margin-top: 1em; - margin-bottom: 1em; - -} - -/* force full width of table within div */ -.tip table, -.warning table, -.caution table, -.note table { - border: none; - width: 100%; -} - - -.tip table th, -.warning table th, -.caution table th, -.note table th { - padding: 0.8em 0.0em 0.0em 0.0em; - margin : 0em 0em 0em 0em; -} - -.tip p, -.warning p, -.caution p, -.note p { - margin-top: 0.5em; - margin-bottom: 0.5em; - padding-right: 1em; - text-align: left; -} - -.acronym { - text-transform: uppercase; -} - -b.keycap, -.keycap { - padding: 0.09em 0.3em; - margin: 0em; -} - -.itemizedlist li { - clear: none; -} - -.filename { - font-size: medium; - font-family: Courier, monospace; -} - - -div.navheader, div.heading{ - position: absolute; - left: 0em; - top: 0em; - width: 100%; - background-color: #cdf; - width: 100%; -} - -div.navfooter, div.footing{ - position: fixed; - left: 0em; - bottom: 0em; - background-color: #eee; - width: 100%; -} - - -div.navheader td, -div.navfooter td { - font-size: 66%; -} - -div.navheader table th { - /*font-family: Georgia, Times, serif;*/ - /*font-size: x-large;*/ - font-size: 80%; -} - -div.navheader table { - border-left: 0em; - border-right: 0em; - border-top: 0em; - width: 100%; -} - -div.navfooter table { - border-left: 0em; - border-right: 0em; - border-bottom: 0em; - width: 100%; -} - -div.navheader table td a, -div.navfooter table td a { - color: #777; - text-decoration: none; -} - -/* normal text in the footer */ -div.navfooter table td { - color: black; -} - -div.navheader table td a:visited, -div.navfooter table td a:visited { - color: #444; -} - - -/* links in header and footer */ -div.navheader table td a:hover, -div.navfooter table td a:hover { - text-decoration: underline; - background-color: transparent; - color: #33a; -} - -div.navheader hr, -div.navfooter hr { - display: none; -} - - -.qandaset tr.question td p { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; -} - -.qandaset tr.answer td p { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; -} -.answer td { - padding-bottom: 1.5em; -} - -.emphasis { - font-weight: bold; -} - - - /************* / - / decorations / -/ *************/ - -.titlepage { -} - -.part .title { -} - -.subtitle { - border: none; -} - -/* -h1 { - border: none; -} - -h2 { - border-top: solid 0.2em; - border-bottom: solid 0.06em; -} - -h3 { - border-top: 0em; - border-bottom: solid 0.06em; -} - -h4 { - border: 0em; - border-bottom: solid 0.06em; -} - -h5 { - border: 0em; -} -*/ - -.programlisting { - border: solid 1px; -} - -div.figure, -div.table, -div.informalfigure, -div.informaltable, -div.informalexample, -div.example { - border: 1px solid; -} - - - -.tip, -.warning, -.caution, -.note { - border: 1px solid; -} - -.tip table th, -.warning table th, -.caution table th, -.note table th { - border-bottom: 1px solid; -} - -.question td { - border-top: 1px solid black; -} - -.answer { -} - - -b.keycap, -.keycap { - border: 1px solid; -} - - -div.navheader, div.heading{ - border-bottom: 1px solid; -} - - -div.navfooter, div.footing{ - border-top: 1px solid; -} - - /********* / - / colors / -/ *********/ - -body { - color: #333; - background: white; -} - -a { - background: transparent; -} - -a:hover { - background-color: #dedede; -} - - -h1, -h2, -h3, -h4, -h5, -h6, -h7, -h8 { - background-color: transparent; -} - -hr { - border-color: #aaa; -} - - -.tip, .warning, .caution, .note { - border-color: #fff; -} - - -.tip table th, -.warning table th, -.caution table th, -.note table th { - border-bottom-color: #fff; -} - - -.warning { - background-color: #f0f0f2; -} - -.caution { - background-color: #f0f0f2; -} - -.tip { - background-color: #f0f0f2; -} - -.note { - background-color: #f0f0f2; -} - -.glossary dl dt, -.variablelist dl dt, -.variablelist dl dt span.term { - color: #044; -} - -div.figure, -div.table, -div.example, -div.informalfigure, -div.informaltable, -div.informalexample { - border-color: #aaa; -} - -pre.programlisting { - color: black; - background-color: #fff; - border-color: #aaa; - border-width: 2px; -} - -.guimenu, -.guilabel, -.guimenuitem { - background-color: #eee; -} - - -b.keycap, -.keycap { - background-color: #eee; - border-color: #999; -} - - -div.navheader { - border-color: black; -} - - -div.navfooter { - border-color: black; -} - -.writernotes { - color: red; -} - - - /*********** / - / graphics / -/ ***********/ - -/* -body { - background-image: url("images/body_bg.jpg"); - background-attachment: fixed; -} - -.navheader, -.note, -.tip { - background-image: url("images/note_bg.jpg"); - background-attachment: fixed; -} - -.warning, -.caution { - background-image: url("images/warning_bg.jpg"); - background-attachment: fixed; -} - -.figure, -.informalfigure, -.example, -.informalexample, -.table, -.informaltable { - background-image: url("images/figure_bg.jpg"); - background-attachment: fixed; -} - -*/ -h1, -h2, -h3, -h4, -h5, -h6, -h7{ -} - -/* -Example of how to stick an image as part of the title. - -div.article .titlepage .title -{ - background-image: url("figures/white-on-black.png"); - background-position: center; - background-repeat: repeat-x; -} -*/ - -div.preface .titlepage .title, -div.colophon .title, -div.chapter .titlepage .title, -div.article .titlepage .title -{ -} - -div.section div.section .titlepage .title, -div.sect2 .titlepage .title { - background: none; -} - - -h1.title { - background-color: transparent; - background-image: url("figures/test-title.png"); - background-repeat: no-repeat; - height: 256px; - text-indent: -9000px; - overflow:hidden; -} - -h2.subtitle { - background-color: transparent; - text-indent: -9000px; - overflow:hidden; - width: 0px; - display: none; -} - - /*************************************** / - / pippin.gimp.org specific alterations / -/ ***************************************/ - -/* -div.heading, div.navheader { - color: #777; - font-size: 80%; - padding: 0; - margin: 0; - text-align: left; - position: absolute; - top: 0px; - left: 0px; - width: 100%; - height: 50px; - background: url('/gfx/heading_bg.png') transparent; - background-repeat: repeat-x; - background-attachment: fixed; - border: none; -} - -div.heading a { - color: #444; -} - -div.footing, div.navfooter { - border: none; - color: #ddd; - font-size: 80%; - text-align:right; - - width: 100%; - padding-top: 10px; - position: absolute; - bottom: 0px; - left: 0px; - - background: url('/gfx/footing_bg.png') transparent; -} -*/ - - - - /****************** / - / nasty ie tweaks / -/ ******************/ - -/* -div.heading, div.navheader { - width:expression(document.body.clientWidth + "px"); -} - -div.footing, div.navfooter { - width:expression(document.body.clientWidth + "px"); - margin-left:expression("-5em"); -} -body { - padding:expression("4em 5em 0em 5em"); -} -*/ - - /**************************************** / - / mozilla vendor specific css extensions / -/ ****************************************/ -/* -div.navfooter, div.footing{ - -moz-opacity: 0.8em; -} - -div.figure, -div.table, -div.informalfigure, -div.informaltable, -div.informalexample, -div.example, -.tip, -.warning, -.caution, -.note { - -moz-border-radius: 0.5em; -} - -b.keycap, -.keycap { - -moz-border-radius: 0.3em; -} -*/ - -table tr td table tr td { - display: none; -} - - -hr { - display: none; -} - -table { - border: 0em; -} - - .photo { - float: right; - margin-left: 1.5em; - margin-bottom: 1.5em; - margin-top: 0em; - max-width: 17em; - border: 1px solid gray; - padding: 3px; - background: white; -} - .seperator { - padding-top: 2em; - clear: both; - } - - #validators { - margin-top: 5em; - text-align: right; - color: #777; - } - @media print { - body { - font-size: 8pt; - } - .noprint { - display: none; - } - } - - -.tip, -.note { - background: #f0f0f2; - color: #333; - padding: 20px; - margin: 20px; -} - -.tip h3, -.note h3 { - padding: 0em; - margin: 0em; - font-size: 2em; - font-weight: bold; - color: #333; -} - -.tip a, -.note a { - color: #333; - text-decoration: underline; -} - -.footnote { - font-size: small; - color: #333; -} - -/* Changes the announcement text */ -.tip h3, -.warning h3, -.caution h3, -.note h3 { - font-size:large; - color: #00557D; -} diff --git a/documentation/test-manual/test-manual-test-process.rst b/documentation/test-manual/test-manual-test-process.rst deleted file mode 100644 index 96e71bf314..0000000000 --- a/documentation/test-manual/test-manual-test-process.rst +++ /dev/null @@ -1,103 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.0-UK - -*********************************** -Project Testing and Release Process -*********************************** - -.. _test-daily-devel: - -Day to Day Development -====================== - -This section details how the project tests changes, through automation -on the Autobuilder or with the assistance of QA teams, through to making -releases. - -The project aims to test changes against our test matrix before those -changes are merged into the master branch. As such, changes are queued -up in batches either in the ``master-next`` branch in the main trees, or -in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton -helps review and test patches and this is his testing tree). - -We have two broad categories of test builds, including "full" and -"quick". On the Autobuilder, these can be seen as "a-quick" and -"a-full", simply for ease of sorting in the UI. Use our Autobuilder -console view to see where me manage most test-related items, available -at: :yocto_ab:`/typhoon/#/console`. - -Builds are triggered manually when the test branches are ready. The -builds are monitored by the SWAT team. For additional information, see -:yocto_wiki:`/wiki/Yocto_Build_Failure_Swat_Team`. -If successful, the changes would usually be merged to the ``master`` -branch. If not successful, someone would respond to the changes on the -mailing list explaining that there was a failure in testing. The choice -of quick or full would depend on the type of changes and the speed with -which the result was required. - -The Autobuilder does build the ``master`` branch once daily for several -reasons, in particular, to ensure the current ``master`` branch does -build, but also to keep ``yocto-testresults`` -(:yocto_git:`/cgit.cgi/yocto-testresults/`), -buildhistory -(:yocto_git:`/cgit.cgi/poky-buildhistory/`), and -our sstate up to date. On the weekend, there is a master-next build -instead to ensure the test results are updated for the less frequently -run targets. - -Performance builds (buildperf-\* targets in the console) are triggered -separately every six hours and automatically push their results to the -buildstats repository at: -:yocto_git:`/cgit.cgi/yocto-buildstats/`. - -The 'quick' targets have been selected to be the ones which catch the -most failures or give the most valuable data. We run 'fast' ptests in -this case for example but not the ones which take a long time. The quick -target doesn't include \*-lsb builds for all architectures, some world -builds and doesn't trigger performance tests or ltp testing. The full -build includes all these things and is slower but more comprehensive. - -Release Builds -============== - -The project typically has two major releases a year with a six month -cadence in April and October. Between these there would be a number of -milestone releases (usually four) with the final one being stablization -only along with point releases of our stable branches. - -The build and release process for these project releases is similar to -that in `Day to Day Development <#test-daily-devel>`__, in that the -a-full target of the Autobuilder is used but in addition the form is -configured to generate and publish artefacts and the milestone number, -version, release candidate number and other information is entered. The -box to "generate an email to QA"is also checked. - -When the build completes, an email is sent out using the send-qa-email -script in the ``yocto-autobuilder-helper`` repository to the list of -people configured for that release. Release builds are placed into a -directory in https://autobuilder.yocto.io/pub/releases on the -Autobuilder which is included in the email. The process from here is -more manual and control is effectively passed to release engineering. -The next steps include: - -- QA teams respond to the email saying which tests they plan to run and - when the results will be available. - -- QA teams run their tests and share their results in the yocto- - testresults-contrib repository, along with a summary of their - findings. - -- Release engineering prepare the release as per their process. - -- Test results from the QA teams are included into the release in - separate directories and also uploaded to the yocto-testresults - repository alongside the other test results for the given revision. - -- The QA report in the final release is regenerated using resulttool to - include the new test results and the test summaries from the teams - (as headers to the generated report). - -- The release is checked against the release checklist and release - readiness criteria. - -- A final decision on whether to release is made by the YP TSC who have - final oversight on release readiness. diff --git a/documentation/test-manual/test-manual-test-process.xml b/documentation/test-manual/test-manual-test-process.xml deleted file mode 100644 index 6e2157c621..0000000000 --- a/documentation/test-manual/test-manual-test-process.xml +++ /dev/null @@ -1,110 +0,0 @@ - %poky; ] > - - - - -Project Testing and Release Process -
- Day to Day Development - - This section details how the project tests changes, through automation on the - Autobuilder or with the assistance of QA teams, through to making releases. - - The project aims to test changes against our test matrix before those changes are - merged into the master branch. As such, changes are queued up in batches either in the - master-next branch in the main trees, or in user trees such as - ross/mut in poky-contrib (Ross Burton - helps review and test patches and this is his testing tree). - We have two broad categories of test builds, including "full" and "quick". On the - Autobuilder, these can be seen as "a-quick" and "a-full", simply for ease of sorting in - the UI. Use our Autobuilder console view to see where me manage most test-related items, - available at: https://autobuilder.yoctoproject.org/typhoon/#/console. - Builds are triggered manually when the test branches are ready. The builds are - monitored by the SWAT team. For additional information, see https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team. If - successful, the changes would usually be merged to the master - branch. If not successful, someone would respond to the changes on the mailing list - explaining that there was a failure in testing. The choice of quick or full would depend - on the type of changes and the speed with which the result was required. - The Autobuilder does build the master branch once daily for - several reasons, in particular, to ensure the current master branch - does build, but also to keep yocto-testresults (http://git.yoctoproject.org/cgit.cgi/yocto-testresults/), buildhistory - (http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/), - and our sstate up to date. On the weekend, there is a master-next build instead to - ensure the test results are updated for the less frequently run targets. - Performance builds (buildperf-* targets in the console) are triggered separately every - six hours and automatically push their results to the buildstats repository at: http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/. - The 'quick' targets have been selected to be the ones which catch the most failures or - give the most valuable data. We run 'fast' ptests in this case for example but not the - ones which take a long time. The quick target doesn't include *-lsb builds for all - architectures, some world builds and doesn't trigger performance tests or ltp testing. - The full build includes all these things and is slower but more comprehensive. -
- -
- Release Builds - - The project typically has two major releases a year with a six month cadence in April - and October. Between these there would be a number of milestone releases (usually four) - with the final one being stablization only along with point releases of our stable - branches. - The build and release process for these project releases is similar to that in Day to Day Development, in that the a-full target - of the Autobuilder is used but in addition the form is configured to generate and - publish artefacts and the milestone number, version, release candidate number and other - information is entered. The box to "generate an email to QA"is also checked. - When the build completes, an email is sent out using the send-qa-email script in the - yocto-autobuilder-helper repository to the list of people - configured for that release. Release builds are placed into a directory in https://autobuilder.yocto.io/pub/releases on the Autobuilder which - is included in the email. The process from here is more manual and control is - effectively passed to release engineering. The next steps include: - - QA teams respond to the email saying which tests they plan to run and when - the results will be available. - - - QA teams run their tests and share their results in the yocto- - testresults-contrib repository, along with a summary of their findings. - - - - Release engineering prepare the release as per their process. - - - Test results from the QA teams are included into the release in separate - directories and also uploaded to the yocto-testresults repository alongside - the other test results for the given revision. - - - The QA report in the final release is regenerated using resulttool to - include the new test results and the test summaries from the teams (as - headers to the generated report). - - - The release is checked against the release checklist and release readiness - criteria. - - - A final decision on whether to release is made by the YP TSC who have - final oversight on release readiness. - - -
- - - - - - - - - - diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst deleted file mode 100644 index 2fcae5000e..0000000000 --- a/documentation/test-manual/test-manual-understand-autobuilder.rst +++ /dev/null @@ -1,305 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.0-UK - -******************************************* -Understanding the Yocto Project Autobuilder -******************************************* - -Execution Flow within the Autobuilder -===================================== - -The "a-full" and "a-quick" targets are the usual entry points into the -Autobuilder and it makes sense to follow the process through the system -starting there. This is best visualised from the Autobuilder Console -view (:yocto_ab:`/typhoon/#/console`). - -Each item along the top of that view represents some "target build" and -these targets are all run in parallel. The 'full' build will trigger the -majority of them, the "quick" build will trigger some subset of them. -The Autobuilder effectively runs whichever configuration is defined for -each of those targets on a seperate buildbot worker. To understand the -configuration, you need to look at the entry on ``config.json`` file -within the ``yocto-autobuilder-helper`` repository. The targets are -defined in the ‘overrides' section, a quick example could be qemux86-64 -which looks like:: - - "qemux86-64" : { - "MACHINE" : "qemux86-64", - "TEMPLATE" : "arch-qemu", - "step1" : { - "extravars" : [ - "IMAGE_FSTYPES_append = ' wic wic.bmap'" - ] - } - }, - -And to expand that, you need the "arch-qemu" entry from -the "templates" section, which looks like:: - - "arch-qemu" : { - "BUILDINFO" : true, - "BUILDHISTORY" : true, - "step1" : { - "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", - "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" - }, - "step2" : { - "SDKMACHINE" : "x86_64", - "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", - "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" - }, - "step3" : { - "BUILDHISTORY" : false, - "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], - "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] - } - }, - -Combining these two entries you can see that "qemux86-64" is a three step build where the -``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for -``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step -1 an extra variable is added to the ``auto.conf`` file to enable wic -image generation. - -While not every detail of this is covered here, you can see how the -template mechanism allows quite complex configurations to be built up -yet allows duplication and repetition to be kept to a minimum. - -The different build targets are designed to allow for parallelisation, -so different machines are usually built in parallel, operations using -the same machine and metadata are built sequentially, with the aim of -trying to optimise build efficiency as much as possible. - -The ``config.json`` file is processed by the scripts in the Helper -repository in the ``scripts`` directory. The following section details -how this works. - -.. _test-autobuilder-target-exec-overview: - -Autobuilder Target Execution Overview -===================================== - -For each given target in a build, the Autobuilder executes several -steps. These are configured in ``yocto-autobuilder2/builders.py`` and -roughly consist of: - -#. *Run clobberdir*. - - This cleans out any previous build. Old builds are left around to - allow easier debugging of failed builds. For additional information, - see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`. - -#. *Obtain yocto-autobuilder-helper* - - This step clones the ``yocto-autobuilder-helper`` git repository. - This is necessary to prevent the requirement to maintain all the - release or project-specific code within Buildbot. The branch chosen - matches the release being built so we can support older releases and - still make changes in newer ones. - -#. *Write layerinfo.json* - - This transfers data in the Buildbot UI when the build was configured - to the Helper. - -#. *Call scripts/shared-repo-unpack* - - This is a call into the Helper scripts to set up a checkout of all - the pieces this build might need. It might clone the BitBake - repository and the OpenEmbedded-Core repository. It may clone the - Poky repository, as well as additional layers. It will use the data - from the ``layerinfo.json`` file to help understand the - configuration. It will also use a local cache of repositories to - speed up the clone checkouts. For additional information, see - :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`. - - This step has two possible modes of operation. If the build is part - of a parent build, its possible that all the repositories needed may - already be available, ready in a pre-prepared directory. An "a-quick" - or "a-full" build would prepare this before starting the other - sub-target builds. This is done for two reasons: - - - the upstream may change during a build, for example, from a forced - push and this ensures we have matching content for the whole build - - - if 15 Workers all tried to pull the same data from the same repos, - we can hit resource limits on upstream servers as they can think - they are under some kind of network attack - - This pre-prepared directory is shared among the Workers over NFS. If - the build is an individual build and there is no "shared" directory - available, it would clone from the cache and the upstreams as - necessary. This is considered the fallback mode. - -#. *Call scripts/run-config* - - This is another call into the Helper scripts where its expected that - the main functionality of this target will be executed. - -.. _test-autobuilder-tech: - -Autobuilder Technology -====================== - -The Autobuilder has Yocto Project-specific functionality to allow builds -to operate with increased efficiency and speed. - -.. _test-clobberdir: - -clobberdir ----------- - -When deleting files, the Autobuilder uses ``clobberdir``, which is a -special script that moves files to a special location, rather than -deleting them. Files in this location are deleted by an ``rm`` command, -which is run under ``ionice -c 3``. For example, the deletion only -happens when there is idle IO capacity on the Worker. The Autobuilder -Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. - -.. _test-autobuilder-clone-cache: - -Autobuilder Clone Cache ------------------------ - -Cloning repositories from scratch each time they are required was slow -on the Autobuilder. We therefore have a stash of commonly used -repositories pre-cloned on the Workers. Data is fetched from these -during clones first, then "topped up" with later revisions from any -upstream when necesary. The cache is maintained by the Autobuilder -Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. - -.. _test-autobuilder-worker-janitor: - -Autobuilder Worker Janitor --------------------------- - -This is a process running on each Worker that performs two basic -operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and -maintainenance of a cache of cloned repositories to improve the speed -the system can checkout repositories. - -.. _test-shared-dl-dir: - -Shared DL_DIR -------------- - -The Workers are all connected over NFS which allows DL_DIR to be shared -between them. This reduces network accesses from the system and allows -the build to be sped up. Usage of the directory within the build system -is designed to be able to be shared over NFS. - -.. _test-shared-sstate-cache: - -Shared SSTATE_DIR ------------------ - -The Workers are all connected over NFS which allows the ``sstate`` -directory to be shared between them. This means once a Worker has built -an artifact, all the others can benefit from it. Usage of the directory -within the directory is designed for sharing over NFS. - -.. _test-resulttool: - -Resulttool ----------- - -All of the different tests run as part of the build generate output into -``testresults.json`` files. This allows us to determine which tests ran -in a given build and their status. Additional information, such as -failure logs or the time taken to run the tests, may also be included. - -Resulttool is part of OpenEmbedded-Core and is used to manipulate these -json results files. It has the ability to merge files together, display -reports of the test results and compare different result files. - -For details, see :yocto_wiki:`/wiki/Resulttool`. - -.. _test-run-config-tgt-execution: - -run-config Target Execution -=========================== - -The ``scripts/run-config`` execution is where most of the work within -the Autobuilder happens. It runs through a number of steps; the first -are general setup steps that are run once and include: - -#. Set up any ``buildtools-tarball`` if configured. - -#. Call "buildhistory-init" if buildhistory is configured. - -For each step that is configured in ``config.json``, it will perform the -following: - -#. Add any layers that are specified using the - ``bitbake-layers add-layer`` command (logging as stepXa) - -#. Call the ``scripts/setup-config`` script to generate the necessary - ``auto.conf`` configuration file for the build - -#. Run the ``bitbake BBTARGETS`` command (logging as stepXb) - -#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) - -#. Run the ``EXTRACMDS`` command, which are run within the BitBake build - environment (logging as stepXd) - -#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the - BitBake build environment (logging as stepXd) - -#. Remove any layers added in step - 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa) - -Once the execution steps above complete, ``run-config`` executes a set -of post-build steps, including: - -#. Call ``scripts/publish-artifacts`` to collect any output which is to - be saved from the build. - -#. Call ``scripts/collect-results`` to collect any test results to be - saved from the build. - -#. Call ``scripts/upload-error-reports`` to send any error reports - generated to the remote server. - -#. Cleanup the build directory using - :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful, - else rename it to "build-renamed" for potential future debugging. - -.. _test-deploying-yp-autobuilder: - -Deploying Yocto Autobuilder -=========================== - -The most up to date information about how to setup and deploy your own -Autbuilder can be found in README.md in the ``yocto-autobuilder2`` -repository. - -We hope that people can use the ``yocto-autobuilder2`` code directly but -it is inevitable that users will end up needing to heavily customise the -``yocto-autobuilder-helper`` repository, particularly the -``config.json`` file as they will want to define their own test matrix. - -The Autobuilder supports wo customization options: - -- variable substitution - -- overlaying configuration files - -The standard ``config.json`` minimally attempts to allow substitution of -the paths. The Helper script repository includes a -``local-example.json`` file to show how you could override these from a -separate configuration file. Pass the following into the environment of -the Autobuilder:: - - $ ABHELPER_JSON="config.json local-example.json" - -As another example, you could also pass the following into the -environment:: - - $ ABHELPER_JSON="config.json /some/location/local.json" - -One issue users often run into is validation of the ``config.json`` files. A -tip for minimizing issues from invalid json files is to use a Git -``pre-commit-hook.sh`` script to verify the JSON file before committing -it. Create a symbolic link as follows:: - - $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit diff --git a/documentation/test-manual/test-manual-understand-autobuilder.xml b/documentation/test-manual/test-manual-understand-autobuilder.xml deleted file mode 100644 index 8600367be7..0000000000 --- a/documentation/test-manual/test-manual-understand-autobuilder.xml +++ /dev/null @@ -1,314 +0,0 @@ - %poky; ] > - - - - -Understanding the Yocto Project Autobuilder -
- Execution Flow within the Autobuilder - The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and - it makes sense to follow the process through the system starting there. This is best - visualised from the Autobuilder Console view (https://autobuilder.yoctoproject.org/typhoon/#/console). - Each item along the top of that view represents some "target build" and these targets - are all run in parallel. The 'full' build will trigger the majority of them, the "quick" - build will trigger some subset of them. The Autobuilder effectively runs whichever - configuration is defined for each of those targets on a seperate buildbot worker. To - understand the configuration, you need to look at the entry on - config.json file within the - yocto-autobuilder-helper repository. The targets are defined in - the ‘overrides' section, a quick example could be qemux86-64 which looks - like: - "qemux86-64" : { - "MACHINE" : "qemux86-64", - "TEMPLATE" : "arch-qemu", - "step1" : { - "extravars" : [ - "IMAGE_FSTYPES_append = ' wic wic.bmap'" - ] - } - }, - And - to expand that, you need the "arch-qemu" entry from the "templates" section, which looks - like: - "arch-qemu" : { - "BUILDINFO" : true, - "BUILDHISTORY" : true, - "step1" : { - "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", - "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" - }, - "step2" : { - "SDKMACHINE" : "x86_64", - "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", - "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" - }, - "step3" : { - "BUILDHISTORY" : false, - "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], - "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] - } - }, - Combining - these two entries you can see that "qemux86-64" is a three step build where the - bitbake BBTARGETS would be run, then bitbake - SANITYTARGETS for each step; all for - MACHINE="qemx86-64" but with differing SDKMACHINE settings. In - step 1 an extra variable is added to the auto.conf file to enable - wic image generation. - While not every detail of this is covered here, you can see how the templating - mechanism allows quite complex configurations to be built up yet allows duplication and - repetition to be kept to a minimum. - The different build targets are designed to allow for parallelisation, so different - machines are usually built in parallel, operations using the same machine and metadata - are built sequentially, with the aim of trying to optimise build efficiency as much as - possible. - The config.json file is processed by the scripts in the Helper - repository in the scripts directory. The following section details - how this works. -
- -
- Autobuilder Target Execution Overview - - For each given target in a build, the Autobuilder executes several steps. These are - configured in yocto-autobuilder2/builders.py and roughly consist - of: - - Run clobberdir - This cleans out any previous build. Old builds are left around to allow - easier debugging of failed builds. For additional information, see clobberdir. - - - Obtain yocto-autobuilder-helper - This step clones the yocto-autobuilder-helper git - repository. This is necessary to prevent the requirement to maintain all the - release or project-specific code within Buildbot. The branch chosen matches - the release being built so we can support older releases and still make - changes in newer ones. - - - Write layerinfo.json - This transfers data in the Buildbot UI when the build was configured to - the Helper. - - - Call scripts/shared-repo-unpack - This is a call into the Helper scripts to set up a checkout of all the - pieces this build might need. It might clone the BitBake repository and the - OpenEmbedded-Core repository. It may clone the Poky repository, as well as - additional layers. It will use the data from the - layerinfo.json file to help understand the - configuration. It will also use a local cache of repositories to speed up - the clone checkouts. For additional information, see Autobuilder Clone - Cache. - This step has two possible modes of operation. If the build is part of a - parent build, its possible that all the repositories needed may already be - available, ready in a pre-prepared directory. An "a-quick" or "a-full" build - would prepare this before starting the other sub-target builds. This is done - for two reasons: - - the upstream may change during a build, for example, from a - forced push and this ensures we have matching content for the - whole build - - - if 15 Workers all tried to pull the same data from the same - repos, we can hit resource limits on upstream servers as they - can think they are under some kind of network attack - - This pre-prepared directory is shared among the Workers over - NFS. If the build is an individual build and there is no "shared" directory - available, it would clone from the cache and the upstreams as necessary. - This is considered the fallback mode. - - - Call scripts/run-config - This is another call into the Helper scripts where its expected that the - main functionality of this target will be executed. - - -
-
- Autobuilder Technology - The Autobuilder has Yocto Project-specific functionality to allow builds to operate - with increased efficiency and speed. -
- clobberdir - When deleting files, the Autobuilder uses clobberdir, which - is a special script that moves files to a special location, rather than deleting - them. Files in this location are deleted by an rm command, - which is run under ionice -c 3. For example, the deletion only - happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor - runs this deletion. See Autobuilder - Worker Janitor. -
-
- Autobuilder Clone Cache - Cloning repositories from scratch each time they are required was slow on the - Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on - the Workers. Data is fetched from these during clones first, then "topped up" with - later revisions from any upstream when necesary. The cache is maintained by the - Autobuilder Worker Janitor. See Autobuilder Worker Janitor. -
-
- Autobuilder Worker Janitor - This is a process running on each Worker that performs two basic operations, - including background file deletion at IO idle (see Target Execution: clobberdir) and - maintainenance of a cache of cloned repositories to improve the speed the system can - checkout repositories. -
-
- Shared DL_DIR - The Workers are all connected over NFS which allows DL_DIR to be shared between - them. This reduces network accesses from the system and allows the build to be sped - up. Usage of the directory within the build system is designed to be able to be - shared over NFS. -
-
- Shared SSTATE_DIR - The Workers are all connected over NFS which allows the - sstate directory to be shared between them. This means once - a Worker has built an artefact, all the others can benefit from it. Usage of the - directory within the directory is designed for sharing over NFS. -
-
- Resulttool - All of the different tests run as part of the build generate output into - testresults.json files. This allows us to determine which - tests ran in a given build and their status. Additional information, such as failure - logs or the time taken to run the tests, may also be included. - Resulttool is part of OpenEmbedded-Core and is used to manipulate these json - results files. It has the ability to merge files together, display reports of the - test results and compare different result files. - For details, see https://wiki.yoctoproject.org/wiki/Resulttool. -
-
-
- run-config Target Execution - The scripts/run-config execution is where most of the work within - the Autobuilder happens. It runs through a number of steps; the first are general setup - steps that are run once and include: - - Set up any buildtools-tarball if configured. - - - Call "buildhistory-init" if buildhistory is configured. - - - For each step that is configured in config.json, it will perform - the following: - - ## WRITER's question: What does "logging in as stepXa" and others refer to - below? ## - - - Add any layers that are specified using the - bitbake-layers add-layer command (logging as - stepXa) - - - Call the scripts/setup-config script to - generate the necessary auto.conf configuration file for - the build - - - Run the bitbake BBTARGETS command (logging - as stepXb) - - - Run the bitbake SANITYTARGETS command - (logging as stepXc) - - - Run the EXTRACMDS command, which are run - within the BitBake build environment (logging as stepXd) - - - Run the EXTRAPLAINCMDS command(s), which - are run outside the BitBake build environment (logging as stepXd) - - - Remove any layers added in step 1 using the - bitbake-layers remove-layer command (logging as - stepXa) - - - - Once the execution steps above complete, run-config executes a - set of post-build steps, including: - - Call scripts/publish-artifacts to collect - any output which is to be saved from the build. - - - Call scripts/collect-results to collect any - test results to be saved from the build. - - - Call scripts/upload-error-reports to send - any error reports generated to the remote server. - - - Cleanup the build directory using clobberdir if the - build was successful, else rename it to "build-renamed" for potential future - debugging. - - -
-
- Deploying Yocto Autobuilder - The most up to date information about how to setup and deploy your own Autbuilder can - be found in README.md in the yocto-autobuilder2 repository. - We hope that people can use the yocto-autobuilder2 code directly - but it is inevitable that users will end up needing to heavily customise the - yocto-autobuilder-helper repository, particularly the - config.json file as they will want to define their own test - matrix. - The Autobuilder supports wo customization options: - - variable substitution - - - overlaying configuration files - - The standard config.json minimally attempts to allow - substitution of the paths. The Helper script repository includes a - local-example.json file to show how you could override these - from a separate configuration file. Pass the following into the environment of the - Autobuilder: - $ ABHELPER_JSON="config.json local-example.json" - As - another example, you could also pass the following into the - environment: - $ ABHELPER_JSON="config.json /some/location/local.json" - One - issue users often run into is validation of the config.json files. - A tip for minimizing issues from invalid json files is to use a Git - pre-commit-hook.sh script to verify the JSON file before - committing it. Create a symbolic link as - follows: - $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit - -
- - - - - - - - - - diff --git a/documentation/test-manual/test-manual.rst b/documentation/test-manual/test-manual.rst deleted file mode 100644 index bd5b1b0968..0000000000 --- a/documentation/test-manual/test-manual.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.0-UK - -===================================== -Yocto Project Test Environment Manual -===================================== - -| - -.. toctree:: - :caption: Table of Contents - :numbered: - - test-manual-intro - test-manual-test-process - test-manual-understand-autobuilder - history - -.. include:: /boilerplate.rst diff --git a/documentation/test-manual/test-manual.xml b/documentation/test-manual/test-manual.xml deleted file mode 100755 index d454566c5f..0000000000 --- a/documentation/test-manual/test-manual.xml +++ /dev/null @@ -1,104 +0,0 @@ - %poky; ] > - - - - - - - - - - - - - Yocto Project Test Environment Manual - - - - - - &ORGNAME; - - &ORGEMAIL; - - - - - - 3.1.1 - TBD - DRAFT - Work-in-Progress - posted June 16, 2020 - - - - - ©RIGHT_YEAR; - Linux Foundation - - - - - Permission is granted to copy, distribute and/or modify this document under - the terms of the - Creative Commons Attribution-Share Alike 2.0 UK: England & Wales as published by - Creative Commons. - - Manual Notes - - - This version of the - Yocto Project Test Environment Manual - is for the &YOCTO_DOC_VERSION; release of the - Yocto Project. - To be sure you have the latest version of the manual - for this release, go to the - Yocto Project documentation page - and select the manual from that site. - Manuals from the site are more up-to-date than manuals - derived from the Yocto Project released TAR files. - - - If you located this manual through a web search, the - version of the manual might not be the one you want - (e.g. the search might have returned a manual much - older than the Yocto Project version with which you - are working). - You can see all Yocto Project major releases by - visiting the - Releases - page. - If you need a version of this manual for a different - Yocto Project release, visit the - Yocto Project documentation page - and select the manual set by using the - "ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARCHIVE" - pull-down menus. - - - To report any inaccuracies or problems with this - manual, send an email to the Yocto Project - discussion group at - yocto@yoctoproject.com or log into - the freenode #yocto channel. - - - - - - - - - - - - - -- cgit v1.2.3-54-g00ecf