summaryrefslogtreecommitdiffstats
path: root/documentation
diff options
context:
space:
mode:
authorNicolas Dechesne <nicolas.dechesne@linaro.org>2020-11-13 01:25:13 +0100
committerRichard Purdie <richard.purdie@linuxfoundation.org>2021-01-04 10:55:00 +0000
commit6f248db9a26e963ae8379e40371ed478ed8e3d3f (patch)
treec4d166a1c872835cdedd3896a814ecdd316eed02 /documentation
parent8408c03763f3565ad08da271ea126ca2b001f9bd (diff)
downloadpoky-6f248db9a26e963ae8379e40371ed478ed8e3d3f.tar.gz
sphinx: remove test-manual
This is a new manual in YP 3.2, let's remove it from the 3.1 Sphinx backport. (From yocto-docs rev: baf4410abad44b7f1fc6e4f1c445060b87c21c32) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation')
-rw-r--r--documentation/index.rst1
-rw-r--r--documentation/overview-manual/overview-manual-yp-intro.rst2
-rw-r--r--documentation/test-manual/figures/ab-test-cluster.pngbin18684 -> 0 bytes
-rw-r--r--documentation/test-manual/figures/test-manual-title.pngbin15382 -> 0 bytes
-rw-r--r--documentation/test-manual/history.rst16
-rw-r--r--documentation/test-manual/test-manual-customization.xsl29
-rw-r--r--documentation/test-manual/test-manual-intro.rst550
-rw-r--r--documentation/test-manual/test-manual-intro.xml624
-rw-r--r--documentation/test-manual/test-manual-style.css991
-rw-r--r--documentation/test-manual/test-manual-test-process.rst103
-rw-r--r--documentation/test-manual/test-manual-test-process.xml110
-rw-r--r--documentation/test-manual/test-manual-understand-autobuilder.rst305
-rw-r--r--documentation/test-manual/test-manual-understand-autobuilder.xml314
-rw-r--r--documentation/test-manual/test-manual.rst18
-rwxr-xr-xdocumentation/test-manual/test-manual.xml104
15 files changed, 1 insertions, 3166 deletions
diff --git a/documentation/index.rst b/documentation/index.rst
index 258ecb81a7..a1605ade91 100644
--- a/documentation/index.rst
+++ b/documentation/index.rst
@@ -33,7 +33,6 @@ Welcome to The Yocto Project's documentation!
33 Profile and Tracing Manual <profile-manual/profile-manual> 33 Profile and Tracing Manual <profile-manual/profile-manual>
34 Application Development and the Extensible SDK (eSDK) <sdk-manual/sdk-manual> 34 Application Development and the Extensible SDK (eSDK) <sdk-manual/sdk-manual>
35 Toaster Manual <toaster-manual/toaster-manual> 35 Toaster Manual <toaster-manual/toaster-manual>
36 Test Environment Manual <test-manual/test-manual>
37 Bitbake User Manual <https://docs.yoctoproject.org/bitbake> 36 Bitbake User Manual <https://docs.yoctoproject.org/bitbake>
38 37
39.. toctree:: 38.. toctree::
diff --git a/documentation/overview-manual/overview-manual-yp-intro.rst b/documentation/overview-manual/overview-manual-yp-intro.rst
index 5cdab7ca4a..2b3c3b07c7 100644
--- a/documentation/overview-manual/overview-manual-yp-intro.rst
+++ b/documentation/overview-manual/overview-manual-yp-intro.rst
@@ -401,7 +401,7 @@ activities using the Yocto Project:
401 benefit of the development community. 401 benefit of the development community.
402 402
403 You can learn more about the AutoBuilder used by the Yocto Project 403 You can learn more about the AutoBuilder used by the Yocto Project
404 Autobuilder :doc:`here <../test-manual/test-manual-understand-autobuilder>`. 404 Autobuilder `here <&YOCTO_AB_URL;>`__.
405 405
406- *Cross-Prelink:* Prelinking is the process of pre-computing the load 406- *Cross-Prelink:* Prelinking is the process of pre-computing the load
407 addresses and link tables generated by the dynamic linker as compared 407 addresses and link tables generated by the dynamic linker as compared
diff --git a/documentation/test-manual/figures/ab-test-cluster.png b/documentation/test-manual/figures/ab-test-cluster.png
deleted file mode 100644
index 6a6a7882b4..0000000000
--- a/documentation/test-manual/figures/ab-test-cluster.png
+++ /dev/null
Binary files differ
diff --git a/documentation/test-manual/figures/test-manual-title.png b/documentation/test-manual/figures/test-manual-title.png
deleted file mode 100644
index c709cb9d09..0000000000
--- a/documentation/test-manual/figures/test-manual-title.png
+++ /dev/null
Binary files differ
diff --git a/documentation/test-manual/history.rst b/documentation/test-manual/history.rst
deleted file mode 100644
index 76d43091a5..0000000000
--- a/documentation/test-manual/history.rst
+++ /dev/null
@@ -1,16 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-2.0-UK
2
3***********************
4Manual Revision History
5***********************
6
7.. list-table::
8 :widths: 10 15 40
9 :header-rows: 1
10
11 * - Revision
12 - Date
13 - Note
14 * - 3.2
15 - October 2020
16 - The initial document released with the Yocto Project 3.2 Release
diff --git a/documentation/test-manual/test-manual-customization.xsl b/documentation/test-manual/test-manual-customization.xsl
deleted file mode 100644
index 17bf57c2d1..0000000000
--- a/documentation/test-manual/test-manual-customization.xsl
+++ /dev/null
@@ -1,29 +0,0 @@
1<?xml version='1.0'?>
2<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
3
4<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
5
6 <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
7
8<!--
9
10 <xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
11
12 <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/xhtml/docbook.xsl" />
13
14-->
15
16 <xsl:include href="../template/permalinks.xsl"/>
17 <xsl:include href="../template/section.title.xsl"/>
18 <xsl:include href="../template/component.title.xsl"/>
19 <xsl:include href="../template/division.title.xsl"/>
20 <xsl:include href="../template/formal.object.heading.xsl"/>
21
22 <xsl:param name="html.stylesheet" select="'test-manual-style.css'" />
23 <xsl:param name="chapter.autolabel" select="1" />
24 <xsl:param name="appendix.autolabel" select="A" />
25 <xsl:param name="section.autolabel" select="1" />
26 <xsl:param name="section.label.includes.component.label" select="1" />
27 <xsl:param name="generate.id.attributes" select="1" />
28
29</xsl:stylesheet>
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst
deleted file mode 100644
index 53ad650b35..0000000000
--- a/documentation/test-manual/test-manual-intro.rst
+++ /dev/null
@@ -1,550 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-2.0-UK
2
3*****************************************
4The Yocto Project Test Environment Manual
5*****************************************
6
7.. _test-welcome:
8
9Welcome
10=======
11
12Welcome to the Yocto Project Test Environment Manual! This manual is a
13work in progress. The manual contains information about the testing
14environment used by the Yocto Project to make sure each major and minor
15release works as intended. All the project's testing infrastructure and
16processes are publicly visible and available so that the community can
17see what testing is being performed, how it's being done and the current
18status of the tests and the project at any given time. It is intended
19that Other organizations can leverage off the process and testing
20environment used by the Yocto Project to create their own automated,
21production test environment, building upon the foundations from the
22project core.
23
24Currently, the Yocto Project Test Environment Manual has no projected
25release date. This manual is a work-in-progress and is being initially
26loaded with information from the README files and notes from key
27engineers:
28
29- *yocto-autobuilder2:* This
30 :yocto_git:`README.md </cgit.cgi/yocto-autobuilder2/tree/README.md>`
31 is the main README which detials how to set up the Yocto Project
32 Autobuilder. The ``yocto-autobuilder2`` repository represents the
33 Yocto Project's console UI plugin to Buildbot and the configuration
34 necessary to configure Buildbot to perform the testing the project
35 requires.
36
37- *yocto-autobuilder-helper:* This :yocto_git:`README </cgit.cgi/yocto-autobuilder-helper/tree/README/>`
38 and repository contains Yocto Project Autobuilder Helper scripts and
39 configuration. The ``yocto-autobuilder-helper`` repository contains
40 the "glue" logic that defines which tests to run and how to run them.
41 As a result, it can be used by any Continuous Improvement (CI) system
42 to run builds, support getting the correct code revisions, configure
43 builds and layers, run builds, and collect results. The code is
44 independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
45 Jenkins, or others. This repository has a branch per release of the
46 project defining the tests to run on a per release basis.
47
48.. _test-yocto-project-autobuilder-overview:
49
50Yocto Project Autobuilder Overview
51==================================
52
53The Yocto Project Autobuilder collectively refers to the software,
54tools, scripts, and procedures used by the Yocto Project to test
55released software across supported hardware in an automated and regular
56fashion. Basically, during the development of a Yocto Project release,
57the Autobuilder tests if things work. The Autobuilder builds all test
58targets and runs all the tests.
59
60The Yocto Project uses now uses standard upstream
61`Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to
62drive its integration and testing. Buildbot Nine has a plug-in interface
63that the Yocto Project customizes using code from the
64``yocto-autobuilder2`` repository, adding its own console UI plugin. The
65resulting UI plug-in allows you to visualize builds in a way suited to
66the project's needs.
67
68A ``helper`` layer provides configuration and job management through
69scripts found in the ``yocto-autobuilder-helper`` repository. The
70``helper`` layer contains the bulk of the build configuration
71information and is release-specific, which makes it highly customizable
72on a per-project basis. The layer is CI system-agnostic and contains a
73number of Helper scripts that can generate build configurations from
74simple JSON files.
75
76.. note::
77
78 The project uses Buildbot for historical reasons but also because
79 many of the project developers have knowledge of python. It is
80 possible to use the outer layers from another Continuous Integration
81 (CI) system such as
82 `Jenkins <https://en.wikipedia.org/wiki/Jenkins_(software)>`__
83 instead of Buildbot.
84
85The following figure shows the Yocto Project Autobuilder stack with a
86topology that includes a controller and a cluster of workers:
87
88.. image:: figures/ab-test-cluster.png
89 :align: center
90
91.. _test-project-tests:
92
93Yocto Project Tests - Types of Testing Overview
94===============================================
95
96The Autobuilder tests different elements of the project by using
97thefollowing types of tests:
98
99- *Build Testing:* Tests whether specific configurations build by
100 varying :term:`MACHINE`,
101 :term:`DISTRO`, other configuration
102 options, and the specific target images being built (or world). Used
103 to trigger builds of all the different test configurations on the
104 Autobuilder. Builds usually cover many different targets for
105 different architectures, machines, and distributions, as well as
106 different configurations, such as different init systems. The
107 Autobuilder tests literally hundreds of configurations and targets.
108
109 - *Sanity Checks During the Build Process:* Tests initiated through
110 the :ref:`insane <ref-classes-insane>`
111 class. These checks ensure the output of the builds are correct.
112 For example, does the ELF architecture in the generated binaries
113 match the target system? ARM binaries would not work in a MIPS
114 system!
115
116- *Build Performance Testing:* Tests whether or not commonly used steps
117 during builds work efficiently and avoid regressions. Tests to time
118 commonly used usage scenarios are run through ``oe-build-perf-test``.
119 These tests are run on isolated machines so that the time
120 measurements of the tests are accurate and no other processes
121 interfere with the timing results. The project currently tests
122 performance on two different distributions, Fedora and Ubuntu, to
123 ensure we have no single point of failure and can ensure the
124 different distros work effectively.
125
126- *eSDK Testing:* Image tests initiated through the following command::
127
128 $ bitbake image -c testsdkext
129
130 The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
131
132- *Feature Testing:* Various scenario-based tests are run through the
133 :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
134 we support.
135
136- *Image Testing:* Image tests initiated through the following command::
137
138 $ bitbake image -c testimage
139
140 The tests utilize the :ref:`testimage* <ref-classes-testimage*>`
141 classes and the :ref:`ref-tasks-testimage` task.
142
143- *Layer Testing:* The Autobuilder has the possibility to test whether
144 specific layers work with the test of the system. The layers tested
145 may be selected by members of the project. Some key community layers
146 are also tested periodically.
147
148- *Package Testing:* A Package Test (ptest) runs tests against packages
149 built by the OpenEmbedded build system on the target machine. See the
150 :ref:`Testing Packages With
151 ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section
152 in the Yocto Project Development Tasks Manual and the
153 ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
154 information on Ptest.
155
156- *SDK Testing:* Image tests initiated through the following command::
157
158 $ bitbake image -c testsdk
159
160 The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and
161 the ``do_testsdk`` task.
162
163- *Unit Testing:* Unit tests on various components of the system run
164 through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and
165 :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`.
166
167- *Automatic Upgrade Helper:* This target tests whether new versions of
168 software are available and whether we can automatically upgrade to
169 those new versions. If so, this target emails the maintainers with a
170 patch to let them know this is possible.
171
172.. _test-test-mapping:
173
174How Tests Map to Areas of Code
175==============================
176
177Tests map into the codebase as follows:
178
179- *bitbake-selftest:*
180
181 These tests are self-contained and test BitBake as well as its APIs,
182 which include the fetchers. The tests are located in
183 ``bitbake/lib/*/tests``.
184
185 From within the BitBake repository, run the following::
186
187 $ bitbake-selftest
188
189 To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
190 variable when running "bitbake-selftest" as follows::
191
192 $ BB_SKIP_NETTEST=yes bitbake-selftest
193
194 The default output is quiet and just prints a summary of what was
195 run. To see more information, there is a verbose option::
196
197 $ bitbake-selftest -v
198
199 Use this option when you wish to skip tests that access the network,
200 which are mostly necessary to test the fetcher modules. To specify
201 individual test modules to run, append the test module name to the
202 "bitbake-selftest" command. For example, to specify the tests for the
203 bb.data.module, run::
204
205 $ bitbake-selftest bb.test.data.module
206
207 You can also specify individual tests by defining the full name and module
208 plus the class path of the test, for example::
209
210 $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
211
212 The tests are based on `Python
213 unittest <https://docs.python.org/3/library/unittest.html>`__.
214
215- *oe-selftest:*
216
217 - These tests use OE to test the workflows, which include testing
218 specific features, behaviors of tasks, and API unit tests.
219
220 - The tests can take advantage of parallelism through the "-j"
221 option, which can specify a number of threads to spread the tests
222 across. Note that all tests from a given class of tests will run
223 in the same thread. To parallelize large numbers of tests you can
224 split the class into multiple units.
225
226 - The tests are based on Python unittest.
227
228 - The code for the tests resides in
229 ``meta/lib/oeqa/selftest/cases/``.
230
231 - To run all the tests, enter the following command::
232
233 $ oe-selftest -a
234
235 - To run a specific test, use the following command form where
236 testname is the name of the specific test::
237
238 $ oe-selftest -r <testname>
239
240 For example, the following command would run the tinfoil
241 getVar API test::
242
243 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
244
245 It is also possible to run a set
246 of tests. For example the following command will run all of the
247 tinfoil tests::
248
249 $ oe-selftest -r tinfoil
250
251- *testimage:*
252
253 - These tests build an image, boot it, and run tests against the
254 image's content.
255
256 - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``.
257
258 - You need to set the :term:`IMAGE_CLASSES` variable as follows::
259
260 IMAGE_CLASSES += "testimage"
261
262 - Run the tests using the following command form::
263
264 $ bitbake image -c testimage
265
266- *testsdk:*
267
268 - These tests build an SDK, install it, and then run tests against
269 that SDK.
270
271 - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
272
273 - Run the test using the following command form::
274
275 $ bitbake image -c testsdk
276
277- *testsdk_ext:*
278
279 - These tests build an extended SDK (eSDK), install that eSDK, and
280 run tests against the eSDK.
281
282 - The code for these tests resides in ``meta/lib/oeqa/esdk``.
283
284 - To run the tests, use the following command form::
285
286 $ bitbake image -c testsdkext
287
288- *oe-build-perf-test:*
289
290 - These tests run through commonly used usage scenarios and measure
291 the performance times.
292
293 - The code for these tests resides in ``meta/lib/oeqa/buildperf``.
294
295 - To run the tests, use the following command form::
296
297 $ oe-build-perf-test <options>
298
299 The command takes a number of options,
300 such as where to place the test results. The Autobuilder Helper
301 Scripts include the ``build-perf-test-wrapper`` script with
302 examples of how to use the oe-build-perf-test from the command
303 line.
304
305 Use the ``oe-git-archive`` command to store test results into a
306 Git repository.
307
308 Use the ``oe-build-perf-report`` command to generate text reports
309 and HTML reports with graphs of the performance data. For
310 examples, see
311 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html`
312 and
313 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`.
314
315 - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``.
316
317Test Examples
318=============
319
320This section provides example tests for each of the tests listed in the
321:ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section.
322
323For oeqa tests, testcases for each area reside in the main test
324directory at ``meta/lib/oeqa/selftest/cases`` directory.
325
326For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/``
327directory.
328
329.. _bitbake-selftest-example:
330
331``bitbake-selftest``
332--------------------
333
334A simple test example from ``lib/bb/tests/data.py`` is::
335
336 class DataExpansions(unittest.TestCase):
337 def setUp(self):
338 self.d = bb.data.init()
339 self.d["foo"] = "value_of_foo"
340 self.d["bar"] = "value_of_bar"
341 self.d["value_of_foo"] = "value_of_'value_of_foo'"
342
343 def test_one_var(self):
344 val = self.d.expand("${foo}")
345 self.assertEqual(str(val), "value_of_foo")
346
347In this example, a ``DataExpansions`` class of tests is created,
348derived from standard python unittest. The class has a common ``setUp``
349function which is shared by all the tests in the class. A simple test is
350then added to test that when a variable is expanded, the correct value
351is found.
352
353Bitbake selftests are straightforward python unittest. Refer to the
354Python unittest documentation for additional information on writing
355these tests at: https://docs.python.org/3/library/unittest.html.
356
357.. _oe-selftest-example:
358
359``oe-selftest``
360---------------
361
362These tests are more complex due to the setup required behind the scenes
363for full builds. Rather than directly using Python's unittest, the code
364wraps most of the standard objects. The tests can be simple, such as
365testing a command from within the OE build environment using the
366following example::
367
368 class BitbakeLayers(OESelftestTestCase):
369 def test_bitbakelayers_showcrossdepends(self):
370 result = runCmd('bitbake-layers show-cross-depends')
371 self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
372
373This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
374creates a testcase from the ``OESelftestTestCase`` class, derived
375from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
376and checks the output to ensure it contains something we know should be
377here.
378
379The ``oeqa.utils.commands`` module contains Helpers which can assist
380with common tasks, including:
381
382- *Obtaining the value of a bitbake variable:* Use
383 ``oeqa.utils.commands.get_bb_var()`` or use
384 ``oeqa.utils.commands.get_bb_vars()`` for more than one variable
385
386- *Running a bitbake invocation for a build:* Use
387 ``oeqa.utils.commands.bitbake()``
388
389- *Running a command:* Use ``oeqa.utils.commandsrunCmd()``
390
391There is also a ``oeqa.utils.commands.runqemu()`` function for launching
392the ``runqemu`` command for testing things within a running, virtualized
393image.
394
395You can run these tests in parallel. Parallelism works per test class,
396so tests within a given test class should always run in the same build,
397while tests in different classes or modules may be split into different
398builds. There is no data store available for these tests since the tests
399launch the ``bitbake`` command and exist outside of its context. As a
400result, common bitbake library functions (bb.\*) are also unavailable.
401
402.. _testimage-example:
403
404``testimage``
405-------------
406
407These tests are run once an image is up and running, either on target
408hardware or under QEMU. As a result, they are assumed to be running in a
409target image environment, as opposed to a host build environment. A
410simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
411the following::
412
413 class PythonTest(OERuntimeTestCase):
414 @OETestDepends(['ssh.SSHTest.test_ssh'])
415 @OEHasPackage(['python3-core'])
416 def test_python3(self):
417 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
418 status, output = self.target.run(cmd)
419 msg = 'Exit status was not 0. Output: %s' % output
420 self.assertEqual(status, 0, msg=msg)
421
422In this example, the ``OERuntimeTestCase`` class wraps
423``unittest.TestCase``. Within the test, ``self.target`` represents the
424target system, where commands can be run on it using the ``run()``
425method.
426
427To ensure certain test or package dependencies are met, you can use the
428``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
429in this example would only make sense if python3-core is installed in
430the image.
431
432.. _testsdk_ext-example:
433
434``testsdk_ext``
435---------------
436
437These tests are run against built extensible SDKs (eSDKs). The tests can
438assume that the eSDK environment has already been setup. An example from
439``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
440
441 class DevtoolTest(OESDKExtTestCase):
442 @classmethod def setUpClass(cls):
443 myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
444 cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
445 shutil.copytree(myapp_src, cls.myapp_dst)
446 subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
447 subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
448 subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
449
450 @classmethod
451 def tearDownClass(cls):
452 shutil.rmtree(cls.myapp_dst)
453 def _test_devtool_build(self, directory):
454 self._run('devtool add myapp %s' % directory)
455 try:
456 self._run('devtool build myapp')
457 finally:
458 self._run('devtool reset myapp')
459 def test_devtool_build_make(self):
460 self._test_devtool_build(self.myapp_dst)
461
462In this example, the ``devtool``
463command is tested to see whether a sample application can be built with
464the ``devtool build`` command within the eSDK.
465
466.. _testsdk-example:
467
468``testsdk``
469-----------
470
471These tests are run against built SDKs. The tests can assume that an SDK
472has already been extracted and its environment file has been sourced. A
473simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
474following::
475
476 class Python3Test(OESDKTestCase):
477 def setUp(self):
478 if not (self.tc.hasHostPackage("nativesdk-python3-core") or
479 self.tc.hasHostPackage("python3-core-native")):
480 raise unittest.SkipTest("No python3 package in the SDK")
481
482 def test_python3(self):
483 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
484 output = self._run(cmd)
485 self.assertEqual(output, "Hello, world\n")
486
487In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
488the python3 interpreter with a basic command to check it is working
489correctly. The test would only run if python3 is installed in the SDK.
490
491.. _oe-build-perf-test-example:
492
493``oe-build-perf-test``
494----------------------
495
496The performance tests usually measure how long operations take and the
497resource utilisation as that happens. An example from
498``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
499
500 class Test3(BuildPerfTestCase):
501 def test3(self):
502 """Bitbake parsing (bitbake -p)"""
503 # Drop all caches and parse
504 self.rm_cache()
505 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
506 self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
507 'bitbake -p (no caches)')
508 # Drop tmp/cache
509 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
510 self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
511 'bitbake -p (no tmp/cache)')
512 # Parse with fully cached data
513 self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
514 'bitbake -p (cached)')
515
516This example shows how three specific parsing timings are
517measured, with and without various caches, to show how BitBake's parsing
518performance trends over time.
519
520.. _test-writing-considerations:
521
522Considerations When Writing Tests
523=================================
524
525When writing good tests, there are several things to keep in mind. Since
526things running on the Autobuilder are accessed concurrently by multiple
527workers, consider the following:
528
529**Running "cleanall" is not permitted.**
530
531This can delete files from DL_DIR which would potentially break other
532builds running in parallel. If this is required, DL_DIR must be set to
533an isolated directory.
534
535**Running "cleansstate" is not permitted.**
536
537This can delete files from SSTATE_DIR which would potentially break
538other builds running in parallel. If this is required, SSTATE_DIR must
539be set to an isolated directory. Alternatively, you can use the "-f"
540option with the ``bitbake`` command to "taint" tasks by changing the
541sstate checksums to ensure sstate cache items will not be reused.
542
543**Tests should not change the metadata.**
544
545This is particularly true for oe-selftests since these can run in
546parallel and changing metadata leads to changing checksums, which
547confuses BitBake while running in parallel. If this is necessary, copy
548layers to a temporary location and modify them. Some tests need to
549change metadata, such as the devtool tests. To prevent the metadate from
550changes, set up temporary copies of that data first.
diff --git a/documentation/test-manual/test-manual-intro.xml b/documentation/test-manual/test-manual-intro.xml
deleted file mode 100644
index 0cdbee4d8e..0000000000
--- a/documentation/test-manual/test-manual-intro.xml
+++ /dev/null
@@ -1,624 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='test-manual-intro'>
7
8<title>The Yocto Project Test Environment Manual</title>
9 <section id='test-welcome'>
10 <title>Welcome</title>
11
12 <para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in
13 progress. The manual contains information about the testing environment used by the
14 Yocto Project to make sure each major and minor release works as intended. All the
15 project's testing infrastructure and processes are publicly visible and available so
16 that the community can see what testing is being performed, how it's being done and the
17 current status of the tests and the project at any given time. It is intended that Other
18 organizations can leverage off the process and testing environment used by the Yocto
19 Project to create their own automated, production test environment, building upon the
20 foundations from the project core. </para>
21
22 <para> Currently, the Yocto Project Test Environment Manual has no projected release date.
23 This manual is a work-in-progress and is being initially loaded with information from
24 the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
25 <listitem>
26 <para>
27 <emphasis><filename>yocto-autobuilder2</filename>:</emphasis> This <ulink
28 url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md"
29 ><filename>README.md</filename></ulink> is the main README which
30 detials how to set up the Yocto Project Autobuilder. The
31 <filename>yocto-autobuilder2</filename> repository represents the Yocto
32 Project's console UI plugin to Buildbot and the configuration necessary to
33 configure Buildbot to perform the testing the project requires. </para>
34 </listitem>
35 <listitem>
36 <para>
37 <emphasis><filename>yocto-autobuilder-helper</filename>:</emphasis> This
38 <ulink
39 url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README"
40 ><filename>README</filename></ulink> and repository contains Yocto
41 Project Autobuilder Helper scripts and configuration. The
42 <filename>yocto-autobuilder-helper</filename> repository contains the
43 "glue" logic that defines which tests to run and how to run them. As a
44 result, it can be used by any Continuous Improvement (CI) system to run
45 builds, support getting the correct code revisions, configure builds and
46 layers, run builds, and collect results. The code is independent of any CI
47 system, which means the code can work Buildbot, Jenkins, or others. This
48 repository has a branch per release of the project defining the tests to run
49 on a per release basis.</para>
50 </listitem>
51 </itemizedlist>
52 </para>
53 </section>
54
55 <section id='test-yocto-project-autobuilder-overview'>
56 <title>Yocto Project Autobuilder Overview</title>
57
58 <para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and
59 procedures used by the Yocto Project to test released software across supported hardware
60 in an automated and regular fashion. Basically, during the development of a Yocto
61 Project release, the Autobuilder tests if things work. The Autobuilder builds all test
62 targets and runs all the tests. </para>
63
64 <para>The Yocto Project uses now uses standard upstream <ulink
65 url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive
66 its integration and testing. Buildbot Nine has a plug-in interface that the Yocto
67 Project customizes using code from the <filename>yocto-autobuilder2</filename>
68 repository, adding its own console UI plugin. The resulting UI plug-in allows you to
69 visualize builds in a way suited to the project's needs.</para>
70
71 <para>A <filename>helper</filename> layer provides configuration and job management through
72 scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The
73 <filename>helper</filename> layer contains the bulk of the build configuration
74 information and is release-specific, which makes it highly customizable on a per-project
75 basis. The layer is CI system-agnostic and contains a number of Helper scripts that can
76 generate build configurations from simple JSON files. <note>
77 <para>The project uses Buildbot for historical reasons but also because many of the
78 project developers have knowledge of python. It is possible to use the outer
79 layers from another Continuous Integration (CI) system such as <ulink
80 url="https://en.wikipedia.org/wiki/Jenkins_(software)">Jenkins</ulink>
81 instead of Buildbot. </para>
82 </note>
83 </para>
84
85 <para> The following figure shows the Yocto Project Autobuilder stack with a topology that
86 includes a controller and a cluster of workers: <imagedata
87 fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center"
88 scalefit="1"/>
89 </para>
90 </section>
91
92 <section id='test-project-tests'>
93 <title>Yocto Project Tests - Types of Testing Overview</title>
94
95 <para>The Autobuilder tests different elements of the project by using thefollowing types of
96 tests: <itemizedlist>
97 <listitem>
98 <para>
99 <emphasis>Build Testing:</emphasis> Tests whether specific configurations
100 build by varying <ulink url="&YOCTO_DOCS_REF_URL;#var-MACHINE"
101 ><filename>MACHINE</filename></ulink>, <ulink
102 url="&YOCTO_DOCS_REF_URL;#var-DISTRO"
103 ><filename>DISTRO</filename></ulink>, other configuration options, and
104 the specific target images being built (or world). Used to trigger builds of
105 all the different test configurations on the Autobuilder. Builds usually
106 cover many different targets for different architectures, machines, and
107 distributions, as well as different configurations, such as different init
108 systems. The Autobuilder tests literally hundreds of configurations and
109 targets. <itemizedlist>
110 <listitem>
111 <para>
112 <emphasis>Sanity Checks During the Build Process:</emphasis>
113 Tests initiated through the <ulink
114 url="&YOCTO_DOCS_REF_URL;#ref-classes-insane"
115 ><filename>insane</filename></ulink> class. These checks
116 ensure the output of the builds are correct. For example, does
117 the ELF architecture in the generated binaries match the target
118 system? ARM binaries would not work in a MIPS system! </para>
119 </listitem>
120 </itemizedlist></para>
121 </listitem>
122 <listitem>
123 <para>
124 <emphasis>Build Performance Testing:</emphasis> Tests whether or not
125 commonly used steps during builds work efficiently and avoid regressions.
126 Tests to time commonly used usage scenarios are run through
127 <filename>oe-build-perf-test</filename>. These tests are run on isolated
128 machines so that the time measurements of the tests are accurate and no
129 other processes interfere with the timing results. The project currently
130 tests performance on two different distributions, Fedora and Ubuntu, to
131 ensure we have no single point of failure and can ensure the different
132 distros work effectively. </para>
133 </listitem>
134 <listitem>
135 <para>
136 <emphasis>eSDK Testing:</emphasis> Image tests initiated through the
137 following command:
138 <literallayout class="monospaced">
139 $ bitbake <replaceable>image</replaceable> -c testsdkext
140 </literallayout>
141 The tests utilize the <filename>testsdkext</filename> class and the
142 <filename>do_testsdkext</filename> task. </para>
143 </listitem>
144 <listitem>
145 <para>
146 <emphasis>Feature Testing:</emphasis> Various scenario-based tests are run
147 through the <ulink url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
148 >OpenEmbedded Self-Test</ulink> (oe-selftest). We test oe-selftest on
149 each of the main distrubutions we support. </para>
150 </listitem>
151 <listitem>
152 <para>
153 <emphasis>Image Testing:</emphasis> Image tests initiated through the
154 following command:
155 <literallayout class="monospaced">
156 $ bitbake <replaceable>image</replaceable> -c testimage
157 </literallayout>
158 The tests utilize the <ulink
159 url="&YOCTO_DOCS_REF_URL;#ref-classes-testimage*"
160 ><filename>testimage*</filename></ulink> classes and the <ulink
161 url="&YOCTO_DOCS_REF_URL;#ref-tasks-testimage"
162 ><filename>do_testimage</filename></ulink> task. </para>
163 </listitem>
164 <listitem>
165 <para>
166 <emphasis>Layer Testing:</emphasis> The Autobuilder has the possibility to
167 test whether specific layers work with the test of the system. The layers
168 tested may be selected by members of the project. Some key community layers
169 are also tested periodically.</para>
170 </listitem>
171 <listitem>
172 <para>
173 <emphasis>Package Testing:</emphasis> A Package Test (ptest) runs tests
174 against packages built by the OpenEmbedded build system on the target
175 machine. See the "<ulink
176 url="&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest">Testing Packages
177 With ptest</ulink>" section in the Yocto Project Development Tasks
178 Manual and the "<ulink url="&YOCTO_WIKI_URL;/wiki/Ptest">Ptest</ulink>" Wiki
179 page for more information on Ptest. </para>
180 </listitem>
181 <listitem>
182 <para>
183 <emphasis>SDK Testing:</emphasis> Image tests initiated through the
184 following command:
185 <literallayout class="monospaced">
186 $ bitbake <replaceable>image</replaceable> -c testsdk
187 </literallayout>
188 The tests utilize the <ulink url="&YOCTO_DOCS_REF_URL;#ref-classes-testsdk"
189 ><filename>testsdk</filename></ulink> class and the
190 <filename>do_testsdk</filename> task. </para>
191 </listitem>
192 <listitem>
193 <para>
194 <emphasis>Unit Testing:</emphasis> Unit tests on various components of the
195 system run through <filename>oe-selftest</filename> and <ulink
196 url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
197 ><filename>bitbake-selftest</filename></ulink>. </para>
198 </listitem>
199 <listitem>
200 <para>
201 <emphasis>Automatic Upgrade Helper:</emphasis> This target tests whether new
202 versions of software are available and whether we can automatically upgrade
203 to those new versions. If so, this target emails the maintainers with a
204 patch to let them know this is possible.</para>
205 </listitem>
206 </itemizedlist>
207 </para>
208 </section>
209
210 <section id='test-test-mapping'>
211 <title>How Tests Map to Areas of Code</title>
212
213 <para>
214 Tests map into the codebase as follows:
215 <itemizedlist>
216 <listitem><para>
217 <emphasis>bitbake-selftest</emphasis>: </para>
218 <para>These tests are self-contained and test BitBake as well as its APIs, which
219 include the fetchers. The tests are located in
220 <filename>bitbake/lib/*/tests</filename>. </para>
221 <para>From within the BitBake repository, run the following:
222 <literallayout class="monospaced">
223 $ bitbake-selftest
224 </literallayout>
225 </para>
226 <para>To skip tests that access the Internet, use the
227 <filename>BB_SKIP_NETTEST</filename> variable when running
228 "bitbake-selftest" as follows:
229 <literallayout class="monospaced">
230 $ BB_SKIP_NETTEST=yes bitbake-selftest
231 </literallayout></para>
232 <para>The default output is quiet and just prints a summary of what was run. To
233 see more information, there is a verbose
234 option:<literallayout class="monospaced">
235 $ bitbake-selftest -v
236 </literallayout></para>
237 <para>Use this option when you wish to skip tests that access the network, which
238 are mostly necessary to test the fetcher modules. To specify individual test
239 modules to run, append the test module name to the "bitbake-selftest"
240 command. For example, to specify the tests for the bb.data.module, run:
241 <literallayout class="monospaced">
242 $ bitbake-selftest bb.test.data.module
243 </literallayout>You
244 can also specify individual tests by defining the full name and module plus
245 the class path of the test, for example:
246 <literallayout class="monospaced">
247 $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
248 </literallayout></para>
249 <para>The tests are based on <ulink
250 url="https://docs.python.org/3/library/unittest.html">Python
251 unittest</ulink>. </para></listitem>
252 <listitem><para>
253 <emphasis>oe-selftest</emphasis>: <itemizedlist>
254 <listitem>
255 <para>These tests use OE to test the workflows, which include
256 testing specific features, behaviors of tasks, and API unit
257 tests. </para>
258 </listitem>
259 <listitem>
260 <para>The tests can take advantage of parallelism through the "-j"
261 option, which can specify a number of threads to spread the
262 tests across. Note that all tests from a given class of tests
263 will run in the same thread. To parallelize large numbers of
264 tests you can split the class into multiple units.</para>
265 </listitem>
266 <listitem>
267 <para>The tests are based on Python unittest. </para>
268 </listitem>
269 <listitem>
270 <para>The code for the tests resides in
271 <filename>meta/lib/oeqa/selftest/cases/</filename>. </para>
272 </listitem>
273 <listitem>
274 <para>To run all the tests, enter the following command:
275 <literallayout class="monospaced">
276 $ oe-selftest -a
277 </literallayout>
278 </para>
279 </listitem>
280 <listitem>
281 <para>To run a specific test, use the following command form where
282 <replaceable>testname</replaceable> is the name of the
283 specific test:
284 <literallayout class="monospaced">
285 $ oe-selftest -r <replaceable>testname</replaceable>
286 </literallayout>
287 For example, the following command would run the tinfoil getVar
288 API
289 test:<literallayout class="monospaced">
290 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
291 </literallayout>It
292 is also possible to run a set of tests. For example the
293 following command will run all of the tinfoil
294 tests:<literallayout class="monospaced">
295 $ oe-selftest -r tinfoil
296 </literallayout></para>
297 </listitem>
298 </itemizedlist>
299 </para></listitem>
300 <listitem><para>
301 <emphasis>testimage:</emphasis>
302 <itemizedlist>
303 <listitem><para>
304 These tests build an image, boot it, and run tests
305 against the image's content.
306 </para></listitem>
307 <listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem>
308 <listitem><para>
309 You need to set the
310 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink>
311 variable as follows:
312 <literallayout class='monospaced'>
313 IMAGE_CLASSES += "testimage"
314 </literallayout>
315 </para></listitem>
316 <listitem><para>
317 Run the tests using the following command form:
318 <literallayout class='monospaced'>
319 $ bitbake <replaceable>image</replaceable> -c testimage
320 </literallayout>
321 </para></listitem>
322 </itemizedlist>
323 </para></listitem>
324 <listitem><para>
325 <emphasis>testsdk:</emphasis>
326 <itemizedlist>
327 <listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem>
328 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem>
329 <listitem><para>Run the test using the following command form:
330 <literallayout class="monospaced">
331 $ bitbake <replaceable>image</replaceable> -c testsdk
332 </literallayout>
333 </para></listitem>
334 </itemizedlist>
335 </para></listitem>
336 <listitem><para>
337 <emphasis>testsdk_ext:</emphasis>
338 <itemizedlist>
339 <listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem>
340 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem>
341 <listitem><para>To run the tests, use the following command form:
342 <literallayout class="monospaced">
343 $ bitbake <replaceable>image</replaceable> -c testsdkext
344 </literallayout>
345 </para></listitem>
346 </itemizedlist>
347 </para></listitem>
348
349
350 <listitem><para>
351 <emphasis>oe-build-perf-test:</emphasis>
352 <itemizedlist>
353 <listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem>
354 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem>
355 <listitem><para>To run the tests, use the following command form:
356 <literallayout class="monospaced">
357 $ oe-build-perf-test <replaceable>options</replaceable>
358 </literallayout>The
359 command takes a number of options, such as where to place the
360 test results. The Autobuilder Helper Scripts include the
361 <filename>build-perf-test-wrapper</filename> script with
362 examples of how to use the oe-build-perf-test from the command
363 line.</para>
364 <para>Use the <filename>oe-git-archive</filename> command to store
365 test results into a Git repository. </para>
366 <para>Use the <filename>oe-build-perf-report</filename> command to
367 generate text reports and HTML reports with graphs of the
368 performance data. For examples, see <link linkend=""
369 >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>
370 and <link linkend=""
371 >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>
372 <listitem>
373 <para>The tests are contained in
374 <filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>
375 </listitem>
376 </itemizedlist>
377 </para></listitem>
378
379
380
381
382 </itemizedlist>
383 </para>
384 </section>
385
386 <section id='test-examples'>
387 <title>Test Examples</title>
388
389 <para>This section provides example tests for each of the tests listed in the <link
390 linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>
391 <para>For oeqa tests, testcases for each area reside in the main test directory at
392 <filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>
393 <para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>
394 directory. </para>
395
396 <section id='bitbake-selftest-example'>
397 <title><filename>bitbake-selftest</filename></title>
398
399 <para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:
400 <literallayout class="monospaced">
401 class DataExpansions(unittest.TestCase):
402 def setUp(self):
403 self.d = bb.data.init()
404 self.d["foo"] = "value_of_foo"
405 self.d["bar"] = "value_of_bar"
406 self.d["value_of_foo"] = "value_of_'value_of_foo'"
407
408 def test_one_var(self):
409 val = self.d.expand("${foo}")
410 self.assertEqual(str(val), "value_of_foo")
411 </literallayout>
412 </para>
413 <para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class
414 of tests is created, derived from standard python unittest. The class has a common
415 <filename>setUp</filename> function which is shared by all the tests in the
416 class. A simple test is then added to test that when a variable is expanded, the
417 correct value is found.</para>
418 <para>Bitbake selftests are straightforward python unittest. Refer to the Python
419 unittest documentation for additional information on writing these tests at: <link
420 linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>
421 </section>
422
423 <section id='oe-selftest-example'>
424 <title><filename>oe-selftest</filename></title>
425
426 <para>These tests are more complex due to the setup required behind the scenes for full
427 builds. Rather than directly using Python's unittest, the code wraps most of the
428 standard objects. The tests can be simple, such as testing a command from within the
429 OE build environment using the following
430 example:<literallayout class="monospaced">
431 class BitbakeLayers(OESelftestTestCase):
432 def test_bitbakelayers_showcrossdepends(self):
433 result = runCmd('bitbake-layers show-cross-depends')
434 self.assertTrue('aspell' in result.output, msg = "No dependencies
435 were shown. bitbake-layers show-cross-depends output:
436 %s"% result.output)
437 </literallayout></para>
438 <para>This example, taken from
439 <filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
440 testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>
441 class, derived from <filename>unittest.TestCase</filename>, which runs the
442 <filename>bitbake-layers</filename> command and checks the output to ensure it
443 contains something we know should be here.</para>
444 <para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can
445 assist with common tasks, including:<itemizedlist>
446 <listitem>
447 <para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use
448 <filename>oeqa.utils.commands.get_bb_var()</filename> or use
449 <filename>oeqa.utils.commands.get_bb_vars()</filename> for more than
450 one variable</para>
451 </listitem>
452 <listitem>
453 <para><emphasis>Running a bitbake invocation for a build:</emphasis> Use
454 <filename>oeqa.utils.commands.bitbake()</filename></para>
455 </listitem>
456 <listitem>
457 <para><emphasis>Running a command:</emphasis> Use
458 <filename>oeqa.utils.commandsrunCmd()</filename></para>
459 </listitem>
460 </itemizedlist></para>
461 <para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for
462 launching the <filename>runqemu</filename> command for testing things within a
463 running, virtualized image.</para>
464 <para>You can run these tests in parallel. Parallelism works per test class, so tests
465 within a given test class should always run in the same build, while tests in
466 different classes or modules may be split into different builds. There is no data
467 store available for these tests since the tests launch the
468 <filename>bitbake</filename> command and exist outside of its context. As a
469 result, common bitbake library functions (bb.*) are also unavailable.</para>
470 </section>
471
472 <section id='testimage-example'>
473 <title><filename>testimage</filename></title>
474
475 <para>These tests are run once an image is up and running, either on target hardware or
476 under QEMU. As a result, they are assumed to be running in a target image
477 environment, as opposed to a host build environment. A simple example from
478 <filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the
479 following:<literallayout class="monospaced">
480 class PythonTest(OERuntimeTestCase):
481 @OETestDepends(['ssh.SSHTest.test_ssh'])
482 @OEHasPackage(['python3-core'])
483 def test_python3(self):
484 cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,
485 jbeyq', 'rot13'))\""
486 status, output = self.target.run(cmd)
487 msg = 'Exit status was not 0. Output: %s' % output
488 self.assertEqual(status, 0, msg=msg)
489 </literallayout></para>
490 <para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>
491 class wraps <filename>unittest.TestCase</filename>. Within the test,
492 <filename>self.target</filename> represents the target system, where commands
493 can be run on it using the <filename>run()</filename> method. </para>
494 <para>To ensure certain test or package dependencies are met, you can use the
495 <filename>OETestDepends</filename> and <filename>OEHasPackage</filename>
496 decorators. For example, the test in this example would only make sense if
497 python3-core is installed in the image.</para>
498 </section>
499
500 <section id='testsdk_ext-example'>
501 <title><filename>testsdk_ext</filename></title>
502
503 <para>These tests are run against built extensible SDKs (eSDKs). The tests can assume
504 that the eSDK environment has already been setup. An example from
505 <filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
506 following:<literallayout class="monospaced">
507 class DevtoolTest(OESDKExtTestCase):
508 @classmethod
509 def setUpClass(cls):
510 myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
511 cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
512 shutil.copytree(myapp_src, cls.myapp_dst)
513 subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
514 subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
515 subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
516
517 @classmethod
518 def tearDownClass(cls):
519 shutil.rmtree(cls.myapp_dst)
520 def _test_devtool_build(self, directory):
521 self._run('devtool add myapp %s' % directory)
522 try:
523 self._run('devtool build myapp')
524 finally:
525 self._run('devtool reset myapp')
526 def test_devtool_build_make(self):
527 self._test_devtool_build(self.myapp_dst)
528 </literallayout>In
529 this example, the <filename>devtool</filename> command is tested to see whether a
530 sample application can be built with the <filename>devtool build</filename> command
531 within the eSDK.</para>
532 </section>
533
534 <section id='testsdk-example'>
535 <title><filename>testsdk</filename></title>
536
537 <para>These tests are run against built SDKs. The tests can assume that an SDK has
538 already been extracted and its environment file has been sourced. A simple example
539 from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
540 following:<literallayout class="monospaced">
541 class Python3Test(OESDKTestCase):
542 def setUp(self):
543 if not (self.tc.hasHostPackage("nativesdk-python3-core") or
544 self.tc.hasHostPackage("python3-core-native")):
545 raise unittest.SkipTest("No python3 package in the SDK")
546
547 def test_python3(self):
548 cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
549 output = self._run(cmd)
550 self.assertEqual(output, "Hello, world\n")
551 </literallayout>In
552 this example, if nativesdk-python3-core has been installed into the SDK, the code
553 runs the python3 interpreter with a basic command to check it is working correctly.
554 The test would only run if python3 is installed in the SDK.</para>
555 </section>
556
557 <section id='oe-build-perf-test-example'>
558 <title><filename>oe-build-perf-test</filename></title>
559
560 <para>The performance tests usually measure how long operations take and the resource
561 utilisation as that happens. An example from
562 <filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
563 following:<literallayout class="monospaced">
564 class Test3(BuildPerfTestCase):
565
566 def test3(self):
567 """Bitbake parsing (bitbake -p)"""
568 # Drop all caches and parse
569 self.rm_cache()
570 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
571 self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
572 'bitbake -p (no caches)')
573 # Drop tmp/cache
574 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
575 self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
576 'bitbake -p (no tmp/cache)')
577 # Parse with fully cached data
578 self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
579 'bitbake -p (cached)')
580 </literallayout>This
581 example shows how three specific parsing timings are measured, with and without
582 various caches, to show how BitBake's parsing performance trends over time.</para>
583 </section>
584 </section>
585 <section id='test-writing-considerations'>
586 <title>Considerations When Writing Tests</title>
587 <para>When writing good tests, there are several things to keep in mind. Since things
588 running on the Autobuilder are accessed concurrently by multiple workers, consider the
589 following:</para>
590 <formalpara>
591 <title>Running "cleanall" is not permitted</title>
592 <para>This can delete files from DL_DIR which would potentially break other builds
593 running in parallel. If this is required, DL_DIR must be set to an isolated
594 directory.</para>
595 </formalpara>
596 <formalpara>
597 <title>Running "cleansstate" is not permitted</title>
598 <para>This can delete files from SSTATE_DIR which would potentially break other builds
599 running in parallel. If this is required, SSTATE_DIR must be set to an isolated
600 directory. Alternatively, you can use the "-f" option with the
601 <filename>bitbake</filename> command to "taint" tasks by changing the sstate
602 checksums to ensure sstate cache items will not be reused.</para>
603 </formalpara>
604 <formalpara>
605 <title>Tests should not change the metadata</title>
606 <para>This is particularly true for oe-selftests since these can run in parallel and
607 changing metadata leads to changing checksums, which confuses BitBake while running
608 in parallel. If this is necessary, copy layers to a temporary location and modify
609 them. Some tests need to change metadata, such as the devtool tests. To prevent the
610 metadate from changes, set up temporary copies of that data first.</para>
611 </formalpara>
612 </section>
613
614
615
616
617
618
619
620
621</chapter>
622<!--
623vim: expandtab tw=80 ts=4
624-->
diff --git a/documentation/test-manual/test-manual-style.css b/documentation/test-manual/test-manual-style.css
deleted file mode 100644
index 10ee4c79cf..0000000000
--- a/documentation/test-manual/test-manual-style.css
+++ /dev/null
@@ -1,991 +0,0 @@
1/*
2 Generic XHTML / DocBook XHTML CSS Stylesheet.
3
4 SPDX-License-Identifier: CC-BY-2.0-UK
5
6 Browser wrangling and typographic design by
7 Oyvind Kolas / pippin@gimp.org
8
9 Customised for Poky by
10 Matthew Allum / mallum@o-hand.com
11
12 Thanks to:
13 Liam R. E. Quin
14 William Skaggs
15 Jakub Steiner
16
17 Structure
18 ---------
19
20 The stylesheet is divided into the following sections:
21
22 Positioning
23 Margins, paddings, width, font-size, clearing.
24 Decorations
25 Borders, style
26 Colors
27 Colors
28 Graphics
29 Graphical backgrounds
30 Nasty IE tweaks
31 Workarounds needed to make it work in internet explorer,
32 currently makes the stylesheet non validating, but up until
33 this point it is validating.
34 Mozilla extensions
35 Transparency for footer
36 Rounded corners on boxes
37
38*/
39
40
41 /*************** /
42 / Positioning /
43/ ***************/
44
45body {
46 font-family: Verdana, Sans, sans-serif;
47
48 min-width: 640px;
49 width: 80%;
50 margin: 0em auto;
51 padding: 2em 5em 5em 5em;
52 color: #333;
53}
54
55h1,h2,h3,h4,h5,h6,h7 {
56 font-family: Arial, Sans;
57 color: #00557D;
58 clear: both;
59}
60
61h1 {
62 font-size: 2em;
63 text-align: left;
64 padding: 0em 0em 0em 0em;
65 margin: 2em 0em 0em 0em;
66}
67
68h2.subtitle {
69 margin: 0.10em 0em 3.0em 0em;
70 padding: 0em 0em 0em 0em;
71 font-size: 1.8em;
72 padding-left: 20%;
73 font-weight: normal;
74 font-style: italic;
75}
76
77h2 {
78 margin: 2em 0em 0.66em 0em;
79 padding: 0.5em 0em 0em 0em;
80 font-size: 1.5em;
81 font-weight: bold;
82}
83
84h3.subtitle {
85 margin: 0em 0em 1em 0em;
86 padding: 0em 0em 0em 0em;
87 font-size: 142.14%;
88 text-align: right;
89}
90
91h3 {
92 margin: 1em 0em 0.5em 0em;
93 padding: 1em 0em 0em 0em;
94 font-size: 140%;
95 font-weight: bold;
96}
97
98h4 {
99 margin: 1em 0em 0.5em 0em;
100 padding: 1em 0em 0em 0em;
101 font-size: 120%;
102 font-weight: bold;
103}
104
105h5 {
106 margin: 1em 0em 0.5em 0em;
107 padding: 1em 0em 0em 0em;
108 font-size: 110%;
109 font-weight: bold;
110}
111
112h6 {
113 margin: 1em 0em 0em 0em;
114 padding: 1em 0em 0em 0em;
115 font-size: 110%;
116 font-weight: bold;
117}
118
119.authorgroup {
120 background-color: transparent;
121 background-repeat: no-repeat;
122 padding-top: 256px;
123 background-image: url("figures/test-manual-title.png");
124 background-position: left top;
125 margin-top: -256px;
126 padding-right: 50px;
127 margin-left: 0px;
128 text-align: right;
129 width: 740px;
130}
131
132h3.author {
133 margin: 0em 0me 0em 0em;
134 padding: 0em 0em 0em 0em;
135 font-weight: normal;
136 font-size: 100%;
137 color: #333;
138 clear: both;
139}
140
141.author tt.email {
142 font-size: 66%;
143}
144
145.titlepage hr {
146 width: 0em;
147 clear: both;
148}
149
150.revhistory {
151 padding-top: 2em;
152 clear: both;
153}
154
155.toc,
156.list-of-tables,
157.list-of-examples,
158.list-of-figures {
159 padding: 1.33em 0em 2.5em 0em;
160 color: #00557D;
161}
162
163.toc p,
164.list-of-tables p,
165.list-of-figures p,
166.list-of-examples p {
167 padding: 0em 0em 0em 0em;
168 padding: 0em 0em 0.3em;
169 margin: 1.5em 0em 0em 0em;
170}
171
172.toc p b,
173.list-of-tables p b,
174.list-of-figures p b,
175.list-of-examples p b{
176 font-size: 100.0%;
177 font-weight: bold;
178}
179
180.toc dl,
181.list-of-tables dl,
182.list-of-figures dl,
183.list-of-examples dl {
184 margin: 0em 0em 0.5em 0em;
185 padding: 0em 0em 0em 0em;
186}
187
188.toc dt {
189 margin: 0em 0em 0em 0em;
190 padding: 0em 0em 0em 0em;
191}
192
193.toc dd {
194 margin: 0em 0em 0em 2.6em;
195 padding: 0em 0em 0em 0em;
196}
197
198div.glossary dl,
199div.variablelist dl {
200}
201
202.glossary dl dt,
203.variablelist dl dt,
204.variablelist dl dt span.term {
205 font-weight: normal;
206 width: 20em;
207 text-align: right;
208}
209
210.variablelist dl dt {
211 margin-top: 0.5em;
212}
213
214.glossary dl dd,
215.variablelist dl dd {
216 margin-top: 0em;
217 margin-left: 25.5em;
218}
219
220.glossary dd p,
221.variablelist dd p {
222 margin-top: 0em;
223 margin-bottom: 1em;
224}
225
226
227div.calloutlist table td {
228 padding: 0em 0em 0em 0em;
229 margin: 0em 0em 0em 0em;
230}
231
232div.calloutlist table td p {
233 margin-top: 0em;
234 margin-bottom: 1em;
235}
236
237div p.copyright {
238 text-align: left;
239}
240
241div.legalnotice p.legalnotice-title {
242 margin-bottom: 0em;
243}
244
245p {
246 line-height: 1.5em;
247 margin-top: 0em;
248
249}
250
251dl {
252 padding-top: 0em;
253}
254
255hr {
256 border: solid 1px;
257}
258
259
260.mediaobject,
261.mediaobjectco {
262 text-align: center;
263}
264
265img {
266 border: none;
267}
268
269ul {
270 padding: 0em 0em 0em 1.5em;
271}
272
273ul li {
274 padding: 0em 0em 0em 0em;
275}
276
277ul li p {
278 text-align: left;
279}
280
281table {
282 width :100%;
283}
284
285th {
286 padding: 0.25em;
287 text-align: left;
288 font-weight: normal;
289 vertical-align: top;
290}
291
292td {
293 padding: 0.25em;
294 vertical-align: top;
295}
296
297p a[id] {
298 margin: 0px;
299 padding: 0px;
300 display: inline;
301 background-image: none;
302}
303
304a {
305 text-decoration: underline;
306 color: #444;
307}
308
309pre {
310 overflow: auto;
311}
312
313a:hover {
314 text-decoration: underline;
315 /*font-weight: bold;*/
316}
317
318/* This style defines how the permalink character
319 appears by itself and when hovered over with
320 the mouse. */
321
322[alt='Permalink'] { color: #eee; }
323[alt='Permalink']:hover { color: black; }
324
325
326div.informalfigure,
327div.informalexample,
328div.informaltable,
329div.figure,
330div.table,
331div.example {
332 margin: 1em 0em;
333 padding: 1em;
334 page-break-inside: avoid;
335}
336
337
338div.informalfigure p.title b,
339div.informalexample p.title b,
340div.informaltable p.title b,
341div.figure p.title b,
342div.example p.title b,
343div.table p.title b{
344 padding-top: 0em;
345 margin-top: 0em;
346 font-size: 100%;
347 font-weight: normal;
348}
349
350.mediaobject .caption,
351.mediaobject .caption p {
352 text-align: center;
353 font-size: 80%;
354 padding-top: 0.5em;
355 padding-bottom: 0.5em;
356}
357
358.epigraph {
359 padding-left: 55%;
360 margin-bottom: 1em;
361}
362
363.epigraph p {
364 text-align: left;
365}
366
367.epigraph .quote {
368 font-style: italic;
369}
370.epigraph .attribution {
371 font-style: normal;
372 text-align: right;
373}
374
375span.application {
376 font-style: italic;
377}
378
379.programlisting {
380 font-family: monospace;
381 font-size: 80%;
382 white-space: pre;
383 margin: 1.33em 0em;
384 padding: 1.33em;
385}
386
387.tip,
388.warning,
389.caution,
390.note {
391 margin-top: 1em;
392 margin-bottom: 1em;
393
394}
395
396/* force full width of table within div */
397.tip table,
398.warning table,
399.caution table,
400.note table {
401 border: none;
402 width: 100%;
403}
404
405
406.tip table th,
407.warning table th,
408.caution table th,
409.note table th {
410 padding: 0.8em 0.0em 0.0em 0.0em;
411 margin : 0em 0em 0em 0em;
412}
413
414.tip p,
415.warning p,
416.caution p,
417.note p {
418 margin-top: 0.5em;
419 margin-bottom: 0.5em;
420 padding-right: 1em;
421 text-align: left;
422}
423
424.acronym {
425 text-transform: uppercase;
426}
427
428b.keycap,
429.keycap {
430 padding: 0.09em 0.3em;
431 margin: 0em;
432}
433
434.itemizedlist li {
435 clear: none;
436}
437
438.filename {
439 font-size: medium;
440 font-family: Courier, monospace;
441}
442
443
444div.navheader, div.heading{
445 position: absolute;
446 left: 0em;
447 top: 0em;
448 width: 100%;
449 background-color: #cdf;
450 width: 100%;
451}
452
453div.navfooter, div.footing{
454 position: fixed;
455 left: 0em;
456 bottom: 0em;
457 background-color: #eee;
458 width: 100%;
459}
460
461
462div.navheader td,
463div.navfooter td {
464 font-size: 66%;
465}
466
467div.navheader table th {
468 /*font-family: Georgia, Times, serif;*/
469 /*font-size: x-large;*/
470 font-size: 80%;
471}
472
473div.navheader table {
474 border-left: 0em;
475 border-right: 0em;
476 border-top: 0em;
477 width: 100%;
478}
479
480div.navfooter table {
481 border-left: 0em;
482 border-right: 0em;
483 border-bottom: 0em;
484 width: 100%;
485}
486
487div.navheader table td a,
488div.navfooter table td a {
489 color: #777;
490 text-decoration: none;
491}
492
493/* normal text in the footer */
494div.navfooter table td {
495 color: black;
496}
497
498div.navheader table td a:visited,
499div.navfooter table td a:visited {
500 color: #444;
501}
502
503
504/* links in header and footer */
505div.navheader table td a:hover,
506div.navfooter table td a:hover {
507 text-decoration: underline;
508 background-color: transparent;
509 color: #33a;
510}
511
512div.navheader hr,
513div.navfooter hr {
514 display: none;
515}
516
517
518.qandaset tr.question td p {
519 margin: 0em 0em 1em 0em;
520 padding: 0em 0em 0em 0em;
521}
522
523.qandaset tr.answer td p {
524 margin: 0em 0em 1em 0em;
525 padding: 0em 0em 0em 0em;
526}
527.answer td {
528 padding-bottom: 1.5em;
529}
530
531.emphasis {
532 font-weight: bold;
533}
534
535
536 /************* /
537 / decorations /
538/ *************/
539
540.titlepage {
541}
542
543.part .title {
544}
545
546.subtitle {
547 border: none;
548}
549
550/*
551h1 {
552 border: none;
553}
554
555h2 {
556 border-top: solid 0.2em;
557 border-bottom: solid 0.06em;
558}
559
560h3 {
561 border-top: 0em;
562 border-bottom: solid 0.06em;
563}
564
565h4 {
566 border: 0em;
567 border-bottom: solid 0.06em;
568}
569
570h5 {
571 border: 0em;
572}
573*/
574
575.programlisting {
576 border: solid 1px;
577}
578
579div.figure,
580div.table,
581div.informalfigure,
582div.informaltable,
583div.informalexample,
584div.example {
585 border: 1px solid;
586}
587
588
589
590.tip,
591.warning,
592.caution,
593.note {
594 border: 1px solid;
595}
596
597.tip table th,
598.warning table th,
599.caution table th,
600.note table th {
601 border-bottom: 1px solid;
602}
603
604.question td {
605 border-top: 1px solid black;
606}
607
608.answer {
609}
610
611
612b.keycap,
613.keycap {
614 border: 1px solid;
615}
616
617
618div.navheader, div.heading{
619 border-bottom: 1px solid;
620}
621
622
623div.navfooter, div.footing{
624 border-top: 1px solid;
625}
626
627 /********* /
628 / colors /
629/ *********/
630
631body {
632 color: #333;
633 background: white;
634}
635
636a {
637 background: transparent;
638}
639
640a:hover {
641 background-color: #dedede;
642}
643
644
645h1,
646h2,
647h3,
648h4,
649h5,
650h6,
651h7,
652h8 {
653 background-color: transparent;
654}
655
656hr {
657 border-color: #aaa;
658}
659
660
661.tip, .warning, .caution, .note {
662 border-color: #fff;
663}
664
665
666.tip table th,
667.warning table th,
668.caution table th,
669.note table th {
670 border-bottom-color: #fff;
671}
672
673
674.warning {
675 background-color: #f0f0f2;
676}
677
678.caution {
679 background-color: #f0f0f2;
680}
681
682.tip {
683 background-color: #f0f0f2;
684}
685
686.note {
687 background-color: #f0f0f2;
688}
689
690.glossary dl dt,
691.variablelist dl dt,
692.variablelist dl dt span.term {
693 color: #044;
694}
695
696div.figure,
697div.table,
698div.example,
699div.informalfigure,
700div.informaltable,
701div.informalexample {
702 border-color: #aaa;
703}
704
705pre.programlisting {
706 color: black;
707 background-color: #fff;
708 border-color: #aaa;
709 border-width: 2px;
710}
711
712.guimenu,
713.guilabel,
714.guimenuitem {
715 background-color: #eee;
716}
717
718
719b.keycap,
720.keycap {
721 background-color: #eee;
722 border-color: #999;
723}
724
725
726div.navheader {
727 border-color: black;
728}
729
730
731div.navfooter {
732 border-color: black;
733}
734
735.writernotes {
736 color: red;
737}
738
739
740 /*********** /
741 / graphics /
742/ ***********/
743
744/*
745body {
746 background-image: url("images/body_bg.jpg");
747 background-attachment: fixed;
748}
749
750.navheader,
751.note,
752.tip {
753 background-image: url("images/note_bg.jpg");
754 background-attachment: fixed;
755}
756
757.warning,
758.caution {
759 background-image: url("images/warning_bg.jpg");
760 background-attachment: fixed;
761}
762
763.figure,
764.informalfigure,
765.example,
766.informalexample,
767.table,
768.informaltable {
769 background-image: url("images/figure_bg.jpg");
770 background-attachment: fixed;
771}
772
773*/
774h1,
775h2,
776h3,
777h4,
778h5,
779h6,
780h7{
781}
782
783/*
784Example of how to stick an image as part of the title.
785
786div.article .titlepage .title
787{
788 background-image: url("figures/white-on-black.png");
789 background-position: center;
790 background-repeat: repeat-x;
791}
792*/
793
794div.preface .titlepage .title,
795div.colophon .title,
796div.chapter .titlepage .title,
797div.article .titlepage .title
798{
799}
800
801div.section div.section .titlepage .title,
802div.sect2 .titlepage .title {
803 background: none;
804}
805
806
807h1.title {
808 background-color: transparent;
809 background-image: url("figures/test-title.png");
810 background-repeat: no-repeat;
811 height: 256px;
812 text-indent: -9000px;
813 overflow:hidden;
814}
815
816h2.subtitle {
817 background-color: transparent;
818 text-indent: -9000px;
819 overflow:hidden;
820 width: 0px;
821 display: none;
822}
823
824 /*************************************** /
825 / pippin.gimp.org specific alterations /
826/ ***************************************/
827
828/*
829div.heading, div.navheader {
830 color: #777;
831 font-size: 80%;
832 padding: 0;
833 margin: 0;
834 text-align: left;
835 position: absolute;
836 top: 0px;
837 left: 0px;
838 width: 100%;
839 height: 50px;
840 background: url('/gfx/heading_bg.png') transparent;
841 background-repeat: repeat-x;
842 background-attachment: fixed;
843 border: none;
844}
845
846div.heading a {
847 color: #444;
848}
849
850div.footing, div.navfooter {
851 border: none;
852 color: #ddd;
853 font-size: 80%;
854 text-align:right;
855
856 width: 100%;
857 padding-top: 10px;
858 position: absolute;
859 bottom: 0px;
860 left: 0px;
861
862 background: url('/gfx/footing_bg.png') transparent;
863}
864*/
865
866
867
868 /****************** /
869 / nasty ie tweaks /
870/ ******************/
871
872/*
873div.heading, div.navheader {
874 width:expression(document.body.clientWidth + "px");
875}
876
877div.footing, div.navfooter {
878 width:expression(document.body.clientWidth + "px");
879 margin-left:expression("-5em");
880}
881body {
882 padding:expression("4em 5em 0em 5em");
883}
884*/
885
886 /**************************************** /
887 / mozilla vendor specific css extensions /
888/ ****************************************/
889/*
890div.navfooter, div.footing{
891 -moz-opacity: 0.8em;
892}
893
894div.figure,
895div.table,
896div.informalfigure,
897div.informaltable,
898div.informalexample,
899div.example,
900.tip,
901.warning,
902.caution,
903.note {
904 -moz-border-radius: 0.5em;
905}
906
907b.keycap,
908.keycap {
909 -moz-border-radius: 0.3em;
910}
911*/
912
913table tr td table tr td {
914 display: none;
915}
916
917
918hr {
919 display: none;
920}
921
922table {
923 border: 0em;
924}
925
926 .photo {
927 float: right;
928 margin-left: 1.5em;
929 margin-bottom: 1.5em;
930 margin-top: 0em;
931 max-width: 17em;
932 border: 1px solid gray;
933 padding: 3px;
934 background: white;
935}
936 .seperator {
937 padding-top: 2em;
938 clear: both;
939 }
940
941 #validators {
942 margin-top: 5em;
943 text-align: right;
944 color: #777;
945 }
946 @media print {
947 body {
948 font-size: 8pt;
949 }
950 .noprint {
951 display: none;
952 }
953 }
954
955
956.tip,
957.note {
958 background: #f0f0f2;
959 color: #333;
960 padding: 20px;
961 margin: 20px;
962}
963
964.tip h3,
965.note h3 {
966 padding: 0em;
967 margin: 0em;
968 font-size: 2em;
969 font-weight: bold;
970 color: #333;
971}
972
973.tip a,
974.note a {
975 color: #333;
976 text-decoration: underline;
977}
978
979.footnote {
980 font-size: small;
981 color: #333;
982}
983
984/* Changes the announcement text */
985.tip h3,
986.warning h3,
987.caution h3,
988.note h3 {
989 font-size:large;
990 color: #00557D;
991}
diff --git a/documentation/test-manual/test-manual-test-process.rst b/documentation/test-manual/test-manual-test-process.rst
deleted file mode 100644
index 96e71bf314..0000000000
--- a/documentation/test-manual/test-manual-test-process.rst
+++ /dev/null
@@ -1,103 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-2.0-UK
2
3***********************************
4Project Testing and Release Process
5***********************************
6
7.. _test-daily-devel:
8
9Day to Day Development
10======================
11
12This section details how the project tests changes, through automation
13on the Autobuilder or with the assistance of QA teams, through to making
14releases.
15
16The project aims to test changes against our test matrix before those
17changes are merged into the master branch. As such, changes are queued
18up in batches either in the ``master-next`` branch in the main trees, or
19in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton
20helps review and test patches and this is his testing tree).
21
22We have two broad categories of test builds, including "full" and
23"quick". On the Autobuilder, these can be seen as "a-quick" and
24"a-full", simply for ease of sorting in the UI. Use our Autobuilder
25console view to see where me manage most test-related items, available
26at: :yocto_ab:`/typhoon/#/console`.
27
28Builds are triggered manually when the test branches are ready. The
29builds are monitored by the SWAT team. For additional information, see
30:yocto_wiki:`/wiki/Yocto_Build_Failure_Swat_Team`.
31If successful, the changes would usually be merged to the ``master``
32branch. If not successful, someone would respond to the changes on the
33mailing list explaining that there was a failure in testing. The choice
34of quick or full would depend on the type of changes and the speed with
35which the result was required.
36
37The Autobuilder does build the ``master`` branch once daily for several
38reasons, in particular, to ensure the current ``master`` branch does
39build, but also to keep ``yocto-testresults``
40(:yocto_git:`/cgit.cgi/yocto-testresults/`),
41buildhistory
42(:yocto_git:`/cgit.cgi/poky-buildhistory/`), and
43our sstate up to date. On the weekend, there is a master-next build
44instead to ensure the test results are updated for the less frequently
45run targets.
46
47Performance builds (buildperf-\* targets in the console) are triggered
48separately every six hours and automatically push their results to the
49buildstats repository at:
50:yocto_git:`/cgit.cgi/yocto-buildstats/`.
51
52The 'quick' targets have been selected to be the ones which catch the
53most failures or give the most valuable data. We run 'fast' ptests in
54this case for example but not the ones which take a long time. The quick
55target doesn't include \*-lsb builds for all architectures, some world
56builds and doesn't trigger performance tests or ltp testing. The full
57build includes all these things and is slower but more comprehensive.
58
59Release Builds
60==============
61
62The project typically has two major releases a year with a six month
63cadence in April and October. Between these there would be a number of
64milestone releases (usually four) with the final one being stablization
65only along with point releases of our stable branches.
66
67The build and release process for these project releases is similar to
68that in `Day to Day Development <#test-daily-devel>`__, in that the
69a-full target of the Autobuilder is used but in addition the form is
70configured to generate and publish artefacts and the milestone number,
71version, release candidate number and other information is entered. The
72box to "generate an email to QA"is also checked.
73
74When the build completes, an email is sent out using the send-qa-email
75script in the ``yocto-autobuilder-helper`` repository to the list of
76people configured for that release. Release builds are placed into a
77directory in https://autobuilder.yocto.io/pub/releases on the
78Autobuilder which is included in the email. The process from here is
79more manual and control is effectively passed to release engineering.
80The next steps include:
81
82- QA teams respond to the email saying which tests they plan to run and
83 when the results will be available.
84
85- QA teams run their tests and share their results in the yocto-
86 testresults-contrib repository, along with a summary of their
87 findings.
88
89- Release engineering prepare the release as per their process.
90
91- Test results from the QA teams are included into the release in
92 separate directories and also uploaded to the yocto-testresults
93 repository alongside the other test results for the given revision.
94
95- The QA report in the final release is regenerated using resulttool to
96 include the new test results and the test summaries from the teams
97 (as headers to the generated report).
98
99- The release is checked against the release checklist and release
100 readiness criteria.
101
102- A final decision on whether to release is made by the YP TSC who have
103 final oversight on release readiness.
diff --git a/documentation/test-manual/test-manual-test-process.xml b/documentation/test-manual/test-manual-test-process.xml
deleted file mode 100644
index 6e2157c621..0000000000
--- a/documentation/test-manual/test-manual-test-process.xml
+++ /dev/null
@@ -1,110 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='test-manual-test-process'>
7
8<title>Project Testing and Release Process</title>
9 <section id='test-daily-devel'>
10 <title>Day to Day Development</title>
11
12 <para>This section details how the project tests changes, through automation on the
13 Autobuilder or with the assistance of QA teams, through to making releases.</para>
14
15 <para>The project aims to test changes against our test matrix before those changes are
16 merged into the master branch. As such, changes are queued up in batches either in the
17 <filename>master-next</filename> branch in the main trees, or in user trees such as
18 <filename>ross/mut</filename> in <filename>poky-contrib</filename> (Ross Burton
19 helps review and test patches and this is his testing tree).</para>
20 <para>We have two broad categories of test builds, including "full" and "quick". On the
21 Autobuilder, these can be seen as "a-quick" and "a-full", simply for ease of sorting in
22 the UI. Use our Autobuilder console view to see where me manage most test-related items,
23 available at: <link linkend=""
24 >https://autobuilder.yoctoproject.org/typhoon/#/console</link>.</para>
25 <para>Builds are triggered manually when the test branches are ready. The builds are
26 monitored by the SWAT team. For additional information, see <link linkend=""
27 >https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team</link>. If
28 successful, the changes would usually be merged to the <filename>master</filename>
29 branch. If not successful, someone would respond to the changes on the mailing list
30 explaining that there was a failure in testing. The choice of quick or full would depend
31 on the type of changes and the speed with which the result was required.</para>
32 <para>The Autobuilder does build the <filename>master</filename> branch once daily for
33 several reasons, in particular, to ensure the current <filename>master</filename> branch
34 does build, but also to keep <filename>yocto-testresults</filename> (<link linkend=""
35 >http://git.yoctoproject.org/cgit.cgi/yocto-testresults/</link>), buildhistory
36 (<link linkend="">http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/</link>),
37 and our sstate up to date. On the weekend, there is a master-next build instead to
38 ensure the test results are updated for the less frequently run targets.</para>
39 <para>Performance builds (buildperf-* targets in the console) are triggered separately every
40 six hours and automatically push their results to the buildstats repository at: <link
41 linkend="">http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/</link>. </para>
42 <para>The 'quick' targets have been selected to be the ones which catch the most failures or
43 give the most valuable data. We run 'fast' ptests in this case for example but not the
44 ones which take a long time. The quick target doesn't include *-lsb builds for all
45 architectures, some world builds and doesn't trigger performance tests or ltp testing.
46 The full build includes all these things and is slower but more comprehensive.</para>
47 </section>
48
49 <section id='test-yocto-project-autobuilder-overview'>
50 <title>Release Builds</title>
51
52 <para>The project typically has two major releases a year with a six month cadence in April
53 and October. Between these there would be a number of milestone releases (usually four)
54 with the final one being stablization only along with point releases of our stable
55 branches.</para>
56 <para>The build and release process for these project releases is similar to that in <link
57 linkend="test-daily-devel">Day to Day Development</link>, in that the a-full target
58 of the Autobuilder is used but in addition the form is configured to generate and
59 publish artefacts and the milestone number, version, release candidate number and other
60 information is entered. The box to "generate an email to QA"is also checked.</para>
61 <para>When the build completes, an email is sent out using the send-qa-email script in the
62 <filename>yocto-autobuilder-helper</filename> repository to the list of people
63 configured for that release. Release builds are placed into a directory in <link
64 linkend="">https://autobuilder.yocto.io/pub/releases</link> on the Autobuilder which
65 is included in the email. The process from here is more manual and control is
66 effectively passed to release engineering. The next steps include:<itemizedlist>
67 <listitem>
68 <para>QA teams respond to the email saying which tests they plan to run and when
69 the results will be available.</para>
70 </listitem>
71 <listitem>
72 <para>QA teams run their tests and share their results in the yocto-
73 testresults-contrib repository, along with a summary of their findings.
74 </para>
75 </listitem>
76 <listitem>
77 <para>Release engineering prepare the release as per their process. </para>
78 </listitem>
79 <listitem>
80 <para>Test results from the QA teams are included into the release in separate
81 directories and also uploaded to the yocto-testresults repository alongside
82 the other test results for the given revision.</para>
83 </listitem>
84 <listitem>
85 <para>The QA report in the final release is regenerated using resulttool to
86 include the new test results and the test summaries from the teams (as
87 headers to the generated report).</para>
88 </listitem>
89 <listitem>
90 <para>The release is checked against the release checklist and release readiness
91 criteria.</para>
92 </listitem>
93 <listitem>
94 <para>A final decision on whether to release is made by the YP TSC who have
95 final oversight on release readiness.</para>
96 </listitem>
97 </itemizedlist></para>
98 </section>
99
100
101
102
103
104
105
106
107</chapter>
108<!--
109vim: expandtab tw=80 ts=4
110-->
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst
deleted file mode 100644
index 2fcae5000e..0000000000
--- a/documentation/test-manual/test-manual-understand-autobuilder.rst
+++ /dev/null
@@ -1,305 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-2.0-UK
2
3*******************************************
4Understanding the Yocto Project Autobuilder
5*******************************************
6
7Execution Flow within the Autobuilder
8=====================================
9
10The "a-full" and "a-quick" targets are the usual entry points into the
11Autobuilder and it makes sense to follow the process through the system
12starting there. This is best visualised from the Autobuilder Console
13view (:yocto_ab:`/typhoon/#/console`).
14
15Each item along the top of that view represents some "target build" and
16these targets are all run in parallel. The 'full' build will trigger the
17majority of them, the "quick" build will trigger some subset of them.
18The Autobuilder effectively runs whichever configuration is defined for
19each of those targets on a seperate buildbot worker. To understand the
20configuration, you need to look at the entry on ``config.json`` file
21within the ``yocto-autobuilder-helper`` repository. The targets are
22defined in the ‘overrides' section, a quick example could be qemux86-64
23which looks like::
24
25 "qemux86-64" : {
26 "MACHINE" : "qemux86-64",
27 "TEMPLATE" : "arch-qemu",
28 "step1" : {
29 "extravars" : [
30 "IMAGE_FSTYPES_append = ' wic wic.bmap'"
31 ]
32 }
33 },
34
35And to expand that, you need the "arch-qemu" entry from
36the "templates" section, which looks like::
37
38 "arch-qemu" : {
39 "BUILDINFO" : true,
40 "BUILDHISTORY" : true,
41 "step1" : {
42 "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
43 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
44 },
45 "step2" : {
46 "SDKMACHINE" : "x86_64",
47 "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
48 "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
49 },
50 "step3" : {
51 "BUILDHISTORY" : false,
52 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
53 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
54 }
55 },
56
57Combining these two entries you can see that "qemux86-64" is a three step build where the
58``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
59``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step
601 an extra variable is added to the ``auto.conf`` file to enable wic
61image generation.
62
63While not every detail of this is covered here, you can see how the
64template mechanism allows quite complex configurations to be built up
65yet allows duplication and repetition to be kept to a minimum.
66
67The different build targets are designed to allow for parallelisation,
68so different machines are usually built in parallel, operations using
69the same machine and metadata are built sequentially, with the aim of
70trying to optimise build efficiency as much as possible.
71
72The ``config.json`` file is processed by the scripts in the Helper
73repository in the ``scripts`` directory. The following section details
74how this works.
75
76.. _test-autobuilder-target-exec-overview:
77
78Autobuilder Target Execution Overview
79=====================================
80
81For each given target in a build, the Autobuilder executes several
82steps. These are configured in ``yocto-autobuilder2/builders.py`` and
83roughly consist of:
84
85#. *Run clobberdir*.
86
87 This cleans out any previous build. Old builds are left around to
88 allow easier debugging of failed builds. For additional information,
89 see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`.
90
91#. *Obtain yocto-autobuilder-helper*
92
93 This step clones the ``yocto-autobuilder-helper`` git repository.
94 This is necessary to prevent the requirement to maintain all the
95 release or project-specific code within Buildbot. The branch chosen
96 matches the release being built so we can support older releases and
97 still make changes in newer ones.
98
99#. *Write layerinfo.json*
100
101 This transfers data in the Buildbot UI when the build was configured
102 to the Helper.
103
104#. *Call scripts/shared-repo-unpack*
105
106 This is a call into the Helper scripts to set up a checkout of all
107 the pieces this build might need. It might clone the BitBake
108 repository and the OpenEmbedded-Core repository. It may clone the
109 Poky repository, as well as additional layers. It will use the data
110 from the ``layerinfo.json`` file to help understand the
111 configuration. It will also use a local cache of repositories to
112 speed up the clone checkouts. For additional information, see
113 :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`.
114
115 This step has two possible modes of operation. If the build is part
116 of a parent build, its possible that all the repositories needed may
117 already be available, ready in a pre-prepared directory. An "a-quick"
118 or "a-full" build would prepare this before starting the other
119 sub-target builds. This is done for two reasons:
120
121 - the upstream may change during a build, for example, from a forced
122 push and this ensures we have matching content for the whole build
123
124 - if 15 Workers all tried to pull the same data from the same repos,
125 we can hit resource limits on upstream servers as they can think
126 they are under some kind of network attack
127
128 This pre-prepared directory is shared among the Workers over NFS. If
129 the build is an individual build and there is no "shared" directory
130 available, it would clone from the cache and the upstreams as
131 necessary. This is considered the fallback mode.
132
133#. *Call scripts/run-config*
134
135 This is another call into the Helper scripts where its expected that
136 the main functionality of this target will be executed.
137
138.. _test-autobuilder-tech:
139
140Autobuilder Technology
141======================
142
143The Autobuilder has Yocto Project-specific functionality to allow builds
144to operate with increased efficiency and speed.
145
146.. _test-clobberdir:
147
148clobberdir
149----------
150
151When deleting files, the Autobuilder uses ``clobberdir``, which is a
152special script that moves files to a special location, rather than
153deleting them. Files in this location are deleted by an ``rm`` command,
154which is run under ``ionice -c 3``. For example, the deletion only
155happens when there is idle IO capacity on the Worker. The Autobuilder
156Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
157
158.. _test-autobuilder-clone-cache:
159
160Autobuilder Clone Cache
161-----------------------
162
163Cloning repositories from scratch each time they are required was slow
164on the Autobuilder. We therefore have a stash of commonly used
165repositories pre-cloned on the Workers. Data is fetched from these
166during clones first, then "topped up" with later revisions from any
167upstream when necesary. The cache is maintained by the Autobuilder
168Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
169
170.. _test-autobuilder-worker-janitor:
171
172Autobuilder Worker Janitor
173--------------------------
174
175This is a process running on each Worker that performs two basic
176operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and
177maintainenance of a cache of cloned repositories to improve the speed
178the system can checkout repositories.
179
180.. _test-shared-dl-dir:
181
182Shared DL_DIR
183-------------
184
185The Workers are all connected over NFS which allows DL_DIR to be shared
186between them. This reduces network accesses from the system and allows
187the build to be sped up. Usage of the directory within the build system
188is designed to be able to be shared over NFS.
189
190.. _test-shared-sstate-cache:
191
192Shared SSTATE_DIR
193-----------------
194
195The Workers are all connected over NFS which allows the ``sstate``
196directory to be shared between them. This means once a Worker has built
197an artifact, all the others can benefit from it. Usage of the directory
198within the directory is designed for sharing over NFS.
199
200.. _test-resulttool:
201
202Resulttool
203----------
204
205All of the different tests run as part of the build generate output into
206``testresults.json`` files. This allows us to determine which tests ran
207in a given build and their status. Additional information, such as
208failure logs or the time taken to run the tests, may also be included.
209
210Resulttool is part of OpenEmbedded-Core and is used to manipulate these
211json results files. It has the ability to merge files together, display
212reports of the test results and compare different result files.
213
214For details, see :yocto_wiki:`/wiki/Resulttool`.
215
216.. _test-run-config-tgt-execution:
217
218run-config Target Execution
219===========================
220
221The ``scripts/run-config`` execution is where most of the work within
222the Autobuilder happens. It runs through a number of steps; the first
223are general setup steps that are run once and include:
224
225#. Set up any ``buildtools-tarball`` if configured.
226
227#. Call "buildhistory-init" if buildhistory is configured.
228
229For each step that is configured in ``config.json``, it will perform the
230following:
231
232#. Add any layers that are specified using the
233 ``bitbake-layers add-layer`` command (logging as stepXa)
234
235#. Call the ``scripts/setup-config`` script to generate the necessary
236 ``auto.conf`` configuration file for the build
237
238#. Run the ``bitbake BBTARGETS`` command (logging as stepXb)
239
240#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc)
241
242#. Run the ``EXTRACMDS`` command, which are run within the BitBake build
243 environment (logging as stepXd)
244
245#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
246 BitBake build environment (logging as stepXd)
247
248#. Remove any layers added in step
249 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
250
251Once the execution steps above complete, ``run-config`` executes a set
252of post-build steps, including:
253
254#. Call ``scripts/publish-artifacts`` to collect any output which is to
255 be saved from the build.
256
257#. Call ``scripts/collect-results`` to collect any test results to be
258 saved from the build.
259
260#. Call ``scripts/upload-error-reports`` to send any error reports
261 generated to the remote server.
262
263#. Cleanup the build directory using
264 :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful,
265 else rename it to "build-renamed" for potential future debugging.
266
267.. _test-deploying-yp-autobuilder:
268
269Deploying Yocto Autobuilder
270===========================
271
272The most up to date information about how to setup and deploy your own
273Autbuilder can be found in README.md in the ``yocto-autobuilder2``
274repository.
275
276We hope that people can use the ``yocto-autobuilder2`` code directly but
277it is inevitable that users will end up needing to heavily customise the
278``yocto-autobuilder-helper`` repository, particularly the
279``config.json`` file as they will want to define their own test matrix.
280
281The Autobuilder supports wo customization options:
282
283- variable substitution
284
285- overlaying configuration files
286
287The standard ``config.json`` minimally attempts to allow substitution of
288the paths. The Helper script repository includes a
289``local-example.json`` file to show how you could override these from a
290separate configuration file. Pass the following into the environment of
291the Autobuilder::
292
293 $ ABHELPER_JSON="config.json local-example.json"
294
295As another example, you could also pass the following into the
296environment::
297
298 $ ABHELPER_JSON="config.json /some/location/local.json"
299
300One issue users often run into is validation of the ``config.json`` files. A
301tip for minimizing issues from invalid json files is to use a Git
302``pre-commit-hook.sh`` script to verify the JSON file before committing
303it. Create a symbolic link as follows::
304
305 $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.xml b/documentation/test-manual/test-manual-understand-autobuilder.xml
deleted file mode 100644
index 8600367be7..0000000000
--- a/documentation/test-manual/test-manual-understand-autobuilder.xml
+++ /dev/null
@@ -1,314 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='test-manual-understand-autobuilder'>
7
8<title>Understanding the Yocto Project Autobuilder</title>
9 <section>
10 <title>Execution Flow within the Autobuilder</title>
11 <para>The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and
12 it makes sense to follow the process through the system starting there. This is best
13 visualised from the Autobuilder Console view (<link linkend=""
14 >https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para>
15 <para>Each item along the top of that view represents some "target build" and these targets
16 are all run in parallel. The 'full' build will trigger the majority of them, the "quick"
17 build will trigger some subset of them. The Autobuilder effectively runs whichever
18 configuration is defined for each of those targets on a seperate buildbot worker. To
19 understand the configuration, you need to look at the entry on
20 <filename>config.json</filename> file within the
21 <filename>yocto-autobuilder-helper</filename> repository. The targets are defined in
22 the ‘overrides' section, a quick example could be qemux86-64 which looks
23 like:<literallayout class="monospaced">
24 "qemux86-64" : {
25 "MACHINE" : "qemux86-64",
26 "TEMPLATE" : "arch-qemu",
27 "step1" : {
28 "extravars" : [
29 "IMAGE_FSTYPES_append = ' wic wic.bmap'"
30 ]
31 }
32 },
33 </literallayout>And
34 to expand that, you need the "arch-qemu" entry from the "templates" section, which looks
35 like:<literallayout class="monospaced">
36 "arch-qemu" : {
37 "BUILDINFO" : true,
38 "BUILDHISTORY" : true,
39 "step1" : {
40 "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
41 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
42 },
43 "step2" : {
44 "SDKMACHINE" : "x86_64",
45 "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
46 "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
47 },
48 "step3" : {
49 "BUILDHISTORY" : false,
50 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
51 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
52 }
53 },
54 </literallayout>Combining
55 these two entries you can see that "qemux86-64" is a three step build where the
56 <filename>bitbake BBTARGETS</filename> would be run, then <filename>bitbake
57 SANITYTARGETS</filename> for each step; all for
58 <filename>MACHINE="qemx86-64"</filename> but with differing SDKMACHINE settings. In
59 step 1 an extra variable is added to the <filename>auto.conf</filename> file to enable
60 wic image generation.</para>
61 <para>While not every detail of this is covered here, you can see how the templating
62 mechanism allows quite complex configurations to be built up yet allows duplication and
63 repetition to be kept to a minimum.</para>
64 <para>The different build targets are designed to allow for parallelisation, so different
65 machines are usually built in parallel, operations using the same machine and metadata
66 are built sequentially, with the aim of trying to optimise build efficiency as much as
67 possible.</para>
68 <para>The <filename>config.json</filename> file is processed by the scripts in the Helper
69 repository in the <filename>scripts</filename> directory. The following section details
70 how this works.</para>
71 </section>
72
73 <section id='test-autobuilder-target-exec-overview'>
74 <title>Autobuilder Target Execution Overview</title>
75
76 <para>For each given target in a build, the Autobuilder executes several steps. These are
77 configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist
78 of: <orderedlist>
79 <listitem id='test-list-tgt-exec-clobberdir'>
80 <para><emphasis>Run <filename>clobberdir</filename></emphasis></para>
81 <para>This cleans out any previous build. Old builds are left around to allow
82 easier debugging of failed builds. For additional information, see <link
83 linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para>
84 </listitem>
85 <listitem>
86 <para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para>
87 <para>This step clones the <filename>yocto-autobuilder-helper</filename> git
88 repository. This is necessary to prevent the requirement to maintain all the
89 release or project-specific code within Buildbot. The branch chosen matches
90 the release being built so we can support older releases and still make
91 changes in newer ones.</para>
92 </listitem>
93 <listitem>
94 <para><emphasis>Write layerinfo.json</emphasis></para>
95 <para>This transfers data in the Buildbot UI when the build was configured to
96 the Helper.</para>
97 </listitem>
98 <listitem>
99 <para><emphasis>Call scripts/shared-repo-unpack</emphasis></para>
100 <para>This is a call into the Helper scripts to set up a checkout of all the
101 pieces this build might need. It might clone the BitBake repository and the
102 OpenEmbedded-Core repository. It may clone the Poky repository, as well as
103 additional layers. It will use the data from the
104 <filename>layerinfo.json</filename> file to help understand the
105 configuration. It will also use a local cache of repositories to speed up
106 the clone checkouts. For additional information, see <link
107 linkend="test-autobuilder-clone-cache">Autobuilder Clone
108 Cache</link>.</para>
109 <para>This step has two possible modes of operation. If the build is part of a
110 parent build, its possible that all the repositories needed may already be
111 available, ready in a pre-prepared directory. An "a-quick" or "a-full" build
112 would prepare this before starting the other sub-target builds. This is done
113 for two reasons:<itemizedlist>
114 <listitem>
115 <para>the upstream may change during a build, for example, from a
116 forced push and this ensures we have matching content for the
117 whole build</para>
118 </listitem>
119 <listitem>
120 <para>if 15 Workers all tried to pull the same data from the same
121 repos, we can hit resource limits on upstream servers as they
122 can think they are under some kind of network attack</para>
123 </listitem>
124 </itemizedlist>This pre-prepared directory is shared among the Workers over
125 NFS. If the build is an individual build and there is no "shared" directory
126 available, it would clone from the cache and the upstreams as necessary.
127 This is considered the fallback mode.</para>
128 </listitem>
129 <listitem>
130 <para><emphasis>Call scripts/run-config</emphasis></para>
131 <para>This is another call into the Helper scripts where its expected that the
132 main functionality of this target will be executed.</para>
133 </listitem>
134 </orderedlist></para>
135 </section>
136 <section id='test-autobuilder-tech'>
137 <title>Autobuilder Technology</title>
138 <para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate
139 with increased efficiency and speed.</para>
140 <section id='test-clobberdir'>
141 <title>clobberdir</title>
142 <para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which
143 is a special script that moves files to a special location, rather than deleting
144 them. Files in this location are deleted by an <filename>rm</filename> command,
145 which is run under <filename>ionice -c 3</filename>. For example, the deletion only
146 happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor
147 runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder
148 Worker Janitor</link>.</para>
149 </section>
150 <section id='test-autobuilder-clone-cache'>
151 <title>Autobuilder Clone Cache</title>
152 <para>Cloning repositories from scratch each time they are required was slow on the
153 Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on
154 the Workers. Data is fetched from these during clones first, then "topped up" with
155 later revisions from any upstream when necesary. The cache is maintained by the
156 Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor"
157 >Autobuilder Worker Janitor</link>.</para>
158 </section>
159 <section id='test-autobuilder-worker-janitor'>
160 <title>Autobuilder Worker Janitor</title>
161 <para>This is a process running on each Worker that performs two basic operations,
162 including background file deletion at IO idle (see <link
163 linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and
164 maintainenance of a cache of cloned repositories to improve the speed the system can
165 checkout repositories.</para>
166 </section>
167 <section id='test-shared-dl-dir'>
168 <title>Shared DL_DIR</title>
169 <para>The Workers are all connected over NFS which allows DL_DIR to be shared between
170 them. This reduces network accesses from the system and allows the build to be sped
171 up. Usage of the directory within the build system is designed to be able to be
172 shared over NFS.</para>
173 </section>
174 <section id='test-shared-sstate-cache'>
175 <title>Shared SSTATE_DIR</title>
176 <para>The Workers are all connected over NFS which allows the
177 <filename>sstate</filename> directory to be shared between them. This means once
178 a Worker has built an artefact, all the others can benefit from it. Usage of the
179 directory within the directory is designed for sharing over NFS.</para>
180 </section>
181 <section id='test-resulttool'>
182 <title>Resulttool</title>
183 <para>All of the different tests run as part of the build generate output into
184 <filename>testresults.json</filename> files. This allows us to determine which
185 tests ran in a given build and their status. Additional information, such as failure
186 logs or the time taken to run the tests, may also be included.</para>
187 <para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json
188 results files. It has the ability to merge files together, display reports of the
189 test results and compare different result files.</para>
190 <para>For details, see <link linkend=""
191 >https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para>
192 </section>
193 </section>
194 <section id='test-run-config-tgt-execution'>
195 <title>run-config Target Execution</title>
196 <para>The <filename>scripts/run-config</filename> execution is where most of the work within
197 the Autobuilder happens. It runs through a number of steps; the first are general setup
198 steps that are run once and include:<orderedlist>
199 <listitem>
200 <para>Set up any <filename>buildtools-tarball</filename> if configured.</para>
201 </listitem>
202 <listitem>
203 <para>Call "buildhistory-init" if buildhistory is configured.</para>
204 </listitem>
205 </orderedlist></para>
206 <para>For each step that is configured in <filename>config.json</filename>, it will perform
207 the following:</para>
208 <para>
209 <remark>## WRITER's question: What does "logging in as stepXa" and others refer to
210 below? ##</remark>
211 <orderedlist>
212 <listitem id="test-run-config-add-layers-step">
213 <para dir="ltr">Add any layers that are specified using the
214 <filename>bitbake-layers add-layer</filename> command (logging as
215 stepXa)</para>
216 </listitem>
217 <listitem>
218 <para dir="ltr">Call the <filename>scripts/setup-config</filename> script to
219 generate the necessary <filename>auto.conf</filename> configuration file for
220 the build</para>
221 </listitem>
222 <listitem>
223 <para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging
224 as stepXb)</para>
225 </listitem>
226 <listitem>
227 <para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command
228 (logging as stepXc)</para>
229 </listitem>
230 <listitem>
231 <para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run
232 within the BitBake build environment (logging as stepXd)</para>
233 </listitem>
234 <listitem>
235 <para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which
236 are run outside the BitBake build environment (logging as stepXd)</para>
237 </listitem>
238 <listitem>
239 <para dir="ltr">Remove any layers added in <link
240 linkend="test-run-config-add-layers-step">step 1</link> using the
241 <filename>bitbake-layers remove-layer</filename> command (logging as
242 stepXa)</para>
243 </listitem>
244 </orderedlist>
245 </para>
246 <para>Once the execution steps above complete, <filename>run-config</filename> executes a
247 set of post-build steps, including:<orderedlist>
248 <listitem>
249 <para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect
250 any output which is to be saved from the build.</para>
251 </listitem>
252 <listitem>
253 <para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any
254 test results to be saved from the build.</para>
255 </listitem>
256 <listitem>
257 <para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send
258 any error reports generated to the remote server.</para>
259 </listitem>
260 <listitem>
261 <para dir="ltr">Cleanup the build directory using <link
262 linkend="test-clobberdir"><filename>clobberdir</filename></link> if the
263 build was successful, else rename it to "build-renamed" for potential future
264 debugging.</para>
265 </listitem>
266 </orderedlist></para>
267 </section>
268 <section id='test-deploying-yp-autobuilder'>
269 <title>Deploying Yocto Autobuilder</title>
270 <para>The most up to date information about how to setup and deploy your own Autbuilder can
271 be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para>
272 <para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly
273 but it is inevitable that users will end up needing to heavily customise the
274 <filename>yocto-autobuilder-helper</filename> repository, particularly the
275 <filename>config.json</filename> file as they will want to define their own test
276 matrix.</para>
277 <para>The Autobuilder supports wo customization options: <itemizedlist>
278 <listitem>
279 <para>variable substitution</para>
280 </listitem>
281 <listitem>
282 <para>overlaying configuration files</para>
283 </listitem>
284 </itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow
285 substitution of the paths. The Helper script repository includes a
286 <filename>local-example.json</filename> file to show how you could override these
287 from a separate configuration file. Pass the following into the environment of the
288 Autobuilder:<literallayout class="monospaced">
289 $ ABHELPER_JSON="config.json local-example.json"
290 </literallayout>As
291 another example, you could also pass the following into the
292 environment:<literallayout class="monospaced">
293 $ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json"
294 </literallayout>One
295 issue users often run into is validation of the <filename>config.json</filename> files.
296 A tip for minimizing issues from invalid json files is to use a Git
297 <filename>pre-commit-hook.sh</filename> script to verify the JSON file before
298 committing it. Create a symbolic link as
299 follows:<literallayout class="monospaced">
300 $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
301 </literallayout></para>
302 </section>
303
304
305
306
307
308
309
310
311</chapter>
312<!--
313vim: expandtab tw=80 ts=4
314-->
diff --git a/documentation/test-manual/test-manual.rst b/documentation/test-manual/test-manual.rst
deleted file mode 100644
index bd5b1b0968..0000000000
--- a/documentation/test-manual/test-manual.rst
+++ /dev/null
@@ -1,18 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-2.0-UK
2
3=====================================
4Yocto Project Test Environment Manual
5=====================================
6
7|
8
9.. toctree::
10 :caption: Table of Contents
11 :numbered:
12
13 test-manual-intro
14 test-manual-test-process
15 test-manual-understand-autobuilder
16 history
17
18.. include:: /boilerplate.rst
diff --git a/documentation/test-manual/test-manual.xml b/documentation/test-manual/test-manual.xml
deleted file mode 100755
index d454566c5f..0000000000
--- a/documentation/test-manual/test-manual.xml
+++ /dev/null
@@ -1,104 +0,0 @@
1<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<book id='test-manual' lang='en'
7 xmlns:xi="http://www.w3.org/2003/XInclude"
8 xmlns="http://docbook.org/ns/docbook"
9 >
10 <bookinfo>
11
12 <mediaobject>
13 <imageobject>
14 <imagedata fileref='figures/test-manual-title.png'
15 format='SVG'
16 align='left' scalefit='1' width='100%'/>
17 </imageobject>
18 </mediaobject>
19
20 <title>
21 Yocto Project Test Environment Manual
22 </title>
23
24 <authorgroup>
25 <author>
26 <affiliation>
27 <orgname>&ORGNAME;</orgname>
28 </affiliation>
29 <email>&ORGEMAIL;</email>
30 </author>
31 </authorgroup>
32
33 <revhistory>
34 <revision>
35 <revnumber>3.1.1</revnumber>
36 <date>TBD</date>
37 <revremark>DRAFT - Work-in-Progress - posted June 16, 2020</revremark>
38 </revision>
39 </revhistory>
40
41 <copyright>
42 <year>&COPYRIGHT_YEAR;</year>
43 <holder>Linux Foundation</holder>
44 </copyright>
45
46 <legalnotice>
47 <para>
48 Permission is granted to copy, distribute and/or modify this document under
49 the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">
50 Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by
51 Creative Commons.
52 </para>
53 <note><title>Manual Notes</title>
54 <itemizedlist>
55 <listitem><para>
56 This version of the
57 <emphasis>Yocto Project Test Environment Manual</emphasis>
58 is for the &YOCTO_DOC_VERSION; release of the
59 Yocto Project.
60 To be sure you have the latest version of the manual
61 for this release, go to the
62 <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
63 and select the manual from that site.
64 Manuals from the site are more up-to-date than manuals
65 derived from the Yocto Project released TAR files.
66 </para></listitem>
67 <listitem><para>
68 If you located this manual through a web search, the
69 version of the manual might not be the one you want
70 (e.g. the search might have returned a manual much
71 older than the Yocto Project version with which you
72 are working).
73 You can see all Yocto Project major releases by
74 visiting the
75 <ulink url='&YOCTO_WIKI_URL;/wiki/Releases'>Releases</ulink>
76 page.
77 If you need a version of this manual for a different
78 Yocto Project release, visit the
79 <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
80 and select the manual set by using the
81 "ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARCHIVE"
82 pull-down menus.
83 </para></listitem>
84 <listitem><para>
85 To report any inaccuracies or problems with this
86 manual, send an email to the Yocto Project
87 discussion group at
88 <filename>yocto@yoctoproject.com</filename> or log into
89 the freenode <filename>#yocto</filename> channel.
90 </para></listitem>
91 </itemizedlist>
92 </note>
93 </legalnotice>
94
95 </bookinfo>
96
97 <xi:include href="test-manual-intro.xml"/>
98 <xi:include href="test-manual-test-process.xml"/>
99 <xi:include href="test-manual-understand-autobuilder.xml"/>
100
101</book>
102<!--
103vim: expandtab tw=80 ts=4
104-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-04-30 12:58:31 +0000