summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMichael Opdenacker <michael.opdenacker@bootlin.com>2023-12-06 16:36:00 +0100
committerRichard Purdie <richard.purdie@linuxfoundation.org>2023-12-18 13:18:55 +0000
commit84b242ef8bc103e49d83dcc25d968ab59524e1f3 (patch)
tree0fd504622c840757a2995a158ebc422605a8d5e9
parentc24e3b9f676ea68c100154f1aa865430ed2c99f1 (diff)
downloadpoky-84b242ef8bc103e49d83dcc25d968ab59524e1f3.tar.gz
test-manual: text and formatting fixes
(From yocto-docs rev: 7c4f616f773bb9071b395e977b2ca9f8ac8f7e2a) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-rw-r--r--documentation/test-manual/intro.rst44
-rw-r--r--documentation/test-manual/test-process.rst12
-rw-r--r--documentation/test-manual/understand-autobuilder.rst33
3 files changed, 45 insertions, 44 deletions
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst
index aaf64ae017..f27210642c 100644
--- a/documentation/test-manual/intro.rst
+++ b/documentation/test-manual/intro.rst
@@ -14,7 +14,7 @@ release works as intended. All the project's testing infrastructure and
14processes are publicly visible and available so that the community can 14processes are publicly visible and available so that the community can
15see what testing is being performed, how it's being done and the current 15see what testing is being performed, how it's being done and the current
16status of the tests and the project at any given time. It is intended 16status of the tests and the project at any given time. It is intended
17that Other organizations can leverage off the process and testing 17that other organizations can leverage off the process and testing
18environment used by the Yocto Project to create their own automated, 18environment used by the Yocto Project to create their own automated,
19production test environment, building upon the foundations from the 19production test environment, building upon the foundations from the
20project core. 20project core.
@@ -93,8 +93,8 @@ the following types of tests:
93- *Build Testing:* Tests whether specific configurations build by 93- *Build Testing:* Tests whether specific configurations build by
94 varying :term:`MACHINE`, 94 varying :term:`MACHINE`,
95 :term:`DISTRO`, other configuration 95 :term:`DISTRO`, other configuration
96 options, and the specific target images being built (or world). Used 96 options, and the specific target images being built (or ``world``). This is
97 to trigger builds of all the different test configurations on the 97 used to trigger builds of all the different test configurations on the
98 Autobuilder. Builds usually cover many different targets for 98 Autobuilder. Builds usually cover many different targets for
99 different architectures, machines, and distributions, as well as 99 different architectures, machines, and distributions, as well as
100 different configurations, such as different init systems. The 100 different configurations, such as different init systems. The
@@ -120,7 +120,7 @@ the following types of tests:
120 120
121 $ bitbake image -c testsdkext 121 $ bitbake image -c testsdkext
122 122
123 The tests utilize the :ref:`ref-classes-testsdk` class and the 123 The tests use the :ref:`ref-classes-testsdk` class and the
124 ``do_testsdkext`` task. 124 ``do_testsdkext`` task.
125 125
126- *Feature Testing:* Various scenario-based tests are run through the 126- *Feature Testing:* Various scenario-based tests are run through the
@@ -131,7 +131,7 @@ the following types of tests:
131 131
132 $ bitbake image -c testimage 132 $ bitbake image -c testimage
133 133
134 The tests utilize the :ref:`ref-classes-testimage` 134 The tests use the :ref:`ref-classes-testimage`
135 class and the :ref:`ref-tasks-testimage` task. 135 class and the :ref:`ref-tasks-testimage` task.
136 136
137- *Layer Testing:* The Autobuilder has the possibility to test whether 137- *Layer Testing:* The Autobuilder has the possibility to test whether
@@ -151,7 +151,7 @@ the following types of tests:
151 151
152 $ bitbake image -c testsdk 152 $ bitbake image -c testsdk
153 153
154 The tests utilize the :ref:`ref-classes-testsdk` class and 154 The tests use the :ref:`ref-classes-testsdk` class and
155 the ``do_testsdk`` task. 155 the ``do_testsdk`` task.
156 156
157- *Unit Testing:* Unit tests on various components of the system run 157- *Unit Testing:* Unit tests on various components of the system run
@@ -190,7 +190,7 @@ Tests map into the codebase as follows:
190 $ bitbake-selftest -v 190 $ bitbake-selftest -v
191 191
192 To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` 192 To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS``
193 variable when running "bitbake-selftest" as follows:: 193 variable when running ``bitbake-selftest`` as follows::
194 194
195 $ BB_SKIP_NETTESTS=yes bitbake-selftest 195 $ BB_SKIP_NETTESTS=yes bitbake-selftest
196 196
@@ -215,7 +215,7 @@ Tests map into the codebase as follows:
215 - These tests use OE to test the workflows, which include testing 215 - These tests use OE to test the workflows, which include testing
216 specific features, behaviors of tasks, and API unit tests. 216 specific features, behaviors of tasks, and API unit tests.
217 217
218 - The tests can take advantage of parallelism through the "-j" 218 - The tests can take advantage of parallelism through the ``-j``
219 option, which can specify a number of threads to spread the tests 219 option, which can specify a number of threads to spread the tests
220 across. Note that all tests from a given class of tests will run 220 across. Note that all tests from a given class of tests will run
221 in the same thread. To parallelize large numbers of tests you can 221 in the same thread. To parallelize large numbers of tests you can
@@ -231,18 +231,18 @@ Tests map into the codebase as follows:
231 $ oe-selftest -a 231 $ oe-selftest -a
232 232
233 - To run a specific test, use the following command form where 233 - To run a specific test, use the following command form where
234 testname is the name of the specific test:: 234 ``testname`` is the name of the specific test::
235 235
236 $ oe-selftest -r <testname> 236 $ oe-selftest -r <testname>
237 237
238 For example, the following command would run the tinfoil 238 For example, the following command would run the ``tinfoil``
239 getVar API test:: 239 ``getVar`` API test::
240 240
241 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar 241 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
242 242
243 It is also possible to run a set 243 It is also possible to run a set
244 of tests. For example the following command will run all of the 244 of tests. For example the following command will run all of the
245 tinfoil tests:: 245 ``tinfoil`` tests::
246 246
247 $ oe-selftest -r tinfoil 247 $ oe-selftest -r tinfoil
248 248
@@ -391,14 +391,14 @@ so tests within a given test class should always run in the same build,
391while tests in different classes or modules may be split into different 391while tests in different classes or modules may be split into different
392builds. There is no data store available for these tests since the tests 392builds. There is no data store available for these tests since the tests
393launch the ``bitbake`` command and exist outside of its context. As a 393launch the ``bitbake`` command and exist outside of its context. As a
394result, common bitbake library functions (bb.\*) are also unavailable. 394result, common BitBake library functions (``bb.\*``) are also unavailable.
395 395
396``testimage`` 396``testimage``
397------------- 397-------------
398 398
399These tests are run once an image is up and running, either on target 399These tests are run once an image is up and running, either on target
400hardware or under QEMU. As a result, they are assumed to be running in a 400hardware or under QEMU. As a result, they are assumed to be running in a
401target image environment, as opposed to a host build environment. A 401target image environment, as opposed to in a host build environment. A
402simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains 402simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
403the following:: 403the following::
404 404
@@ -413,19 +413,19 @@ the following::
413 413
414In this example, the ``OERuntimeTestCase`` class wraps 414In this example, the ``OERuntimeTestCase`` class wraps
415``unittest.TestCase``. Within the test, ``self.target`` represents the 415``unittest.TestCase``. Within the test, ``self.target`` represents the
416target system, where commands can be run on it using the ``run()`` 416target system, where commands can be run using the ``run()``
417method. 417method.
418 418
419To ensure certain test or package dependencies are met, you can use the 419To ensure certain tests or package dependencies are met, you can use the
420``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test 420``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
421in this example would only make sense if python3-core is installed in 421in this example would only make sense if ``python3-core`` is installed in
422the image. 422the image.
423 423
424``testsdk_ext`` 424``testsdk_ext``
425--------------- 425---------------
426 426
427These tests are run against built extensible SDKs (eSDKs). The tests can 427These tests are run against built extensible SDKs (eSDKs). The tests can
428assume that the eSDK environment has already been setup. An example from 428assume that the eSDK environment has already been set up. An example from
429``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: 429``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
430 430
431 class DevtoolTest(OESDKExtTestCase): 431 class DevtoolTest(OESDKExtTestCase):
@@ -472,9 +472,9 @@ following::
472 output = self._run(cmd) 472 output = self._run(cmd)
473 self.assertEqual(output, "Hello, world\n") 473 self.assertEqual(output, "Hello, world\n")
474 474
475In this example, if nativesdk-python3-core has been installed into the SDK, the code runs 475In this example, if ``nativesdk-python3-core`` has been installed into the SDK,
476the python3 interpreter with a basic command to check it is working 476the code runs the ``python3`` interpreter with a basic command to check it is
477correctly. The test would only run if Python3 is installed in the SDK. 477working correctly. The test would only run if Python3 is installed in the SDK.
478 478
479``oe-build-perf-test`` 479``oe-build-perf-test``
480---------------------- 480----------------------
@@ -520,7 +520,7 @@ an isolated directory.
520 520
521This can delete files from :term:`SSTATE_DIR` which would potentially break 521This can delete files from :term:`SSTATE_DIR` which would potentially break
522other builds running in parallel. If this is required, :term:`SSTATE_DIR` must 522other builds running in parallel. If this is required, :term:`SSTATE_DIR` must
523be set to an isolated directory. Alternatively, you can use the "-f" 523be set to an isolated directory. Alternatively, you can use the ``-f``
524option with the ``bitbake`` command to "taint" tasks by changing the 524option with the ``bitbake`` command to "taint" tasks by changing the
525sstate checksums to ensure sstate cache items will not be reused. 525sstate checksums to ensure sstate cache items will not be reused.
526 526
diff --git a/documentation/test-manual/test-process.rst b/documentation/test-manual/test-process.rst
index 4c3b32bfea..0c560efe39 100644
--- a/documentation/test-manual/test-process.rst
+++ b/documentation/test-manual/test-process.rst
@@ -42,16 +42,16 @@ our sstate up to date. On the weekend, there is a master-next build
42instead to ensure the test results are updated for the less frequently 42instead to ensure the test results are updated for the less frequently
43run targets. 43run targets.
44 44
45Performance builds (buildperf-\* targets in the console) are triggered 45Performance builds (``buildperf-\*`` targets in the console) are triggered
46separately every six hours and automatically push their results to the 46separately every six hours and automatically push their results to the
47buildstats repository at: 47buildstats repository at:
48:yocto_git:`/yocto-buildstats/`. 48:yocto_git:`/yocto-buildstats/`.
49 49
50The 'quick' targets have been selected to be the ones which catch the 50The "quick" targets have been selected to be the ones which catch the
51most failures or give the most valuable data. We run 'fast' ptests in 51most failures or give the most valuable data. We run "fast" ptests in
52this case for example but not the ones which take a long time. The quick 52this case for example but not the ones which take a long time. The quick
53target doesn't include \*-lsb builds for all architectures, some world 53target doesn't include ``\*-lsb`` builds for all architectures, some ``world``
54builds and doesn't trigger performance tests or ltp testing. The full 54builds and doesn't trigger performance tests or ``ltp`` testing. The full
55build includes all these things and is slower but more comprehensive. 55build includes all these things and is slower but more comprehensive.
56 56
57Release Builds 57Release Builds
@@ -67,7 +67,7 @@ that in :ref:`test-manual/test-process:day to day development`, in that the
67a-full target of the Autobuilder is used but in addition the form is 67a-full target of the Autobuilder is used but in addition the form is
68configured to generate and publish artifacts and the milestone number, 68configured to generate and publish artifacts and the milestone number,
69version, release candidate number and other information is entered. The 69version, release candidate number and other information is entered. The
70box to "generate an email to QA"is also checked. 70box to "generate an email to QA" is also checked.
71 71
72When the build completes, an email is sent out using the send-qa-email 72When the build completes, an email is sent out using the send-qa-email
73script in the ``yocto-autobuilder-helper`` repository to the list of 73script in the ``yocto-autobuilder-helper`` repository to the list of
diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst
index 7a6cb2443b..a3fff29aca 100644
--- a/documentation/test-manual/understand-autobuilder.rst
+++ b/documentation/test-manual/understand-autobuilder.rst
@@ -32,8 +32,8 @@ which looks like::
32 } 32 }
33 }, 33 },
34 34
35And to expand that, you need the "arch-qemu" entry from 35And to expand that, you need the ``arch-qemu`` entry from
36the "templates" section, which looks like:: 36the ``templates`` section, which looks like::
37 37
38 "arch-qemu" : { 38 "arch-qemu" : {
39 "BUILDINFO" : true, 39 "BUILDINFO" : true,
@@ -54,11 +54,11 @@ the "templates" section, which looks like::
54 } 54 }
55 }, 55 },
56 56
57Combining these two entries you can see that "qemux86-64" is a three step build where the 57Combining these two entries you can see that ``qemux86-64`` is a three step
58``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for 58build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS``
59``MACHINE="qemux86-64"`` but with differing :term:`SDKMACHINE` settings. In step 59for each step; all for ``MACHINE="qemux86-64"`` but with differing
601 an extra variable is added to the ``auto.conf`` file to enable wic 60:term:`SDKMACHINE` settings. In step 1, an extra variable is added to the
61image generation. 61``auto.conf`` file to enable wic image generation.
62 62
63While not every detail of this is covered here, you can see how the 63While not every detail of this is covered here, you can see how the
64template mechanism allows quite complex configurations to be built up 64template mechanism allows quite complex configurations to be built up
@@ -163,8 +163,9 @@ Autobuilder Worker Janitor
163-------------------------- 163--------------------------
164 164
165This is a process running on each Worker that performs two basic 165This is a process running on each Worker that performs two basic
166operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and 166operations, including background file deletion at IO idle (see
167maintenance of a cache of cloned repositories to improve the speed 167"Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`)
168and maintenance of a cache of cloned repositories to improve the speed
168the system can checkout repositories. 169the system can checkout repositories.
169 170
170Shared DL_DIR 171Shared DL_DIR
@@ -172,7 +173,7 @@ Shared DL_DIR
172 173
173The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared 174The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared
174between them. This reduces network accesses from the system and allows 175between them. This reduces network accesses from the system and allows
175the build to be sped up. Usage of the directory within the build system 176the build to be sped up. The usage of the directory within the build system
176is designed to be able to be shared over NFS. 177is designed to be able to be shared over NFS.
177 178
178Shared SSTATE_DIR 179Shared SSTATE_DIR
@@ -180,8 +181,8 @@ Shared SSTATE_DIR
180 181
181The Workers are all connected over NFS which allows the ``sstate`` 182The Workers are all connected over NFS which allows the ``sstate``
182directory to be shared between them. This means once a Worker has built 183directory to be shared between them. This means once a Worker has built
183an artifact, all the others can benefit from it. Usage of the directory 184an artifact, all the others can benefit from it. The usage of the directory
184within the directory is designed for sharing over NFS. 185within the build system is designed for sharing over NFS.
185 186
186Resulttool 187Resulttool
187---------- 188----------
@@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as
192failure logs or the time taken to run the tests, may also be included. 193failure logs or the time taken to run the tests, may also be included.
193 194
194Resulttool is part of OpenEmbedded-Core and is used to manipulate these 195Resulttool is part of OpenEmbedded-Core and is used to manipulate these
195json results files. It has the ability to merge files together, display 196JSON results files. It has the ability to merge files together, display
196reports of the test results and compare different result files. 197reports of the test results and compare different result files.
197 198
198For details, see :yocto_wiki:`/Resulttool`. 199For details, see :yocto_wiki:`/Resulttool`.
@@ -206,7 +207,7 @@ are general setup steps that are run once and include:
206 207
207#. Set up any :term:`buildtools` tarball if configured. 208#. Set up any :term:`buildtools` tarball if configured.
208 209
209#. Call "buildhistory-init" if :ref:`ref-classes-buildhistory` is configured. 210#. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured.
210 211
211For each step that is configured in ``config.json``, it will perform the 212For each step that is configured in ``config.json``, it will perform the
212following: 213following:
@@ -258,7 +259,7 @@ it is inevitable that users will end up needing to heavily customise the
258``yocto-autobuilder-helper`` repository, particularly the 259``yocto-autobuilder-helper`` repository, particularly the
259``config.json`` file as they will want to define their own test matrix. 260``config.json`` file as they will want to define their own test matrix.
260 261
261The Autobuilder supports wo customization options: 262The Autobuilder supports two customization options:
262 263
263- variable substitution 264- variable substitution
264 265
@@ -278,7 +279,7 @@ environment::
278 $ ABHELPER_JSON="config.json /some/location/local.json" 279 $ ABHELPER_JSON="config.json /some/location/local.json"
279 280
280One issue users often run into is validation of the ``config.json`` files. A 281One issue users often run into is validation of the ``config.json`` files. A
281tip for minimizing issues from invalid json files is to use a Git 282tip for minimizing issues from invalid JSON files is to use a Git
282``pre-commit-hook.sh`` script to verify the JSON file before committing 283``pre-commit-hook.sh`` script to verify the JSON file before committing
283it. Create a symbolic link as follows:: 284it. Create a symbolic link as follows::
284 285