diff options
author | Michael Opdenacker <michael.opdenacker@bootlin.com> | 2023-12-06 16:36:00 +0100 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2023-12-18 13:18:55 +0000 |
commit | 84b242ef8bc103e49d83dcc25d968ab59524e1f3 (patch) | |
tree | 0fd504622c840757a2995a158ebc422605a8d5e9 | |
parent | c24e3b9f676ea68c100154f1aa865430ed2c99f1 (diff) | |
download | poky-84b242ef8bc103e49d83dcc25d968ab59524e1f3.tar.gz |
test-manual: text and formatting fixes
(From yocto-docs rev: 7c4f616f773bb9071b395e977b2ca9f8ac8f7e2a)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-rw-r--r-- | documentation/test-manual/intro.rst | 44 | ||||
-rw-r--r-- | documentation/test-manual/test-process.rst | 12 | ||||
-rw-r--r-- | documentation/test-manual/understand-autobuilder.rst | 33 |
3 files changed, 45 insertions, 44 deletions
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index aaf64ae017..f27210642c 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst | |||
@@ -14,7 +14,7 @@ release works as intended. All the project's testing infrastructure and | |||
14 | processes are publicly visible and available so that the community can | 14 | processes are publicly visible and available so that the community can |
15 | see what testing is being performed, how it's being done and the current | 15 | see what testing is being performed, how it's being done and the current |
16 | status of the tests and the project at any given time. It is intended | 16 | status of the tests and the project at any given time. It is intended |
17 | that Other organizations can leverage off the process and testing | 17 | that other organizations can leverage off the process and testing |
18 | environment used by the Yocto Project to create their own automated, | 18 | environment used by the Yocto Project to create their own automated, |
19 | production test environment, building upon the foundations from the | 19 | production test environment, building upon the foundations from the |
20 | project core. | 20 | project core. |
@@ -93,8 +93,8 @@ the following types of tests: | |||
93 | - *Build Testing:* Tests whether specific configurations build by | 93 | - *Build Testing:* Tests whether specific configurations build by |
94 | varying :term:`MACHINE`, | 94 | varying :term:`MACHINE`, |
95 | :term:`DISTRO`, other configuration | 95 | :term:`DISTRO`, other configuration |
96 | options, and the specific target images being built (or world). Used | 96 | options, and the specific target images being built (or ``world``). This is |
97 | to trigger builds of all the different test configurations on the | 97 | used to trigger builds of all the different test configurations on the |
98 | Autobuilder. Builds usually cover many different targets for | 98 | Autobuilder. Builds usually cover many different targets for |
99 | different architectures, machines, and distributions, as well as | 99 | different architectures, machines, and distributions, as well as |
100 | different configurations, such as different init systems. The | 100 | different configurations, such as different init systems. The |
@@ -120,7 +120,7 @@ the following types of tests: | |||
120 | 120 | ||
121 | $ bitbake image -c testsdkext | 121 | $ bitbake image -c testsdkext |
122 | 122 | ||
123 | The tests utilize the :ref:`ref-classes-testsdk` class and the | 123 | The tests use the :ref:`ref-classes-testsdk` class and the |
124 | ``do_testsdkext`` task. | 124 | ``do_testsdkext`` task. |
125 | 125 | ||
126 | - *Feature Testing:* Various scenario-based tests are run through the | 126 | - *Feature Testing:* Various scenario-based tests are run through the |
@@ -131,7 +131,7 @@ the following types of tests: | |||
131 | 131 | ||
132 | $ bitbake image -c testimage | 132 | $ bitbake image -c testimage |
133 | 133 | ||
134 | The tests utilize the :ref:`ref-classes-testimage` | 134 | The tests use the :ref:`ref-classes-testimage` |
135 | class and the :ref:`ref-tasks-testimage` task. | 135 | class and the :ref:`ref-tasks-testimage` task. |
136 | 136 | ||
137 | - *Layer Testing:* The Autobuilder has the possibility to test whether | 137 | - *Layer Testing:* The Autobuilder has the possibility to test whether |
@@ -151,7 +151,7 @@ the following types of tests: | |||
151 | 151 | ||
152 | $ bitbake image -c testsdk | 152 | $ bitbake image -c testsdk |
153 | 153 | ||
154 | The tests utilize the :ref:`ref-classes-testsdk` class and | 154 | The tests use the :ref:`ref-classes-testsdk` class and |
155 | the ``do_testsdk`` task. | 155 | the ``do_testsdk`` task. |
156 | 156 | ||
157 | - *Unit Testing:* Unit tests on various components of the system run | 157 | - *Unit Testing:* Unit tests on various components of the system run |
@@ -190,7 +190,7 @@ Tests map into the codebase as follows: | |||
190 | $ bitbake-selftest -v | 190 | $ bitbake-selftest -v |
191 | 191 | ||
192 | To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` | 192 | To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` |
193 | variable when running "bitbake-selftest" as follows:: | 193 | variable when running ``bitbake-selftest`` as follows:: |
194 | 194 | ||
195 | $ BB_SKIP_NETTESTS=yes bitbake-selftest | 195 | $ BB_SKIP_NETTESTS=yes bitbake-selftest |
196 | 196 | ||
@@ -215,7 +215,7 @@ Tests map into the codebase as follows: | |||
215 | - These tests use OE to test the workflows, which include testing | 215 | - These tests use OE to test the workflows, which include testing |
216 | specific features, behaviors of tasks, and API unit tests. | 216 | specific features, behaviors of tasks, and API unit tests. |
217 | 217 | ||
218 | - The tests can take advantage of parallelism through the "-j" | 218 | - The tests can take advantage of parallelism through the ``-j`` |
219 | option, which can specify a number of threads to spread the tests | 219 | option, which can specify a number of threads to spread the tests |
220 | across. Note that all tests from a given class of tests will run | 220 | across. Note that all tests from a given class of tests will run |
221 | in the same thread. To parallelize large numbers of tests you can | 221 | in the same thread. To parallelize large numbers of tests you can |
@@ -231,18 +231,18 @@ Tests map into the codebase as follows: | |||
231 | $ oe-selftest -a | 231 | $ oe-selftest -a |
232 | 232 | ||
233 | - To run a specific test, use the following command form where | 233 | - To run a specific test, use the following command form where |
234 | testname is the name of the specific test:: | 234 | ``testname`` is the name of the specific test:: |
235 | 235 | ||
236 | $ oe-selftest -r <testname> | 236 | $ oe-selftest -r <testname> |
237 | 237 | ||
238 | For example, the following command would run the tinfoil | 238 | For example, the following command would run the ``tinfoil`` |
239 | getVar API test:: | 239 | ``getVar`` API test:: |
240 | 240 | ||
241 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar | 241 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar |
242 | 242 | ||
243 | It is also possible to run a set | 243 | It is also possible to run a set |
244 | of tests. For example the following command will run all of the | 244 | of tests. For example the following command will run all of the |
245 | tinfoil tests:: | 245 | ``tinfoil`` tests:: |
246 | 246 | ||
247 | $ oe-selftest -r tinfoil | 247 | $ oe-selftest -r tinfoil |
248 | 248 | ||
@@ -391,14 +391,14 @@ so tests within a given test class should always run in the same build, | |||
391 | while tests in different classes or modules may be split into different | 391 | while tests in different classes or modules may be split into different |
392 | builds. There is no data store available for these tests since the tests | 392 | builds. There is no data store available for these tests since the tests |
393 | launch the ``bitbake`` command and exist outside of its context. As a | 393 | launch the ``bitbake`` command and exist outside of its context. As a |
394 | result, common bitbake library functions (bb.\*) are also unavailable. | 394 | result, common BitBake library functions (``bb.\*``) are also unavailable. |
395 | 395 | ||
396 | ``testimage`` | 396 | ``testimage`` |
397 | ------------- | 397 | ------------- |
398 | 398 | ||
399 | These tests are run once an image is up and running, either on target | 399 | These tests are run once an image is up and running, either on target |
400 | hardware or under QEMU. As a result, they are assumed to be running in a | 400 | hardware or under QEMU. As a result, they are assumed to be running in a |
401 | target image environment, as opposed to a host build environment. A | 401 | target image environment, as opposed to in a host build environment. A |
402 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains | 402 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains |
403 | the following:: | 403 | the following:: |
404 | 404 | ||
@@ -413,19 +413,19 @@ the following:: | |||
413 | 413 | ||
414 | In this example, the ``OERuntimeTestCase`` class wraps | 414 | In this example, the ``OERuntimeTestCase`` class wraps |
415 | ``unittest.TestCase``. Within the test, ``self.target`` represents the | 415 | ``unittest.TestCase``. Within the test, ``self.target`` represents the |
416 | target system, where commands can be run on it using the ``run()`` | 416 | target system, where commands can be run using the ``run()`` |
417 | method. | 417 | method. |
418 | 418 | ||
419 | To ensure certain test or package dependencies are met, you can use the | 419 | To ensure certain tests or package dependencies are met, you can use the |
420 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test | 420 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test |
421 | in this example would only make sense if python3-core is installed in | 421 | in this example would only make sense if ``python3-core`` is installed in |
422 | the image. | 422 | the image. |
423 | 423 | ||
424 | ``testsdk_ext`` | 424 | ``testsdk_ext`` |
425 | --------------- | 425 | --------------- |
426 | 426 | ||
427 | These tests are run against built extensible SDKs (eSDKs). The tests can | 427 | These tests are run against built extensible SDKs (eSDKs). The tests can |
428 | assume that the eSDK environment has already been setup. An example from | 428 | assume that the eSDK environment has already been set up. An example from |
429 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: | 429 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: |
430 | 430 | ||
431 | class DevtoolTest(OESDKExtTestCase): | 431 | class DevtoolTest(OESDKExtTestCase): |
@@ -472,9 +472,9 @@ following:: | |||
472 | output = self._run(cmd) | 472 | output = self._run(cmd) |
473 | self.assertEqual(output, "Hello, world\n") | 473 | self.assertEqual(output, "Hello, world\n") |
474 | 474 | ||
475 | In this example, if nativesdk-python3-core has been installed into the SDK, the code runs | 475 | In this example, if ``nativesdk-python3-core`` has been installed into the SDK, |
476 | the python3 interpreter with a basic command to check it is working | 476 | the code runs the ``python3`` interpreter with a basic command to check it is |
477 | correctly. The test would only run if Python3 is installed in the SDK. | 477 | working correctly. The test would only run if Python3 is installed in the SDK. |
478 | 478 | ||
479 | ``oe-build-perf-test`` | 479 | ``oe-build-perf-test`` |
480 | ---------------------- | 480 | ---------------------- |
@@ -520,7 +520,7 @@ an isolated directory. | |||
520 | 520 | ||
521 | This can delete files from :term:`SSTATE_DIR` which would potentially break | 521 | This can delete files from :term:`SSTATE_DIR` which would potentially break |
522 | other builds running in parallel. If this is required, :term:`SSTATE_DIR` must | 522 | other builds running in parallel. If this is required, :term:`SSTATE_DIR` must |
523 | be set to an isolated directory. Alternatively, you can use the "-f" | 523 | be set to an isolated directory. Alternatively, you can use the ``-f`` |
524 | option with the ``bitbake`` command to "taint" tasks by changing the | 524 | option with the ``bitbake`` command to "taint" tasks by changing the |
525 | sstate checksums to ensure sstate cache items will not be reused. | 525 | sstate checksums to ensure sstate cache items will not be reused. |
526 | 526 | ||
diff --git a/documentation/test-manual/test-process.rst b/documentation/test-manual/test-process.rst index 4c3b32bfea..0c560efe39 100644 --- a/documentation/test-manual/test-process.rst +++ b/documentation/test-manual/test-process.rst | |||
@@ -42,16 +42,16 @@ our sstate up to date. On the weekend, there is a master-next build | |||
42 | instead to ensure the test results are updated for the less frequently | 42 | instead to ensure the test results are updated for the less frequently |
43 | run targets. | 43 | run targets. |
44 | 44 | ||
45 | Performance builds (buildperf-\* targets in the console) are triggered | 45 | Performance builds (``buildperf-\*`` targets in the console) are triggered |
46 | separately every six hours and automatically push their results to the | 46 | separately every six hours and automatically push their results to the |
47 | buildstats repository at: | 47 | buildstats repository at: |
48 | :yocto_git:`/yocto-buildstats/`. | 48 | :yocto_git:`/yocto-buildstats/`. |
49 | 49 | ||
50 | The 'quick' targets have been selected to be the ones which catch the | 50 | The "quick" targets have been selected to be the ones which catch the |
51 | most failures or give the most valuable data. We run 'fast' ptests in | 51 | most failures or give the most valuable data. We run "fast" ptests in |
52 | this case for example but not the ones which take a long time. The quick | 52 | this case for example but not the ones which take a long time. The quick |
53 | target doesn't include \*-lsb builds for all architectures, some world | 53 | target doesn't include ``\*-lsb`` builds for all architectures, some ``world`` |
54 | builds and doesn't trigger performance tests or ltp testing. The full | 54 | builds and doesn't trigger performance tests or ``ltp`` testing. The full |
55 | build includes all these things and is slower but more comprehensive. | 55 | build includes all these things and is slower but more comprehensive. |
56 | 56 | ||
57 | Release Builds | 57 | Release Builds |
@@ -67,7 +67,7 @@ that in :ref:`test-manual/test-process:day to day development`, in that the | |||
67 | a-full target of the Autobuilder is used but in addition the form is | 67 | a-full target of the Autobuilder is used but in addition the form is |
68 | configured to generate and publish artifacts and the milestone number, | 68 | configured to generate and publish artifacts and the milestone number, |
69 | version, release candidate number and other information is entered. The | 69 | version, release candidate number and other information is entered. The |
70 | box to "generate an email to QA"is also checked. | 70 | box to "generate an email to QA" is also checked. |
71 | 71 | ||
72 | When the build completes, an email is sent out using the send-qa-email | 72 | When the build completes, an email is sent out using the send-qa-email |
73 | script in the ``yocto-autobuilder-helper`` repository to the list of | 73 | script in the ``yocto-autobuilder-helper`` repository to the list of |
diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst index 7a6cb2443b..a3fff29aca 100644 --- a/documentation/test-manual/understand-autobuilder.rst +++ b/documentation/test-manual/understand-autobuilder.rst | |||
@@ -32,8 +32,8 @@ which looks like:: | |||
32 | } | 32 | } |
33 | }, | 33 | }, |
34 | 34 | ||
35 | And to expand that, you need the "arch-qemu" entry from | 35 | And to expand that, you need the ``arch-qemu`` entry from |
36 | the "templates" section, which looks like:: | 36 | the ``templates`` section, which looks like:: |
37 | 37 | ||
38 | "arch-qemu" : { | 38 | "arch-qemu" : { |
39 | "BUILDINFO" : true, | 39 | "BUILDINFO" : true, |
@@ -54,11 +54,11 @@ the "templates" section, which looks like:: | |||
54 | } | 54 | } |
55 | }, | 55 | }, |
56 | 56 | ||
57 | Combining these two entries you can see that "qemux86-64" is a three step build where the | 57 | Combining these two entries you can see that ``qemux86-64`` is a three step |
58 | ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for | 58 | build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` |
59 | ``MACHINE="qemux86-64"`` but with differing :term:`SDKMACHINE` settings. In step | 59 | for each step; all for ``MACHINE="qemux86-64"`` but with differing |
60 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | 60 | :term:`SDKMACHINE` settings. In step 1, an extra variable is added to the |
61 | image generation. | 61 | ``auto.conf`` file to enable wic image generation. |
62 | 62 | ||
63 | While not every detail of this is covered here, you can see how the | 63 | While not every detail of this is covered here, you can see how the |
64 | template mechanism allows quite complex configurations to be built up | 64 | template mechanism allows quite complex configurations to be built up |
@@ -163,8 +163,9 @@ Autobuilder Worker Janitor | |||
163 | -------------------------- | 163 | -------------------------- |
164 | 164 | ||
165 | This is a process running on each Worker that performs two basic | 165 | This is a process running on each Worker that performs two basic |
166 | operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and | 166 | operations, including background file deletion at IO idle (see |
167 | maintenance of a cache of cloned repositories to improve the speed | 167 | "Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`) |
168 | and maintenance of a cache of cloned repositories to improve the speed | ||
168 | the system can checkout repositories. | 169 | the system can checkout repositories. |
169 | 170 | ||
170 | Shared DL_DIR | 171 | Shared DL_DIR |
@@ -172,7 +173,7 @@ Shared DL_DIR | |||
172 | 173 | ||
173 | The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared | 174 | The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared |
174 | between them. This reduces network accesses from the system and allows | 175 | between them. This reduces network accesses from the system and allows |
175 | the build to be sped up. Usage of the directory within the build system | 176 | the build to be sped up. The usage of the directory within the build system |
176 | is designed to be able to be shared over NFS. | 177 | is designed to be able to be shared over NFS. |
177 | 178 | ||
178 | Shared SSTATE_DIR | 179 | Shared SSTATE_DIR |
@@ -180,8 +181,8 @@ Shared SSTATE_DIR | |||
180 | 181 | ||
181 | The Workers are all connected over NFS which allows the ``sstate`` | 182 | The Workers are all connected over NFS which allows the ``sstate`` |
182 | directory to be shared between them. This means once a Worker has built | 183 | directory to be shared between them. This means once a Worker has built |
183 | an artifact, all the others can benefit from it. Usage of the directory | 184 | an artifact, all the others can benefit from it. The usage of the directory |
184 | within the directory is designed for sharing over NFS. | 185 | within the build system is designed for sharing over NFS. |
185 | 186 | ||
186 | Resulttool | 187 | Resulttool |
187 | ---------- | 188 | ---------- |
@@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as | |||
192 | failure logs or the time taken to run the tests, may also be included. | 193 | failure logs or the time taken to run the tests, may also be included. |
193 | 194 | ||
194 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these | 195 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these |
195 | json results files. It has the ability to merge files together, display | 196 | JSON results files. It has the ability to merge files together, display |
196 | reports of the test results and compare different result files. | 197 | reports of the test results and compare different result files. |
197 | 198 | ||
198 | For details, see :yocto_wiki:`/Resulttool`. | 199 | For details, see :yocto_wiki:`/Resulttool`. |
@@ -206,7 +207,7 @@ are general setup steps that are run once and include: | |||
206 | 207 | ||
207 | #. Set up any :term:`buildtools` tarball if configured. | 208 | #. Set up any :term:`buildtools` tarball if configured. |
208 | 209 | ||
209 | #. Call "buildhistory-init" if :ref:`ref-classes-buildhistory` is configured. | 210 | #. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured. |
210 | 211 | ||
211 | For each step that is configured in ``config.json``, it will perform the | 212 | For each step that is configured in ``config.json``, it will perform the |
212 | following: | 213 | following: |
@@ -258,7 +259,7 @@ it is inevitable that users will end up needing to heavily customise the | |||
258 | ``yocto-autobuilder-helper`` repository, particularly the | 259 | ``yocto-autobuilder-helper`` repository, particularly the |
259 | ``config.json`` file as they will want to define their own test matrix. | 260 | ``config.json`` file as they will want to define their own test matrix. |
260 | 261 | ||
261 | The Autobuilder supports wo customization options: | 262 | The Autobuilder supports two customization options: |
262 | 263 | ||
263 | - variable substitution | 264 | - variable substitution |
264 | 265 | ||
@@ -278,7 +279,7 @@ environment:: | |||
278 | $ ABHELPER_JSON="config.json /some/location/local.json" | 279 | $ ABHELPER_JSON="config.json /some/location/local.json" |
279 | 280 | ||
280 | One issue users often run into is validation of the ``config.json`` files. A | 281 | One issue users often run into is validation of the ``config.json`` files. A |
281 | tip for minimizing issues from invalid json files is to use a Git | 282 | tip for minimizing issues from invalid JSON files is to use a Git |
282 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | 283 | ``pre-commit-hook.sh`` script to verify the JSON file before committing |
283 | it. Create a symbolic link as follows:: | 284 | it. Create a symbolic link as follows:: |
284 | 285 | ||