diff options
| author | Michael Opdenacker <michael.opdenacker@bootlin.com> | 2023-12-06 16:36:00 +0100 |
|---|---|---|
| committer | Steve Sakoman <steve@sakoman.com> | 2023-12-21 06:14:00 -1000 |
| commit | ac4914f538bfd4214369c1cf73c3d2e009806e07 (patch) | |
| tree | 55e6e54103e0d8f89d7066f97c8adb9aa85df08b /documentation | |
| parent | f4d36b60f8c46c6c55d183ddbadd80377dffc96a (diff) | |
| download | poky-ac4914f538bfd4214369c1cf73c3d2e009806e07.tar.gz | |
test-manual: text and formatting fixes
(From yocto-docs rev: 330092ae9e624cef0fb8494a0deab896fb1b9026)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Diffstat (limited to 'documentation')
| -rw-r--r-- | documentation/test-manual/intro.rst | 51 | ||||
| -rw-r--r-- | documentation/test-manual/test-process.rst | 12 | ||||
| -rw-r--r-- | documentation/test-manual/understand-autobuilder.rst | 35 |
3 files changed, 52 insertions, 46 deletions
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index 8cfd6be87d..3a3fc2270e 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst | |||
| @@ -14,7 +14,7 @@ release works as intended. All the project's testing infrastructure and | |||
| 14 | processes are publicly visible and available so that the community can | 14 | processes are publicly visible and available so that the community can |
| 15 | see what testing is being performed, how it's being done and the current | 15 | see what testing is being performed, how it's being done and the current |
| 16 | status of the tests and the project at any given time. It is intended | 16 | status of the tests and the project at any given time. It is intended |
| 17 | that Other organizations can leverage off the process and testing | 17 | that other organizations can leverage off the process and testing |
| 18 | environment used by the Yocto Project to create their own automated, | 18 | environment used by the Yocto Project to create their own automated, |
| 19 | production test environment, building upon the foundations from the | 19 | production test environment, building upon the foundations from the |
| 20 | project core. | 20 | project core. |
| @@ -93,8 +93,8 @@ the following types of tests: | |||
| 93 | - *Build Testing:* Tests whether specific configurations build by | 93 | - *Build Testing:* Tests whether specific configurations build by |
| 94 | varying :term:`MACHINE`, | 94 | varying :term:`MACHINE`, |
| 95 | :term:`DISTRO`, other configuration | 95 | :term:`DISTRO`, other configuration |
| 96 | options, and the specific target images being built (or world). Used | 96 | options, and the specific target images being built (or ``world``). This is |
| 97 | to trigger builds of all the different test configurations on the | 97 | used to trigger builds of all the different test configurations on the |
| 98 | Autobuilder. Builds usually cover many different targets for | 98 | Autobuilder. Builds usually cover many different targets for |
| 99 | different architectures, machines, and distributions, as well as | 99 | different architectures, machines, and distributions, as well as |
| 100 | different configurations, such as different init systems. The | 100 | different configurations, such as different init systems. The |
| @@ -121,7 +121,8 @@ the following types of tests: | |||
| 121 | 121 | ||
| 122 | $ bitbake image -c testsdkext | 122 | $ bitbake image -c testsdkext |
| 123 | 123 | ||
| 124 | The tests utilize the :ref:`testsdkext <ref-classes-testsdk>` class and the ``do_testsdkext`` task. | 124 | The tests use the :ref:`ref-classes-testsdk` class and the |
| 125 | ``do_testsdkext`` task. | ||
| 125 | 126 | ||
| 126 | - *Feature Testing:* Various scenario-based tests are run through the | 127 | - *Feature Testing:* Various scenario-based tests are run through the |
| 127 | :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distributions | 128 | :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distributions |
| @@ -131,8 +132,8 @@ the following types of tests: | |||
| 131 | 132 | ||
| 132 | $ bitbake image -c testimage | 133 | $ bitbake image -c testimage |
| 133 | 134 | ||
| 134 | The tests utilize the :ref:`testimage* <ref-classes-testimage>` | 135 | The tests use the :ref:`ref-classes-testimage` |
| 135 | classes and the :ref:`ref-tasks-testimage` task. | 136 | class and the :ref:`ref-tasks-testimage` task. |
| 136 | 137 | ||
| 137 | - *Layer Testing:* The Autobuilder has the possibility to test whether | 138 | - *Layer Testing:* The Autobuilder has the possibility to test whether |
| 138 | specific layers work with the test of the system. The layers tested | 139 | specific layers work with the test of the system. The layers tested |
| @@ -151,7 +152,7 @@ the following types of tests: | |||
| 151 | 152 | ||
| 152 | $ bitbake image -c testsdk | 153 | $ bitbake image -c testsdk |
| 153 | 154 | ||
| 154 | The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and | 155 | The tests use the :ref:`ref-classes-testsdk` class and |
| 155 | the ``do_testsdk`` task. | 156 | the ``do_testsdk`` task. |
| 156 | 157 | ||
| 157 | - *Unit Testing:* Unit tests on various components of the system run | 158 | - *Unit Testing:* Unit tests on various components of the system run |
| @@ -179,7 +180,7 @@ Tests map into the codebase as follows: | |||
| 179 | $ bitbake-selftest | 180 | $ bitbake-selftest |
| 180 | 181 | ||
| 181 | To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` | 182 | To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` |
| 182 | variable when running "bitbake-selftest" as follows:: | 183 | variable when running ``bitbake-selftest`` as follows:: |
| 183 | 184 | ||
| 184 | $ BB_SKIP_NETTESTS=yes bitbake-selftest | 185 | $ BB_SKIP_NETTESTS=yes bitbake-selftest |
| 185 | 186 | ||
| @@ -209,7 +210,7 @@ Tests map into the codebase as follows: | |||
| 209 | - These tests use OE to test the workflows, which include testing | 210 | - These tests use OE to test the workflows, which include testing |
| 210 | specific features, behaviors of tasks, and API unit tests. | 211 | specific features, behaviors of tasks, and API unit tests. |
| 211 | 212 | ||
| 212 | - The tests can take advantage of parallelism through the "-j" | 213 | - The tests can take advantage of parallelism through the ``-j`` |
| 213 | option, which can specify a number of threads to spread the tests | 214 | option, which can specify a number of threads to spread the tests |
| 214 | across. Note that all tests from a given class of tests will run | 215 | across. Note that all tests from a given class of tests will run |
| 215 | in the same thread. To parallelize large numbers of tests you can | 216 | in the same thread. To parallelize large numbers of tests you can |
| @@ -225,18 +226,18 @@ Tests map into the codebase as follows: | |||
| 225 | $ oe-selftest -a | 226 | $ oe-selftest -a |
| 226 | 227 | ||
| 227 | - To run a specific test, use the following command form where | 228 | - To run a specific test, use the following command form where |
| 228 | testname is the name of the specific test:: | 229 | ``testname`` is the name of the specific test:: |
| 229 | 230 | ||
| 230 | $ oe-selftest -r <testname> | 231 | $ oe-selftest -r <testname> |
| 231 | 232 | ||
| 232 | For example, the following command would run the tinfoil | 233 | For example, the following command would run the ``tinfoil`` |
| 233 | getVar API test:: | 234 | ``getVar`` API test:: |
| 234 | 235 | ||
| 235 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar | 236 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar |
| 236 | 237 | ||
| 237 | It is also possible to run a set | 238 | It is also possible to run a set |
| 238 | of tests. For example the following command will run all of the | 239 | of tests. For example the following command will run all of the |
| 239 | tinfoil tests:: | 240 | ``tinfoil`` tests:: |
| 240 | 241 | ||
| 241 | $ oe-selftest -r tinfoil | 242 | $ oe-selftest -r tinfoil |
| 242 | 243 | ||
| @@ -385,14 +386,14 @@ so tests within a given test class should always run in the same build, | |||
| 385 | while tests in different classes or modules may be split into different | 386 | while tests in different classes or modules may be split into different |
| 386 | builds. There is no data store available for these tests since the tests | 387 | builds. There is no data store available for these tests since the tests |
| 387 | launch the ``bitbake`` command and exist outside of its context. As a | 388 | launch the ``bitbake`` command and exist outside of its context. As a |
| 388 | result, common bitbake library functions (bb.\*) are also unavailable. | 389 | result, common BitBake library functions (``bb.\*``) are also unavailable. |
| 389 | 390 | ||
| 390 | ``testimage`` | 391 | ``testimage`` |
| 391 | ------------- | 392 | ------------- |
| 392 | 393 | ||
| 393 | These tests are run once an image is up and running, either on target | 394 | These tests are run once an image is up and running, either on target |
| 394 | hardware or under QEMU. As a result, they are assumed to be running in a | 395 | hardware or under QEMU. As a result, they are assumed to be running in a |
| 395 | target image environment, as opposed to a host build environment. A | 396 | target image environment, as opposed to in a host build environment. A |
| 396 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains | 397 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains |
| 397 | the following:: | 398 | the following:: |
| 398 | 399 | ||
| @@ -407,19 +408,19 @@ the following:: | |||
| 407 | 408 | ||
| 408 | In this example, the ``OERuntimeTestCase`` class wraps | 409 | In this example, the ``OERuntimeTestCase`` class wraps |
| 409 | ``unittest.TestCase``. Within the test, ``self.target`` represents the | 410 | ``unittest.TestCase``. Within the test, ``self.target`` represents the |
| 410 | target system, where commands can be run on it using the ``run()`` | 411 | target system, where commands can be run using the ``run()`` |
| 411 | method. | 412 | method. |
| 412 | 413 | ||
| 413 | To ensure certain test or package dependencies are met, you can use the | 414 | To ensure certain tests or package dependencies are met, you can use the |
| 414 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test | 415 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test |
| 415 | in this example would only make sense if python3-core is installed in | 416 | in this example would only make sense if ``python3-core`` is installed in |
| 416 | the image. | 417 | the image. |
| 417 | 418 | ||
| 418 | ``testsdk_ext`` | 419 | ``testsdk_ext`` |
| 419 | --------------- | 420 | --------------- |
| 420 | 421 | ||
| 421 | These tests are run against built extensible SDKs (eSDKs). The tests can | 422 | These tests are run against built extensible SDKs (eSDKs). The tests can |
| 422 | assume that the eSDK environment has already been setup. An example from | 423 | assume that the eSDK environment has already been set up. An example from |
| 423 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: | 424 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: |
| 424 | 425 | ||
| 425 | class DevtoolTest(OESDKExtTestCase): | 426 | class DevtoolTest(OESDKExtTestCase): |
| @@ -466,9 +467,9 @@ following:: | |||
| 466 | output = self._run(cmd) | 467 | output = self._run(cmd) |
| 467 | self.assertEqual(output, "Hello, world\n") | 468 | self.assertEqual(output, "Hello, world\n") |
| 468 | 469 | ||
| 469 | In this example, if nativesdk-python3-core has been installed into the SDK, the code runs | 470 | In this example, if ``nativesdk-python3-core`` has been installed into the SDK, |
| 470 | the python3 interpreter with a basic command to check it is working | 471 | the code runs the ``python3`` interpreter with a basic command to check it is |
| 471 | correctly. The test would only run if python3 is installed in the SDK. | 472 | working correctly. The test would only run if Python3 is installed in the SDK. |
| 472 | 473 | ||
| 473 | ``oe-build-perf-test`` | 474 | ``oe-build-perf-test`` |
| 474 | ---------------------- | 475 | ---------------------- |
| @@ -512,9 +513,9 @@ an isolated directory. | |||
| 512 | 513 | ||
| 513 | **Running "cleansstate" is not permitted.** | 514 | **Running "cleansstate" is not permitted.** |
| 514 | 515 | ||
| 515 | This can delete files from SSTATE_DIR which would potentially break | 516 | This can delete files from :term:`SSTATE_DIR` which would potentially break |
| 516 | other builds running in parallel. If this is required, SSTATE_DIR must | 517 | other builds running in parallel. If this is required, :term:`SSTATE_DIR` must |
| 517 | be set to an isolated directory. Alternatively, you can use the "-f" | 518 | be set to an isolated directory. Alternatively, you can use the ``-f`` |
| 518 | option with the ``bitbake`` command to "taint" tasks by changing the | 519 | option with the ``bitbake`` command to "taint" tasks by changing the |
| 519 | sstate checksums to ensure sstate cache items will not be reused. | 520 | sstate checksums to ensure sstate cache items will not be reused. |
| 520 | 521 | ||
diff --git a/documentation/test-manual/test-process.rst b/documentation/test-manual/test-process.rst index 4c3b32bfea..0c560efe39 100644 --- a/documentation/test-manual/test-process.rst +++ b/documentation/test-manual/test-process.rst | |||
| @@ -42,16 +42,16 @@ our sstate up to date. On the weekend, there is a master-next build | |||
| 42 | instead to ensure the test results are updated for the less frequently | 42 | instead to ensure the test results are updated for the less frequently |
| 43 | run targets. | 43 | run targets. |
| 44 | 44 | ||
| 45 | Performance builds (buildperf-\* targets in the console) are triggered | 45 | Performance builds (``buildperf-\*`` targets in the console) are triggered |
| 46 | separately every six hours and automatically push their results to the | 46 | separately every six hours and automatically push their results to the |
| 47 | buildstats repository at: | 47 | buildstats repository at: |
| 48 | :yocto_git:`/yocto-buildstats/`. | 48 | :yocto_git:`/yocto-buildstats/`. |
| 49 | 49 | ||
| 50 | The 'quick' targets have been selected to be the ones which catch the | 50 | The "quick" targets have been selected to be the ones which catch the |
| 51 | most failures or give the most valuable data. We run 'fast' ptests in | 51 | most failures or give the most valuable data. We run "fast" ptests in |
| 52 | this case for example but not the ones which take a long time. The quick | 52 | this case for example but not the ones which take a long time. The quick |
| 53 | target doesn't include \*-lsb builds for all architectures, some world | 53 | target doesn't include ``\*-lsb`` builds for all architectures, some ``world`` |
| 54 | builds and doesn't trigger performance tests or ltp testing. The full | 54 | builds and doesn't trigger performance tests or ``ltp`` testing. The full |
| 55 | build includes all these things and is slower but more comprehensive. | 55 | build includes all these things and is slower but more comprehensive. |
| 56 | 56 | ||
| 57 | Release Builds | 57 | Release Builds |
| @@ -67,7 +67,7 @@ that in :ref:`test-manual/test-process:day to day development`, in that the | |||
| 67 | a-full target of the Autobuilder is used but in addition the form is | 67 | a-full target of the Autobuilder is used but in addition the form is |
| 68 | configured to generate and publish artifacts and the milestone number, | 68 | configured to generate and publish artifacts and the milestone number, |
| 69 | version, release candidate number and other information is entered. The | 69 | version, release candidate number and other information is entered. The |
| 70 | box to "generate an email to QA"is also checked. | 70 | box to "generate an email to QA" is also checked. |
| 71 | 71 | ||
| 72 | When the build completes, an email is sent out using the send-qa-email | 72 | When the build completes, an email is sent out using the send-qa-email |
| 73 | script in the ``yocto-autobuilder-helper`` repository to the list of | 73 | script in the ``yocto-autobuilder-helper`` repository to the list of |
diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst index b6809ce7bd..ef03955954 100644 --- a/documentation/test-manual/understand-autobuilder.rst +++ b/documentation/test-manual/understand-autobuilder.rst | |||
| @@ -32,8 +32,8 @@ which looks like:: | |||
| 32 | } | 32 | } |
| 33 | }, | 33 | }, |
| 34 | 34 | ||
| 35 | And to expand that, you need the "arch-qemu" entry from | 35 | And to expand that, you need the ``arch-qemu`` entry from |
| 36 | the "templates" section, which looks like:: | 36 | the ``templates`` section, which looks like:: |
| 37 | 37 | ||
| 38 | "arch-qemu" : { | 38 | "arch-qemu" : { |
| 39 | "BUILDINFO" : true, | 39 | "BUILDINFO" : true, |
| @@ -54,11 +54,11 @@ the "templates" section, which looks like:: | |||
| 54 | } | 54 | } |
| 55 | }, | 55 | }, |
| 56 | 56 | ||
| 57 | Combining these two entries you can see that "qemux86-64" is a three step build where the | 57 | Combining these two entries you can see that ``qemux86-64`` is a three step |
| 58 | ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for | 58 | build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` |
| 59 | ``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step | 59 | for each step; all for ``MACHINE="qemux86-64"`` but with differing |
| 60 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | 60 | :term:`SDKMACHINE` settings. In step 1, an extra variable is added to the |
| 61 | image generation. | 61 | ``auto.conf`` file to enable wic image generation. |
| 62 | 62 | ||
| 63 | While not every detail of this is covered here, you can see how the | 63 | While not every detail of this is covered here, you can see how the |
| 64 | template mechanism allows quite complex configurations to be built up | 64 | template mechanism allows quite complex configurations to be built up |
| @@ -163,8 +163,9 @@ Autobuilder Worker Janitor | |||
| 163 | -------------------------- | 163 | -------------------------- |
| 164 | 164 | ||
| 165 | This is a process running on each Worker that performs two basic | 165 | This is a process running on each Worker that performs two basic |
| 166 | operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and | 166 | operations, including background file deletion at IO idle (see |
| 167 | maintenance of a cache of cloned repositories to improve the speed | 167 | "Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`) |
| 168 | and maintenance of a cache of cloned repositories to improve the speed | ||
| 168 | the system can checkout repositories. | 169 | the system can checkout repositories. |
| 169 | 170 | ||
| 170 | Shared DL_DIR | 171 | Shared DL_DIR |
| @@ -172,7 +173,7 @@ Shared DL_DIR | |||
| 172 | 173 | ||
| 173 | The Workers are all connected over NFS which allows DL_DIR to be shared | 174 | The Workers are all connected over NFS which allows DL_DIR to be shared |
| 174 | between them. This reduces network accesses from the system and allows | 175 | between them. This reduces network accesses from the system and allows |
| 175 | the build to be sped up. Usage of the directory within the build system | 176 | the build to be sped up. The usage of the directory within the build system |
| 176 | is designed to be able to be shared over NFS. | 177 | is designed to be able to be shared over NFS. |
| 177 | 178 | ||
| 178 | Shared SSTATE_DIR | 179 | Shared SSTATE_DIR |
| @@ -180,8 +181,8 @@ Shared SSTATE_DIR | |||
| 180 | 181 | ||
| 181 | The Workers are all connected over NFS which allows the ``sstate`` | 182 | The Workers are all connected over NFS which allows the ``sstate`` |
| 182 | directory to be shared between them. This means once a Worker has built | 183 | directory to be shared between them. This means once a Worker has built |
| 183 | an artifact, all the others can benefit from it. Usage of the directory | 184 | an artifact, all the others can benefit from it. The usage of the directory |
| 184 | within the directory is designed for sharing over NFS. | 185 | within the build system is designed for sharing over NFS. |
| 185 | 186 | ||
| 186 | Resulttool | 187 | Resulttool |
| 187 | ---------- | 188 | ---------- |
| @@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as | |||
| 192 | failure logs or the time taken to run the tests, may also be included. | 193 | failure logs or the time taken to run the tests, may also be included. |
| 193 | 194 | ||
| 194 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these | 195 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these |
| 195 | json results files. It has the ability to merge files together, display | 196 | JSON results files. It has the ability to merge files together, display |
| 196 | reports of the test results and compare different result files. | 197 | reports of the test results and compare different result files. |
| 197 | 198 | ||
| 198 | For details, see :yocto_wiki:`/Resulttool`. | 199 | For details, see :yocto_wiki:`/Resulttool`. |
| @@ -206,7 +207,11 @@ are general setup steps that are run once and include: | |||
| 206 | 207 | ||
| 207 | #. Set up any ``buildtools-tarball`` if configured. | 208 | #. Set up any ``buildtools-tarball`` if configured. |
| 208 | 209 | ||
| 210 | <<<<<<< HEAD | ||
| 209 | #. Call "buildhistory-init" if buildhistory is configured. | 211 | #. Call "buildhistory-init" if buildhistory is configured. |
| 212 | ======= | ||
| 213 | #. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured. | ||
| 214 | >>>>>>> 7c4f616f77 (test-manual: text and formatting fixes) | ||
| 210 | 215 | ||
| 211 | For each step that is configured in ``config.json``, it will perform the | 216 | For each step that is configured in ``config.json``, it will perform the |
| 212 | following: | 217 | following: |
| @@ -258,7 +263,7 @@ it is inevitable that users will end up needing to heavily customise the | |||
| 258 | ``yocto-autobuilder-helper`` repository, particularly the | 263 | ``yocto-autobuilder-helper`` repository, particularly the |
| 259 | ``config.json`` file as they will want to define their own test matrix. | 264 | ``config.json`` file as they will want to define their own test matrix. |
| 260 | 265 | ||
| 261 | The Autobuilder supports wo customization options: | 266 | The Autobuilder supports two customization options: |
| 262 | 267 | ||
| 263 | - variable substitution | 268 | - variable substitution |
| 264 | 269 | ||
| @@ -278,7 +283,7 @@ environment:: | |||
| 278 | $ ABHELPER_JSON="config.json /some/location/local.json" | 283 | $ ABHELPER_JSON="config.json /some/location/local.json" |
| 279 | 284 | ||
| 280 | One issue users often run into is validation of the ``config.json`` files. A | 285 | One issue users often run into is validation of the ``config.json`` files. A |
| 281 | tip for minimizing issues from invalid json files is to use a Git | 286 | tip for minimizing issues from invalid JSON files is to use a Git |
| 282 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | 287 | ``pre-commit-hook.sh`` script to verify the JSON file before committing |
| 283 | it. Create a symbolic link as follows:: | 288 | it. Create a symbolic link as follows:: |
| 284 | 289 | ||
