diff options
| -rw-r--r-- | documentation/test-manual/intro.rst | 44 | ||||
| -rw-r--r-- | documentation/test-manual/test-process.rst | 12 | ||||
| -rw-r--r-- | documentation/test-manual/understand-autobuilder.rst | 33 |
3 files changed, 45 insertions, 44 deletions
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index aaf64ae017..f27210642c 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst | |||
| @@ -14,7 +14,7 @@ release works as intended. All the project's testing infrastructure and | |||
| 14 | processes are publicly visible and available so that the community can | 14 | processes are publicly visible and available so that the community can |
| 15 | see what testing is being performed, how it's being done and the current | 15 | see what testing is being performed, how it's being done and the current |
| 16 | status of the tests and the project at any given time. It is intended | 16 | status of the tests and the project at any given time. It is intended |
| 17 | that Other organizations can leverage off the process and testing | 17 | that other organizations can leverage off the process and testing |
| 18 | environment used by the Yocto Project to create their own automated, | 18 | environment used by the Yocto Project to create their own automated, |
| 19 | production test environment, building upon the foundations from the | 19 | production test environment, building upon the foundations from the |
| 20 | project core. | 20 | project core. |
| @@ -93,8 +93,8 @@ the following types of tests: | |||
| 93 | - *Build Testing:* Tests whether specific configurations build by | 93 | - *Build Testing:* Tests whether specific configurations build by |
| 94 | varying :term:`MACHINE`, | 94 | varying :term:`MACHINE`, |
| 95 | :term:`DISTRO`, other configuration | 95 | :term:`DISTRO`, other configuration |
| 96 | options, and the specific target images being built (or world). Used | 96 | options, and the specific target images being built (or ``world``). This is |
| 97 | to trigger builds of all the different test configurations on the | 97 | used to trigger builds of all the different test configurations on the |
| 98 | Autobuilder. Builds usually cover many different targets for | 98 | Autobuilder. Builds usually cover many different targets for |
| 99 | different architectures, machines, and distributions, as well as | 99 | different architectures, machines, and distributions, as well as |
| 100 | different configurations, such as different init systems. The | 100 | different configurations, such as different init systems. The |
| @@ -120,7 +120,7 @@ the following types of tests: | |||
| 120 | 120 | ||
| 121 | $ bitbake image -c testsdkext | 121 | $ bitbake image -c testsdkext |
| 122 | 122 | ||
| 123 | The tests utilize the :ref:`ref-classes-testsdk` class and the | 123 | The tests use the :ref:`ref-classes-testsdk` class and the |
| 124 | ``do_testsdkext`` task. | 124 | ``do_testsdkext`` task. |
| 125 | 125 | ||
| 126 | - *Feature Testing:* Various scenario-based tests are run through the | 126 | - *Feature Testing:* Various scenario-based tests are run through the |
| @@ -131,7 +131,7 @@ the following types of tests: | |||
| 131 | 131 | ||
| 132 | $ bitbake image -c testimage | 132 | $ bitbake image -c testimage |
| 133 | 133 | ||
| 134 | The tests utilize the :ref:`ref-classes-testimage` | 134 | The tests use the :ref:`ref-classes-testimage` |
| 135 | class and the :ref:`ref-tasks-testimage` task. | 135 | class and the :ref:`ref-tasks-testimage` task. |
| 136 | 136 | ||
| 137 | - *Layer Testing:* The Autobuilder has the possibility to test whether | 137 | - *Layer Testing:* The Autobuilder has the possibility to test whether |
| @@ -151,7 +151,7 @@ the following types of tests: | |||
| 151 | 151 | ||
| 152 | $ bitbake image -c testsdk | 152 | $ bitbake image -c testsdk |
| 153 | 153 | ||
| 154 | The tests utilize the :ref:`ref-classes-testsdk` class and | 154 | The tests use the :ref:`ref-classes-testsdk` class and |
| 155 | the ``do_testsdk`` task. | 155 | the ``do_testsdk`` task. |
| 156 | 156 | ||
| 157 | - *Unit Testing:* Unit tests on various components of the system run | 157 | - *Unit Testing:* Unit tests on various components of the system run |
| @@ -190,7 +190,7 @@ Tests map into the codebase as follows: | |||
| 190 | $ bitbake-selftest -v | 190 | $ bitbake-selftest -v |
| 191 | 191 | ||
| 192 | To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` | 192 | To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` |
| 193 | variable when running "bitbake-selftest" as follows:: | 193 | variable when running ``bitbake-selftest`` as follows:: |
| 194 | 194 | ||
| 195 | $ BB_SKIP_NETTESTS=yes bitbake-selftest | 195 | $ BB_SKIP_NETTESTS=yes bitbake-selftest |
| 196 | 196 | ||
| @@ -215,7 +215,7 @@ Tests map into the codebase as follows: | |||
| 215 | - These tests use OE to test the workflows, which include testing | 215 | - These tests use OE to test the workflows, which include testing |
| 216 | specific features, behaviors of tasks, and API unit tests. | 216 | specific features, behaviors of tasks, and API unit tests. |
| 217 | 217 | ||
| 218 | - The tests can take advantage of parallelism through the "-j" | 218 | - The tests can take advantage of parallelism through the ``-j`` |
| 219 | option, which can specify a number of threads to spread the tests | 219 | option, which can specify a number of threads to spread the tests |
| 220 | across. Note that all tests from a given class of tests will run | 220 | across. Note that all tests from a given class of tests will run |
| 221 | in the same thread. To parallelize large numbers of tests you can | 221 | in the same thread. To parallelize large numbers of tests you can |
| @@ -231,18 +231,18 @@ Tests map into the codebase as follows: | |||
| 231 | $ oe-selftest -a | 231 | $ oe-selftest -a |
| 232 | 232 | ||
| 233 | - To run a specific test, use the following command form where | 233 | - To run a specific test, use the following command form where |
| 234 | testname is the name of the specific test:: | 234 | ``testname`` is the name of the specific test:: |
| 235 | 235 | ||
| 236 | $ oe-selftest -r <testname> | 236 | $ oe-selftest -r <testname> |
| 237 | 237 | ||
| 238 | For example, the following command would run the tinfoil | 238 | For example, the following command would run the ``tinfoil`` |
| 239 | getVar API test:: | 239 | ``getVar`` API test:: |
| 240 | 240 | ||
| 241 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar | 241 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar |
| 242 | 242 | ||
| 243 | It is also possible to run a set | 243 | It is also possible to run a set |
| 244 | of tests. For example the following command will run all of the | 244 | of tests. For example the following command will run all of the |
| 245 | tinfoil tests:: | 245 | ``tinfoil`` tests:: |
| 246 | 246 | ||
| 247 | $ oe-selftest -r tinfoil | 247 | $ oe-selftest -r tinfoil |
| 248 | 248 | ||
| @@ -391,14 +391,14 @@ so tests within a given test class should always run in the same build, | |||
| 391 | while tests in different classes or modules may be split into different | 391 | while tests in different classes or modules may be split into different |
| 392 | builds. There is no data store available for these tests since the tests | 392 | builds. There is no data store available for these tests since the tests |
| 393 | launch the ``bitbake`` command and exist outside of its context. As a | 393 | launch the ``bitbake`` command and exist outside of its context. As a |
| 394 | result, common bitbake library functions (bb.\*) are also unavailable. | 394 | result, common BitBake library functions (``bb.\*``) are also unavailable. |
| 395 | 395 | ||
| 396 | ``testimage`` | 396 | ``testimage`` |
| 397 | ------------- | 397 | ------------- |
| 398 | 398 | ||
| 399 | These tests are run once an image is up and running, either on target | 399 | These tests are run once an image is up and running, either on target |
| 400 | hardware or under QEMU. As a result, they are assumed to be running in a | 400 | hardware or under QEMU. As a result, they are assumed to be running in a |
| 401 | target image environment, as opposed to a host build environment. A | 401 | target image environment, as opposed to in a host build environment. A |
| 402 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains | 402 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains |
| 403 | the following:: | 403 | the following:: |
| 404 | 404 | ||
| @@ -413,19 +413,19 @@ the following:: | |||
| 413 | 413 | ||
| 414 | In this example, the ``OERuntimeTestCase`` class wraps | 414 | In this example, the ``OERuntimeTestCase`` class wraps |
| 415 | ``unittest.TestCase``. Within the test, ``self.target`` represents the | 415 | ``unittest.TestCase``. Within the test, ``self.target`` represents the |
| 416 | target system, where commands can be run on it using the ``run()`` | 416 | target system, where commands can be run using the ``run()`` |
| 417 | method. | 417 | method. |
| 418 | 418 | ||
| 419 | To ensure certain test or package dependencies are met, you can use the | 419 | To ensure certain tests or package dependencies are met, you can use the |
| 420 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test | 420 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test |
| 421 | in this example would only make sense if python3-core is installed in | 421 | in this example would only make sense if ``python3-core`` is installed in |
| 422 | the image. | 422 | the image. |
| 423 | 423 | ||
| 424 | ``testsdk_ext`` | 424 | ``testsdk_ext`` |
| 425 | --------------- | 425 | --------------- |
| 426 | 426 | ||
| 427 | These tests are run against built extensible SDKs (eSDKs). The tests can | 427 | These tests are run against built extensible SDKs (eSDKs). The tests can |
| 428 | assume that the eSDK environment has already been setup. An example from | 428 | assume that the eSDK environment has already been set up. An example from |
| 429 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: | 429 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: |
| 430 | 430 | ||
| 431 | class DevtoolTest(OESDKExtTestCase): | 431 | class DevtoolTest(OESDKExtTestCase): |
| @@ -472,9 +472,9 @@ following:: | |||
| 472 | output = self._run(cmd) | 472 | output = self._run(cmd) |
| 473 | self.assertEqual(output, "Hello, world\n") | 473 | self.assertEqual(output, "Hello, world\n") |
| 474 | 474 | ||
| 475 | In this example, if nativesdk-python3-core has been installed into the SDK, the code runs | 475 | In this example, if ``nativesdk-python3-core`` has been installed into the SDK, |
| 476 | the python3 interpreter with a basic command to check it is working | 476 | the code runs the ``python3`` interpreter with a basic command to check it is |
| 477 | correctly. The test would only run if Python3 is installed in the SDK. | 477 | working correctly. The test would only run if Python3 is installed in the SDK. |
| 478 | 478 | ||
| 479 | ``oe-build-perf-test`` | 479 | ``oe-build-perf-test`` |
| 480 | ---------------------- | 480 | ---------------------- |
| @@ -520,7 +520,7 @@ an isolated directory. | |||
| 520 | 520 | ||
| 521 | This can delete files from :term:`SSTATE_DIR` which would potentially break | 521 | This can delete files from :term:`SSTATE_DIR` which would potentially break |
| 522 | other builds running in parallel. If this is required, :term:`SSTATE_DIR` must | 522 | other builds running in parallel. If this is required, :term:`SSTATE_DIR` must |
| 523 | be set to an isolated directory. Alternatively, you can use the "-f" | 523 | be set to an isolated directory. Alternatively, you can use the ``-f`` |
| 524 | option with the ``bitbake`` command to "taint" tasks by changing the | 524 | option with the ``bitbake`` command to "taint" tasks by changing the |
| 525 | sstate checksums to ensure sstate cache items will not be reused. | 525 | sstate checksums to ensure sstate cache items will not be reused. |
| 526 | 526 | ||
diff --git a/documentation/test-manual/test-process.rst b/documentation/test-manual/test-process.rst index 4c3b32bfea..0c560efe39 100644 --- a/documentation/test-manual/test-process.rst +++ b/documentation/test-manual/test-process.rst | |||
| @@ -42,16 +42,16 @@ our sstate up to date. On the weekend, there is a master-next build | |||
| 42 | instead to ensure the test results are updated for the less frequently | 42 | instead to ensure the test results are updated for the less frequently |
| 43 | run targets. | 43 | run targets. |
| 44 | 44 | ||
| 45 | Performance builds (buildperf-\* targets in the console) are triggered | 45 | Performance builds (``buildperf-\*`` targets in the console) are triggered |
| 46 | separately every six hours and automatically push their results to the | 46 | separately every six hours and automatically push their results to the |
| 47 | buildstats repository at: | 47 | buildstats repository at: |
| 48 | :yocto_git:`/yocto-buildstats/`. | 48 | :yocto_git:`/yocto-buildstats/`. |
| 49 | 49 | ||
| 50 | The 'quick' targets have been selected to be the ones which catch the | 50 | The "quick" targets have been selected to be the ones which catch the |
| 51 | most failures or give the most valuable data. We run 'fast' ptests in | 51 | most failures or give the most valuable data. We run "fast" ptests in |
| 52 | this case for example but not the ones which take a long time. The quick | 52 | this case for example but not the ones which take a long time. The quick |
| 53 | target doesn't include \*-lsb builds for all architectures, some world | 53 | target doesn't include ``\*-lsb`` builds for all architectures, some ``world`` |
| 54 | builds and doesn't trigger performance tests or ltp testing. The full | 54 | builds and doesn't trigger performance tests or ``ltp`` testing. The full |
| 55 | build includes all these things and is slower but more comprehensive. | 55 | build includes all these things and is slower but more comprehensive. |
| 56 | 56 | ||
| 57 | Release Builds | 57 | Release Builds |
| @@ -67,7 +67,7 @@ that in :ref:`test-manual/test-process:day to day development`, in that the | |||
| 67 | a-full target of the Autobuilder is used but in addition the form is | 67 | a-full target of the Autobuilder is used but in addition the form is |
| 68 | configured to generate and publish artifacts and the milestone number, | 68 | configured to generate and publish artifacts and the milestone number, |
| 69 | version, release candidate number and other information is entered. The | 69 | version, release candidate number and other information is entered. The |
| 70 | box to "generate an email to QA"is also checked. | 70 | box to "generate an email to QA" is also checked. |
| 71 | 71 | ||
| 72 | When the build completes, an email is sent out using the send-qa-email | 72 | When the build completes, an email is sent out using the send-qa-email |
| 73 | script in the ``yocto-autobuilder-helper`` repository to the list of | 73 | script in the ``yocto-autobuilder-helper`` repository to the list of |
diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst index 7a6cb2443b..a3fff29aca 100644 --- a/documentation/test-manual/understand-autobuilder.rst +++ b/documentation/test-manual/understand-autobuilder.rst | |||
| @@ -32,8 +32,8 @@ which looks like:: | |||
| 32 | } | 32 | } |
| 33 | }, | 33 | }, |
| 34 | 34 | ||
| 35 | And to expand that, you need the "arch-qemu" entry from | 35 | And to expand that, you need the ``arch-qemu`` entry from |
| 36 | the "templates" section, which looks like:: | 36 | the ``templates`` section, which looks like:: |
| 37 | 37 | ||
| 38 | "arch-qemu" : { | 38 | "arch-qemu" : { |
| 39 | "BUILDINFO" : true, | 39 | "BUILDINFO" : true, |
| @@ -54,11 +54,11 @@ the "templates" section, which looks like:: | |||
| 54 | } | 54 | } |
| 55 | }, | 55 | }, |
| 56 | 56 | ||
| 57 | Combining these two entries you can see that "qemux86-64" is a three step build where the | 57 | Combining these two entries you can see that ``qemux86-64`` is a three step |
| 58 | ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for | 58 | build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` |
| 59 | ``MACHINE="qemux86-64"`` but with differing :term:`SDKMACHINE` settings. In step | 59 | for each step; all for ``MACHINE="qemux86-64"`` but with differing |
| 60 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | 60 | :term:`SDKMACHINE` settings. In step 1, an extra variable is added to the |
| 61 | image generation. | 61 | ``auto.conf`` file to enable wic image generation. |
| 62 | 62 | ||
| 63 | While not every detail of this is covered here, you can see how the | 63 | While not every detail of this is covered here, you can see how the |
| 64 | template mechanism allows quite complex configurations to be built up | 64 | template mechanism allows quite complex configurations to be built up |
| @@ -163,8 +163,9 @@ Autobuilder Worker Janitor | |||
| 163 | -------------------------- | 163 | -------------------------- |
| 164 | 164 | ||
| 165 | This is a process running on each Worker that performs two basic | 165 | This is a process running on each Worker that performs two basic |
| 166 | operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and | 166 | operations, including background file deletion at IO idle (see |
| 167 | maintenance of a cache of cloned repositories to improve the speed | 167 | "Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`) |
| 168 | and maintenance of a cache of cloned repositories to improve the speed | ||
| 168 | the system can checkout repositories. | 169 | the system can checkout repositories. |
| 169 | 170 | ||
| 170 | Shared DL_DIR | 171 | Shared DL_DIR |
| @@ -172,7 +173,7 @@ Shared DL_DIR | |||
| 172 | 173 | ||
| 173 | The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared | 174 | The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared |
| 174 | between them. This reduces network accesses from the system and allows | 175 | between them. This reduces network accesses from the system and allows |
| 175 | the build to be sped up. Usage of the directory within the build system | 176 | the build to be sped up. The usage of the directory within the build system |
| 176 | is designed to be able to be shared over NFS. | 177 | is designed to be able to be shared over NFS. |
| 177 | 178 | ||
| 178 | Shared SSTATE_DIR | 179 | Shared SSTATE_DIR |
| @@ -180,8 +181,8 @@ Shared SSTATE_DIR | |||
| 180 | 181 | ||
| 181 | The Workers are all connected over NFS which allows the ``sstate`` | 182 | The Workers are all connected over NFS which allows the ``sstate`` |
| 182 | directory to be shared between them. This means once a Worker has built | 183 | directory to be shared between them. This means once a Worker has built |
| 183 | an artifact, all the others can benefit from it. Usage of the directory | 184 | an artifact, all the others can benefit from it. The usage of the directory |
| 184 | within the directory is designed for sharing over NFS. | 185 | within the build system is designed for sharing over NFS. |
| 185 | 186 | ||
| 186 | Resulttool | 187 | Resulttool |
| 187 | ---------- | 188 | ---------- |
| @@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as | |||
| 192 | failure logs or the time taken to run the tests, may also be included. | 193 | failure logs or the time taken to run the tests, may also be included. |
| 193 | 194 | ||
| 194 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these | 195 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these |
| 195 | json results files. It has the ability to merge files together, display | 196 | JSON results files. It has the ability to merge files together, display |
| 196 | reports of the test results and compare different result files. | 197 | reports of the test results and compare different result files. |
| 197 | 198 | ||
| 198 | For details, see :yocto_wiki:`/Resulttool`. | 199 | For details, see :yocto_wiki:`/Resulttool`. |
| @@ -206,7 +207,7 @@ are general setup steps that are run once and include: | |||
| 206 | 207 | ||
| 207 | #. Set up any :term:`buildtools` tarball if configured. | 208 | #. Set up any :term:`buildtools` tarball if configured. |
| 208 | 209 | ||
| 209 | #. Call "buildhistory-init" if :ref:`ref-classes-buildhistory` is configured. | 210 | #. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured. |
| 210 | 211 | ||
| 211 | For each step that is configured in ``config.json``, it will perform the | 212 | For each step that is configured in ``config.json``, it will perform the |
| 212 | following: | 213 | following: |
| @@ -258,7 +259,7 @@ it is inevitable that users will end up needing to heavily customise the | |||
| 258 | ``yocto-autobuilder-helper`` repository, particularly the | 259 | ``yocto-autobuilder-helper`` repository, particularly the |
| 259 | ``config.json`` file as they will want to define their own test matrix. | 260 | ``config.json`` file as they will want to define their own test matrix. |
| 260 | 261 | ||
| 261 | The Autobuilder supports wo customization options: | 262 | The Autobuilder supports two customization options: |
| 262 | 263 | ||
| 263 | - variable substitution | 264 | - variable substitution |
| 264 | 265 | ||
| @@ -278,7 +279,7 @@ environment:: | |||
| 278 | $ ABHELPER_JSON="config.json /some/location/local.json" | 279 | $ ABHELPER_JSON="config.json /some/location/local.json" |
| 279 | 280 | ||
| 280 | One issue users often run into is validation of the ``config.json`` files. A | 281 | One issue users often run into is validation of the ``config.json`` files. A |
| 281 | tip for minimizing issues from invalid json files is to use a Git | 282 | tip for minimizing issues from invalid JSON files is to use a Git |
| 282 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | 283 | ``pre-commit-hook.sh`` script to verify the JSON file before committing |
| 283 | it. Create a symbolic link as follows:: | 284 | it. Create a symbolic link as follows:: |
| 284 | 285 | ||
