diff options
Diffstat (limited to 'documentation/test-manual/intro.rst')
-rw-r--r-- | documentation/test-manual/intro.rst | 168 |
1 files changed, 87 insertions, 81 deletions
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index 81c24a8c3f..c31fd11c7a 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst | |||
@@ -14,19 +14,17 @@ release works as intended. All the project's testing infrastructure and | |||
14 | processes are publicly visible and available so that the community can | 14 | processes are publicly visible and available so that the community can |
15 | see what testing is being performed, how it's being done and the current | 15 | see what testing is being performed, how it's being done and the current |
16 | status of the tests and the project at any given time. It is intended | 16 | status of the tests and the project at any given time. It is intended |
17 | that Other organizations can leverage off the process and testing | 17 | that other organizations can leverage off the process and testing |
18 | environment used by the Yocto Project to create their own automated, | 18 | environment used by the Yocto Project to create their own automated, |
19 | production test environment, building upon the foundations from the | 19 | production test environment, building upon the foundations from the |
20 | project core. | 20 | project core. |
21 | 21 | ||
22 | Currently, the Yocto Project Test Environment Manual has no projected | 22 | This manual is a work-in-progress and is being initially loaded with |
23 | release date. This manual is a work-in-progress and is being initially | 23 | information from the README files and notes from key engineers: |
24 | loaded with information from the README files and notes from key | ||
25 | engineers: | ||
26 | 24 | ||
27 | - *yocto-autobuilder2:* This | 25 | - *yocto-autobuilder2:* This |
28 | :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>` | 26 | :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>` |
29 | is the main README which detials how to set up the Yocto Project | 27 | is the main README which details how to set up the Yocto Project |
30 | Autobuilder. The ``yocto-autobuilder2`` repository represents the | 28 | Autobuilder. The ``yocto-autobuilder2`` repository represents the |
31 | Yocto Project's console UI plugin to Buildbot and the configuration | 29 | Yocto Project's console UI plugin to Buildbot and the configuration |
32 | necessary to configure Buildbot to perform the testing the project | 30 | necessary to configure Buildbot to perform the testing the project |
@@ -39,7 +37,7 @@ engineers: | |||
39 | As a result, it can be used by any Continuous Improvement (CI) system | 37 | As a result, it can be used by any Continuous Improvement (CI) system |
40 | to run builds, support getting the correct code revisions, configure | 38 | to run builds, support getting the correct code revisions, configure |
41 | builds and layers, run builds, and collect results. The code is | 39 | builds and layers, run builds, and collect results. The code is |
42 | independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__, | 40 | independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/current/>`__, |
43 | Jenkins, or others. This repository has a branch per release of the | 41 | Jenkins, or others. This repository has a branch per release of the |
44 | project defining the tests to run on a per release basis. | 42 | project defining the tests to run on a per release basis. |
45 | 43 | ||
@@ -54,8 +52,8 @@ the Autobuilder tests if things work. The Autobuilder builds all test | |||
54 | targets and runs all the tests. | 52 | targets and runs all the tests. |
55 | 53 | ||
56 | The Yocto Project uses now uses standard upstream | 54 | The Yocto Project uses now uses standard upstream |
57 | `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to | 55 | Buildbot (`version 3.8 <https://docs.buildbot.net/3.8.0/>`__) to |
58 | drive its integration and testing. Buildbot Nine has a plug-in interface | 56 | drive its integration and testing. Buildbot has a plug-in interface |
59 | that the Yocto Project customizes using code from the | 57 | that the Yocto Project customizes using code from the |
60 | ``yocto-autobuilder2`` repository, adding its own console UI plugin. The | 58 | ``yocto-autobuilder2`` repository, adding its own console UI plugin. The |
61 | resulting UI plug-in allows you to visualize builds in a way suited to | 59 | resulting UI plug-in allows you to visualize builds in a way suited to |
@@ -72,10 +70,9 @@ simple JSON files. | |||
72 | .. note:: | 70 | .. note:: |
73 | 71 | ||
74 | The project uses Buildbot for historical reasons but also because | 72 | The project uses Buildbot for historical reasons but also because |
75 | many of the project developers have knowledge of python. It is | 73 | many of the project developers have knowledge of Python. It is |
76 | possible to use the outer layers from another Continuous Integration | 74 | possible to use the outer layers from another Continuous Integration |
77 | (CI) system such as | 75 | (CI) system such as :wikipedia:`Jenkins <Jenkins_(software)>` |
78 | `Jenkins <https://en.wikipedia.org/wiki/Jenkins_(software)>`__ | ||
79 | instead of Buildbot. | 76 | instead of Buildbot. |
80 | 77 | ||
81 | The following figure shows the Yocto Project Autobuilder stack with a | 78 | The following figure shows the Yocto Project Autobuilder stack with a |
@@ -83,29 +80,29 @@ topology that includes a controller and a cluster of workers: | |||
83 | 80 | ||
84 | .. image:: figures/ab-test-cluster.png | 81 | .. image:: figures/ab-test-cluster.png |
85 | :align: center | 82 | :align: center |
83 | :width: 70% | ||
86 | 84 | ||
87 | Yocto Project Tests - Types of Testing Overview | 85 | Yocto Project Tests --- Types of Testing Overview |
88 | =============================================== | 86 | ================================================= |
89 | 87 | ||
90 | The Autobuilder tests different elements of the project by using | 88 | The Autobuilder tests different elements of the project by using |
91 | thefollowing types of tests: | 89 | the following types of tests: |
92 | 90 | ||
93 | - *Build Testing:* Tests whether specific configurations build by | 91 | - *Build Testing:* Tests whether specific configurations build by |
94 | varying :term:`MACHINE`, | 92 | varying :term:`MACHINE`, |
95 | :term:`DISTRO`, other configuration | 93 | :term:`DISTRO`, other configuration |
96 | options, and the specific target images being built (or world). Used | 94 | options, and the specific target images being built (or ``world``). This is |
97 | to trigger builds of all the different test configurations on the | 95 | used to trigger builds of all the different test configurations on the |
98 | Autobuilder. Builds usually cover many different targets for | 96 | Autobuilder. Builds usually cover many different targets for |
99 | different architectures, machines, and distributions, as well as | 97 | different architectures, machines, and distributions, as well as |
100 | different configurations, such as different init systems. The | 98 | different configurations, such as different init systems. The |
101 | Autobuilder tests literally hundreds of configurations and targets. | 99 | Autobuilder tests literally hundreds of configurations and targets. |
102 | 100 | ||
103 | - *Sanity Checks During the Build Process:* Tests initiated through | 101 | - *Sanity Checks During the Build Process:* Tests initiated through the |
104 | the :ref:`insane <ref-classes-insane>` | 102 | :ref:`ref-classes-insane` class. These checks ensure the output of the |
105 | class. These checks ensure the output of the builds are correct. | 103 | builds are correct. For example, does the ELF architecture in the |
106 | For example, does the ELF architecture in the generated binaries | 104 | generated binaries match the target system? ARM binaries would not work |
107 | match the target system? ARM binaries would not work in a MIPS | 105 | in a MIPS system! |
108 | system! | ||
109 | 106 | ||
110 | - *Build Performance Testing:* Tests whether or not commonly used steps | 107 | - *Build Performance Testing:* Tests whether or not commonly used steps |
111 | during builds work efficiently and avoid regressions. Tests to time | 108 | during builds work efficiently and avoid regressions. Tests to time |
@@ -121,18 +118,19 @@ thefollowing types of tests: | |||
121 | 118 | ||
122 | $ bitbake image -c testsdkext | 119 | $ bitbake image -c testsdkext |
123 | 120 | ||
124 | The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task. | 121 | The tests use the :ref:`ref-classes-testsdk` class and the |
122 | ``do_testsdkext`` task. | ||
125 | 123 | ||
126 | - *Feature Testing:* Various scenario-based tests are run through the | 124 | - *Feature Testing:* Various scenario-based tests are run through the |
127 | :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions | 125 | :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distributions |
128 | we support. | 126 | we support. |
129 | 127 | ||
130 | - *Image Testing:* Image tests initiated through the following command:: | 128 | - *Image Testing:* Image tests initiated through the following command:: |
131 | 129 | ||
132 | $ bitbake image -c testimage | 130 | $ bitbake image -c testimage |
133 | 131 | ||
134 | The tests utilize the :ref:`testimage* <ref-classes-testimage*>` | 132 | The tests use the :ref:`ref-classes-testimage` |
135 | classes and the :ref:`ref-tasks-testimage` task. | 133 | class and the :ref:`ref-tasks-testimage` task. |
136 | 134 | ||
137 | - *Layer Testing:* The Autobuilder has the possibility to test whether | 135 | - *Layer Testing:* The Autobuilder has the possibility to test whether |
138 | specific layers work with the test of the system. The layers tested | 136 | specific layers work with the test of the system. The layers tested |
@@ -142,7 +140,7 @@ thefollowing types of tests: | |||
142 | - *Package Testing:* A Package Test (ptest) runs tests against packages | 140 | - *Package Testing:* A Package Test (ptest) runs tests against packages |
143 | built by the OpenEmbedded build system on the target machine. See the | 141 | built by the OpenEmbedded build system on the target machine. See the |
144 | :ref:`Testing Packages With | 142 | :ref:`Testing Packages With |
145 | ptest <dev-manual/common-tasks:Testing Packages With ptest>` section | 143 | ptest <dev-manual/packages:Testing Packages With ptest>` section |
146 | in the Yocto Project Development Tasks Manual and the | 144 | in the Yocto Project Development Tasks Manual and the |
147 | ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more | 145 | ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more |
148 | information on Ptest. | 146 | information on Ptest. |
@@ -151,7 +149,7 @@ thefollowing types of tests: | |||
151 | 149 | ||
152 | $ bitbake image -c testsdk | 150 | $ bitbake image -c testsdk |
153 | 151 | ||
154 | The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and | 152 | The tests use the :ref:`ref-classes-testsdk` class and |
155 | the ``do_testsdk`` task. | 153 | the ``do_testsdk`` task. |
156 | 154 | ||
157 | - *Unit Testing:* Unit tests on various components of the system run | 155 | - *Unit Testing:* Unit tests on various components of the system run |
@@ -174,48 +172,55 @@ Tests map into the codebase as follows: | |||
174 | which include the fetchers. The tests are located in | 172 | which include the fetchers. The tests are located in |
175 | ``bitbake/lib/*/tests``. | 173 | ``bitbake/lib/*/tests``. |
176 | 174 | ||
175 | Some of these tests run the ``bitbake`` command, so ``bitbake/bin`` | ||
176 | must be added to the ``PATH`` before running ``bitbake-selftest``. | ||
177 | From within the BitBake repository, run the following:: | 177 | From within the BitBake repository, run the following:: |
178 | 178 | ||
179 | $ bitbake-selftest | 179 | $ export PATH=$PWD/bin:$PATH |
180 | 180 | ||
181 | To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` | 181 | After that, you can run the selftest script:: |
182 | variable when running "bitbake-selftest" as follows:: | ||
183 | 182 | ||
184 | $ BB_SKIP_NETTEST=yes bitbake-selftest | 183 | $ bitbake-selftest |
185 | 184 | ||
186 | The default output is quiet and just prints a summary of what was | 185 | The default output is quiet and just prints a summary of what was |
187 | run. To see more information, there is a verbose option:: | 186 | run. To see more information, there is a verbose option:: |
188 | 187 | ||
189 | $ bitbake-selftest -v | 188 | $ bitbake-selftest -v |
190 | 189 | ||
190 | To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS`` | ||
191 | variable when running ``bitbake-selftest`` as follows:: | ||
192 | |||
193 | $ BB_SKIP_NETTESTS=yes bitbake-selftest | ||
194 | |||
191 | Use this option when you wish to skip tests that access the network, | 195 | Use this option when you wish to skip tests that access the network, |
192 | which are mostly necessary to test the fetcher modules. To specify | 196 | which are mostly necessary to test the fetcher modules. To specify |
193 | individual test modules to run, append the test module name to the | 197 | individual test modules to run, append the test module name to the |
194 | "bitbake-selftest" command. For example, to specify the tests for the | 198 | ``bitbake-selftest`` command. For example, to specify the tests for |
195 | bb.data.module, run:: | 199 | ``bb.tests.data.DataExpansions``, run:: |
196 | 200 | ||
197 | $ bitbake-selftest bb.test.data.module | 201 | $ bitbake-selftest bb.tests.data.DataExpansions |
198 | 202 | ||
199 | You can also specify individual tests by defining the full name and module | 203 | You can also specify individual tests by defining the full name and module |
200 | plus the class path of the test, for example:: | 204 | plus the class path of the test, for example:: |
201 | 205 | ||
202 | $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override | 206 | $ bitbake-selftest bb.tests.data.DataExpansions.test_one_var |
203 | 207 | ||
204 | The tests are based on `Python | 208 | The tests are based on |
205 | unittest <https://docs.python.org/3/library/unittest.html>`__. | 209 | `Python unittest <https://docs.python.org/3/library/unittest.html>`__. |
206 | 210 | ||
207 | - *oe-selftest:* | 211 | - *oe-selftest:* |
208 | 212 | ||
209 | - These tests use OE to test the workflows, which include testing | 213 | - These tests use OE to test the workflows, which include testing |
210 | specific features, behaviors of tasks, and API unit tests. | 214 | specific features, behaviors of tasks, and API unit tests. |
211 | 215 | ||
212 | - The tests can take advantage of parallelism through the "-j" | 216 | - The tests can take advantage of parallelism through the ``-j`` |
213 | option, which can specify a number of threads to spread the tests | 217 | option, which can specify a number of threads to spread the tests |
214 | across. Note that all tests from a given class of tests will run | 218 | across. Note that all tests from a given class of tests will run |
215 | in the same thread. To parallelize large numbers of tests you can | 219 | in the same thread. To parallelize large numbers of tests you can |
216 | split the class into multiple units. | 220 | split the class into multiple units. |
217 | 221 | ||
218 | - The tests are based on Python unittest. | 222 | - The tests are based on |
223 | `Python unittest <https://docs.python.org/3/library/unittest.html>`__. | ||
219 | 224 | ||
220 | - The code for the tests resides in | 225 | - The code for the tests resides in |
221 | ``meta/lib/oeqa/selftest/cases/``. | 226 | ``meta/lib/oeqa/selftest/cases/``. |
@@ -225,18 +230,18 @@ Tests map into the codebase as follows: | |||
225 | $ oe-selftest -a | 230 | $ oe-selftest -a |
226 | 231 | ||
227 | - To run a specific test, use the following command form where | 232 | - To run a specific test, use the following command form where |
228 | testname is the name of the specific test:: | 233 | ``testname`` is the name of the specific test:: |
229 | 234 | ||
230 | $ oe-selftest -r <testname> | 235 | $ oe-selftest -r <testname> |
231 | 236 | ||
232 | For example, the following command would run the tinfoil | 237 | For example, the following command would run the ``tinfoil`` |
233 | getVar API test:: | 238 | ``getVar`` API test:: |
234 | 239 | ||
235 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar | 240 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar |
236 | 241 | ||
237 | It is also possible to run a set | 242 | It is also possible to run a set |
238 | of tests. For example the following command will run all of the | 243 | of tests. For example the following command will run all of the |
239 | tinfoil tests:: | 244 | ``tinfoil`` tests:: |
240 | 245 | ||
241 | $ oe-selftest -r tinfoil | 246 | $ oe-selftest -r tinfoil |
242 | 247 | ||
@@ -271,7 +276,7 @@ Tests map into the codebase as follows: | |||
271 | - These tests build an extended SDK (eSDK), install that eSDK, and | 276 | - These tests build an extended SDK (eSDK), install that eSDK, and |
272 | run tests against the eSDK. | 277 | run tests against the eSDK. |
273 | 278 | ||
274 | - The code for these tests resides in ``meta/lib/oeqa/esdk``. | 279 | - The code for these tests resides in ``meta/lib/oeqa/sdkext/cases/``. |
275 | 280 | ||
276 | - To run the tests, use the following command form:: | 281 | - To run the tests, use the following command form:: |
277 | 282 | ||
@@ -298,13 +303,13 @@ Tests map into the codebase as follows: | |||
298 | Git repository. | 303 | Git repository. |
299 | 304 | ||
300 | Use the ``oe-build-perf-report`` command to generate text reports | 305 | Use the ``oe-build-perf-report`` command to generate text reports |
301 | and HTML reports with graphs of the performance data. For | 306 | and HTML reports with graphs of the performance data. See |
302 | examples, see | 307 | :yocto_dl:`html </releases/yocto/yocto-4.3/testresults/buildperf-debian11/perf-debian11_nanbield_20231019191258_15b576c410.html>` |
303 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` | ||
304 | and | 308 | and |
305 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. | 309 | :yocto_dl:`txt </releases/yocto/yocto-4.3/testresults/buildperf-debian11/perf-debian11_nanbield_20231019191258_15b576c410.txt>` |
310 | examples. | ||
306 | 311 | ||
307 | - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. | 312 | - The tests are contained in ``meta/lib/oeqa/buildperf/test_basic.py``. |
308 | 313 | ||
309 | Test Examples | 314 | Test Examples |
310 | ============= | 315 | ============= |
@@ -312,16 +317,14 @@ Test Examples | |||
312 | This section provides example tests for each of the tests listed in the | 317 | This section provides example tests for each of the tests listed in the |
313 | :ref:`test-manual/intro:How Tests Map to Areas of Code` section. | 318 | :ref:`test-manual/intro:How Tests Map to Areas of Code` section. |
314 | 319 | ||
315 | For oeqa tests, testcases for each area reside in the main test | 320 | - ``oe-selftest`` testcases reside in the ``meta/lib/oeqa/selftest/cases`` directory. |
316 | directory at ``meta/lib/oeqa/selftest/cases`` directory. | ||
317 | 321 | ||
318 | For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/`` | 322 | - ``bitbake-selftest`` testcases reside in the ``bitbake/lib/bb/tests/`` directory. |
319 | directory. | ||
320 | 323 | ||
321 | ``bitbake-selftest`` | 324 | ``bitbake-selftest`` |
322 | -------------------- | 325 | -------------------- |
323 | 326 | ||
324 | A simple test example from ``lib/bb/tests/data.py`` is:: | 327 | A simple test example from ``bitbake/lib/bb/tests/data.py`` is:: |
325 | 328 | ||
326 | class DataExpansions(unittest.TestCase): | 329 | class DataExpansions(unittest.TestCase): |
327 | def setUp(self): | 330 | def setUp(self): |
@@ -334,21 +337,24 @@ A simple test example from ``lib/bb/tests/data.py`` is:: | |||
334 | val = self.d.expand("${foo}") | 337 | val = self.d.expand("${foo}") |
335 | self.assertEqual(str(val), "value_of_foo") | 338 | self.assertEqual(str(val), "value_of_foo") |
336 | 339 | ||
337 | In this example, a ``DataExpansions`` class of tests is created, | 340 | In this example, a ``DataExpansions`` class of tests is created, derived from |
338 | derived from standard python unittest. The class has a common ``setUp`` | 341 | standard `Python unittest <https://docs.python.org/3/library/unittest.html>`__. |
339 | function which is shared by all the tests in the class. A simple test is | 342 | The class has a common ``setUp`` function which is shared by all the tests in |
340 | then added to test that when a variable is expanded, the correct value | 343 | the class. A simple test is then added to test that when a variable is |
341 | is found. | 344 | expanded, the correct value is found. |
342 | 345 | ||
343 | Bitbake selftests are straightforward python unittest. Refer to the | 346 | BitBake selftests are straightforward |
344 | Python unittest documentation for additional information on writing | 347 | `Python unittest <https://docs.python.org/3/library/unittest.html>`__. |
345 | these tests at: https://docs.python.org/3/library/unittest.html. | 348 | Refer to the `Python unittest documentation |
349 | <https://docs.python.org/3/library/unittest.html>`__ for additional information | ||
350 | on writing such tests. | ||
346 | 351 | ||
347 | ``oe-selftest`` | 352 | ``oe-selftest`` |
348 | --------------- | 353 | --------------- |
349 | 354 | ||
350 | These tests are more complex due to the setup required behind the scenes | 355 | These tests are more complex due to the setup required behind the scenes |
351 | for full builds. Rather than directly using Python's unittest, the code | 356 | for full builds. Rather than directly using `Python unittest |
357 | <https://docs.python.org/3/library/unittest.html>`__, the code | ||
352 | wraps most of the standard objects. The tests can be simple, such as | 358 | wraps most of the standard objects. The tests can be simple, such as |
353 | testing a command from within the OE build environment using the | 359 | testing a command from within the OE build environment using the |
354 | following example:: | 360 | following example:: |
@@ -385,14 +391,14 @@ so tests within a given test class should always run in the same build, | |||
385 | while tests in different classes or modules may be split into different | 391 | while tests in different classes or modules may be split into different |
386 | builds. There is no data store available for these tests since the tests | 392 | builds. There is no data store available for these tests since the tests |
387 | launch the ``bitbake`` command and exist outside of its context. As a | 393 | launch the ``bitbake`` command and exist outside of its context. As a |
388 | result, common bitbake library functions (bb.\*) are also unavailable. | 394 | result, common BitBake library functions (``bb.\*``) are also unavailable. |
389 | 395 | ||
390 | ``testimage`` | 396 | ``testimage`` |
391 | ------------- | 397 | ------------- |
392 | 398 | ||
393 | These tests are run once an image is up and running, either on target | 399 | These tests are run once an image is up and running, either on target |
394 | hardware or under QEMU. As a result, they are assumed to be running in a | 400 | hardware or under QEMU. As a result, they are assumed to be running in a |
395 | target image environment, as opposed to a host build environment. A | 401 | target image environment, as opposed to in a host build environment. A |
396 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains | 402 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains |
397 | the following:: | 403 | the following:: |
398 | 404 | ||
@@ -407,19 +413,19 @@ the following:: | |||
407 | 413 | ||
408 | In this example, the ``OERuntimeTestCase`` class wraps | 414 | In this example, the ``OERuntimeTestCase`` class wraps |
409 | ``unittest.TestCase``. Within the test, ``self.target`` represents the | 415 | ``unittest.TestCase``. Within the test, ``self.target`` represents the |
410 | target system, where commands can be run on it using the ``run()`` | 416 | target system, where commands can be run using the ``run()`` |
411 | method. | 417 | method. |
412 | 418 | ||
413 | To ensure certain test or package dependencies are met, you can use the | 419 | To ensure certain tests or package dependencies are met, you can use the |
414 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test | 420 | ``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test |
415 | in this example would only make sense if python3-core is installed in | 421 | in this example would only make sense if ``python3-core`` is installed in |
416 | the image. | 422 | the image. |
417 | 423 | ||
418 | ``testsdk_ext`` | 424 | ``testsdk_ext`` |
419 | --------------- | 425 | --------------- |
420 | 426 | ||
421 | These tests are run against built extensible SDKs (eSDKs). The tests can | 427 | These tests are run against built extensible SDKs (eSDKs). The tests can |
422 | assume that the eSDK environment has already been setup. An example from | 428 | assume that the eSDK environment has already been set up. An example from |
423 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: | 429 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: |
424 | 430 | ||
425 | class DevtoolTest(OESDKExtTestCase): | 431 | class DevtoolTest(OESDKExtTestCase): |
@@ -466,15 +472,15 @@ following:: | |||
466 | output = self._run(cmd) | 472 | output = self._run(cmd) |
467 | self.assertEqual(output, "Hello, world\n") | 473 | self.assertEqual(output, "Hello, world\n") |
468 | 474 | ||
469 | In this example, if nativesdk-python3-core has been installed into the SDK, the code runs | 475 | In this example, if ``nativesdk-python3-core`` has been installed into the SDK, |
470 | the python3 interpreter with a basic command to check it is working | 476 | the code runs the ``python3`` interpreter with a basic command to check it is |
471 | correctly. The test would only run if python3 is installed in the SDK. | 477 | working correctly. The test would only run if Python3 is installed in the SDK. |
472 | 478 | ||
473 | ``oe-build-perf-test`` | 479 | ``oe-build-perf-test`` |
474 | ---------------------- | 480 | ---------------------- |
475 | 481 | ||
476 | The performance tests usually measure how long operations take and the | 482 | The performance tests usually measure how long operations take and the |
477 | resource utilisation as that happens. An example from | 483 | resource utilization as that happens. An example from |
478 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: | 484 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: |
479 | 485 | ||
480 | class Test3(BuildPerfTestCase): | 486 | class Test3(BuildPerfTestCase): |
@@ -506,15 +512,15 @@ workers, consider the following: | |||
506 | 512 | ||
507 | **Running "cleanall" is not permitted.** | 513 | **Running "cleanall" is not permitted.** |
508 | 514 | ||
509 | This can delete files from DL_DIR which would potentially break other | 515 | This can delete files from :term:`DL_DIR` which would potentially break other |
510 | builds running in parallel. If this is required, DL_DIR must be set to | 516 | builds running in parallel. If this is required, :term:`DL_DIR` must be set to |
511 | an isolated directory. | 517 | an isolated directory. |
512 | 518 | ||
513 | **Running "cleansstate" is not permitted.** | 519 | **Running "cleansstate" is not permitted.** |
514 | 520 | ||
515 | This can delete files from SSTATE_DIR which would potentially break | 521 | This can delete files from :term:`SSTATE_DIR` which would potentially break |
516 | other builds running in parallel. If this is required, SSTATE_DIR must | 522 | other builds running in parallel. If this is required, :term:`SSTATE_DIR` must |
517 | be set to an isolated directory. Alternatively, you can use the "-f" | 523 | be set to an isolated directory. Alternatively, you can use the ``-f`` |
518 | option with the ``bitbake`` command to "taint" tasks by changing the | 524 | option with the ``bitbake`` command to "taint" tasks by changing the |
519 | sstate checksums to ensure sstate cache items will not be reused. | 525 | sstate checksums to ensure sstate cache items will not be reused. |
520 | 526 | ||
@@ -524,5 +530,5 @@ This is particularly true for oe-selftests since these can run in | |||
524 | parallel and changing metadata leads to changing checksums, which | 530 | parallel and changing metadata leads to changing checksums, which |
525 | confuses BitBake while running in parallel. If this is necessary, copy | 531 | confuses BitBake while running in parallel. If this is necessary, copy |
526 | layers to a temporary location and modify them. Some tests need to | 532 | layers to a temporary location and modify them. Some tests need to |
527 | change metadata, such as the devtool tests. To prevent the metadate from | 533 | change metadata, such as the devtool tests. To protect the metadata from |
528 | changes, set up temporary copies of that data first. | 534 | changes, set up temporary copies of that data first. |