diff options
author | Mark Morton <Mark.Morton@windriver.com> | 2020-09-14 16:11:31 -0700 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2020-09-17 10:09:35 +0100 |
commit | 74f212d2e1db87ccfecaf9385ab280123d5fb168 (patch) | |
tree | 79e85fac733b78eb11cb19ca9753ed44dbf8358e /documentation | |
parent | fc876832cb57a1a3e61150778a9cfd7acffbc2b6 (diff) | |
download | poky-74f212d2e1db87ccfecaf9385ab280123d5fb168.tar.gz |
sphinx: test-manual code block, link, and format update
(From yocto-docs rev: 928c212ec4ad6e09524fdf8147aa7daf244677b0)
Signed-off-by: Mark Morton <mark.morton@windriver.com>
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation')
3 files changed, 257 insertions, 184 deletions
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst index 2f11e2beb3..b0bc24c9b0 100644 --- a/documentation/test-manual/test-manual-intro.rst +++ b/documentation/test-manual/test-manual-intro.rst | |||
@@ -23,26 +23,25 @@ project core. | |||
23 | 23 | ||
24 | Currently, the Yocto Project Test Environment Manual has no projected | 24 | Currently, the Yocto Project Test Environment Manual has no projected |
25 | release date. This manual is a work-in-progress and is being initially | 25 | release date. This manual is a work-in-progress and is being initially |
26 | loaded with information from the `README <>`__ files and notes from key | 26 | loaded with information from the README files and notes from key |
27 | engineers: | 27 | engineers: |
28 | 28 | ||
29 | - *``yocto-autobuilder2``:* This | 29 | - *yocto-autobuilder2:* This |
30 | ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md | 30 | :yocto_git:`README.md </cgit.cgi/yocto-autobuilder2/tree/README.md>` |
31 | is the main README which detials how to set up the Yocto Project | 31 | is the main README which detials how to set up the Yocto Project |
32 | Autobuilder. The ``yocto-autobuilder2`` repository represents the | 32 | Autobuilder. The ``yocto-autobuilder2`` repository represents the |
33 | Yocto Project's console UI plugin to Buildbot and the configuration | 33 | Yocto Project's console UI plugin to Buildbot and the configuration |
34 | necessary to configure Buildbot to perform the testing the project | 34 | necessary to configure Buildbot to perform the testing the project |
35 | requires. | 35 | requires. |
36 | 36 | ||
37 | - *``yocto-autobuilder-helper``:* This | 37 | - *yocto-autobuilder-helper:* This :yocto_git:`README </cgit.cgi/yocto-autobuilder-helper/tree/README/>` |
38 | ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README | ||
39 | and repository contains Yocto Project Autobuilder Helper scripts and | 38 | and repository contains Yocto Project Autobuilder Helper scripts and |
40 | configuration. The ``yocto-autobuilder-helper`` repository contains | 39 | configuration. The ``yocto-autobuilder-helper`` repository contains |
41 | the "glue" logic that defines which tests to run and how to run them. | 40 | the "glue" logic that defines which tests to run and how to run them. |
42 | As a result, it can be used by any Continuous Improvement (CI) system | 41 | As a result, it can be used by any Continuous Improvement (CI) system |
43 | to run builds, support getting the correct code revisions, configure | 42 | to run builds, support getting the correct code revisions, configure |
44 | builds and layers, run builds, and collect results. The code is | 43 | builds and layers, run builds, and collect results. The code is |
45 | independent of any CI system, which means the code can work Buildbot, | 44 | independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__, |
46 | Jenkins, or others. This repository has a branch per release of the | 45 | Jenkins, or others. This repository has a branch per release of the |
47 | project defining the tests to run on a per release basis. | 46 | project defining the tests to run on a per release basis. |
48 | 47 | ||
@@ -124,21 +123,22 @@ thefollowing types of tests: | |||
124 | ensure we have no single point of failure and can ensure the | 123 | ensure we have no single point of failure and can ensure the |
125 | different distros work effectively. | 124 | different distros work effectively. |
126 | 125 | ||
127 | - *eSDK Testing:* Image tests initiated through the following command: | 126 | - *eSDK Testing:* Image tests initiated through the following command:: |
128 | $ bitbake image -c testsdkext The tests utilize the ``testsdkext`` | 127 | |
129 | class and the ``do_testsdkext`` task. | 128 | $ bitbake image -c testsdkext |
129 | |||
130 | The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task. | ||
130 | 131 | ||
131 | - *Feature Testing:* Various scenario-based tests are run through the | 132 | - *Feature Testing:* Various scenario-based tests are run through the |
132 | `OpenEmbedded | 133 | :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions |
133 | Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__ | ||
134 | (oe-selftest). We test oe-selftest on each of the main distrubutions | ||
135 | we support. | 134 | we support. |
136 | 135 | ||
137 | - *Image Testing:* Image tests initiated through the following command: | 136 | - *Image Testing:* Image tests initiated through the following command:: |
138 | $ bitbake image -c testimage The tests utilize the | 137 | |
139 | :ref:`testimage* <ref-classes-testimage*>` | 138 | $ bitbake image -c testimage |
140 | classes and the | 139 | |
141 | :ref:`ref-tasks-testimage` task. | 140 | The tests utilize the :ref:`testimage* <ref-classes-testimage*>` |
141 | classes and the :ref:`ref-tasks-testimage` task. | ||
142 | 142 | ||
143 | - *Layer Testing:* The Autobuilder has the possibility to test whether | 143 | - *Layer Testing:* The Autobuilder has the possibility to test whether |
144 | specific layers work with the test of the system. The layers tested | 144 | specific layers work with the test of the system. The layers tested |
@@ -147,20 +147,22 @@ thefollowing types of tests: | |||
147 | 147 | ||
148 | - *Package Testing:* A Package Test (ptest) runs tests against packages | 148 | - *Package Testing:* A Package Test (ptest) runs tests against packages |
149 | built by the OpenEmbedded build system on the target machine. See the | 149 | built by the OpenEmbedded build system on the target machine. See the |
150 | "`Testing Packages With | 150 | :ref:`Testing Packages With |
151 | ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section | 151 | ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section |
152 | in the Yocto Project Development Tasks Manual and the | 152 | in the Yocto Project Development Tasks Manual and the |
153 | ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more | 153 | ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more |
154 | information on Ptest. | 154 | information on Ptest. |
155 | 155 | ||
156 | - *SDK Testing:* Image tests initiated through the following command: $ | 156 | - *SDK Testing:* Image tests initiated through the following command:: |
157 | bitbake image -c testsdk The tests utilize the | 157 | |
158 | :ref:`testsdk <ref-classes-testsdk>` class and | 158 | $ bitbake image -c testsdk |
159 | |||
160 | The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and | ||
159 | the ``do_testsdk`` task. | 161 | the ``do_testsdk`` task. |
160 | 162 | ||
161 | - *Unit Testing:* Unit tests on various components of the system run | 163 | - *Unit Testing:* Unit tests on various components of the system run |
162 | through ``oe-selftest`` and | 164 | through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and |
163 | ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__. | 165 | :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`. |
164 | 166 | ||
165 | - *Automatic Upgrade Helper:* This target tests whether new versions of | 167 | - *Automatic Upgrade Helper:* This target tests whether new versions of |
166 | software are available and whether we can automatically upgrade to | 168 | software are available and whether we can automatically upgrade to |
@@ -174,36 +176,43 @@ How Tests Map to Areas of Code | |||
174 | 176 | ||
175 | Tests map into the codebase as follows: | 177 | Tests map into the codebase as follows: |
176 | 178 | ||
177 | - *bitbake-selftest*: | 179 | - *bitbake-selftest:* |
178 | 180 | ||
179 | These tests are self-contained and test BitBake as well as its APIs, | 181 | These tests are self-contained and test BitBake as well as its APIs, |
180 | which include the fetchers. The tests are located in | 182 | which include the fetchers. The tests are located in |
181 | ``bitbake/lib/*/tests``. | 183 | ``bitbake/lib/*/tests``. |
182 | 184 | ||
183 | From within the BitBake repository, run the following: $ | 185 | From within the BitBake repository, run the following:: |
184 | bitbake-selftest | 186 | |
187 | $ bitbake-selftest | ||
185 | 188 | ||
186 | To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` | 189 | To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` |
187 | variable when running "bitbake-selftest" as follows: $ | 190 | variable when running "bitbake-selftest" as follows:: |
188 | BB_SKIP_NETTEST=yes bitbake-selftest | 191 | |
192 | $ BB_SKIP_NETTEST=yes bitbake-selftest | ||
189 | 193 | ||
190 | The default output is quiet and just prints a summary of what was | 194 | The default output is quiet and just prints a summary of what was |
191 | run. To see more information, there is a verbose option:$ | 195 | run. To see more information, there is a verbose option:: |
192 | bitbake-selftest -v | 196 | |
197 | $ bitbake-selftest -v | ||
193 | 198 | ||
194 | Use this option when you wish to skip tests that access the network, | 199 | Use this option when you wish to skip tests that access the network, |
195 | which are mostly necessary to test the fetcher modules. To specify | 200 | which are mostly necessary to test the fetcher modules. To specify |
196 | individual test modules to run, append the test module name to the | 201 | individual test modules to run, append the test module name to the |
197 | "bitbake-selftest" command. For example, to specify the tests for the | 202 | "bitbake-selftest" command. For example, to specify the tests for the |
198 | bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can | 203 | bb.data.module, run:: |
199 | also specify individual tests by defining the full name and module | 204 | |
200 | plus the class path of the test, for example: $ bitbake-selftest | 205 | $ bitbake-selftest bb.test.data.module |
201 | bb.tests.data.TestOverrides.test_one_override | 206 | |
207 | You can also specify individual tests by defining the full name and module | ||
208 | plus the class path of the test, for example:: | ||
209 | |||
210 | $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override | ||
202 | 211 | ||
203 | The tests are based on `Python | 212 | The tests are based on `Python |
204 | unittest <https://docs.python.org/3/library/unittest.html>`__. | 213 | unittest <https://docs.python.org/3/library/unittest.html>`__. |
205 | 214 | ||
206 | - *oe-selftest*: | 215 | - *oe-selftest:* |
207 | 216 | ||
208 | - These tests use OE to test the workflows, which include testing | 217 | - These tests use OE to test the workflows, which include testing |
209 | specific features, behaviors of tasks, and API unit tests. | 218 | specific features, behaviors of tasks, and API unit tests. |
@@ -219,31 +228,40 @@ Tests map into the codebase as follows: | |||
219 | - The code for the tests resides in | 228 | - The code for the tests resides in |
220 | ``meta/lib/oeqa/selftest/cases/``. | 229 | ``meta/lib/oeqa/selftest/cases/``. |
221 | 230 | ||
222 | - To run all the tests, enter the following command: $ oe-selftest | 231 | - To run all the tests, enter the following command:: |
223 | -a | 232 | |
233 | $ oe-selftest -a | ||
224 | 234 | ||
225 | - To run a specific test, use the following command form where | 235 | - To run a specific test, use the following command form where |
226 | testname is the name of the specific test: $ oe-selftest -r | 236 | testname is the name of the specific test:: |
227 | testname For example, the following command would run the tinfoil | 237 | |
228 | getVar API test:$ oe-selftest -r | 238 | $ oe-selftest -r <testname> |
229 | tinfoil.TinfoilTests.test_getvarIt is also possible to run a set | 239 | |
240 | For example, the following command would run the tinfoil | ||
241 | getVar API test:: | ||
242 | |||
243 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar | ||
244 | |||
245 | It is also possible to run a set | ||
230 | of tests. For example the following command will run all of the | 246 | of tests. For example the following command will run all of the |
231 | tinfoil tests:$ oe-selftest -r tinfoil | 247 | tinfoil tests:: |
248 | |||
249 | $ oe-selftest -r tinfoil | ||
232 | 250 | ||
233 | - *testimage:* | 251 | - *testimage:* |
234 | 252 | ||
235 | - These tests build an image, boot it, and run tests against the | 253 | - These tests build an image, boot it, and run tests against the |
236 | image's content. | 254 | image's content. |
237 | 255 | ||
238 | - The code for these tests resides in | 256 | - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``. |
239 | ``meta/lib/oeqa/runtime/cases/``. | 257 | |
258 | - You need to set the :term:`IMAGE_CLASSES` variable as follows:: | ||
240 | 259 | ||
241 | - You need to set the | 260 | IMAGE_CLASSES += "testimage" |
242 | :term:`IMAGE_CLASSES` | ||
243 | variable as follows: IMAGE_CLASSES += "testimage" | ||
244 | 261 | ||
245 | - Run the tests using the following command form: $ bitbake image -c | 262 | - Run the tests using the following command form:: |
246 | testimage | 263 | |
264 | $ bitbake image -c testimage | ||
247 | 265 | ||
248 | - *testsdk:* | 266 | - *testsdk:* |
249 | 267 | ||
@@ -252,8 +270,9 @@ Tests map into the codebase as follows: | |||
252 | 270 | ||
253 | - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. | 271 | - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. |
254 | 272 | ||
255 | - Run the test using the following command form: $ bitbake image -c | 273 | - Run the test using the following command form:: |
256 | testsdk | 274 | |
275 | $ bitbake image -c testsdk | ||
257 | 276 | ||
258 | - *testsdk_ext:* | 277 | - *testsdk_ext:* |
259 | 278 | ||
@@ -262,8 +281,9 @@ Tests map into the codebase as follows: | |||
262 | 281 | ||
263 | - The code for these tests resides in ``meta/lib/oeqa/esdk``. | 282 | - The code for these tests resides in ``meta/lib/oeqa/esdk``. |
264 | 283 | ||
265 | - To run the tests, use the following command form: $ bitbake image | 284 | - To run the tests, use the following command form:: |
266 | -c testsdkext | 285 | |
286 | $ bitbake image -c testsdkext | ||
267 | 287 | ||
268 | - *oe-build-perf-test:* | 288 | - *oe-build-perf-test:* |
269 | 289 | ||
@@ -272,8 +292,11 @@ Tests map into the codebase as follows: | |||
272 | 292 | ||
273 | - The code for these tests resides in ``meta/lib/oeqa/buildperf``. | 293 | - The code for these tests resides in ``meta/lib/oeqa/buildperf``. |
274 | 294 | ||
275 | - To run the tests, use the following command form: $ | 295 | - To run the tests, use the following command form:: |
276 | oe-build-perf-test optionsThe command takes a number of options, | 296 | |
297 | $ oe-build-perf-test <options> | ||
298 | |||
299 | The command takes a number of options, | ||
277 | such as where to place the test results. The Autobuilder Helper | 300 | such as where to place the test results. The Autobuilder Helper |
278 | Scripts include the ``build-perf-test-wrapper`` script with | 301 | Scripts include the ``build-perf-test-wrapper`` script with |
279 | examples of how to use the oe-build-perf-test from the command | 302 | examples of how to use the oe-build-perf-test from the command |
@@ -285,9 +308,9 @@ Tests map into the codebase as follows: | |||
285 | Use the ``oe-build-perf-report`` command to generate text reports | 308 | Use the ``oe-build-perf-report`` command to generate text reports |
286 | and HTML reports with graphs of the performance data. For | 309 | and HTML reports with graphs of the performance data. For |
287 | examples, see | 310 | examples, see |
288 | `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html <#>`__ | 311 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` |
289 | and | 312 | and |
290 | `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt <#>`__. | 313 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. |
291 | 314 | ||
292 | - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. | 315 | - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. |
293 | 316 | ||
@@ -295,7 +318,7 @@ Test Examples | |||
295 | ============= | 318 | ============= |
296 | 319 | ||
297 | This section provides example tests for each of the tests listed in the | 320 | This section provides example tests for each of the tests listed in the |
298 | `How Tests Map to Areas of Code <#test-test-mapping>`__ section. | 321 | :ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section. |
299 | 322 | ||
300 | For oeqa tests, testcases for each area reside in the main test | 323 | For oeqa tests, testcases for each area reside in the main test |
301 | directory at ``meta/lib/oeqa/selftest/cases`` directory. | 324 | directory at ``meta/lib/oeqa/selftest/cases`` directory. |
@@ -308,14 +331,20 @@ directory. | |||
308 | ``bitbake-selftest`` | 331 | ``bitbake-selftest`` |
309 | -------------------- | 332 | -------------------- |
310 | 333 | ||
311 | A simple test example from ``lib/bb/tests/data.py`` is: class | 334 | A simple test example from ``lib/bb/tests/data.py`` is:: |
312 | DataExpansions(unittest.TestCase): def setUp(self): self.d = | 335 | |
313 | bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] = | 336 | class DataExpansions(unittest.TestCase): |
314 | "value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def | 337 | def setUp(self): |
315 | test_one_var(self): val = self.d.expand("${foo}") | 338 | self.d = bb.data.init() |
316 | self.assertEqual(str(val), "value_of_foo") | 339 | self.d["foo"] = "value_of_foo" |
340 | self.d["bar"] = "value_of_bar" | ||
341 | self.d["value_of_foo"] = "value_of_'value_of_foo'" | ||
317 | 342 | ||
318 | In this example, a ```DataExpansions`` <>`__ class of tests is created, | 343 | def test_one_var(self): |
344 | val = self.d.expand("${foo}") | ||
345 | self.assertEqual(str(val), "value_of_foo") | ||
346 | |||
347 | In this example, a ``DataExpansions`` class of tests is created, | ||
319 | derived from standard python unittest. The class has a common ``setUp`` | 348 | derived from standard python unittest. The class has a common ``setUp`` |
320 | function which is shared by all the tests in the class. A simple test is | 349 | function which is shared by all the tests in the class. A simple test is |
321 | then added to test that when a variable is expanded, the correct value | 350 | then added to test that when a variable is expanded, the correct value |
@@ -323,7 +352,7 @@ is found. | |||
323 | 352 | ||
324 | Bitbake selftests are straightforward python unittest. Refer to the | 353 | Bitbake selftests are straightforward python unittest. Refer to the |
325 | Python unittest documentation for additional information on writing | 354 | Python unittest documentation for additional information on writing |
326 | these tests at: `https://docs.python.org/3/library/unittest.html <#>`__. | 355 | these tests at: https://docs.python.org/3/library/unittest.html. |
327 | 356 | ||
328 | .. _oe-selftest-example: | 357 | .. _oe-selftest-example: |
329 | 358 | ||
@@ -334,14 +363,15 @@ These tests are more complex due to the setup required behind the scenes | |||
334 | for full builds. Rather than directly using Python's unittest, the code | 363 | for full builds. Rather than directly using Python's unittest, the code |
335 | wraps most of the standard objects. The tests can be simple, such as | 364 | wraps most of the standard objects. The tests can be simple, such as |
336 | testing a command from within the OE build environment using the | 365 | testing a command from within the OE build environment using the |
337 | following example:class BitbakeLayers(OESelftestTestCase): def | 366 | following example:: |
338 | test_bitbakelayers_showcrossdepends(self): result = | 367 | |
339 | runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in | 368 | class BitbakeLayers(OESelftestTestCase): |
340 | result.output, msg = "No dependencies were shown. bitbake-layers | 369 | def test_bitbakelayers_showcrossdepends(self): |
341 | show-cross-depends output: %s"% result.output) | 370 | result = runCmd('bitbake-layers show-cross-depends') |
371 | self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output) | ||
342 | 372 | ||
343 | This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, | 373 | This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, |
344 | creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived | 374 | creates a testcase from the ``OESelftestTestCase`` class, derived |
345 | from ``unittest.TestCase``, which runs the ``bitbake-layers`` command | 375 | from ``unittest.TestCase``, which runs the ``bitbake-layers`` command |
346 | and checks the output to ensure it contains something we know should be | 376 | and checks the output to ensure it contains something we know should be |
347 | here. | 377 | here. |
@@ -367,7 +397,7 @@ so tests within a given test class should always run in the same build, | |||
367 | while tests in different classes or modules may be split into different | 397 | while tests in different classes or modules may be split into different |
368 | builds. There is no data store available for these tests since the tests | 398 | builds. There is no data store available for these tests since the tests |
369 | launch the ``bitbake`` command and exist outside of its context. As a | 399 | launch the ``bitbake`` command and exist outside of its context. As a |
370 | result, common bitbake library functions (bb.*) are also unavailable. | 400 | result, common bitbake library functions (bb.\*) are also unavailable. |
371 | 401 | ||
372 | .. _testimage-example: | 402 | .. _testimage-example: |
373 | 403 | ||
@@ -378,14 +408,18 @@ These tests are run once an image is up and running, either on target | |||
378 | hardware or under QEMU. As a result, they are assumed to be running in a | 408 | hardware or under QEMU. As a result, they are assumed to be running in a |
379 | target image environment, as opposed to a host build environment. A | 409 | target image environment, as opposed to a host build environment. A |
380 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains | 410 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains |
381 | the following:class PythonTest(OERuntimeTestCase): | 411 | the following:: |
382 | @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core']) | 412 | |
383 | def test_python3(self): cmd = "python3 -c \\"import codecs; | 413 | class PythonTest(OERuntimeTestCase): |
384 | print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output = | 414 | @OETestDepends(['ssh.SSHTest.test_ssh']) |
385 | self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output | 415 | @OEHasPackage(['python3-core']) |
386 | self.assertEqual(status, 0, msg=msg) | 416 | def test_python3(self): |
387 | 417 | cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" | |
388 | In this example, the ```OERuntimeTestCase`` <>`__ class wraps | 418 | status, output = self.target.run(cmd) |
419 | msg = 'Exit status was not 0. Output: %s' % output | ||
420 | self.assertEqual(status, 0, msg=msg) | ||
421 | |||
422 | In this example, the ``OERuntimeTestCase`` class wraps | ||
389 | ``unittest.TestCase``. Within the test, ``self.target`` represents the | 423 | ``unittest.TestCase``. Within the test, ``self.target`` represents the |
390 | target system, where commands can be run on it using the ``run()`` | 424 | target system, where commands can be run on it using the ``run()`` |
391 | method. | 425 | method. |
@@ -402,19 +436,30 @@ the image. | |||
402 | 436 | ||
403 | These tests are run against built extensible SDKs (eSDKs). The tests can | 437 | These tests are run against built extensible SDKs (eSDKs). The tests can |
404 | assume that the eSDK environment has already been setup. An example from | 438 | assume that the eSDK environment has already been setup. An example from |
405 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class | 439 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: |
406 | DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls): | 440 | |
407 | myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst = | 441 | class DevtoolTest(OESDKExtTestCase): |
408 | os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src, | 442 | @classmethod def setUpClass(cls): |
409 | cls.myapp_dst) subprocess.check_output(['git', 'init', '.'], | 443 | myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") |
410 | cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'], | 444 | cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") |
411 | cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m', | 445 | shutil.copytree(myapp_src, cls.myapp_dst) |
412 | "'test commit'"], cwd=cls.myapp_dst) @classmethod def | 446 | subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) |
413 | tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def | 447 | subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) |
414 | \_test_devtool_build(self, directory): self._run('devtool add myapp %s' | 448 | subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) |
415 | % directory) try: self._run('devtool build myapp') finally: | 449 | |
416 | self._run('devtool reset myapp') def test_devtool_build_make(self): | 450 | @classmethod |
417 | self._test_devtool_build(self.myapp_dst)In this example, the ``devtool`` | 451 | def tearDownClass(cls): |
452 | shutil.rmtree(cls.myapp_dst) | ||
453 | def _test_devtool_build(self, directory): | ||
454 | self._run('devtool add myapp %s' % directory) | ||
455 | try: | ||
456 | self._run('devtool build myapp') | ||
457 | finally: | ||
458 | self._run('devtool reset myapp') | ||
459 | def test_devtool_build_make(self): | ||
460 | self._test_devtool_build(self.myapp_dst) | ||
461 | |||
462 | In this example, the ``devtool`` | ||
418 | command is tested to see whether a sample application can be built with | 463 | command is tested to see whether a sample application can be built with |
419 | the ``devtool build`` command within the eSDK. | 464 | the ``devtool build`` command within the eSDK. |
420 | 465 | ||
@@ -426,14 +471,20 @@ the ``devtool build`` command within the eSDK. | |||
426 | These tests are run against built SDKs. The tests can assume that an SDK | 471 | These tests are run against built SDKs. The tests can assume that an SDK |
427 | has already been extracted and its environment file has been sourced. A | 472 | has already been extracted and its environment file has been sourced. A |
428 | simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the | 473 | simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the |
429 | following:class Python3Test(OESDKTestCase): def setUp(self): if not | 474 | following:: |
430 | (self.tc.hasHostPackage("nativesdk-python3-core") or | 475 | |
431 | self.tc.hasHostPackage("python3-core-native")): raise | 476 | class Python3Test(OESDKTestCase): |
432 | unittest.SkipTest("No python3 package in the SDK") def | 477 | def setUp(self): |
433 | test_python3(self): cmd = "python3 -c \\"import codecs; | 478 | if not (self.tc.hasHostPackage("nativesdk-python3-core") or |
434 | print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd) | 479 | self.tc.hasHostPackage("python3-core-native")): |
435 | self.assertEqual(output, "Hello, world\n")In this example, if | 480 | raise unittest.SkipTest("No python3 package in the SDK") |
436 | nativesdk-python3-core has been installed into the SDK, the code runs | 481 | |
482 | def test_python3(self): | ||
483 | cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" | ||
484 | output = self._run(cmd) | ||
485 | self.assertEqual(output, "Hello, world\n") | ||
486 | |||
487 | In this example, if nativesdk-python3-core has been installed into the SDK, the code runs | ||
437 | the python3 interpreter with a basic command to check it is working | 488 | the python3 interpreter with a basic command to check it is working |
438 | correctly. The test would only run if python3 is installed in the SDK. | 489 | correctly. The test would only run if python3 is installed in the SDK. |
439 | 490 | ||
@@ -444,17 +495,25 @@ correctly. The test would only run if python3 is installed in the SDK. | |||
444 | 495 | ||
445 | The performance tests usually measure how long operations take and the | 496 | The performance tests usually measure how long operations take and the |
446 | resource utilisation as that happens. An example from | 497 | resource utilisation as that happens. An example from |
447 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class | 498 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: |
448 | Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake | 499 | |
449 | -p)""" # Drop all caches and parse self.rm_cache() | 500 | class Test3(BuildPerfTestCase): |
450 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | 501 | def test3(self): |
451 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no | 502 | """Bitbake parsing (bitbake -p)""" |
452 | caches)') # Drop tmp/cache | 503 | # Drop all caches and parse |
453 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | 504 | self.rm_cache() |
454 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no | 505 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) |
455 | tmp/cache)') # Parse with fully cached data | 506 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', |
456 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p | 507 | 'bitbake -p (no caches)') |
457 | (cached)')This example shows how three specific parsing timings are | 508 | # Drop tmp/cache |
509 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | ||
510 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', | ||
511 | 'bitbake -p (no tmp/cache)') | ||
512 | # Parse with fully cached data | ||
513 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', | ||
514 | 'bitbake -p (cached)') | ||
515 | |||
516 | This example shows how three specific parsing timings are | ||
458 | measured, with and without various caches, to show how BitBake’s parsing | 517 | measured, with and without various caches, to show how BitBake’s parsing |
459 | performance trends over time. | 518 | performance trends over time. |
460 | 519 | ||
diff --git a/documentation/test-manual/test-manual-test-process.rst b/documentation/test-manual/test-manual-test-process.rst index df825f8efa..96e71bf314 100644 --- a/documentation/test-manual/test-manual-test-process.rst +++ b/documentation/test-manual/test-manual-test-process.rst | |||
@@ -23,11 +23,11 @@ We have two broad categories of test builds, including "full" and | |||
23 | "quick". On the Autobuilder, these can be seen as "a-quick" and | 23 | "quick". On the Autobuilder, these can be seen as "a-quick" and |
24 | "a-full", simply for ease of sorting in the UI. Use our Autobuilder | 24 | "a-full", simply for ease of sorting in the UI. Use our Autobuilder |
25 | console view to see where me manage most test-related items, available | 25 | console view to see where me manage most test-related items, available |
26 | at: `https://autobuilder.yoctoproject.org/typhoon/#/console <#>`__. | 26 | at: :yocto_ab:`/typhoon/#/console`. |
27 | 27 | ||
28 | Builds are triggered manually when the test branches are ready. The | 28 | Builds are triggered manually when the test branches are ready. The |
29 | builds are monitored by the SWAT team. For additional information, see | 29 | builds are monitored by the SWAT team. For additional information, see |
30 | `https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team <#>`__. | 30 | :yocto_wiki:`/wiki/Yocto_Build_Failure_Swat_Team`. |
31 | If successful, the changes would usually be merged to the ``master`` | 31 | If successful, the changes would usually be merged to the ``master`` |
32 | branch. If not successful, someone would respond to the changes on the | 32 | branch. If not successful, someone would respond to the changes on the |
33 | mailing list explaining that there was a failure in testing. The choice | 33 | mailing list explaining that there was a failure in testing. The choice |
@@ -37,9 +37,9 @@ which the result was required. | |||
37 | The Autobuilder does build the ``master`` branch once daily for several | 37 | The Autobuilder does build the ``master`` branch once daily for several |
38 | reasons, in particular, to ensure the current ``master`` branch does | 38 | reasons, in particular, to ensure the current ``master`` branch does |
39 | build, but also to keep ``yocto-testresults`` | 39 | build, but also to keep ``yocto-testresults`` |
40 | (`http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ <#>`__), | 40 | (:yocto_git:`/cgit.cgi/yocto-testresults/`), |
41 | buildhistory | 41 | buildhistory |
42 | (`http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/ <#>`__), and | 42 | (:yocto_git:`/cgit.cgi/poky-buildhistory/`), and |
43 | our sstate up to date. On the weekend, there is a master-next build | 43 | our sstate up to date. On the weekend, there is a master-next build |
44 | instead to ensure the test results are updated for the less frequently | 44 | instead to ensure the test results are updated for the less frequently |
45 | run targets. | 45 | run targets. |
@@ -47,7 +47,7 @@ run targets. | |||
47 | Performance builds (buildperf-\* targets in the console) are triggered | 47 | Performance builds (buildperf-\* targets in the console) are triggered |
48 | separately every six hours and automatically push their results to the | 48 | separately every six hours and automatically push their results to the |
49 | buildstats repository at: | 49 | buildstats repository at: |
50 | `http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/ <#>`__. | 50 | :yocto_git:`/cgit.cgi/yocto-buildstats/`. |
51 | 51 | ||
52 | The 'quick' targets have been selected to be the ones which catch the | 52 | The 'quick' targets have been selected to be the ones which catch the |
53 | most failures or give the most valuable data. We run 'fast' ptests in | 53 | most failures or give the most valuable data. We run 'fast' ptests in |
@@ -56,8 +56,6 @@ target doesn't include \*-lsb builds for all architectures, some world | |||
56 | builds and doesn't trigger performance tests or ltp testing. The full | 56 | builds and doesn't trigger performance tests or ltp testing. The full |
57 | build includes all these things and is slower but more comprehensive. | 57 | build includes all these things and is slower but more comprehensive. |
58 | 58 | ||
59 | .. _test-yocto-project-autobuilder-overview: | ||
60 | |||
61 | Release Builds | 59 | Release Builds |
62 | ============== | 60 | ============== |
63 | 61 | ||
@@ -76,7 +74,7 @@ box to "generate an email to QA"is also checked. | |||
76 | When the build completes, an email is sent out using the send-qa-email | 74 | When the build completes, an email is sent out using the send-qa-email |
77 | script in the ``yocto-autobuilder-helper`` repository to the list of | 75 | script in the ``yocto-autobuilder-helper`` repository to the list of |
78 | people configured for that release. Release builds are placed into a | 76 | people configured for that release. Release builds are placed into a |
79 | directory in `https://autobuilder.yocto.io/pub/releases <#>`__ on the | 77 | directory in https://autobuilder.yocto.io/pub/releases on the |
80 | Autobuilder which is included in the email. The process from here is | 78 | Autobuilder which is included in the email. The process from here is |
81 | more manual and control is effectively passed to release engineering. | 79 | more manual and control is effectively passed to release engineering. |
82 | The next steps include: | 80 | The next steps include: |
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst index 8d3c844156..0809190b5b 100644 --- a/documentation/test-manual/test-manual-understand-autobuilder.rst +++ b/documentation/test-manual/test-manual-understand-autobuilder.rst | |||
@@ -10,7 +10,7 @@ Execution Flow within the Autobuilder | |||
10 | The “a-full” and “a-quick” targets are the usual entry points into the | 10 | The “a-full” and “a-quick” targets are the usual entry points into the |
11 | Autobuilder and it makes sense to follow the process through the system | 11 | Autobuilder and it makes sense to follow the process through the system |
12 | starting there. This is best visualised from the Autobuilder Console | 12 | starting there. This is best visualised from the Autobuilder Console |
13 | view (`https://autobuilder.yoctoproject.org/typhoon/#/console <#>`__). | 13 | view (:yocto_ab:`/typhoon/#/console`). |
14 | 14 | ||
15 | Each item along the top of that view represents some “target build” and | 15 | Each item along the top of that view represents some “target build” and |
16 | these targets are all run in parallel. The ‘full’ build will trigger the | 16 | these targets are all run in parallel. The ‘full’ build will trigger the |
@@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker. To understand the | |||
20 | configuration, you need to look at the entry on ``config.json`` file | 20 | configuration, you need to look at the entry on ``config.json`` file |
21 | within the ``yocto-autobuilder-helper`` repository. The targets are | 21 | within the ``yocto-autobuilder-helper`` repository. The targets are |
22 | defined in the ‘overrides’ section, a quick example could be qemux86-64 | 22 | defined in the ‘overrides’ section, a quick example could be qemux86-64 |
23 | which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" : | 23 | which looks like:: |
24 | "arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic | 24 | |
25 | wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from | 25 | "qemux86-64" : { |
26 | the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" : | 26 | "MACHINE" : "qemux86-64", |
27 | true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato | 27 | "TEMPLATE" : "arch-qemu", |
28 | core-image-sato-dev core-image-sato-sdk core-image-minimal | 28 | "step1" : { |
29 | core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS" | 29 | "extravars" : [ |
30 | : "core-image-minimal:do_testimage core-image-sato:do_testimage | 30 | "IMAGE_FSTYPES_append = ' wic wic.bmap'" |
31 | core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2" | 31 | ] |
32 | : { "SDKMACHINE" : "x86_64", "BBTARGETS" : | 32 | } |
33 | "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext | 33 | }, |
34 | core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" : | 34 | |
35 | "core-image-sato:do_testsdk core-image-minimal:do_testsdkext | 35 | And to expand that, you need the “arch-qemu” entry from |
36 | core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false, | 36 | the “templates” section, which looks like:: |
37 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest | 37 | |
38 | ${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" : | 38 | "arch-qemu" : { |
39 | ["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can | 39 | "BUILDINFO" : true, |
40 | see that “qemux86-64” is a three step build where the | 40 | "BUILDHISTORY" : true, |
41 | ``bitbake BBTARGETS`` would be run, then ``bitbake | 41 | "step1" : { |
42 | SANITYTARGETS`` for each step; all for | 42 | "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", |
43 | "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" | ||
44 | }, | ||
45 | "step2" : { | ||
46 | "SDKMACHINE" : "x86_64", | ||
47 | "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", | ||
48 | "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" | ||
49 | }, | ||
50 | "step3" : { | ||
51 | "BUILDHISTORY" : false, | ||
52 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], | ||
53 | "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] | ||
54 | } | ||
55 | }, | ||
56 | |||
57 | Combining these two entries you can see that “qemux86-64” is a three step build where the | ||
58 | ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for | ||
43 | ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step | 59 | ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step |
44 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | 60 | 1 an extra variable is added to the ``auto.conf`` file to enable wic |
45 | image generation. | 61 | image generation. |
46 | 62 | ||
47 | While not every detail of this is covered here, you can see how the | 63 | While not every detail of this is covered here, you can see how the |
48 | templating mechanism allows quite complex configurations to be built up | 64 | template mechanism allows quite complex configurations to be built up |
49 | yet allows duplication and repetition to be kept to a minimum. | 65 | yet allows duplication and repetition to be kept to a minimum. |
50 | 66 | ||
51 | The different build targets are designed to allow for parallelisation, | 67 | The different build targets are designed to allow for parallelisation, |
@@ -66,13 +82,13 @@ For each given target in a build, the Autobuilder executes several | |||
66 | steps. These are configured in ``yocto-autobuilder2/builders.py`` and | 82 | steps. These are configured in ``yocto-autobuilder2/builders.py`` and |
67 | roughly consist of: | 83 | roughly consist of: |
68 | 84 | ||
69 | 1. *Run ``clobberdir``* | 85 | #. *Run clobberdir*. |
70 | 86 | ||
71 | This cleans out any previous build. Old builds are left around to | 87 | This cleans out any previous build. Old builds are left around to |
72 | allow easier debugging of failed builds. For additional information, | 88 | allow easier debugging of failed builds. For additional information, |
73 | see ```clobberdir`` <#test-clobberdir>`__. | 89 | see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`. |
74 | 90 | ||
75 | 2. *Obtain yocto-autobuilder-helper* | 91 | #. *Obtain yocto-autobuilder-helper* |
76 | 92 | ||
77 | This step clones the ``yocto-autobuilder-helper`` git repository. | 93 | This step clones the ``yocto-autobuilder-helper`` git repository. |
78 | This is necessary to prevent the requirement to maintain all the | 94 | This is necessary to prevent the requirement to maintain all the |
@@ -80,12 +96,12 @@ roughly consist of: | |||
80 | matches the release being built so we can support older releases and | 96 | matches the release being built so we can support older releases and |
81 | still make changes in newer ones. | 97 | still make changes in newer ones. |
82 | 98 | ||
83 | 3. *Write layerinfo.json* | 99 | #. *Write layerinfo.json* |
84 | 100 | ||
85 | This transfers data in the Buildbot UI when the build was configured | 101 | This transfers data in the Buildbot UI when the build was configured |
86 | to the Helper. | 102 | to the Helper. |
87 | 103 | ||
88 | 4. *Call scripts/shared-repo-unpack* | 104 | #. *Call scripts/shared-repo-unpack* |
89 | 105 | ||
90 | This is a call into the Helper scripts to set up a checkout of all | 106 | This is a call into the Helper scripts to set up a checkout of all |
91 | the pieces this build might need. It might clone the BitBake | 107 | the pieces this build might need. It might clone the BitBake |
@@ -94,7 +110,7 @@ roughly consist of: | |||
94 | from the ``layerinfo.json`` file to help understand the | 110 | from the ``layerinfo.json`` file to help understand the |
95 | configuration. It will also use a local cache of repositories to | 111 | configuration. It will also use a local cache of repositories to |
96 | speed up the clone checkouts. For additional information, see | 112 | speed up the clone checkouts. For additional information, see |
97 | `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__. | 113 | :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`. |
98 | 114 | ||
99 | This step has two possible modes of operation. If the build is part | 115 | This step has two possible modes of operation. If the build is part |
100 | of a parent build, its possible that all the repositories needed may | 116 | of a parent build, its possible that all the repositories needed may |
@@ -114,7 +130,7 @@ roughly consist of: | |||
114 | available, it would clone from the cache and the upstreams as | 130 | available, it would clone from the cache and the upstreams as |
115 | necessary. This is considered the fallback mode. | 131 | necessary. This is considered the fallback mode. |
116 | 132 | ||
117 | 5. *Call scripts/run-config* | 133 | #. *Call scripts/run-config* |
118 | 134 | ||
119 | This is another call into the Helper scripts where its expected that | 135 | This is another call into the Helper scripts where its expected that |
120 | the main functionality of this target will be executed. | 136 | the main functionality of this target will be executed. |
@@ -137,8 +153,7 @@ special script that moves files to a special location, rather than | |||
137 | deleting them. Files in this location are deleted by an ``rm`` command, | 153 | deleting them. Files in this location are deleted by an ``rm`` command, |
138 | which is run under ``ionice -c 3``. For example, the deletion only | 154 | which is run under ``ionice -c 3``. For example, the deletion only |
139 | happens when there is idle IO capacity on the Worker. The Autobuilder | 155 | happens when there is idle IO capacity on the Worker. The Autobuilder |
140 | Worker Janitor runs this deletion. See `Autobuilder Worker | 156 | Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. |
141 | Janitor <#test-autobuilder-worker-janitor>`__. | ||
142 | 157 | ||
143 | .. _test-autobuilder-clone-cache: | 158 | .. _test-autobuilder-clone-cache: |
144 | 159 | ||
@@ -150,8 +165,7 @@ on the Autobuilder. We therefore have a stash of commonly used | |||
150 | repositories pre-cloned on the Workers. Data is fetched from these | 165 | repositories pre-cloned on the Workers. Data is fetched from these |
151 | during clones first, then "topped up" with later revisions from any | 166 | during clones first, then "topped up" with later revisions from any |
152 | upstream when necesary. The cache is maintained by the Autobuilder | 167 | upstream when necesary. The cache is maintained by the Autobuilder |
153 | Worker Janitor. See `Autobuilder Worker | 168 | Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. |
154 | Janitor <#test-autobuilder-worker-janitor>`__. | ||
155 | 169 | ||
156 | .. _test-autobuilder-worker-janitor: | 170 | .. _test-autobuilder-worker-janitor: |
157 | 171 | ||
@@ -159,8 +173,7 @@ Autobuilder Worker Janitor | |||
159 | -------------------------- | 173 | -------------------------- |
160 | 174 | ||
161 | This is a process running on each Worker that performs two basic | 175 | This is a process running on each Worker that performs two basic |
162 | operations, including background file deletion at IO idle (see `Target | 176 | operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and |
163 | Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and | ||
164 | maintainenance of a cache of cloned repositories to improve the speed | 177 | maintainenance of a cache of cloned repositories to improve the speed |
165 | the system can checkout repositories. | 178 | the system can checkout repositories. |
166 | 179 | ||
@@ -181,7 +194,7 @@ Shared SSTATE_DIR | |||
181 | 194 | ||
182 | The Workers are all connected over NFS which allows the ``sstate`` | 195 | The Workers are all connected over NFS which allows the ``sstate`` |
183 | directory to be shared between them. This means once a Worker has built | 196 | directory to be shared between them. This means once a Worker has built |
184 | an artefact, all the others can benefit from it. Usage of the directory | 197 | an artifact, all the others can benefit from it. Usage of the directory |
185 | within the directory is designed for sharing over NFS. | 198 | within the directory is designed for sharing over NFS. |
186 | 199 | ||
187 | .. _test-resulttool: | 200 | .. _test-resulttool: |
@@ -198,7 +211,7 @@ Resulttool is part of OpenEmbedded-Core and is used to manipulate these | |||
198 | json results files. It has the ability to merge files together, display | 211 | json results files. It has the ability to merge files together, display |
199 | reports of the test results and compare different result files. | 212 | reports of the test results and compare different result files. |
200 | 213 | ||
201 | For details, see `https://wiki.yoctoproject.org/wiki/Resulttool <#>`__. | 214 | For details, see :yocto_wiki:`/wiki/Resulttool`. |
202 | 215 | ||
203 | .. _test-run-config-tgt-execution: | 216 | .. _test-run-config-tgt-execution: |
204 | 217 | ||
@@ -209,50 +222,46 @@ The ``scripts/run-config`` execution is where most of the work within | |||
209 | the Autobuilder happens. It runs through a number of steps; the first | 222 | the Autobuilder happens. It runs through a number of steps; the first |
210 | are general setup steps that are run once and include: | 223 | are general setup steps that are run once and include: |
211 | 224 | ||
212 | 1. Set up any ``buildtools-tarball`` if configured. | 225 | #. Set up any ``buildtools-tarball`` if configured. |
213 | 226 | ||
214 | 2. Call "buildhistory-init" if buildhistory is configured. | 227 | #. Call "buildhistory-init" if buildhistory is configured. |
215 | 228 | ||
216 | For each step that is configured in ``config.json``, it will perform the | 229 | For each step that is configured in ``config.json``, it will perform the |
217 | following: | 230 | following: |
218 | 231 | ||
219 | ## WRITER's question: What does "logging in as stepXa" and others refer | 232 | #. Add any layers that are specified using the |
220 | to below? ## | ||
221 | |||
222 | 1. Add any layers that are specified using the | ||
223 | ``bitbake-layers add-layer`` command (logging as stepXa) | 233 | ``bitbake-layers add-layer`` command (logging as stepXa) |
224 | 234 | ||
225 | 2. Call the ``scripts/setup-config`` script to generate the necessary | 235 | #. Call the ``scripts/setup-config`` script to generate the necessary |
226 | ``auto.conf`` configuration file for the build | 236 | ``auto.conf`` configuration file for the build |
227 | 237 | ||
228 | 3. Run the ``bitbake BBTARGETS`` command (logging as stepXb) | 238 | #. Run the ``bitbake BBTARGETS`` command (logging as stepXb) |
229 | 239 | ||
230 | 4. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) | 240 | #. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) |
231 | 241 | ||
232 | 5. Run the ``EXTRACMDS`` command, which are run within the BitBake build | 242 | #. Run the ``EXTRACMDS`` command, which are run within the BitBake build |
233 | environment (logging as stepXd) | 243 | environment (logging as stepXd) |
234 | 244 | ||
235 | 6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the | 245 | #. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the |
236 | BitBake build environment (logging as stepXd) | 246 | BitBake build environment (logging as stepXd) |
237 | 247 | ||
238 | 7. Remove any layers added in `step | 248 | #. Remove any layers added in step |
239 | 1 <#test-run-config-add-layers-step>`__ using the | 249 | 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa) |
240 | ``bitbake-layers remove-layer`` command (logging as stepXa) | ||
241 | 250 | ||
242 | Once the execution steps above complete, ``run-config`` executes a set | 251 | Once the execution steps above complete, ``run-config`` executes a set |
243 | of post-build steps, including: | 252 | of post-build steps, including: |
244 | 253 | ||
245 | 1. Call ``scripts/publish-artifacts`` to collect any output which is to | 254 | #. Call ``scripts/publish-artifacts`` to collect any output which is to |
246 | be saved from the build. | 255 | be saved from the build. |
247 | 256 | ||
248 | 2. Call ``scripts/collect-results`` to collect any test results to be | 257 | #. Call ``scripts/collect-results`` to collect any test results to be |
249 | saved from the build. | 258 | saved from the build. |
250 | 259 | ||
251 | 3. Call ``scripts/upload-error-reports`` to send any error reports | 260 | #. Call ``scripts/upload-error-reports`` to send any error reports |
252 | generated to the remote server. | 261 | generated to the remote server. |
253 | 262 | ||
254 | 4. Cleanup the build directory using | 263 | #. Cleanup the build directory using |
255 | ```clobberdir`` <#test-clobberdir>`__ if the build was successful, | 264 | :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful, |
256 | else rename it to “build-renamed” for potential future debugging. | 265 | else rename it to “build-renamed” for potential future debugging. |
257 | 266 | ||
258 | .. _test-deploying-yp-autobuilder: | 267 | .. _test-deploying-yp-autobuilder: |
@@ -279,11 +288,18 @@ The standard ``config.json`` minimally attempts to allow substitution of | |||
279 | the paths. The Helper script repository includes a | 288 | the paths. The Helper script repository includes a |
280 | ``local-example.json`` file to show how you could override these from a | 289 | ``local-example.json`` file to show how you could override these from a |
281 | separate configuration file. Pass the following into the environment of | 290 | separate configuration file. Pass the following into the environment of |
282 | the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As | 291 | the Autobuilder:: |
283 | another example, you could also pass the following into the | 292 | |
284 | environment:$ ABHELPER_JSON="config.json /some/location/local.json"One | 293 | $ ABHELPER_JSON="config.json local-example.json" |
285 | issue users often run into is validation of the ``config.json`` files. A | 294 | |
295 | As another example, you could also pass the following into the | ||
296 | environment:: | ||
297 | |||
298 | $ ABHELPER_JSON="config.json /some/location/local.json" | ||
299 | |||
300 | One issue users often run into is validation of the ``config.json`` files. A | ||
286 | tip for minimizing issues from invalid json files is to use a Git | 301 | tip for minimizing issues from invalid json files is to use a Git |
287 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | 302 | ``pre-commit-hook.sh`` script to verify the JSON file before committing |
288 | it. Create a symbolic link as follows:$ ln -s | 303 | it. Create a symbolic link as follows:: |
289 | ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | 304 | |
305 | $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | ||