diff options
| author | Mark Morton <Mark.Morton@windriver.com> | 2020-09-14 16:11:31 -0700 |
|---|---|---|
| committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2020-09-17 10:09:35 +0100 |
| commit | 74f212d2e1db87ccfecaf9385ab280123d5fb168 (patch) | |
| tree | 79e85fac733b78eb11cb19ca9753ed44dbf8358e /documentation/test-manual | |
| parent | fc876832cb57a1a3e61150778a9cfd7acffbc2b6 (diff) | |
| download | poky-74f212d2e1db87ccfecaf9385ab280123d5fb168.tar.gz | |
sphinx: test-manual code block, link, and format update
(From yocto-docs rev: 928c212ec4ad6e09524fdf8147aa7daf244677b0)
Signed-off-by: Mark Morton <mark.morton@windriver.com>
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/test-manual')
3 files changed, 257 insertions, 184 deletions
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst index 2f11e2beb3..b0bc24c9b0 100644 --- a/documentation/test-manual/test-manual-intro.rst +++ b/documentation/test-manual/test-manual-intro.rst | |||
| @@ -23,26 +23,25 @@ project core. | |||
| 23 | 23 | ||
| 24 | Currently, the Yocto Project Test Environment Manual has no projected | 24 | Currently, the Yocto Project Test Environment Manual has no projected |
| 25 | release date. This manual is a work-in-progress and is being initially | 25 | release date. This manual is a work-in-progress and is being initially |
| 26 | loaded with information from the `README <>`__ files and notes from key | 26 | loaded with information from the README files and notes from key |
| 27 | engineers: | 27 | engineers: |
| 28 | 28 | ||
| 29 | - *``yocto-autobuilder2``:* This | 29 | - *yocto-autobuilder2:* This |
| 30 | ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md | 30 | :yocto_git:`README.md </cgit.cgi/yocto-autobuilder2/tree/README.md>` |
| 31 | is the main README which detials how to set up the Yocto Project | 31 | is the main README which detials how to set up the Yocto Project |
| 32 | Autobuilder. The ``yocto-autobuilder2`` repository represents the | 32 | Autobuilder. The ``yocto-autobuilder2`` repository represents the |
| 33 | Yocto Project's console UI plugin to Buildbot and the configuration | 33 | Yocto Project's console UI plugin to Buildbot and the configuration |
| 34 | necessary to configure Buildbot to perform the testing the project | 34 | necessary to configure Buildbot to perform the testing the project |
| 35 | requires. | 35 | requires. |
| 36 | 36 | ||
| 37 | - *``yocto-autobuilder-helper``:* This | 37 | - *yocto-autobuilder-helper:* This :yocto_git:`README </cgit.cgi/yocto-autobuilder-helper/tree/README/>` |
| 38 | ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README | ||
| 39 | and repository contains Yocto Project Autobuilder Helper scripts and | 38 | and repository contains Yocto Project Autobuilder Helper scripts and |
| 40 | configuration. The ``yocto-autobuilder-helper`` repository contains | 39 | configuration. The ``yocto-autobuilder-helper`` repository contains |
| 41 | the "glue" logic that defines which tests to run and how to run them. | 40 | the "glue" logic that defines which tests to run and how to run them. |
| 42 | As a result, it can be used by any Continuous Improvement (CI) system | 41 | As a result, it can be used by any Continuous Improvement (CI) system |
| 43 | to run builds, support getting the correct code revisions, configure | 42 | to run builds, support getting the correct code revisions, configure |
| 44 | builds and layers, run builds, and collect results. The code is | 43 | builds and layers, run builds, and collect results. The code is |
| 45 | independent of any CI system, which means the code can work Buildbot, | 44 | independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__, |
| 46 | Jenkins, or others. This repository has a branch per release of the | 45 | Jenkins, or others. This repository has a branch per release of the |
| 47 | project defining the tests to run on a per release basis. | 46 | project defining the tests to run on a per release basis. |
| 48 | 47 | ||
| @@ -124,21 +123,22 @@ thefollowing types of tests: | |||
| 124 | ensure we have no single point of failure and can ensure the | 123 | ensure we have no single point of failure and can ensure the |
| 125 | different distros work effectively. | 124 | different distros work effectively. |
| 126 | 125 | ||
| 127 | - *eSDK Testing:* Image tests initiated through the following command: | 126 | - *eSDK Testing:* Image tests initiated through the following command:: |
| 128 | $ bitbake image -c testsdkext The tests utilize the ``testsdkext`` | 127 | |
| 129 | class and the ``do_testsdkext`` task. | 128 | $ bitbake image -c testsdkext |
| 129 | |||
| 130 | The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task. | ||
| 130 | 131 | ||
| 131 | - *Feature Testing:* Various scenario-based tests are run through the | 132 | - *Feature Testing:* Various scenario-based tests are run through the |
| 132 | `OpenEmbedded | 133 | :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions |
| 133 | Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__ | ||
| 134 | (oe-selftest). We test oe-selftest on each of the main distrubutions | ||
| 135 | we support. | 134 | we support. |
| 136 | 135 | ||
| 137 | - *Image Testing:* Image tests initiated through the following command: | 136 | - *Image Testing:* Image tests initiated through the following command:: |
| 138 | $ bitbake image -c testimage The tests utilize the | 137 | |
| 139 | :ref:`testimage* <ref-classes-testimage*>` | 138 | $ bitbake image -c testimage |
| 140 | classes and the | 139 | |
| 141 | :ref:`ref-tasks-testimage` task. | 140 | The tests utilize the :ref:`testimage* <ref-classes-testimage*>` |
| 141 | classes and the :ref:`ref-tasks-testimage` task. | ||
| 142 | 142 | ||
| 143 | - *Layer Testing:* The Autobuilder has the possibility to test whether | 143 | - *Layer Testing:* The Autobuilder has the possibility to test whether |
| 144 | specific layers work with the test of the system. The layers tested | 144 | specific layers work with the test of the system. The layers tested |
| @@ -147,20 +147,22 @@ thefollowing types of tests: | |||
| 147 | 147 | ||
| 148 | - *Package Testing:* A Package Test (ptest) runs tests against packages | 148 | - *Package Testing:* A Package Test (ptest) runs tests against packages |
| 149 | built by the OpenEmbedded build system on the target machine. See the | 149 | built by the OpenEmbedded build system on the target machine. See the |
| 150 | "`Testing Packages With | 150 | :ref:`Testing Packages With |
| 151 | ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section | 151 | ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section |
| 152 | in the Yocto Project Development Tasks Manual and the | 152 | in the Yocto Project Development Tasks Manual and the |
| 153 | ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more | 153 | ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more |
| 154 | information on Ptest. | 154 | information on Ptest. |
| 155 | 155 | ||
| 156 | - *SDK Testing:* Image tests initiated through the following command: $ | 156 | - *SDK Testing:* Image tests initiated through the following command:: |
| 157 | bitbake image -c testsdk The tests utilize the | 157 | |
| 158 | :ref:`testsdk <ref-classes-testsdk>` class and | 158 | $ bitbake image -c testsdk |
| 159 | |||
| 160 | The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and | ||
| 159 | the ``do_testsdk`` task. | 161 | the ``do_testsdk`` task. |
| 160 | 162 | ||
| 161 | - *Unit Testing:* Unit tests on various components of the system run | 163 | - *Unit Testing:* Unit tests on various components of the system run |
| 162 | through ``oe-selftest`` and | 164 | through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and |
| 163 | ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__. | 165 | :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`. |
| 164 | 166 | ||
| 165 | - *Automatic Upgrade Helper:* This target tests whether new versions of | 167 | - *Automatic Upgrade Helper:* This target tests whether new versions of |
| 166 | software are available and whether we can automatically upgrade to | 168 | software are available and whether we can automatically upgrade to |
| @@ -174,36 +176,43 @@ How Tests Map to Areas of Code | |||
| 174 | 176 | ||
| 175 | Tests map into the codebase as follows: | 177 | Tests map into the codebase as follows: |
| 176 | 178 | ||
| 177 | - *bitbake-selftest*: | 179 | - *bitbake-selftest:* |
| 178 | 180 | ||
| 179 | These tests are self-contained and test BitBake as well as its APIs, | 181 | These tests are self-contained and test BitBake as well as its APIs, |
| 180 | which include the fetchers. The tests are located in | 182 | which include the fetchers. The tests are located in |
| 181 | ``bitbake/lib/*/tests``. | 183 | ``bitbake/lib/*/tests``. |
| 182 | 184 | ||
| 183 | From within the BitBake repository, run the following: $ | 185 | From within the BitBake repository, run the following:: |
| 184 | bitbake-selftest | 186 | |
| 187 | $ bitbake-selftest | ||
| 185 | 188 | ||
| 186 | To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` | 189 | To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` |
| 187 | variable when running "bitbake-selftest" as follows: $ | 190 | variable when running "bitbake-selftest" as follows:: |
| 188 | BB_SKIP_NETTEST=yes bitbake-selftest | 191 | |
| 192 | $ BB_SKIP_NETTEST=yes bitbake-selftest | ||
| 189 | 193 | ||
| 190 | The default output is quiet and just prints a summary of what was | 194 | The default output is quiet and just prints a summary of what was |
| 191 | run. To see more information, there is a verbose option:$ | 195 | run. To see more information, there is a verbose option:: |
| 192 | bitbake-selftest -v | 196 | |
| 197 | $ bitbake-selftest -v | ||
| 193 | 198 | ||
| 194 | Use this option when you wish to skip tests that access the network, | 199 | Use this option when you wish to skip tests that access the network, |
| 195 | which are mostly necessary to test the fetcher modules. To specify | 200 | which are mostly necessary to test the fetcher modules. To specify |
| 196 | individual test modules to run, append the test module name to the | 201 | individual test modules to run, append the test module name to the |
| 197 | "bitbake-selftest" command. For example, to specify the tests for the | 202 | "bitbake-selftest" command. For example, to specify the tests for the |
| 198 | bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can | 203 | bb.data.module, run:: |
| 199 | also specify individual tests by defining the full name and module | 204 | |
| 200 | plus the class path of the test, for example: $ bitbake-selftest | 205 | $ bitbake-selftest bb.test.data.module |
| 201 | bb.tests.data.TestOverrides.test_one_override | 206 | |
| 207 | You can also specify individual tests by defining the full name and module | ||
| 208 | plus the class path of the test, for example:: | ||
| 209 | |||
| 210 | $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override | ||
| 202 | 211 | ||
| 203 | The tests are based on `Python | 212 | The tests are based on `Python |
| 204 | unittest <https://docs.python.org/3/library/unittest.html>`__. | 213 | unittest <https://docs.python.org/3/library/unittest.html>`__. |
| 205 | 214 | ||
| 206 | - *oe-selftest*: | 215 | - *oe-selftest:* |
| 207 | 216 | ||
| 208 | - These tests use OE to test the workflows, which include testing | 217 | - These tests use OE to test the workflows, which include testing |
| 209 | specific features, behaviors of tasks, and API unit tests. | 218 | specific features, behaviors of tasks, and API unit tests. |
| @@ -219,31 +228,40 @@ Tests map into the codebase as follows: | |||
| 219 | - The code for the tests resides in | 228 | - The code for the tests resides in |
| 220 | ``meta/lib/oeqa/selftest/cases/``. | 229 | ``meta/lib/oeqa/selftest/cases/``. |
| 221 | 230 | ||
| 222 | - To run all the tests, enter the following command: $ oe-selftest | 231 | - To run all the tests, enter the following command:: |
| 223 | -a | 232 | |
| 233 | $ oe-selftest -a | ||
| 224 | 234 | ||
| 225 | - To run a specific test, use the following command form where | 235 | - To run a specific test, use the following command form where |
| 226 | testname is the name of the specific test: $ oe-selftest -r | 236 | testname is the name of the specific test:: |
| 227 | testname For example, the following command would run the tinfoil | 237 | |
| 228 | getVar API test:$ oe-selftest -r | 238 | $ oe-selftest -r <testname> |
| 229 | tinfoil.TinfoilTests.test_getvarIt is also possible to run a set | 239 | |
| 240 | For example, the following command would run the tinfoil | ||
| 241 | getVar API test:: | ||
| 242 | |||
| 243 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar | ||
| 244 | |||
| 245 | It is also possible to run a set | ||
| 230 | of tests. For example the following command will run all of the | 246 | of tests. For example the following command will run all of the |
| 231 | tinfoil tests:$ oe-selftest -r tinfoil | 247 | tinfoil tests:: |
| 248 | |||
| 249 | $ oe-selftest -r tinfoil | ||
| 232 | 250 | ||
| 233 | - *testimage:* | 251 | - *testimage:* |
| 234 | 252 | ||
| 235 | - These tests build an image, boot it, and run tests against the | 253 | - These tests build an image, boot it, and run tests against the |
| 236 | image's content. | 254 | image's content. |
| 237 | 255 | ||
| 238 | - The code for these tests resides in | 256 | - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``. |
| 239 | ``meta/lib/oeqa/runtime/cases/``. | 257 | |
| 258 | - You need to set the :term:`IMAGE_CLASSES` variable as follows:: | ||
| 240 | 259 | ||
| 241 | - You need to set the | 260 | IMAGE_CLASSES += "testimage" |
| 242 | :term:`IMAGE_CLASSES` | ||
| 243 | variable as follows: IMAGE_CLASSES += "testimage" | ||
| 244 | 261 | ||
| 245 | - Run the tests using the following command form: $ bitbake image -c | 262 | - Run the tests using the following command form:: |
| 246 | testimage | 263 | |
| 264 | $ bitbake image -c testimage | ||
| 247 | 265 | ||
| 248 | - *testsdk:* | 266 | - *testsdk:* |
| 249 | 267 | ||
| @@ -252,8 +270,9 @@ Tests map into the codebase as follows: | |||
| 252 | 270 | ||
| 253 | - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. | 271 | - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. |
| 254 | 272 | ||
| 255 | - Run the test using the following command form: $ bitbake image -c | 273 | - Run the test using the following command form:: |
| 256 | testsdk | 274 | |
| 275 | $ bitbake image -c testsdk | ||
| 257 | 276 | ||
| 258 | - *testsdk_ext:* | 277 | - *testsdk_ext:* |
| 259 | 278 | ||
| @@ -262,8 +281,9 @@ Tests map into the codebase as follows: | |||
| 262 | 281 | ||
| 263 | - The code for these tests resides in ``meta/lib/oeqa/esdk``. | 282 | - The code for these tests resides in ``meta/lib/oeqa/esdk``. |
| 264 | 283 | ||
| 265 | - To run the tests, use the following command form: $ bitbake image | 284 | - To run the tests, use the following command form:: |
| 266 | -c testsdkext | 285 | |
| 286 | $ bitbake image -c testsdkext | ||
| 267 | 287 | ||
| 268 | - *oe-build-perf-test:* | 288 | - *oe-build-perf-test:* |
| 269 | 289 | ||
| @@ -272,8 +292,11 @@ Tests map into the codebase as follows: | |||
| 272 | 292 | ||
| 273 | - The code for these tests resides in ``meta/lib/oeqa/buildperf``. | 293 | - The code for these tests resides in ``meta/lib/oeqa/buildperf``. |
| 274 | 294 | ||
| 275 | - To run the tests, use the following command form: $ | 295 | - To run the tests, use the following command form:: |
| 276 | oe-build-perf-test optionsThe command takes a number of options, | 296 | |
| 297 | $ oe-build-perf-test <options> | ||
| 298 | |||
| 299 | The command takes a number of options, | ||
| 277 | such as where to place the test results. The Autobuilder Helper | 300 | such as where to place the test results. The Autobuilder Helper |
| 278 | Scripts include the ``build-perf-test-wrapper`` script with | 301 | Scripts include the ``build-perf-test-wrapper`` script with |
| 279 | examples of how to use the oe-build-perf-test from the command | 302 | examples of how to use the oe-build-perf-test from the command |
| @@ -285,9 +308,9 @@ Tests map into the codebase as follows: | |||
| 285 | Use the ``oe-build-perf-report`` command to generate text reports | 308 | Use the ``oe-build-perf-report`` command to generate text reports |
| 286 | and HTML reports with graphs of the performance data. For | 309 | and HTML reports with graphs of the performance data. For |
| 287 | examples, see | 310 | examples, see |
| 288 | `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html <#>`__ | 311 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` |
| 289 | and | 312 | and |
| 290 | `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt <#>`__. | 313 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. |
| 291 | 314 | ||
| 292 | - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. | 315 | - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. |
| 293 | 316 | ||
| @@ -295,7 +318,7 @@ Test Examples | |||
| 295 | ============= | 318 | ============= |
| 296 | 319 | ||
| 297 | This section provides example tests for each of the tests listed in the | 320 | This section provides example tests for each of the tests listed in the |
| 298 | `How Tests Map to Areas of Code <#test-test-mapping>`__ section. | 321 | :ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section. |
| 299 | 322 | ||
| 300 | For oeqa tests, testcases for each area reside in the main test | 323 | For oeqa tests, testcases for each area reside in the main test |
| 301 | directory at ``meta/lib/oeqa/selftest/cases`` directory. | 324 | directory at ``meta/lib/oeqa/selftest/cases`` directory. |
| @@ -308,14 +331,20 @@ directory. | |||
| 308 | ``bitbake-selftest`` | 331 | ``bitbake-selftest`` |
| 309 | -------------------- | 332 | -------------------- |
| 310 | 333 | ||
| 311 | A simple test example from ``lib/bb/tests/data.py`` is: class | 334 | A simple test example from ``lib/bb/tests/data.py`` is:: |
| 312 | DataExpansions(unittest.TestCase): def setUp(self): self.d = | 335 | |
| 313 | bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] = | 336 | class DataExpansions(unittest.TestCase): |
| 314 | "value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def | 337 | def setUp(self): |
| 315 | test_one_var(self): val = self.d.expand("${foo}") | 338 | self.d = bb.data.init() |
| 316 | self.assertEqual(str(val), "value_of_foo") | 339 | self.d["foo"] = "value_of_foo" |
| 340 | self.d["bar"] = "value_of_bar" | ||
| 341 | self.d["value_of_foo"] = "value_of_'value_of_foo'" | ||
| 317 | 342 | ||
| 318 | In this example, a ```DataExpansions`` <>`__ class of tests is created, | 343 | def test_one_var(self): |
| 344 | val = self.d.expand("${foo}") | ||
| 345 | self.assertEqual(str(val), "value_of_foo") | ||
| 346 | |||
| 347 | In this example, a ``DataExpansions`` class of tests is created, | ||
| 319 | derived from standard python unittest. The class has a common ``setUp`` | 348 | derived from standard python unittest. The class has a common ``setUp`` |
| 320 | function which is shared by all the tests in the class. A simple test is | 349 | function which is shared by all the tests in the class. A simple test is |
| 321 | then added to test that when a variable is expanded, the correct value | 350 | then added to test that when a variable is expanded, the correct value |
| @@ -323,7 +352,7 @@ is found. | |||
| 323 | 352 | ||
| 324 | Bitbake selftests are straightforward python unittest. Refer to the | 353 | Bitbake selftests are straightforward python unittest. Refer to the |
| 325 | Python unittest documentation for additional information on writing | 354 | Python unittest documentation for additional information on writing |
| 326 | these tests at: `https://docs.python.org/3/library/unittest.html <#>`__. | 355 | these tests at: https://docs.python.org/3/library/unittest.html. |
| 327 | 356 | ||
| 328 | .. _oe-selftest-example: | 357 | .. _oe-selftest-example: |
| 329 | 358 | ||
| @@ -334,14 +363,15 @@ These tests are more complex due to the setup required behind the scenes | |||
| 334 | for full builds. Rather than directly using Python's unittest, the code | 363 | for full builds. Rather than directly using Python's unittest, the code |
| 335 | wraps most of the standard objects. The tests can be simple, such as | 364 | wraps most of the standard objects. The tests can be simple, such as |
| 336 | testing a command from within the OE build environment using the | 365 | testing a command from within the OE build environment using the |
| 337 | following example:class BitbakeLayers(OESelftestTestCase): def | 366 | following example:: |
| 338 | test_bitbakelayers_showcrossdepends(self): result = | 367 | |
| 339 | runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in | 368 | class BitbakeLayers(OESelftestTestCase): |
| 340 | result.output, msg = "No dependencies were shown. bitbake-layers | 369 | def test_bitbakelayers_showcrossdepends(self): |
| 341 | show-cross-depends output: %s"% result.output) | 370 | result = runCmd('bitbake-layers show-cross-depends') |
| 371 | self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output) | ||
| 342 | 372 | ||
| 343 | This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, | 373 | This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, |
| 344 | creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived | 374 | creates a testcase from the ``OESelftestTestCase`` class, derived |
| 345 | from ``unittest.TestCase``, which runs the ``bitbake-layers`` command | 375 | from ``unittest.TestCase``, which runs the ``bitbake-layers`` command |
| 346 | and checks the output to ensure it contains something we know should be | 376 | and checks the output to ensure it contains something we know should be |
| 347 | here. | 377 | here. |
| @@ -367,7 +397,7 @@ so tests within a given test class should always run in the same build, | |||
| 367 | while tests in different classes or modules may be split into different | 397 | while tests in different classes or modules may be split into different |
| 368 | builds. There is no data store available for these tests since the tests | 398 | builds. There is no data store available for these tests since the tests |
| 369 | launch the ``bitbake`` command and exist outside of its context. As a | 399 | launch the ``bitbake`` command and exist outside of its context. As a |
| 370 | result, common bitbake library functions (bb.*) are also unavailable. | 400 | result, common bitbake library functions (bb.\*) are also unavailable. |
| 371 | 401 | ||
| 372 | .. _testimage-example: | 402 | .. _testimage-example: |
| 373 | 403 | ||
| @@ -378,14 +408,18 @@ These tests are run once an image is up and running, either on target | |||
| 378 | hardware or under QEMU. As a result, they are assumed to be running in a | 408 | hardware or under QEMU. As a result, they are assumed to be running in a |
| 379 | target image environment, as opposed to a host build environment. A | 409 | target image environment, as opposed to a host build environment. A |
| 380 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains | 410 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains |
| 381 | the following:class PythonTest(OERuntimeTestCase): | 411 | the following:: |
| 382 | @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core']) | 412 | |
| 383 | def test_python3(self): cmd = "python3 -c \\"import codecs; | 413 | class PythonTest(OERuntimeTestCase): |
| 384 | print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output = | 414 | @OETestDepends(['ssh.SSHTest.test_ssh']) |
| 385 | self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output | 415 | @OEHasPackage(['python3-core']) |
| 386 | self.assertEqual(status, 0, msg=msg) | 416 | def test_python3(self): |
| 387 | 417 | cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" | |
| 388 | In this example, the ```OERuntimeTestCase`` <>`__ class wraps | 418 | status, output = self.target.run(cmd) |
| 419 | msg = 'Exit status was not 0. Output: %s' % output | ||
| 420 | self.assertEqual(status, 0, msg=msg) | ||
| 421 | |||
| 422 | In this example, the ``OERuntimeTestCase`` class wraps | ||
| 389 | ``unittest.TestCase``. Within the test, ``self.target`` represents the | 423 | ``unittest.TestCase``. Within the test, ``self.target`` represents the |
| 390 | target system, where commands can be run on it using the ``run()`` | 424 | target system, where commands can be run on it using the ``run()`` |
| 391 | method. | 425 | method. |
| @@ -402,19 +436,30 @@ the image. | |||
| 402 | 436 | ||
| 403 | These tests are run against built extensible SDKs (eSDKs). The tests can | 437 | These tests are run against built extensible SDKs (eSDKs). The tests can |
| 404 | assume that the eSDK environment has already been setup. An example from | 438 | assume that the eSDK environment has already been setup. An example from |
| 405 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class | 439 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: |
| 406 | DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls): | 440 | |
| 407 | myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst = | 441 | class DevtoolTest(OESDKExtTestCase): |
| 408 | os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src, | 442 | @classmethod def setUpClass(cls): |
| 409 | cls.myapp_dst) subprocess.check_output(['git', 'init', '.'], | 443 | myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") |
| 410 | cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'], | 444 | cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") |
| 411 | cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m', | 445 | shutil.copytree(myapp_src, cls.myapp_dst) |
| 412 | "'test commit'"], cwd=cls.myapp_dst) @classmethod def | 446 | subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) |
| 413 | tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def | 447 | subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) |
| 414 | \_test_devtool_build(self, directory): self._run('devtool add myapp %s' | 448 | subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) |
| 415 | % directory) try: self._run('devtool build myapp') finally: | 449 | |
| 416 | self._run('devtool reset myapp') def test_devtool_build_make(self): | 450 | @classmethod |
| 417 | self._test_devtool_build(self.myapp_dst)In this example, the ``devtool`` | 451 | def tearDownClass(cls): |
| 452 | shutil.rmtree(cls.myapp_dst) | ||
| 453 | def _test_devtool_build(self, directory): | ||
| 454 | self._run('devtool add myapp %s' % directory) | ||
| 455 | try: | ||
| 456 | self._run('devtool build myapp') | ||
| 457 | finally: | ||
| 458 | self._run('devtool reset myapp') | ||
| 459 | def test_devtool_build_make(self): | ||
| 460 | self._test_devtool_build(self.myapp_dst) | ||
| 461 | |||
| 462 | In this example, the ``devtool`` | ||
| 418 | command is tested to see whether a sample application can be built with | 463 | command is tested to see whether a sample application can be built with |
| 419 | the ``devtool build`` command within the eSDK. | 464 | the ``devtool build`` command within the eSDK. |
| 420 | 465 | ||
| @@ -426,14 +471,20 @@ the ``devtool build`` command within the eSDK. | |||
| 426 | These tests are run against built SDKs. The tests can assume that an SDK | 471 | These tests are run against built SDKs. The tests can assume that an SDK |
| 427 | has already been extracted and its environment file has been sourced. A | 472 | has already been extracted and its environment file has been sourced. A |
| 428 | simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the | 473 | simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the |
| 429 | following:class Python3Test(OESDKTestCase): def setUp(self): if not | 474 | following:: |
| 430 | (self.tc.hasHostPackage("nativesdk-python3-core") or | 475 | |
| 431 | self.tc.hasHostPackage("python3-core-native")): raise | 476 | class Python3Test(OESDKTestCase): |
| 432 | unittest.SkipTest("No python3 package in the SDK") def | 477 | def setUp(self): |
| 433 | test_python3(self): cmd = "python3 -c \\"import codecs; | 478 | if not (self.tc.hasHostPackage("nativesdk-python3-core") or |
| 434 | print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd) | 479 | self.tc.hasHostPackage("python3-core-native")): |
| 435 | self.assertEqual(output, "Hello, world\n")In this example, if | 480 | raise unittest.SkipTest("No python3 package in the SDK") |
| 436 | nativesdk-python3-core has been installed into the SDK, the code runs | 481 | |
| 482 | def test_python3(self): | ||
| 483 | cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" | ||
| 484 | output = self._run(cmd) | ||
| 485 | self.assertEqual(output, "Hello, world\n") | ||
| 486 | |||
| 487 | In this example, if nativesdk-python3-core has been installed into the SDK, the code runs | ||
| 437 | the python3 interpreter with a basic command to check it is working | 488 | the python3 interpreter with a basic command to check it is working |
| 438 | correctly. The test would only run if python3 is installed in the SDK. | 489 | correctly. The test would only run if python3 is installed in the SDK. |
| 439 | 490 | ||
| @@ -444,17 +495,25 @@ correctly. The test would only run if python3 is installed in the SDK. | |||
| 444 | 495 | ||
| 445 | The performance tests usually measure how long operations take and the | 496 | The performance tests usually measure how long operations take and the |
| 446 | resource utilisation as that happens. An example from | 497 | resource utilisation as that happens. An example from |
| 447 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class | 498 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: |
| 448 | Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake | 499 | |
| 449 | -p)""" # Drop all caches and parse self.rm_cache() | 500 | class Test3(BuildPerfTestCase): |
| 450 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | 501 | def test3(self): |
| 451 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no | 502 | """Bitbake parsing (bitbake -p)""" |
| 452 | caches)') # Drop tmp/cache | 503 | # Drop all caches and parse |
| 453 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | 504 | self.rm_cache() |
| 454 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no | 505 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) |
| 455 | tmp/cache)') # Parse with fully cached data | 506 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', |
| 456 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p | 507 | 'bitbake -p (no caches)') |
| 457 | (cached)')This example shows how three specific parsing timings are | 508 | # Drop tmp/cache |
| 509 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | ||
| 510 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', | ||
| 511 | 'bitbake -p (no tmp/cache)') | ||
| 512 | # Parse with fully cached data | ||
| 513 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', | ||
| 514 | 'bitbake -p (cached)') | ||
| 515 | |||
| 516 | This example shows how three specific parsing timings are | ||
| 458 | measured, with and without various caches, to show how BitBake’s parsing | 517 | measured, with and without various caches, to show how BitBake’s parsing |
| 459 | performance trends over time. | 518 | performance trends over time. |
| 460 | 519 | ||
diff --git a/documentation/test-manual/test-manual-test-process.rst b/documentation/test-manual/test-manual-test-process.rst index df825f8efa..96e71bf314 100644 --- a/documentation/test-manual/test-manual-test-process.rst +++ b/documentation/test-manual/test-manual-test-process.rst | |||
| @@ -23,11 +23,11 @@ We have two broad categories of test builds, including "full" and | |||
| 23 | "quick". On the Autobuilder, these can be seen as "a-quick" and | 23 | "quick". On the Autobuilder, these can be seen as "a-quick" and |
| 24 | "a-full", simply for ease of sorting in the UI. Use our Autobuilder | 24 | "a-full", simply for ease of sorting in the UI. Use our Autobuilder |
| 25 | console view to see where me manage most test-related items, available | 25 | console view to see where me manage most test-related items, available |
| 26 | at: `https://autobuilder.yoctoproject.org/typhoon/#/console <#>`__. | 26 | at: :yocto_ab:`/typhoon/#/console`. |
| 27 | 27 | ||
| 28 | Builds are triggered manually when the test branches are ready. The | 28 | Builds are triggered manually when the test branches are ready. The |
| 29 | builds are monitored by the SWAT team. For additional information, see | 29 | builds are monitored by the SWAT team. For additional information, see |
| 30 | `https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team <#>`__. | 30 | :yocto_wiki:`/wiki/Yocto_Build_Failure_Swat_Team`. |
| 31 | If successful, the changes would usually be merged to the ``master`` | 31 | If successful, the changes would usually be merged to the ``master`` |
| 32 | branch. If not successful, someone would respond to the changes on the | 32 | branch. If not successful, someone would respond to the changes on the |
| 33 | mailing list explaining that there was a failure in testing. The choice | 33 | mailing list explaining that there was a failure in testing. The choice |
| @@ -37,9 +37,9 @@ which the result was required. | |||
| 37 | The Autobuilder does build the ``master`` branch once daily for several | 37 | The Autobuilder does build the ``master`` branch once daily for several |
| 38 | reasons, in particular, to ensure the current ``master`` branch does | 38 | reasons, in particular, to ensure the current ``master`` branch does |
| 39 | build, but also to keep ``yocto-testresults`` | 39 | build, but also to keep ``yocto-testresults`` |
| 40 | (`http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ <#>`__), | 40 | (:yocto_git:`/cgit.cgi/yocto-testresults/`), |
| 41 | buildhistory | 41 | buildhistory |
| 42 | (`http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/ <#>`__), and | 42 | (:yocto_git:`/cgit.cgi/poky-buildhistory/`), and |
| 43 | our sstate up to date. On the weekend, there is a master-next build | 43 | our sstate up to date. On the weekend, there is a master-next build |
| 44 | instead to ensure the test results are updated for the less frequently | 44 | instead to ensure the test results are updated for the less frequently |
| 45 | run targets. | 45 | run targets. |
| @@ -47,7 +47,7 @@ run targets. | |||
| 47 | Performance builds (buildperf-\* targets in the console) are triggered | 47 | Performance builds (buildperf-\* targets in the console) are triggered |
| 48 | separately every six hours and automatically push their results to the | 48 | separately every six hours and automatically push their results to the |
| 49 | buildstats repository at: | 49 | buildstats repository at: |
| 50 | `http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/ <#>`__. | 50 | :yocto_git:`/cgit.cgi/yocto-buildstats/`. |
| 51 | 51 | ||
| 52 | The 'quick' targets have been selected to be the ones which catch the | 52 | The 'quick' targets have been selected to be the ones which catch the |
| 53 | most failures or give the most valuable data. We run 'fast' ptests in | 53 | most failures or give the most valuable data. We run 'fast' ptests in |
| @@ -56,8 +56,6 @@ target doesn't include \*-lsb builds for all architectures, some world | |||
| 56 | builds and doesn't trigger performance tests or ltp testing. The full | 56 | builds and doesn't trigger performance tests or ltp testing. The full |
| 57 | build includes all these things and is slower but more comprehensive. | 57 | build includes all these things and is slower but more comprehensive. |
| 58 | 58 | ||
| 59 | .. _test-yocto-project-autobuilder-overview: | ||
| 60 | |||
| 61 | Release Builds | 59 | Release Builds |
| 62 | ============== | 60 | ============== |
| 63 | 61 | ||
| @@ -76,7 +74,7 @@ box to "generate an email to QA"is also checked. | |||
| 76 | When the build completes, an email is sent out using the send-qa-email | 74 | When the build completes, an email is sent out using the send-qa-email |
| 77 | script in the ``yocto-autobuilder-helper`` repository to the list of | 75 | script in the ``yocto-autobuilder-helper`` repository to the list of |
| 78 | people configured for that release. Release builds are placed into a | 76 | people configured for that release. Release builds are placed into a |
| 79 | directory in `https://autobuilder.yocto.io/pub/releases <#>`__ on the | 77 | directory in https://autobuilder.yocto.io/pub/releases on the |
| 80 | Autobuilder which is included in the email. The process from here is | 78 | Autobuilder which is included in the email. The process from here is |
| 81 | more manual and control is effectively passed to release engineering. | 79 | more manual and control is effectively passed to release engineering. |
| 82 | The next steps include: | 80 | The next steps include: |
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst index 8d3c844156..0809190b5b 100644 --- a/documentation/test-manual/test-manual-understand-autobuilder.rst +++ b/documentation/test-manual/test-manual-understand-autobuilder.rst | |||
| @@ -10,7 +10,7 @@ Execution Flow within the Autobuilder | |||
| 10 | The “a-full” and “a-quick” targets are the usual entry points into the | 10 | The “a-full” and “a-quick” targets are the usual entry points into the |
| 11 | Autobuilder and it makes sense to follow the process through the system | 11 | Autobuilder and it makes sense to follow the process through the system |
| 12 | starting there. This is best visualised from the Autobuilder Console | 12 | starting there. This is best visualised from the Autobuilder Console |
| 13 | view (`https://autobuilder.yoctoproject.org/typhoon/#/console <#>`__). | 13 | view (:yocto_ab:`/typhoon/#/console`). |
| 14 | 14 | ||
| 15 | Each item along the top of that view represents some “target build” and | 15 | Each item along the top of that view represents some “target build” and |
| 16 | these targets are all run in parallel. The ‘full’ build will trigger the | 16 | these targets are all run in parallel. The ‘full’ build will trigger the |
| @@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker. To understand the | |||
| 20 | configuration, you need to look at the entry on ``config.json`` file | 20 | configuration, you need to look at the entry on ``config.json`` file |
| 21 | within the ``yocto-autobuilder-helper`` repository. The targets are | 21 | within the ``yocto-autobuilder-helper`` repository. The targets are |
| 22 | defined in the ‘overrides’ section, a quick example could be qemux86-64 | 22 | defined in the ‘overrides’ section, a quick example could be qemux86-64 |
| 23 | which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" : | 23 | which looks like:: |
| 24 | "arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic | 24 | |
| 25 | wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from | 25 | "qemux86-64" : { |
| 26 | the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" : | 26 | "MACHINE" : "qemux86-64", |
| 27 | true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato | 27 | "TEMPLATE" : "arch-qemu", |
| 28 | core-image-sato-dev core-image-sato-sdk core-image-minimal | 28 | "step1" : { |
| 29 | core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS" | 29 | "extravars" : [ |
| 30 | : "core-image-minimal:do_testimage core-image-sato:do_testimage | 30 | "IMAGE_FSTYPES_append = ' wic wic.bmap'" |
| 31 | core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2" | 31 | ] |
| 32 | : { "SDKMACHINE" : "x86_64", "BBTARGETS" : | 32 | } |
| 33 | "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext | 33 | }, |
| 34 | core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" : | 34 | |
| 35 | "core-image-sato:do_testsdk core-image-minimal:do_testsdkext | 35 | And to expand that, you need the “arch-qemu” entry from |
| 36 | core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false, | 36 | the “templates” section, which looks like:: |
| 37 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest | 37 | |
| 38 | ${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" : | 38 | "arch-qemu" : { |
| 39 | ["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can | 39 | "BUILDINFO" : true, |
| 40 | see that “qemux86-64” is a three step build where the | 40 | "BUILDHISTORY" : true, |
| 41 | ``bitbake BBTARGETS`` would be run, then ``bitbake | 41 | "step1" : { |
| 42 | SANITYTARGETS`` for each step; all for | 42 | "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", |
| 43 | "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" | ||
| 44 | }, | ||
| 45 | "step2" : { | ||
| 46 | "SDKMACHINE" : "x86_64", | ||
| 47 | "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", | ||
| 48 | "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" | ||
| 49 | }, | ||
| 50 | "step3" : { | ||
| 51 | "BUILDHISTORY" : false, | ||
| 52 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], | ||
| 53 | "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] | ||
| 54 | } | ||
| 55 | }, | ||
| 56 | |||
| 57 | Combining these two entries you can see that “qemux86-64” is a three step build where the | ||
| 58 | ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for | ||
| 43 | ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step | 59 | ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step |
| 44 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | 60 | 1 an extra variable is added to the ``auto.conf`` file to enable wic |
| 45 | image generation. | 61 | image generation. |
| 46 | 62 | ||
| 47 | While not every detail of this is covered here, you can see how the | 63 | While not every detail of this is covered here, you can see how the |
| 48 | templating mechanism allows quite complex configurations to be built up | 64 | template mechanism allows quite complex configurations to be built up |
| 49 | yet allows duplication and repetition to be kept to a minimum. | 65 | yet allows duplication and repetition to be kept to a minimum. |
| 50 | 66 | ||
| 51 | The different build targets are designed to allow for parallelisation, | 67 | The different build targets are designed to allow for parallelisation, |
| @@ -66,13 +82,13 @@ For each given target in a build, the Autobuilder executes several | |||
| 66 | steps. These are configured in ``yocto-autobuilder2/builders.py`` and | 82 | steps. These are configured in ``yocto-autobuilder2/builders.py`` and |
| 67 | roughly consist of: | 83 | roughly consist of: |
| 68 | 84 | ||
| 69 | 1. *Run ``clobberdir``* | 85 | #. *Run clobberdir*. |
| 70 | 86 | ||
| 71 | This cleans out any previous build. Old builds are left around to | 87 | This cleans out any previous build. Old builds are left around to |
| 72 | allow easier debugging of failed builds. For additional information, | 88 | allow easier debugging of failed builds. For additional information, |
| 73 | see ```clobberdir`` <#test-clobberdir>`__. | 89 | see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`. |
| 74 | 90 | ||
| 75 | 2. *Obtain yocto-autobuilder-helper* | 91 | #. *Obtain yocto-autobuilder-helper* |
| 76 | 92 | ||
| 77 | This step clones the ``yocto-autobuilder-helper`` git repository. | 93 | This step clones the ``yocto-autobuilder-helper`` git repository. |
| 78 | This is necessary to prevent the requirement to maintain all the | 94 | This is necessary to prevent the requirement to maintain all the |
| @@ -80,12 +96,12 @@ roughly consist of: | |||
| 80 | matches the release being built so we can support older releases and | 96 | matches the release being built so we can support older releases and |
| 81 | still make changes in newer ones. | 97 | still make changes in newer ones. |
| 82 | 98 | ||
| 83 | 3. *Write layerinfo.json* | 99 | #. *Write layerinfo.json* |
| 84 | 100 | ||
| 85 | This transfers data in the Buildbot UI when the build was configured | 101 | This transfers data in the Buildbot UI when the build was configured |
| 86 | to the Helper. | 102 | to the Helper. |
| 87 | 103 | ||
| 88 | 4. *Call scripts/shared-repo-unpack* | 104 | #. *Call scripts/shared-repo-unpack* |
| 89 | 105 | ||
| 90 | This is a call into the Helper scripts to set up a checkout of all | 106 | This is a call into the Helper scripts to set up a checkout of all |
| 91 | the pieces this build might need. It might clone the BitBake | 107 | the pieces this build might need. It might clone the BitBake |
| @@ -94,7 +110,7 @@ roughly consist of: | |||
| 94 | from the ``layerinfo.json`` file to help understand the | 110 | from the ``layerinfo.json`` file to help understand the |
| 95 | configuration. It will also use a local cache of repositories to | 111 | configuration. It will also use a local cache of repositories to |
| 96 | speed up the clone checkouts. For additional information, see | 112 | speed up the clone checkouts. For additional information, see |
| 97 | `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__. | 113 | :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`. |
| 98 | 114 | ||
| 99 | This step has two possible modes of operation. If the build is part | 115 | This step has two possible modes of operation. If the build is part |
| 100 | of a parent build, its possible that all the repositories needed may | 116 | of a parent build, its possible that all the repositories needed may |
| @@ -114,7 +130,7 @@ roughly consist of: | |||
| 114 | available, it would clone from the cache and the upstreams as | 130 | available, it would clone from the cache and the upstreams as |
| 115 | necessary. This is considered the fallback mode. | 131 | necessary. This is considered the fallback mode. |
| 116 | 132 | ||
| 117 | 5. *Call scripts/run-config* | 133 | #. *Call scripts/run-config* |
| 118 | 134 | ||
| 119 | This is another call into the Helper scripts where its expected that | 135 | This is another call into the Helper scripts where its expected that |
| 120 | the main functionality of this target will be executed. | 136 | the main functionality of this target will be executed. |
| @@ -137,8 +153,7 @@ special script that moves files to a special location, rather than | |||
| 137 | deleting them. Files in this location are deleted by an ``rm`` command, | 153 | deleting them. Files in this location are deleted by an ``rm`` command, |
| 138 | which is run under ``ionice -c 3``. For example, the deletion only | 154 | which is run under ``ionice -c 3``. For example, the deletion only |
| 139 | happens when there is idle IO capacity on the Worker. The Autobuilder | 155 | happens when there is idle IO capacity on the Worker. The Autobuilder |
| 140 | Worker Janitor runs this deletion. See `Autobuilder Worker | 156 | Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. |
| 141 | Janitor <#test-autobuilder-worker-janitor>`__. | ||
| 142 | 157 | ||
| 143 | .. _test-autobuilder-clone-cache: | 158 | .. _test-autobuilder-clone-cache: |
| 144 | 159 | ||
| @@ -150,8 +165,7 @@ on the Autobuilder. We therefore have a stash of commonly used | |||
| 150 | repositories pre-cloned on the Workers. Data is fetched from these | 165 | repositories pre-cloned on the Workers. Data is fetched from these |
| 151 | during clones first, then "topped up" with later revisions from any | 166 | during clones first, then "topped up" with later revisions from any |
| 152 | upstream when necesary. The cache is maintained by the Autobuilder | 167 | upstream when necesary. The cache is maintained by the Autobuilder |
| 153 | Worker Janitor. See `Autobuilder Worker | 168 | Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. |
| 154 | Janitor <#test-autobuilder-worker-janitor>`__. | ||
| 155 | 169 | ||
| 156 | .. _test-autobuilder-worker-janitor: | 170 | .. _test-autobuilder-worker-janitor: |
| 157 | 171 | ||
| @@ -159,8 +173,7 @@ Autobuilder Worker Janitor | |||
| 159 | -------------------------- | 173 | -------------------------- |
| 160 | 174 | ||
| 161 | This is a process running on each Worker that performs two basic | 175 | This is a process running on each Worker that performs two basic |
| 162 | operations, including background file deletion at IO idle (see `Target | 176 | operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and |
| 163 | Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and | ||
| 164 | maintainenance of a cache of cloned repositories to improve the speed | 177 | maintainenance of a cache of cloned repositories to improve the speed |
| 165 | the system can checkout repositories. | 178 | the system can checkout repositories. |
| 166 | 179 | ||
| @@ -181,7 +194,7 @@ Shared SSTATE_DIR | |||
| 181 | 194 | ||
| 182 | The Workers are all connected over NFS which allows the ``sstate`` | 195 | The Workers are all connected over NFS which allows the ``sstate`` |
| 183 | directory to be shared between them. This means once a Worker has built | 196 | directory to be shared between them. This means once a Worker has built |
| 184 | an artefact, all the others can benefit from it. Usage of the directory | 197 | an artifact, all the others can benefit from it. Usage of the directory |
| 185 | within the directory is designed for sharing over NFS. | 198 | within the directory is designed for sharing over NFS. |
| 186 | 199 | ||
| 187 | .. _test-resulttool: | 200 | .. _test-resulttool: |
| @@ -198,7 +211,7 @@ Resulttool is part of OpenEmbedded-Core and is used to manipulate these | |||
| 198 | json results files. It has the ability to merge files together, display | 211 | json results files. It has the ability to merge files together, display |
| 199 | reports of the test results and compare different result files. | 212 | reports of the test results and compare different result files. |
| 200 | 213 | ||
| 201 | For details, see `https://wiki.yoctoproject.org/wiki/Resulttool <#>`__. | 214 | For details, see :yocto_wiki:`/wiki/Resulttool`. |
| 202 | 215 | ||
| 203 | .. _test-run-config-tgt-execution: | 216 | .. _test-run-config-tgt-execution: |
| 204 | 217 | ||
| @@ -209,50 +222,46 @@ The ``scripts/run-config`` execution is where most of the work within | |||
| 209 | the Autobuilder happens. It runs through a number of steps; the first | 222 | the Autobuilder happens. It runs through a number of steps; the first |
| 210 | are general setup steps that are run once and include: | 223 | are general setup steps that are run once and include: |
| 211 | 224 | ||
| 212 | 1. Set up any ``buildtools-tarball`` if configured. | 225 | #. Set up any ``buildtools-tarball`` if configured. |
| 213 | 226 | ||
| 214 | 2. Call "buildhistory-init" if buildhistory is configured. | 227 | #. Call "buildhistory-init" if buildhistory is configured. |
| 215 | 228 | ||
| 216 | For each step that is configured in ``config.json``, it will perform the | 229 | For each step that is configured in ``config.json``, it will perform the |
| 217 | following: | 230 | following: |
| 218 | 231 | ||
| 219 | ## WRITER's question: What does "logging in as stepXa" and others refer | 232 | #. Add any layers that are specified using the |
| 220 | to below? ## | ||
| 221 | |||
| 222 | 1. Add any layers that are specified using the | ||
| 223 | ``bitbake-layers add-layer`` command (logging as stepXa) | 233 | ``bitbake-layers add-layer`` command (logging as stepXa) |
| 224 | 234 | ||
| 225 | 2. Call the ``scripts/setup-config`` script to generate the necessary | 235 | #. Call the ``scripts/setup-config`` script to generate the necessary |
| 226 | ``auto.conf`` configuration file for the build | 236 | ``auto.conf`` configuration file for the build |
| 227 | 237 | ||
| 228 | 3. Run the ``bitbake BBTARGETS`` command (logging as stepXb) | 238 | #. Run the ``bitbake BBTARGETS`` command (logging as stepXb) |
| 229 | 239 | ||
| 230 | 4. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) | 240 | #. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) |
| 231 | 241 | ||
| 232 | 5. Run the ``EXTRACMDS`` command, which are run within the BitBake build | 242 | #. Run the ``EXTRACMDS`` command, which are run within the BitBake build |
| 233 | environment (logging as stepXd) | 243 | environment (logging as stepXd) |
| 234 | 244 | ||
| 235 | 6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the | 245 | #. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the |
| 236 | BitBake build environment (logging as stepXd) | 246 | BitBake build environment (logging as stepXd) |
| 237 | 247 | ||
| 238 | 7. Remove any layers added in `step | 248 | #. Remove any layers added in step |
| 239 | 1 <#test-run-config-add-layers-step>`__ using the | 249 | 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa) |
| 240 | ``bitbake-layers remove-layer`` command (logging as stepXa) | ||
| 241 | 250 | ||
| 242 | Once the execution steps above complete, ``run-config`` executes a set | 251 | Once the execution steps above complete, ``run-config`` executes a set |
| 243 | of post-build steps, including: | 252 | of post-build steps, including: |
| 244 | 253 | ||
| 245 | 1. Call ``scripts/publish-artifacts`` to collect any output which is to | 254 | #. Call ``scripts/publish-artifacts`` to collect any output which is to |
| 246 | be saved from the build. | 255 | be saved from the build. |
| 247 | 256 | ||
| 248 | 2. Call ``scripts/collect-results`` to collect any test results to be | 257 | #. Call ``scripts/collect-results`` to collect any test results to be |
| 249 | saved from the build. | 258 | saved from the build. |
| 250 | 259 | ||
| 251 | 3. Call ``scripts/upload-error-reports`` to send any error reports | 260 | #. Call ``scripts/upload-error-reports`` to send any error reports |
| 252 | generated to the remote server. | 261 | generated to the remote server. |
| 253 | 262 | ||
| 254 | 4. Cleanup the build directory using | 263 | #. Cleanup the build directory using |
| 255 | ```clobberdir`` <#test-clobberdir>`__ if the build was successful, | 264 | :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful, |
| 256 | else rename it to “build-renamed” for potential future debugging. | 265 | else rename it to “build-renamed” for potential future debugging. |
| 257 | 266 | ||
| 258 | .. _test-deploying-yp-autobuilder: | 267 | .. _test-deploying-yp-autobuilder: |
| @@ -279,11 +288,18 @@ The standard ``config.json`` minimally attempts to allow substitution of | |||
| 279 | the paths. The Helper script repository includes a | 288 | the paths. The Helper script repository includes a |
| 280 | ``local-example.json`` file to show how you could override these from a | 289 | ``local-example.json`` file to show how you could override these from a |
| 281 | separate configuration file. Pass the following into the environment of | 290 | separate configuration file. Pass the following into the environment of |
| 282 | the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As | 291 | the Autobuilder:: |
| 283 | another example, you could also pass the following into the | 292 | |
| 284 | environment:$ ABHELPER_JSON="config.json /some/location/local.json"One | 293 | $ ABHELPER_JSON="config.json local-example.json" |
| 285 | issue users often run into is validation of the ``config.json`` files. A | 294 | |
| 295 | As another example, you could also pass the following into the | ||
| 296 | environment:: | ||
| 297 | |||
| 298 | $ ABHELPER_JSON="config.json /some/location/local.json" | ||
| 299 | |||
| 300 | One issue users often run into is validation of the ``config.json`` files. A | ||
| 286 | tip for minimizing issues from invalid json files is to use a Git | 301 | tip for minimizing issues from invalid json files is to use a Git |
| 287 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | 302 | ``pre-commit-hook.sh`` script to verify the JSON file before committing |
| 288 | it. Create a symbolic link as follows:$ ln -s | 303 | it. Create a symbolic link as follows:: |
| 289 | ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | 304 | |
| 305 | $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | ||
