summaryrefslogtreecommitdiffstats
path: root/documentation/test-manual/understand-autobuilder.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/test-manual/understand-autobuilder.rst')
-rw-r--r--documentation/test-manual/understand-autobuilder.rst80
1 files changed, 41 insertions, 39 deletions
diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst
index 199cc97a85..6b4fab4f0b 100644
--- a/documentation/test-manual/understand-autobuilder.rst
+++ b/documentation/test-manual/understand-autobuilder.rst
@@ -9,31 +9,31 @@ Execution Flow within the Autobuilder
9 9
10The "a-full" and "a-quick" targets are the usual entry points into the 10The "a-full" and "a-quick" targets are the usual entry points into the
11Autobuilder and it makes sense to follow the process through the system 11Autobuilder and it makes sense to follow the process through the system
12starting there. This is best visualised from the Autobuilder Console 12starting there. This is best visualized from the :yocto_ab:`Autobuilder
13view (:yocto_ab:`/typhoon/#/console`). 13Console view </typhoon/#/console>`.
14 14
15Each item along the top of that view represents some "target build" and 15Each item along the top of that view represents some "target build" and
16these targets are all run in parallel. The 'full' build will trigger the 16these targets are all run in parallel. The 'full' build will trigger the
17majority of them, the "quick" build will trigger some subset of them. 17majority of them, the "quick" build will trigger some subset of them.
18The Autobuilder effectively runs whichever configuration is defined for 18The Autobuilder effectively runs whichever configuration is defined for
19each of those targets on a seperate buildbot worker. To understand the 19each of those targets on a separate buildbot worker. To understand the
20configuration, you need to look at the entry on ``config.json`` file 20configuration, you need to look at the entry on ``config.json`` file
21within the ``yocto-autobuilder-helper`` repository. The targets are 21within the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>`
22defined in the overrides' section, a quick example could be qemux86-64 22repository. The targets are defined in the ``overrides`` section, a quick
23which looks like:: 23example could be ``qemux86-64`` which looks like::
24 24
25 "qemux86-64" : { 25 "qemux86-64" : {
26 "MACHINE" : "qemux86-64", 26 "MACHINE" : "qemux86-64",
27 "TEMPLATE" : "arch-qemu", 27 "TEMPLATE" : "arch-qemu",
28 "step1" : { 28 "step1" : {
29 "extravars" : [ 29 "extravars" : [
30 "IMAGE_FSTYPES_append = ' wic wic.bmap'" 30 "IMAGE_FSTYPES:append = ' wic wic.bmap'"
31 ] 31 ]
32 } 32 }
33 }, 33 },
34 34
35And to expand that, you need the "arch-qemu" entry from 35And to expand that, you need the ``arch-qemu`` entry from
36the "templates" section, which looks like:: 36the ``templates`` section, which looks like::
37 37
38 "arch-qemu" : { 38 "arch-qemu" : {
39 "BUILDINFO" : true, 39 "BUILDINFO" : true,
@@ -54,20 +54,20 @@ the "templates" section, which looks like::
54 } 54 }
55 }, 55 },
56 56
57Combining these two entries you can see that "qemux86-64" is a three step build where the 57Combining these two entries you can see that ``qemux86-64`` is a three step
58``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for 58build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS``
59``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step 59for each step; all for ``MACHINE="qemux86-64"`` but with differing
601 an extra variable is added to the ``auto.conf`` file to enable wic 60:term:`SDKMACHINE` settings. In step 1, an extra variable is added to the
61image generation. 61``auto.conf`` file to enable wic image generation.
62 62
63While not every detail of this is covered here, you can see how the 63While not every detail of this is covered here, you can see how the
64template mechanism allows quite complex configurations to be built up 64template mechanism allows quite complex configurations to be built up
65yet allows duplication and repetition to be kept to a minimum. 65yet allows duplication and repetition to be kept to a minimum.
66 66
67The different build targets are designed to allow for parallelisation, 67The different build targets are designed to allow for parallelization,
68so different machines are usually built in parallel, operations using 68so different machines are usually built in parallel, operations using
69the same machine and metadata are built sequentially, with the aim of 69the same machine and metadata are built sequentially, with the aim of
70trying to optimise build efficiency as much as possible. 70trying to optimize build efficiency as much as possible.
71 71
72The ``config.json`` file is processed by the scripts in the Helper 72The ``config.json`` file is processed by the scripts in the Helper
73repository in the ``scripts`` directory. The following section details 73repository in the ``scripts`` directory. The following section details
@@ -88,9 +88,9 @@ roughly consist of:
88 88
89#. *Obtain yocto-autobuilder-helper* 89#. *Obtain yocto-autobuilder-helper*
90 90
91 This step clones the ``yocto-autobuilder-helper`` git repository. 91 This step clones the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>`
92 This is necessary to prevent the requirement to maintain all the 92 git repository. This is necessary to avoid the requirement to maintain all
93 release or project-specific code within Buildbot. The branch chosen 93 the release or project-specific code within Buildbot. The branch chosen
94 matches the release being built so we can support older releases and 94 matches the release being built so we can support older releases and
95 still make changes in newer ones. 95 still make changes in newer ones.
96 96
@@ -111,7 +111,7 @@ roughly consist of:
111 :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`. 111 :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`.
112 112
113 This step has two possible modes of operation. If the build is part 113 This step has two possible modes of operation. If the build is part
114 of a parent build, its possible that all the repositories needed may 114 of a parent build, it's possible that all the repositories needed may
115 already be available, ready in a pre-prepared directory. An "a-quick" 115 already be available, ready in a pre-prepared directory. An "a-quick"
116 or "a-full" build would prepare this before starting the other 116 or "a-full" build would prepare this before starting the other
117 sub-target builds. This is done for two reasons: 117 sub-target builds. This is done for two reasons:
@@ -130,7 +130,7 @@ roughly consist of:
130 130
131#. *Call scripts/run-config* 131#. *Call scripts/run-config*
132 132
133 This is another call into the Helper scripts where its expected that 133 This is another call into the Helper scripts where it's expected that
134 the main functionality of this target will be executed. 134 the main functionality of this target will be executed.
135 135
136Autobuilder Technology 136Autobuilder Technology
@@ -163,16 +163,17 @@ Autobuilder Worker Janitor
163-------------------------- 163--------------------------
164 164
165This is a process running on each Worker that performs two basic 165This is a process running on each Worker that performs two basic
166operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and 166operations, including background file deletion at IO idle (see
167maintainenance of a cache of cloned repositories to improve the speed 167"Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`)
168and maintenance of a cache of cloned repositories to improve the speed
168the system can checkout repositories. 169the system can checkout repositories.
169 170
170Shared DL_DIR 171Shared DL_DIR
171------------- 172-------------
172 173
173The Workers are all connected over NFS which allows DL_DIR to be shared 174The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared
174between them. This reduces network accesses from the system and allows 175between them. This reduces network accesses from the system and allows
175the build to be sped up. Usage of the directory within the build system 176the build to be sped up. The usage of the directory within the build system
176is designed to be able to be shared over NFS. 177is designed to be able to be shared over NFS.
177 178
178Shared SSTATE_DIR 179Shared SSTATE_DIR
@@ -180,8 +181,8 @@ Shared SSTATE_DIR
180 181
181The Workers are all connected over NFS which allows the ``sstate`` 182The Workers are all connected over NFS which allows the ``sstate``
182directory to be shared between them. This means once a Worker has built 183directory to be shared between them. This means once a Worker has built
183an artifact, all the others can benefit from it. Usage of the directory 184an artifact, all the others can benefit from it. The usage of the directory
184within the directory is designed for sharing over NFS. 185within the build system is designed for sharing over NFS.
185 186
186Resulttool 187Resulttool
187---------- 188----------
@@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as
192failure logs or the time taken to run the tests, may also be included. 193failure logs or the time taken to run the tests, may also be included.
193 194
194Resulttool is part of OpenEmbedded-Core and is used to manipulate these 195Resulttool is part of OpenEmbedded-Core and is used to manipulate these
195json results files. It has the ability to merge files together, display 196JSON results files. It has the ability to merge files together, display
196reports of the test results and compare different result files. 197reports of the test results and compare different result files.
197 198
198For details, see :yocto_wiki:`/Resulttool`. 199For details, see :yocto_wiki:`/Resulttool`.
@@ -204,9 +205,9 @@ The ``scripts/run-config`` execution is where most of the work within
204the Autobuilder happens. It runs through a number of steps; the first 205the Autobuilder happens. It runs through a number of steps; the first
205are general setup steps that are run once and include: 206are general setup steps that are run once and include:
206 207
207#. Set up any ``buildtools-tarball`` if configured. 208#. Set up any :term:`buildtools` tarball if configured.
208 209
209#. Call "buildhistory-init" if buildhistory is configured. 210#. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured.
210 211
211For each step that is configured in ``config.json``, it will perform the 212For each step that is configured in ``config.json``, it will perform the
212following: 213following:
@@ -242,7 +243,7 @@ of post-build steps, including:
242#. Call ``scripts/upload-error-reports`` to send any error reports 243#. Call ``scripts/upload-error-reports`` to send any error reports
243 generated to the remote server. 244 generated to the remote server.
244 245
245#. Cleanup the build directory using 246#. Cleanup the :term:`Build Directory` using
246 :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful, 247 :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful,
247 else rename it to "build-renamed" for potential future debugging. 248 else rename it to "build-renamed" for potential future debugging.
248 249
@@ -250,15 +251,16 @@ Deploying Yocto Autobuilder
250=========================== 251===========================
251 252
252The most up to date information about how to setup and deploy your own 253The most up to date information about how to setup and deploy your own
253Autbuilder can be found in README.md in the ``yocto-autobuilder2`` 254Autobuilder can be found in :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>`
254repository. 255in the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>` repository.
255 256
256We hope that people can use the ``yocto-autobuilder2`` code directly but 257We hope that people can use the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>`
257it is inevitable that users will end up needing to heavily customise the 258code directly but it is inevitable that users will end up needing to heavily
258``yocto-autobuilder-helper`` repository, particularly the 259customize the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>`
259``config.json`` file as they will want to define their own test matrix. 260repository, particularly the ``config.json`` file as they will want to define
261their own test matrix.
260 262
261The Autobuilder supports wo customization options: 263The Autobuilder supports two customization options:
262 264
263- variable substitution 265- variable substitution
264 266
@@ -278,7 +280,7 @@ environment::
278 $ ABHELPER_JSON="config.json /some/location/local.json" 280 $ ABHELPER_JSON="config.json /some/location/local.json"
279 281
280One issue users often run into is validation of the ``config.json`` files. A 282One issue users often run into is validation of the ``config.json`` files. A
281tip for minimizing issues from invalid json files is to use a Git 283tip for minimizing issues from invalid JSON files is to use a Git
282``pre-commit-hook.sh`` script to verify the JSON file before committing 284``pre-commit-hook.sh`` script to verify the JSON file before committing
283it. Create a symbolic link as follows:: 285it. Create a symbolic link as follows::
284 286