summaryrefslogtreecommitdiffstats
path: root/documentation/test-manual/understand-autobuilder.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/test-manual/understand-autobuilder.rst')
-rw-r--r--documentation/test-manual/understand-autobuilder.rst86
1 files changed, 47 insertions, 39 deletions
diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst
index 199cc97a85..23b3ef7d7f 100644
--- a/documentation/test-manual/understand-autobuilder.rst
+++ b/documentation/test-manual/understand-autobuilder.rst
@@ -9,31 +9,31 @@ Execution Flow within the Autobuilder
9 9
10The "a-full" and "a-quick" targets are the usual entry points into the 10The "a-full" and "a-quick" targets are the usual entry points into the
11Autobuilder and it makes sense to follow the process through the system 11Autobuilder and it makes sense to follow the process through the system
12starting there. This is best visualised from the Autobuilder Console 12starting there. This is best visualized from the :yocto_ab:`Autobuilder
13view (:yocto_ab:`/typhoon/#/console`). 13Console view </valkyrie/#/console>`.
14 14
15Each item along the top of that view represents some "target build" and 15Each item along the top of that view represents some "target build" and
16these targets are all run in parallel. The 'full' build will trigger the 16these targets are all run in parallel. The 'full' build will trigger the
17majority of them, the "quick" build will trigger some subset of them. 17majority of them, the "quick" build will trigger some subset of them.
18The Autobuilder effectively runs whichever configuration is defined for 18The Autobuilder effectively runs whichever configuration is defined for
19each of those targets on a seperate buildbot worker. To understand the 19each of those targets on a separate buildbot worker. To understand the
20configuration, you need to look at the entry on ``config.json`` file 20configuration, you need to look at the entry on ``config.json`` file
21within the ``yocto-autobuilder-helper`` repository. The targets are 21within the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>`
22defined in the overrides' section, a quick example could be qemux86-64 22repository. The targets are defined in the ``overrides`` section, a quick
23which looks like:: 23example could be ``qemux86-64`` which looks like::
24 24
25 "qemux86-64" : { 25 "qemux86-64" : {
26 "MACHINE" : "qemux86-64", 26 "MACHINE" : "qemux86-64",
27 "TEMPLATE" : "arch-qemu", 27 "TEMPLATE" : "arch-qemu",
28 "step1" : { 28 "step1" : {
29 "extravars" : [ 29 "extravars" : [
30 "IMAGE_FSTYPES_append = ' wic wic.bmap'" 30 "IMAGE_FSTYPES:append = ' wic wic.bmap'"
31 ] 31 ]
32 } 32 }
33 }, 33 },
34 34
35And to expand that, you need the "arch-qemu" entry from 35And to expand that, you need the ``arch-qemu`` entry from
36the "templates" section, which looks like:: 36the ``templates`` section, which looks like::
37 37
38 "arch-qemu" : { 38 "arch-qemu" : {
39 "BUILDINFO" : true, 39 "BUILDINFO" : true,
@@ -54,20 +54,20 @@ the "templates" section, which looks like::
54 } 54 }
55 }, 55 },
56 56
57Combining these two entries you can see that "qemux86-64" is a three step build where the 57Combining these two entries you can see that ``qemux86-64`` is a three step
58``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for 58build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS``
59``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step 59for each step; all for ``MACHINE="qemux86-64"`` but with differing
601 an extra variable is added to the ``auto.conf`` file to enable wic 60:term:`SDKMACHINE` settings. In step 1, an extra variable is added to the
61image generation. 61``auto.conf`` file to enable wic image generation.
62 62
63While not every detail of this is covered here, you can see how the 63While not every detail of this is covered here, you can see how the
64template mechanism allows quite complex configurations to be built up 64template mechanism allows quite complex configurations to be built up
65yet allows duplication and repetition to be kept to a minimum. 65yet allows duplication and repetition to be kept to a minimum.
66 66
67The different build targets are designed to allow for parallelisation, 67The different build targets are designed to allow for parallelization,
68so different machines are usually built in parallel, operations using 68so different machines are usually built in parallel, operations using
69the same machine and metadata are built sequentially, with the aim of 69the same machine and metadata are built sequentially, with the aim of
70trying to optimise build efficiency as much as possible. 70trying to optimize build efficiency as much as possible.
71 71
72The ``config.json`` file is processed by the scripts in the Helper 72The ``config.json`` file is processed by the scripts in the Helper
73repository in the ``scripts`` directory. The following section details 73repository in the ``scripts`` directory. The following section details
@@ -88,9 +88,9 @@ roughly consist of:
88 88
89#. *Obtain yocto-autobuilder-helper* 89#. *Obtain yocto-autobuilder-helper*
90 90
91 This step clones the ``yocto-autobuilder-helper`` git repository. 91 This step clones the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>`
92 This is necessary to prevent the requirement to maintain all the 92 git repository. This is necessary to avoid the requirement to maintain all
93 release or project-specific code within Buildbot. The branch chosen 93 the release or project-specific code within Buildbot. The branch chosen
94 matches the release being built so we can support older releases and 94 matches the release being built so we can support older releases and
95 still make changes in newer ones. 95 still make changes in newer ones.
96 96
@@ -111,7 +111,7 @@ roughly consist of:
111 :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`. 111 :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`.
112 112
113 This step has two possible modes of operation. If the build is part 113 This step has two possible modes of operation. If the build is part
114 of a parent build, its possible that all the repositories needed may 114 of a parent build, it's possible that all the repositories needed may
115 already be available, ready in a pre-prepared directory. An "a-quick" 115 already be available, ready in a pre-prepared directory. An "a-quick"
116 or "a-full" build would prepare this before starting the other 116 or "a-full" build would prepare this before starting the other
117 sub-target builds. This is done for two reasons: 117 sub-target builds. This is done for two reasons:
@@ -130,7 +130,7 @@ roughly consist of:
130 130
131#. *Call scripts/run-config* 131#. *Call scripts/run-config*
132 132
133 This is another call into the Helper scripts where its expected that 133 This is another call into the Helper scripts where it's expected that
134 the main functionality of this target will be executed. 134 the main functionality of this target will be executed.
135 135
136Autobuilder Technology 136Autobuilder Technology
@@ -163,16 +163,17 @@ Autobuilder Worker Janitor
163-------------------------- 163--------------------------
164 164
165This is a process running on each Worker that performs two basic 165This is a process running on each Worker that performs two basic
166operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and 166operations, including background file deletion at IO idle (see
167maintainenance of a cache of cloned repositories to improve the speed 167"Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`)
168and maintenance of a cache of cloned repositories to improve the speed
168the system can checkout repositories. 169the system can checkout repositories.
169 170
170Shared DL_DIR 171Shared DL_DIR
171------------- 172-------------
172 173
173The Workers are all connected over NFS which allows DL_DIR to be shared 174The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared
174between them. This reduces network accesses from the system and allows 175between them. This reduces network accesses from the system and allows
175the build to be sped up. Usage of the directory within the build system 176the build to be sped up. The usage of the directory within the build system
176is designed to be able to be shared over NFS. 177is designed to be able to be shared over NFS.
177 178
178Shared SSTATE_DIR 179Shared SSTATE_DIR
@@ -180,8 +181,14 @@ Shared SSTATE_DIR
180 181
181The Workers are all connected over NFS which allows the ``sstate`` 182The Workers are all connected over NFS which allows the ``sstate``
182directory to be shared between them. This means once a Worker has built 183directory to be shared between them. This means once a Worker has built
183an artifact, all the others can benefit from it. Usage of the directory 184an artifact, all the others can benefit from it. The usage of the directory
184within the directory is designed for sharing over NFS. 185within the build system is designed for sharing over NFS.
186
187Shared Hash Equivalence Server
188------------------------------
189
190The Workers all use the same Hash Equivalence server, through a common
191definition for :term:`BB_HASHSERVE`.
185 192
186Resulttool 193Resulttool
187---------- 194----------
@@ -192,7 +199,7 @@ in a given build and their status. Additional information, such as
192failure logs or the time taken to run the tests, may also be included. 199failure logs or the time taken to run the tests, may also be included.
193 200
194Resulttool is part of OpenEmbedded-Core and is used to manipulate these 201Resulttool is part of OpenEmbedded-Core and is used to manipulate these
195json results files. It has the ability to merge files together, display 202JSON results files. It has the ability to merge files together, display
196reports of the test results and compare different result files. 203reports of the test results and compare different result files.
197 204
198For details, see :yocto_wiki:`/Resulttool`. 205For details, see :yocto_wiki:`/Resulttool`.
@@ -204,9 +211,9 @@ The ``scripts/run-config`` execution is where most of the work within
204the Autobuilder happens. It runs through a number of steps; the first 211the Autobuilder happens. It runs through a number of steps; the first
205are general setup steps that are run once and include: 212are general setup steps that are run once and include:
206 213
207#. Set up any ``buildtools-tarball`` if configured. 214#. Set up any :term:`buildtools` tarball if configured.
208 215
209#. Call "buildhistory-init" if buildhistory is configured. 216#. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured.
210 217
211For each step that is configured in ``config.json``, it will perform the 218For each step that is configured in ``config.json``, it will perform the
212following: 219following:
@@ -242,7 +249,7 @@ of post-build steps, including:
242#. Call ``scripts/upload-error-reports`` to send any error reports 249#. Call ``scripts/upload-error-reports`` to send any error reports
243 generated to the remote server. 250 generated to the remote server.
244 251
245#. Cleanup the build directory using 252#. Cleanup the :term:`Build Directory` using
246 :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful, 253 :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful,
247 else rename it to "build-renamed" for potential future debugging. 254 else rename it to "build-renamed" for potential future debugging.
248 255
@@ -250,15 +257,16 @@ Deploying Yocto Autobuilder
250=========================== 257===========================
251 258
252The most up to date information about how to setup and deploy your own 259The most up to date information about how to setup and deploy your own
253Autbuilder can be found in README.md in the ``yocto-autobuilder2`` 260Autobuilder can be found in :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>`
254repository. 261in the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>` repository.
255 262
256We hope that people can use the ``yocto-autobuilder2`` code directly but 263We hope that people can use the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>`
257it is inevitable that users will end up needing to heavily customise the 264code directly but it is inevitable that users will end up needing to heavily
258``yocto-autobuilder-helper`` repository, particularly the 265customize the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>`
259``config.json`` file as they will want to define their own test matrix. 266repository, particularly the ``config.json`` file as they will want to define
267their own test matrix.
260 268
261The Autobuilder supports wo customization options: 269The Autobuilder supports two customization options:
262 270
263- variable substitution 271- variable substitution
264 272
@@ -278,7 +286,7 @@ environment::
278 $ ABHELPER_JSON="config.json /some/location/local.json" 286 $ ABHELPER_JSON="config.json /some/location/local.json"
279 287
280One issue users often run into is validation of the ``config.json`` files. A 288One issue users often run into is validation of the ``config.json`` files. A
281tip for minimizing issues from invalid json files is to use a Git 289tip for minimizing issues from invalid JSON files is to use a Git
282``pre-commit-hook.sh`` script to verify the JSON file before committing 290``pre-commit-hook.sh`` script to verify the JSON file before committing
283it. Create a symbolic link as follows:: 291it. Create a symbolic link as follows::
284 292