summaryrefslogtreecommitdiffstats
path: root/documentation/test-manual/test-manual-understand-autobuilder.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/test-manual/test-manual-understand-autobuilder.rst')
-rw-r--r--documentation/test-manual/test-manual-understand-autobuilder.rst140
1 files changed, 78 insertions, 62 deletions
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst
index 8d3c844156..0809190b5b 100644
--- a/documentation/test-manual/test-manual-understand-autobuilder.rst
+++ b/documentation/test-manual/test-manual-understand-autobuilder.rst
@@ -10,7 +10,7 @@ Execution Flow within the Autobuilder
10The “a-full” and “a-quick” targets are the usual entry points into the 10The “a-full” and “a-quick” targets are the usual entry points into the
11Autobuilder and it makes sense to follow the process through the system 11Autobuilder and it makes sense to follow the process through the system
12starting there. This is best visualised from the Autobuilder Console 12starting there. This is best visualised from the Autobuilder Console
13view (`https://autobuilder.yoctoproject.org/typhoon/#/console <#>`__). 13view (:yocto_ab:`/typhoon/#/console`).
14 14
15Each item along the top of that view represents some “target build” and 15Each item along the top of that view represents some “target build” and
16these targets are all run in parallel. The ‘full’ build will trigger the 16these targets are all run in parallel. The ‘full’ build will trigger the
@@ -20,32 +20,48 @@ each of those targets on a seperate buildbot worker. To understand the
20configuration, you need to look at the entry on ``config.json`` file 20configuration, you need to look at the entry on ``config.json`` file
21within the ``yocto-autobuilder-helper`` repository. The targets are 21within the ``yocto-autobuilder-helper`` repository. The targets are
22defined in the ‘overrides’ section, a quick example could be qemux86-64 22defined in the ‘overrides’ section, a quick example could be qemux86-64
23which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" : 23which looks like::
24"arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic 24
25wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from 25 "qemux86-64" : {
26the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" : 26 "MACHINE" : "qemux86-64",
27true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato 27 "TEMPLATE" : "arch-qemu",
28core-image-sato-dev core-image-sato-sdk core-image-minimal 28 "step1" : {
29core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS" 29 "extravars" : [
30: "core-image-minimal:do_testimage core-image-sato:do_testimage 30 "IMAGE_FSTYPES_append = ' wic wic.bmap'"
31core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2" 31 ]
32: { "SDKMACHINE" : "x86_64", "BBTARGETS" : 32 }
33"core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext 33 },
34core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" : 34
35"core-image-sato:do_testsdk core-image-minimal:do_testsdkext 35And to expand that, you need the “arch-qemu” entry from
36core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false, 36the “templates” section, which looks like::
37"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest 37
38${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" : 38 "arch-qemu" : {
39["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can 39 "BUILDINFO" : true,
40see that “qemux86-64” is a three step build where the 40 "BUILDHISTORY" : true,
41``bitbake BBTARGETS`` would be run, then ``bitbake 41 "step1" : {
42 SANITYTARGETS`` for each step; all for 42 "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
43 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
44 },
45 "step2" : {
46 "SDKMACHINE" : "x86_64",
47 "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
48 "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
49 },
50 "step3" : {
51 "BUILDHISTORY" : false,
52 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
53 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
54 }
55 },
56
57Combining these two entries you can see that “qemux86-64” is a three step build where the
58``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
43``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step 59``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step
441 an extra variable is added to the ``auto.conf`` file to enable wic 601 an extra variable is added to the ``auto.conf`` file to enable wic
45image generation. 61image generation.
46 62
47While not every detail of this is covered here, you can see how the 63While not every detail of this is covered here, you can see how the
48templating mechanism allows quite complex configurations to be built up 64template mechanism allows quite complex configurations to be built up
49yet allows duplication and repetition to be kept to a minimum. 65yet allows duplication and repetition to be kept to a minimum.
50 66
51The different build targets are designed to allow for parallelisation, 67The different build targets are designed to allow for parallelisation,
@@ -66,13 +82,13 @@ For each given target in a build, the Autobuilder executes several
66steps. These are configured in ``yocto-autobuilder2/builders.py`` and 82steps. These are configured in ``yocto-autobuilder2/builders.py`` and
67roughly consist of: 83roughly consist of:
68 84
691. *Run ``clobberdir``* 85#. *Run clobberdir*.
70 86
71 This cleans out any previous build. Old builds are left around to 87 This cleans out any previous build. Old builds are left around to
72 allow easier debugging of failed builds. For additional information, 88 allow easier debugging of failed builds. For additional information,
73 see ```clobberdir`` <#test-clobberdir>`__. 89 see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`.
74 90
752. *Obtain yocto-autobuilder-helper* 91#. *Obtain yocto-autobuilder-helper*
76 92
77 This step clones the ``yocto-autobuilder-helper`` git repository. 93 This step clones the ``yocto-autobuilder-helper`` git repository.
78 This is necessary to prevent the requirement to maintain all the 94 This is necessary to prevent the requirement to maintain all the
@@ -80,12 +96,12 @@ roughly consist of:
80 matches the release being built so we can support older releases and 96 matches the release being built so we can support older releases and
81 still make changes in newer ones. 97 still make changes in newer ones.
82 98
833. *Write layerinfo.json* 99#. *Write layerinfo.json*
84 100
85 This transfers data in the Buildbot UI when the build was configured 101 This transfers data in the Buildbot UI when the build was configured
86 to the Helper. 102 to the Helper.
87 103
884. *Call scripts/shared-repo-unpack* 104#. *Call scripts/shared-repo-unpack*
89 105
90 This is a call into the Helper scripts to set up a checkout of all 106 This is a call into the Helper scripts to set up a checkout of all
91 the pieces this build might need. It might clone the BitBake 107 the pieces this build might need. It might clone the BitBake
@@ -94,7 +110,7 @@ roughly consist of:
94 from the ``layerinfo.json`` file to help understand the 110 from the ``layerinfo.json`` file to help understand the
95 configuration. It will also use a local cache of repositories to 111 configuration. It will also use a local cache of repositories to
96 speed up the clone checkouts. For additional information, see 112 speed up the clone checkouts. For additional information, see
97 `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__. 113 :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`.
98 114
99 This step has two possible modes of operation. If the build is part 115 This step has two possible modes of operation. If the build is part
100 of a parent build, its possible that all the repositories needed may 116 of a parent build, its possible that all the repositories needed may
@@ -114,7 +130,7 @@ roughly consist of:
114 available, it would clone from the cache and the upstreams as 130 available, it would clone from the cache and the upstreams as
115 necessary. This is considered the fallback mode. 131 necessary. This is considered the fallback mode.
116 132
1175. *Call scripts/run-config* 133#. *Call scripts/run-config*
118 134
119 This is another call into the Helper scripts where its expected that 135 This is another call into the Helper scripts where its expected that
120 the main functionality of this target will be executed. 136 the main functionality of this target will be executed.
@@ -137,8 +153,7 @@ special script that moves files to a special location, rather than
137deleting them. Files in this location are deleted by an ``rm`` command, 153deleting them. Files in this location are deleted by an ``rm`` command,
138which is run under ``ionice -c 3``. For example, the deletion only 154which is run under ``ionice -c 3``. For example, the deletion only
139happens when there is idle IO capacity on the Worker. The Autobuilder 155happens when there is idle IO capacity on the Worker. The Autobuilder
140Worker Janitor runs this deletion. See `Autobuilder Worker 156Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
141Janitor <#test-autobuilder-worker-janitor>`__.
142 157
143.. _test-autobuilder-clone-cache: 158.. _test-autobuilder-clone-cache:
144 159
@@ -150,8 +165,7 @@ on the Autobuilder. We therefore have a stash of commonly used
150repositories pre-cloned on the Workers. Data is fetched from these 165repositories pre-cloned on the Workers. Data is fetched from these
151during clones first, then "topped up" with later revisions from any 166during clones first, then "topped up" with later revisions from any
152upstream when necesary. The cache is maintained by the Autobuilder 167upstream when necesary. The cache is maintained by the Autobuilder
153Worker Janitor. See `Autobuilder Worker 168Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
154Janitor <#test-autobuilder-worker-janitor>`__.
155 169
156.. _test-autobuilder-worker-janitor: 170.. _test-autobuilder-worker-janitor:
157 171
@@ -159,8 +173,7 @@ Autobuilder Worker Janitor
159-------------------------- 173--------------------------
160 174
161This is a process running on each Worker that performs two basic 175This is a process running on each Worker that performs two basic
162operations, including background file deletion at IO idle (see `Target 176operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and
163Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and
164maintainenance of a cache of cloned repositories to improve the speed 177maintainenance of a cache of cloned repositories to improve the speed
165the system can checkout repositories. 178the system can checkout repositories.
166 179
@@ -181,7 +194,7 @@ Shared SSTATE_DIR
181 194
182The Workers are all connected over NFS which allows the ``sstate`` 195The Workers are all connected over NFS which allows the ``sstate``
183directory to be shared between them. This means once a Worker has built 196directory to be shared between them. This means once a Worker has built
184an artefact, all the others can benefit from it. Usage of the directory 197an artifact, all the others can benefit from it. Usage of the directory
185within the directory is designed for sharing over NFS. 198within the directory is designed for sharing over NFS.
186 199
187.. _test-resulttool: 200.. _test-resulttool:
@@ -198,7 +211,7 @@ Resulttool is part of OpenEmbedded-Core and is used to manipulate these
198json results files. It has the ability to merge files together, display 211json results files. It has the ability to merge files together, display
199reports of the test results and compare different result files. 212reports of the test results and compare different result files.
200 213
201For details, see `https://wiki.yoctoproject.org/wiki/Resulttool <#>`__. 214For details, see :yocto_wiki:`/wiki/Resulttool`.
202 215
203.. _test-run-config-tgt-execution: 216.. _test-run-config-tgt-execution:
204 217
@@ -209,50 +222,46 @@ The ``scripts/run-config`` execution is where most of the work within
209the Autobuilder happens. It runs through a number of steps; the first 222the Autobuilder happens. It runs through a number of steps; the first
210are general setup steps that are run once and include: 223are general setup steps that are run once and include:
211 224
2121. Set up any ``buildtools-tarball`` if configured. 225#. Set up any ``buildtools-tarball`` if configured.
213 226
2142. Call "buildhistory-init" if buildhistory is configured. 227#. Call "buildhistory-init" if buildhistory is configured.
215 228
216For each step that is configured in ``config.json``, it will perform the 229For each step that is configured in ``config.json``, it will perform the
217following: 230following:
218 231
219## WRITER's question: What does "logging in as stepXa" and others refer 232#. Add any layers that are specified using the
220to below? ##
221
2221. Add any layers that are specified using the
223 ``bitbake-layers add-layer`` command (logging as stepXa) 233 ``bitbake-layers add-layer`` command (logging as stepXa)
224 234
2252. Call the ``scripts/setup-config`` script to generate the necessary 235#. Call the ``scripts/setup-config`` script to generate the necessary
226 ``auto.conf`` configuration file for the build 236 ``auto.conf`` configuration file for the build
227 237
2283. Run the ``bitbake BBTARGETS`` command (logging as stepXb) 238#. Run the ``bitbake BBTARGETS`` command (logging as stepXb)
229 239
2304. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) 240#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc)
231 241
2325. Run the ``EXTRACMDS`` command, which are run within the BitBake build 242#. Run the ``EXTRACMDS`` command, which are run within the BitBake build
233 environment (logging as stepXd) 243 environment (logging as stepXd)
234 244
2356. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the 245#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
236 BitBake build environment (logging as stepXd) 246 BitBake build environment (logging as stepXd)
237 247
2387. Remove any layers added in `step 248#. Remove any layers added in step
239 1 <#test-run-config-add-layers-step>`__ using the 249 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
240 ``bitbake-layers remove-layer`` command (logging as stepXa)
241 250
242Once the execution steps above complete, ``run-config`` executes a set 251Once the execution steps above complete, ``run-config`` executes a set
243of post-build steps, including: 252of post-build steps, including:
244 253
2451. Call ``scripts/publish-artifacts`` to collect any output which is to 254#. Call ``scripts/publish-artifacts`` to collect any output which is to
246 be saved from the build. 255 be saved from the build.
247 256
2482. Call ``scripts/collect-results`` to collect any test results to be 257#. Call ``scripts/collect-results`` to collect any test results to be
249 saved from the build. 258 saved from the build.
250 259
2513. Call ``scripts/upload-error-reports`` to send any error reports 260#. Call ``scripts/upload-error-reports`` to send any error reports
252 generated to the remote server. 261 generated to the remote server.
253 262
2544. Cleanup the build directory using 263#. Cleanup the build directory using
255 ```clobberdir`` <#test-clobberdir>`__ if the build was successful, 264 :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful,
256 else rename it to “build-renamed” for potential future debugging. 265 else rename it to “build-renamed” for potential future debugging.
257 266
258.. _test-deploying-yp-autobuilder: 267.. _test-deploying-yp-autobuilder:
@@ -279,11 +288,18 @@ The standard ``config.json`` minimally attempts to allow substitution of
279the paths. The Helper script repository includes a 288the paths. The Helper script repository includes a
280``local-example.json`` file to show how you could override these from a 289``local-example.json`` file to show how you could override these from a
281separate configuration file. Pass the following into the environment of 290separate configuration file. Pass the following into the environment of
282the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As 291the Autobuilder::
283another example, you could also pass the following into the 292
284environment:$ ABHELPER_JSON="config.json /some/location/local.json"One 293 $ ABHELPER_JSON="config.json local-example.json"
285issue users often run into is validation of the ``config.json`` files. A 294
295As another example, you could also pass the following into the
296environment::
297
298 $ ABHELPER_JSON="config.json /some/location/local.json"
299
300One issue users often run into is validation of the ``config.json`` files. A
286tip for minimizing issues from invalid json files is to use a Git 301tip for minimizing issues from invalid json files is to use a Git
287``pre-commit-hook.sh`` script to verify the JSON file before committing 302``pre-commit-hook.sh`` script to verify the JSON file before committing
288it. Create a symbolic link as follows:$ ln -s 303it. Create a symbolic link as follows::
289../../scripts/pre-commit-hook.sh .git/hooks/pre-commit 304
305 $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit