diff options
Diffstat (limited to 'documentation/test-manual/test-manual-understand-autobuilder.rst')
-rw-r--r-- | documentation/test-manual/test-manual-understand-autobuilder.rst | 287 |
1 files changed, 287 insertions, 0 deletions
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst new file mode 100644 index 0000000000..69700088aa --- /dev/null +++ b/documentation/test-manual/test-manual-understand-autobuilder.rst | |||
@@ -0,0 +1,287 @@ | |||
1 | ******************************************* | ||
2 | Understanding the Yocto Project Autobuilder | ||
3 | ******************************************* | ||
4 | |||
5 | Execution Flow within the Autobuilder | ||
6 | ===================================== | ||
7 | |||
8 | The “a-full” and “a-quick” targets are the usual entry points into the | ||
9 | Autobuilder and it makes sense to follow the process through the system | ||
10 | starting there. This is best visualised from the Autobuilder Console | ||
11 | view (`https://autobuilder.yoctoproject.org/typhoon/#/console <#>`__). | ||
12 | |||
13 | Each item along the top of that view represents some “target build” and | ||
14 | these targets are all run in parallel. The ‘full’ build will trigger the | ||
15 | majority of them, the “quick” build will trigger some subset of them. | ||
16 | The Autobuilder effectively runs whichever configuration is defined for | ||
17 | each of those targets on a seperate buildbot worker. To understand the | ||
18 | configuration, you need to look at the entry on ``config.json`` file | ||
19 | within the ``yocto-autobuilder-helper`` repository. The targets are | ||
20 | defined in the ‘overrides’ section, a quick example could be qemux86-64 | ||
21 | which looks like:"qemux86-64" : { "MACHINE" : "qemux86-64", "TEMPLATE" : | ||
22 | "arch-qemu", "step1" : { "extravars" : [ "IMAGE_FSTYPES_append = ' wic | ||
23 | wic.bmap'" ] } },And to expand that, you need the “arch-qemu” entry from | ||
24 | the “templates” section, which looks like:"arch-qemu" : { "BUILDINFO" : | ||
25 | true, "BUILDHISTORY" : true, "step1" : { "BBTARGETS" : "core-image-sato | ||
26 | core-image-sato-dev core-image-sato-sdk core-image-minimal | ||
27 | core-image-minimal-dev core-image-sato:do_populate_sdk", "SANITYTARGETS" | ||
28 | : "core-image-minimal:do_testimage core-image-sato:do_testimage | ||
29 | core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" }, "step2" | ||
30 | : { "SDKMACHINE" : "x86_64", "BBTARGETS" : | ||
31 | "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext | ||
32 | core-image-sato:do_populate_sdk_ext", "SANITYTARGETS" : | ||
33 | "core-image-sato:do_testsdk core-image-minimal:do_testsdkext | ||
34 | core-image-sato:do_testsdkext" }, "step3" : { "BUILDHISTORY" : false, | ||
35 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest | ||
36 | ${HELPERSTMACHTARGS} -j 15"], "ADDLAYER" : | ||
37 | ["${BUILDDIR}/../meta-selftest"] } },Combining these two entries you can | ||
38 | see that “qemux86-64” is a three step build where the | ||
39 | ``bitbake BBTARGETS`` would be run, then ``bitbake | ||
40 | SANITYTARGETS`` for each step; all for | ||
41 | ``MACHINE=”qemx86-64”`` but with differing SDKMACHINE settings. In step | ||
42 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | ||
43 | image generation. | ||
44 | |||
45 | While not every detail of this is covered here, you can see how the | ||
46 | templating mechanism allows quite complex configurations to be built up | ||
47 | yet allows duplication and repetition to be kept to a minimum. | ||
48 | |||
49 | The different build targets are designed to allow for parallelisation, | ||
50 | so different machines are usually built in parallel, operations using | ||
51 | the same machine and metadata are built sequentially, with the aim of | ||
52 | trying to optimise build efficiency as much as possible. | ||
53 | |||
54 | The ``config.json`` file is processed by the scripts in the Helper | ||
55 | repository in the ``scripts`` directory. The following section details | ||
56 | how this works. | ||
57 | |||
58 | .. _test-autobuilder-target-exec-overview: | ||
59 | |||
60 | Autobuilder Target Execution Overview | ||
61 | ===================================== | ||
62 | |||
63 | For each given target in a build, the Autobuilder executes several | ||
64 | steps. These are configured in ``yocto-autobuilder2/builders.py`` and | ||
65 | roughly consist of: | ||
66 | |||
67 | 1. *Run ``clobberdir``* | ||
68 | |||
69 | This cleans out any previous build. Old builds are left around to | ||
70 | allow easier debugging of failed builds. For additional information, | ||
71 | see ```clobberdir`` <#test-clobberdir>`__. | ||
72 | |||
73 | 2. *Obtain yocto-autobuilder-helper* | ||
74 | |||
75 | This step clones the ``yocto-autobuilder-helper`` git repository. | ||
76 | This is necessary to prevent the requirement to maintain all the | ||
77 | release or project-specific code within Buildbot. The branch chosen | ||
78 | matches the release being built so we can support older releases and | ||
79 | still make changes in newer ones. | ||
80 | |||
81 | 3. *Write layerinfo.json* | ||
82 | |||
83 | This transfers data in the Buildbot UI when the build was configured | ||
84 | to the Helper. | ||
85 | |||
86 | 4. *Call scripts/shared-repo-unpack* | ||
87 | |||
88 | This is a call into the Helper scripts to set up a checkout of all | ||
89 | the pieces this build might need. It might clone the BitBake | ||
90 | repository and the OpenEmbedded-Core repository. It may clone the | ||
91 | Poky repository, as well as additional layers. It will use the data | ||
92 | from the ``layerinfo.json`` file to help understand the | ||
93 | configuration. It will also use a local cache of repositories to | ||
94 | speed up the clone checkouts. For additional information, see | ||
95 | `Autobuilder Clone Cache <#test-autobuilder-clone-cache>`__. | ||
96 | |||
97 | This step has two possible modes of operation. If the build is part | ||
98 | of a parent build, its possible that all the repositories needed may | ||
99 | already be available, ready in a pre-prepared directory. An "a-quick" | ||
100 | or "a-full" build would prepare this before starting the other | ||
101 | sub-target builds. This is done for two reasons: | ||
102 | |||
103 | - the upstream may change during a build, for example, from a forced | ||
104 | push and this ensures we have matching content for the whole build | ||
105 | |||
106 | - if 15 Workers all tried to pull the same data from the same repos, | ||
107 | we can hit resource limits on upstream servers as they can think | ||
108 | they are under some kind of network attack | ||
109 | |||
110 | This pre-prepared directory is shared among the Workers over NFS. If | ||
111 | the build is an individual build and there is no "shared" directory | ||
112 | available, it would clone from the cache and the upstreams as | ||
113 | necessary. This is considered the fallback mode. | ||
114 | |||
115 | 5. *Call scripts/run-config* | ||
116 | |||
117 | This is another call into the Helper scripts where its expected that | ||
118 | the main functionality of this target will be executed. | ||
119 | |||
120 | .. _test-autobuilder-tech: | ||
121 | |||
122 | Autobuilder Technology | ||
123 | ====================== | ||
124 | |||
125 | The Autobuilder has Yocto Project-specific functionality to allow builds | ||
126 | to operate with increased efficiency and speed. | ||
127 | |||
128 | .. _test-clobberdir: | ||
129 | |||
130 | clobberdir | ||
131 | ---------- | ||
132 | |||
133 | When deleting files, the Autobuilder uses ``clobberdir``, which is a | ||
134 | special script that moves files to a special location, rather than | ||
135 | deleting them. Files in this location are deleted by an ``rm`` command, | ||
136 | which is run under ``ionice -c 3``. For example, the deletion only | ||
137 | happens when there is idle IO capacity on the Worker. The Autobuilder | ||
138 | Worker Janitor runs this deletion. See `Autobuilder Worker | ||
139 | Janitor <#test-autobuilder-worker-janitor>`__. | ||
140 | |||
141 | .. _test-autobuilder-clone-cache: | ||
142 | |||
143 | Autobuilder Clone Cache | ||
144 | ----------------------- | ||
145 | |||
146 | Cloning repositories from scratch each time they are required was slow | ||
147 | on the Autobuilder. We therefore have a stash of commonly used | ||
148 | repositories pre-cloned on the Workers. Data is fetched from these | ||
149 | during clones first, then "topped up" with later revisions from any | ||
150 | upstream when necesary. The cache is maintained by the Autobuilder | ||
151 | Worker Janitor. See `Autobuilder Worker | ||
152 | Janitor <#test-autobuilder-worker-janitor>`__. | ||
153 | |||
154 | .. _test-autobuilder-worker-janitor: | ||
155 | |||
156 | Autobuilder Worker Janitor | ||
157 | -------------------------- | ||
158 | |||
159 | This is a process running on each Worker that performs two basic | ||
160 | operations, including background file deletion at IO idle (see `Target | ||
161 | Execution: clobberdir <#test-list-tgt-exec-clobberdir>`__) and | ||
162 | maintainenance of a cache of cloned repositories to improve the speed | ||
163 | the system can checkout repositories. | ||
164 | |||
165 | .. _test-shared-dl-dir: | ||
166 | |||
167 | Shared DL_DIR | ||
168 | ------------- | ||
169 | |||
170 | The Workers are all connected over NFS which allows DL_DIR to be shared | ||
171 | between them. This reduces network accesses from the system and allows | ||
172 | the build to be sped up. Usage of the directory within the build system | ||
173 | is designed to be able to be shared over NFS. | ||
174 | |||
175 | .. _test-shared-sstate-cache: | ||
176 | |||
177 | Shared SSTATE_DIR | ||
178 | ----------------- | ||
179 | |||
180 | The Workers are all connected over NFS which allows the ``sstate`` | ||
181 | directory to be shared between them. This means once a Worker has built | ||
182 | an artefact, all the others can benefit from it. Usage of the directory | ||
183 | within the directory is designed for sharing over NFS. | ||
184 | |||
185 | .. _test-resulttool: | ||
186 | |||
187 | Resulttool | ||
188 | ---------- | ||
189 | |||
190 | All of the different tests run as part of the build generate output into | ||
191 | ``testresults.json`` files. This allows us to determine which tests ran | ||
192 | in a given build and their status. Additional information, such as | ||
193 | failure logs or the time taken to run the tests, may also be included. | ||
194 | |||
195 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these | ||
196 | json results files. It has the ability to merge files together, display | ||
197 | reports of the test results and compare different result files. | ||
198 | |||
199 | For details, see `https://wiki.yoctoproject.org/wiki/Resulttool <#>`__. | ||
200 | |||
201 | .. _test-run-config-tgt-execution: | ||
202 | |||
203 | run-config Target Execution | ||
204 | =========================== | ||
205 | |||
206 | The ``scripts/run-config`` execution is where most of the work within | ||
207 | the Autobuilder happens. It runs through a number of steps; the first | ||
208 | are general setup steps that are run once and include: | ||
209 | |||
210 | 1. Set up any ``buildtools-tarball`` if configured. | ||
211 | |||
212 | 2. Call "buildhistory-init" if buildhistory is configured. | ||
213 | |||
214 | For each step that is configured in ``config.json``, it will perform the | ||
215 | following: | ||
216 | |||
217 | ## WRITER's question: What does "logging in as stepXa" and others refer | ||
218 | to below? ## | ||
219 | |||
220 | 1. Add any layers that are specified using the | ||
221 | ``bitbake-layers add-layer`` command (logging as stepXa) | ||
222 | |||
223 | 2. Call the ``scripts/setup-config`` script to generate the necessary | ||
224 | ``auto.conf`` configuration file for the build | ||
225 | |||
226 | 3. Run the ``bitbake BBTARGETS`` command (logging as stepXb) | ||
227 | |||
228 | 4. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) | ||
229 | |||
230 | 5. Run the ``EXTRACMDS`` command, which are run within the BitBake build | ||
231 | environment (logging as stepXd) | ||
232 | |||
233 | 6. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the | ||
234 | BitBake build environment (logging as stepXd) | ||
235 | |||
236 | 7. Remove any layers added in `step | ||
237 | 1 <#test-run-config-add-layers-step>`__ using the | ||
238 | ``bitbake-layers remove-layer`` command (logging as stepXa) | ||
239 | |||
240 | Once the execution steps above complete, ``run-config`` executes a set | ||
241 | of post-build steps, including: | ||
242 | |||
243 | 1. Call ``scripts/publish-artifacts`` to collect any output which is to | ||
244 | be saved from the build. | ||
245 | |||
246 | 2. Call ``scripts/collect-results`` to collect any test results to be | ||
247 | saved from the build. | ||
248 | |||
249 | 3. Call ``scripts/upload-error-reports`` to send any error reports | ||
250 | generated to the remote server. | ||
251 | |||
252 | 4. Cleanup the build directory using | ||
253 | ```clobberdir`` <#test-clobberdir>`__ if the build was successful, | ||
254 | else rename it to “build-renamed” for potential future debugging. | ||
255 | |||
256 | .. _test-deploying-yp-autobuilder: | ||
257 | |||
258 | Deploying Yocto Autobuilder | ||
259 | =========================== | ||
260 | |||
261 | The most up to date information about how to setup and deploy your own | ||
262 | Autbuilder can be found in README.md in the ``yocto-autobuilder2`` | ||
263 | repository. | ||
264 | |||
265 | We hope that people can use the ``yocto-autobuilder2`` code directly but | ||
266 | it is inevitable that users will end up needing to heavily customise the | ||
267 | ``yocto-autobuilder-helper`` repository, particularly the | ||
268 | ``config.json`` file as they will want to define their own test matrix. | ||
269 | |||
270 | The Autobuilder supports wo customization options: | ||
271 | |||
272 | - variable substitution | ||
273 | |||
274 | - overlaying configuration files | ||
275 | |||
276 | The standard ``config.json`` minimally attempts to allow substitution of | ||
277 | the paths. The Helper script repository includes a | ||
278 | ``local-example.json`` file to show how you could override these from a | ||
279 | separate configuration file. Pass the following into the environment of | ||
280 | the Autobuilder:$ ABHELPER_JSON="config.json local-example.json"As | ||
281 | another example, you could also pass the following into the | ||
282 | environment:$ ABHELPER_JSON="config.json /some/location/local.json"One | ||
283 | issue users often run into is validation of the ``config.json`` files. A | ||
284 | tip for minimizing issues from invalid json files is to use a Git | ||
285 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | ||
286 | it. Create a symbolic link as follows:$ ln -s | ||
287 | ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | ||