diff options
Diffstat (limited to 'documentation/test-manual/test-manual-understand-autobuilder.rst')
-rw-r--r-- | documentation/test-manual/test-manual-understand-autobuilder.rst | 305 |
1 files changed, 0 insertions, 305 deletions
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.rst b/documentation/test-manual/test-manual-understand-autobuilder.rst deleted file mode 100644 index 2fcae5000e..0000000000 --- a/documentation/test-manual/test-manual-understand-autobuilder.rst +++ /dev/null | |||
@@ -1,305 +0,0 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ******************************************* | ||
4 | Understanding the Yocto Project Autobuilder | ||
5 | ******************************************* | ||
6 | |||
7 | Execution Flow within the Autobuilder | ||
8 | ===================================== | ||
9 | |||
10 | The "a-full" and "a-quick" targets are the usual entry points into the | ||
11 | Autobuilder and it makes sense to follow the process through the system | ||
12 | starting there. This is best visualised from the Autobuilder Console | ||
13 | view (:yocto_ab:`/typhoon/#/console`). | ||
14 | |||
15 | Each item along the top of that view represents some "target build" and | ||
16 | these targets are all run in parallel. The 'full' build will trigger the | ||
17 | majority of them, the "quick" build will trigger some subset of them. | ||
18 | The Autobuilder effectively runs whichever configuration is defined for | ||
19 | each of those targets on a seperate buildbot worker. To understand the | ||
20 | configuration, you need to look at the entry on ``config.json`` file | ||
21 | within the ``yocto-autobuilder-helper`` repository. The targets are | ||
22 | defined in the ‘overrides' section, a quick example could be qemux86-64 | ||
23 | which looks like:: | ||
24 | |||
25 | "qemux86-64" : { | ||
26 | "MACHINE" : "qemux86-64", | ||
27 | "TEMPLATE" : "arch-qemu", | ||
28 | "step1" : { | ||
29 | "extravars" : [ | ||
30 | "IMAGE_FSTYPES_append = ' wic wic.bmap'" | ||
31 | ] | ||
32 | } | ||
33 | }, | ||
34 | |||
35 | And to expand that, you need the "arch-qemu" entry from | ||
36 | the "templates" section, which looks like:: | ||
37 | |||
38 | "arch-qemu" : { | ||
39 | "BUILDINFO" : true, | ||
40 | "BUILDHISTORY" : true, | ||
41 | "step1" : { | ||
42 | "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", | ||
43 | "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" | ||
44 | }, | ||
45 | "step2" : { | ||
46 | "SDKMACHINE" : "x86_64", | ||
47 | "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", | ||
48 | "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" | ||
49 | }, | ||
50 | "step3" : { | ||
51 | "BUILDHISTORY" : false, | ||
52 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], | ||
53 | "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] | ||
54 | } | ||
55 | }, | ||
56 | |||
57 | Combining these two entries you can see that "qemux86-64" is a three step build where the | ||
58 | ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for | ||
59 | ``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step | ||
60 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | ||
61 | image generation. | ||
62 | |||
63 | While not every detail of this is covered here, you can see how the | ||
64 | template mechanism allows quite complex configurations to be built up | ||
65 | yet allows duplication and repetition to be kept to a minimum. | ||
66 | |||
67 | The different build targets are designed to allow for parallelisation, | ||
68 | so different machines are usually built in parallel, operations using | ||
69 | the same machine and metadata are built sequentially, with the aim of | ||
70 | trying to optimise build efficiency as much as possible. | ||
71 | |||
72 | The ``config.json`` file is processed by the scripts in the Helper | ||
73 | repository in the ``scripts`` directory. The following section details | ||
74 | how this works. | ||
75 | |||
76 | .. _test-autobuilder-target-exec-overview: | ||
77 | |||
78 | Autobuilder Target Execution Overview | ||
79 | ===================================== | ||
80 | |||
81 | For each given target in a build, the Autobuilder executes several | ||
82 | steps. These are configured in ``yocto-autobuilder2/builders.py`` and | ||
83 | roughly consist of: | ||
84 | |||
85 | #. *Run clobberdir*. | ||
86 | |||
87 | This cleans out any previous build. Old builds are left around to | ||
88 | allow easier debugging of failed builds. For additional information, | ||
89 | see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`. | ||
90 | |||
91 | #. *Obtain yocto-autobuilder-helper* | ||
92 | |||
93 | This step clones the ``yocto-autobuilder-helper`` git repository. | ||
94 | This is necessary to prevent the requirement to maintain all the | ||
95 | release or project-specific code within Buildbot. The branch chosen | ||
96 | matches the release being built so we can support older releases and | ||
97 | still make changes in newer ones. | ||
98 | |||
99 | #. *Write layerinfo.json* | ||
100 | |||
101 | This transfers data in the Buildbot UI when the build was configured | ||
102 | to the Helper. | ||
103 | |||
104 | #. *Call scripts/shared-repo-unpack* | ||
105 | |||
106 | This is a call into the Helper scripts to set up a checkout of all | ||
107 | the pieces this build might need. It might clone the BitBake | ||
108 | repository and the OpenEmbedded-Core repository. It may clone the | ||
109 | Poky repository, as well as additional layers. It will use the data | ||
110 | from the ``layerinfo.json`` file to help understand the | ||
111 | configuration. It will also use a local cache of repositories to | ||
112 | speed up the clone checkouts. For additional information, see | ||
113 | :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`. | ||
114 | |||
115 | This step has two possible modes of operation. If the build is part | ||
116 | of a parent build, its possible that all the repositories needed may | ||
117 | already be available, ready in a pre-prepared directory. An "a-quick" | ||
118 | or "a-full" build would prepare this before starting the other | ||
119 | sub-target builds. This is done for two reasons: | ||
120 | |||
121 | - the upstream may change during a build, for example, from a forced | ||
122 | push and this ensures we have matching content for the whole build | ||
123 | |||
124 | - if 15 Workers all tried to pull the same data from the same repos, | ||
125 | we can hit resource limits on upstream servers as they can think | ||
126 | they are under some kind of network attack | ||
127 | |||
128 | This pre-prepared directory is shared among the Workers over NFS. If | ||
129 | the build is an individual build and there is no "shared" directory | ||
130 | available, it would clone from the cache and the upstreams as | ||
131 | necessary. This is considered the fallback mode. | ||
132 | |||
133 | #. *Call scripts/run-config* | ||
134 | |||
135 | This is another call into the Helper scripts where its expected that | ||
136 | the main functionality of this target will be executed. | ||
137 | |||
138 | .. _test-autobuilder-tech: | ||
139 | |||
140 | Autobuilder Technology | ||
141 | ====================== | ||
142 | |||
143 | The Autobuilder has Yocto Project-specific functionality to allow builds | ||
144 | to operate with increased efficiency and speed. | ||
145 | |||
146 | .. _test-clobberdir: | ||
147 | |||
148 | clobberdir | ||
149 | ---------- | ||
150 | |||
151 | When deleting files, the Autobuilder uses ``clobberdir``, which is a | ||
152 | special script that moves files to a special location, rather than | ||
153 | deleting them. Files in this location are deleted by an ``rm`` command, | ||
154 | which is run under ``ionice -c 3``. For example, the deletion only | ||
155 | happens when there is idle IO capacity on the Worker. The Autobuilder | ||
156 | Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. | ||
157 | |||
158 | .. _test-autobuilder-clone-cache: | ||
159 | |||
160 | Autobuilder Clone Cache | ||
161 | ----------------------- | ||
162 | |||
163 | Cloning repositories from scratch each time they are required was slow | ||
164 | on the Autobuilder. We therefore have a stash of commonly used | ||
165 | repositories pre-cloned on the Workers. Data is fetched from these | ||
166 | during clones first, then "topped up" with later revisions from any | ||
167 | upstream when necesary. The cache is maintained by the Autobuilder | ||
168 | Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. | ||
169 | |||
170 | .. _test-autobuilder-worker-janitor: | ||
171 | |||
172 | Autobuilder Worker Janitor | ||
173 | -------------------------- | ||
174 | |||
175 | This is a process running on each Worker that performs two basic | ||
176 | operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and | ||
177 | maintainenance of a cache of cloned repositories to improve the speed | ||
178 | the system can checkout repositories. | ||
179 | |||
180 | .. _test-shared-dl-dir: | ||
181 | |||
182 | Shared DL_DIR | ||
183 | ------------- | ||
184 | |||
185 | The Workers are all connected over NFS which allows DL_DIR to be shared | ||
186 | between them. This reduces network accesses from the system and allows | ||
187 | the build to be sped up. Usage of the directory within the build system | ||
188 | is designed to be able to be shared over NFS. | ||
189 | |||
190 | .. _test-shared-sstate-cache: | ||
191 | |||
192 | Shared SSTATE_DIR | ||
193 | ----------------- | ||
194 | |||
195 | The Workers are all connected over NFS which allows the ``sstate`` | ||
196 | directory to be shared between them. This means once a Worker has built | ||
197 | an artifact, all the others can benefit from it. Usage of the directory | ||
198 | within the directory is designed for sharing over NFS. | ||
199 | |||
200 | .. _test-resulttool: | ||
201 | |||
202 | Resulttool | ||
203 | ---------- | ||
204 | |||
205 | All of the different tests run as part of the build generate output into | ||
206 | ``testresults.json`` files. This allows us to determine which tests ran | ||
207 | in a given build and their status. Additional information, such as | ||
208 | failure logs or the time taken to run the tests, may also be included. | ||
209 | |||
210 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these | ||
211 | json results files. It has the ability to merge files together, display | ||
212 | reports of the test results and compare different result files. | ||
213 | |||
214 | For details, see :yocto_wiki:`/wiki/Resulttool`. | ||
215 | |||
216 | .. _test-run-config-tgt-execution: | ||
217 | |||
218 | run-config Target Execution | ||
219 | =========================== | ||
220 | |||
221 | The ``scripts/run-config`` execution is where most of the work within | ||
222 | the Autobuilder happens. It runs through a number of steps; the first | ||
223 | are general setup steps that are run once and include: | ||
224 | |||
225 | #. Set up any ``buildtools-tarball`` if configured. | ||
226 | |||
227 | #. Call "buildhistory-init" if buildhistory is configured. | ||
228 | |||
229 | For each step that is configured in ``config.json``, it will perform the | ||
230 | following: | ||
231 | |||
232 | #. Add any layers that are specified using the | ||
233 | ``bitbake-layers add-layer`` command (logging as stepXa) | ||
234 | |||
235 | #. Call the ``scripts/setup-config`` script to generate the necessary | ||
236 | ``auto.conf`` configuration file for the build | ||
237 | |||
238 | #. Run the ``bitbake BBTARGETS`` command (logging as stepXb) | ||
239 | |||
240 | #. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) | ||
241 | |||
242 | #. Run the ``EXTRACMDS`` command, which are run within the BitBake build | ||
243 | environment (logging as stepXd) | ||
244 | |||
245 | #. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the | ||
246 | BitBake build environment (logging as stepXd) | ||
247 | |||
248 | #. Remove any layers added in step | ||
249 | 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa) | ||
250 | |||
251 | Once the execution steps above complete, ``run-config`` executes a set | ||
252 | of post-build steps, including: | ||
253 | |||
254 | #. Call ``scripts/publish-artifacts`` to collect any output which is to | ||
255 | be saved from the build. | ||
256 | |||
257 | #. Call ``scripts/collect-results`` to collect any test results to be | ||
258 | saved from the build. | ||
259 | |||
260 | #. Call ``scripts/upload-error-reports`` to send any error reports | ||
261 | generated to the remote server. | ||
262 | |||
263 | #. Cleanup the build directory using | ||
264 | :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful, | ||
265 | else rename it to "build-renamed" for potential future debugging. | ||
266 | |||
267 | .. _test-deploying-yp-autobuilder: | ||
268 | |||
269 | Deploying Yocto Autobuilder | ||
270 | =========================== | ||
271 | |||
272 | The most up to date information about how to setup and deploy your own | ||
273 | Autbuilder can be found in README.md in the ``yocto-autobuilder2`` | ||
274 | repository. | ||
275 | |||
276 | We hope that people can use the ``yocto-autobuilder2`` code directly but | ||
277 | it is inevitable that users will end up needing to heavily customise the | ||
278 | ``yocto-autobuilder-helper`` repository, particularly the | ||
279 | ``config.json`` file as they will want to define their own test matrix. | ||
280 | |||
281 | The Autobuilder supports wo customization options: | ||
282 | |||
283 | - variable substitution | ||
284 | |||
285 | - overlaying configuration files | ||
286 | |||
287 | The standard ``config.json`` minimally attempts to allow substitution of | ||
288 | the paths. The Helper script repository includes a | ||
289 | ``local-example.json`` file to show how you could override these from a | ||
290 | separate configuration file. Pass the following into the environment of | ||
291 | the Autobuilder:: | ||
292 | |||
293 | $ ABHELPER_JSON="config.json local-example.json" | ||
294 | |||
295 | As another example, you could also pass the following into the | ||
296 | environment:: | ||
297 | |||
298 | $ ABHELPER_JSON="config.json /some/location/local.json" | ||
299 | |||
300 | One issue users often run into is validation of the ``config.json`` files. A | ||
301 | tip for minimizing issues from invalid json files is to use a Git | ||
302 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | ||
303 | it. Create a symbolic link as follows:: | ||
304 | |||
305 | $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | ||