diff options
Diffstat (limited to 'documentation/test-manual/test-manual-understand-autobuilder.xml')
-rw-r--r-- | documentation/test-manual/test-manual-understand-autobuilder.xml | 312 |
1 files changed, 312 insertions, 0 deletions
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.xml b/documentation/test-manual/test-manual-understand-autobuilder.xml new file mode 100644 index 0000000000..7541305350 --- /dev/null +++ b/documentation/test-manual/test-manual-understand-autobuilder.xml | |||
@@ -0,0 +1,312 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | ||
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | ||
4 | |||
5 | <chapter id='test-manual-understand-autobuilder'> | ||
6 | |||
7 | <title>Understanding the Yocto Project Autobuilder</title> | ||
8 | <section> | ||
9 | <title>Execution Flow within the Autobuilder</title> | ||
10 | <para>The “a-full” and “a-quick” targets are the usual entry points into the Autobuilder and | ||
11 | it makes sense to follow the process through the system starting there. This is best | ||
12 | visualised from the Autobuilder Console view (<link linkend="" | ||
13 | >https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para> | ||
14 | <para>Each item along the top of that view represents some “target build” and these targets | ||
15 | are all run in parallel. The ‘full’ build will trigger the majority of them, the “quick” | ||
16 | build will trigger some subset of them. The Autobuilder effectively runs whichever | ||
17 | configuration is defined for each of those targets on a seperate buildbot worker. To | ||
18 | understand the configuration, you need to look at the entry on | ||
19 | <filename>config.json</filename> file within the | ||
20 | <filename>yocto-autobuilder-helper</filename> repository. The targets are defined in | ||
21 | the ‘overrides’ section, a quick example could be qemux86-64 which looks | ||
22 | like:<literallayout> | ||
23 | "qemux86-64" : { | ||
24 | "MACHINE" : "qemux86-64", | ||
25 | "TEMPLATE" : "arch-qemu", | ||
26 | "step1" : { | ||
27 | "extravars" : [ | ||
28 | "IMAGE_FSTYPES_append = ' wic wic.bmap'" | ||
29 | ] | ||
30 | } | ||
31 | }, | ||
32 | </literallayout>And | ||
33 | to expand that, you need the “arch-qemu” entry from the “templates” section, which looks | ||
34 | like:<literallayout> | ||
35 | "arch-qemu" : { | ||
36 | "BUILDINFO" : true, | ||
37 | "BUILDHISTORY" : true, | ||
38 | "step1" : { | ||
39 | "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", | ||
40 | "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" | ||
41 | }, | ||
42 | "step2" : { | ||
43 | "SDKMACHINE" : "x86_64", | ||
44 | "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", | ||
45 | "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" | ||
46 | }, | ||
47 | "step3" : { | ||
48 | "BUILDHISTORY" : false, | ||
49 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], | ||
50 | "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] | ||
51 | } | ||
52 | }, | ||
53 | </literallayout>Combining | ||
54 | these two entries you can see that “qemux86-64” is a three step build where the “bitbake | ||
55 | BBTARGETS” would be run, then “bitbake SANITYTARGETS” for each step; all for | ||
56 | MACHINE=”qemx86-64” but with differing SDKMACHINE settings. In step 1 an extra variable | ||
57 | is added to the <filename>auto.conf</filename> file to enable wic image | ||
58 | generation.</para> | ||
59 | <para>While not every detail of this is covered here, you can see how the templating | ||
60 | mechanism allows quite complex configurations to be built up yet allows duplication and | ||
61 | repetition to be kept to a minimum.</para> | ||
62 | <para>The different build targets are designed to allow for parallelisation, so different | ||
63 | machines are usually built in parallel, operations using the same machine and metadata | ||
64 | are built sequentially, with the aim of trying to optimise build efficiency as much as | ||
65 | possible.</para> | ||
66 | <para>The <filename>config.json</filename> file is processed by the scripts in the Helper | ||
67 | repository in the <filename>scripts</filename> directory. The following section details | ||
68 | how this works.</para> | ||
69 | </section> | ||
70 | |||
71 | <section id='test-autobuilder-target-exec-overview'> | ||
72 | <title>Autobuilder Target Execution Overview</title> | ||
73 | |||
74 | <para>For each given target in a build, the Autobuilder executes several steps. These are | ||
75 | configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist | ||
76 | of: <orderedlist> | ||
77 | <listitem id='test-list-tgt-exec-clobberdir'> | ||
78 | <para><emphasis>Run <filename>clobberdir</filename></emphasis></para> | ||
79 | <para>This cleans out any previous build. Old builds are left around to allow | ||
80 | easier debugging of failed builds. For additional information, see <link | ||
81 | linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para> | ||
82 | </listitem> | ||
83 | <listitem> | ||
84 | <para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para> | ||
85 | <para>This step clones the <filename>yocto-autobuilder-helper</filename> git | ||
86 | repository. This is necessary to prevent the requirement to maintain all the | ||
87 | release or project-specific code within Buildbot. The branch chosen matches | ||
88 | the release being built so we can support older releases and still make | ||
89 | changes in newer ones.</para> | ||
90 | </listitem> | ||
91 | <listitem> | ||
92 | <para><emphasis>Write layerinfo.json</emphasis></para> | ||
93 | <para>This transfers data in the Buildbot UI when the build was configured to | ||
94 | the Helper.</para> | ||
95 | </listitem> | ||
96 | <listitem> | ||
97 | <para><emphasis>Call scripts/shared-repo-unpack</emphasis></para> | ||
98 | <para>This is a call into the Helper scripts to set up a checkout of all the | ||
99 | pieces this build might need. It might clone the BitBake repository and the | ||
100 | OpenEmbedded-Core repository. It may clone the Poky repository, as well as | ||
101 | additional layers. It will use the data from the | ||
102 | <filename>layerinfo.json</filename> file to help understand the | ||
103 | configuration. It will also use a local cache of repositories to speed up | ||
104 | the clone checkouts. For additional information, see <link | ||
105 | linkend="test-autobuilder-clone-cache">Autobuilder Clone | ||
106 | Cache</link>.</para> | ||
107 | <para>This step has two possible modes of operation. If the build is part of a | ||
108 | parent build, its possible that all the repositories needed may already be | ||
109 | available, ready in a pre-prepared directory. An "a-quick" or "a-full" build | ||
110 | would prepare this before starting the other sub-target builds. This is done | ||
111 | for two reasons:<itemizedlist> | ||
112 | <listitem> | ||
113 | <para>the upstream may change during a build, for example, from a | ||
114 | forced push and this ensures we have matching content for the | ||
115 | whole build</para> | ||
116 | </listitem> | ||
117 | <listitem> | ||
118 | <para>if 15 Workers all tried to pull the same data from the same | ||
119 | repos, we can hit resource limits on upstream servers as they | ||
120 | can think they are under some kind of network attack</para> | ||
121 | </listitem> | ||
122 | </itemizedlist>This pre-prepared directory is shared among the Workers over | ||
123 | NFS. If the build is an individual build and there is no "shared" directory | ||
124 | available, it would clone from the cache and the upstreams as necessary. | ||
125 | This is considered the fallback mode.</para> | ||
126 | </listitem> | ||
127 | <listitem> | ||
128 | <para><emphasis>Call scripts/run-config</emphasis></para> | ||
129 | <para>This is another call into the Helper scripts where its expected that the | ||
130 | main functionality of this target will be executed.</para> | ||
131 | </listitem> | ||
132 | </orderedlist></para> | ||
133 | </section> | ||
134 | <section id='test-autobuilder-tech'> | ||
135 | <title>Autobuilder Technology</title> | ||
136 | <para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate | ||
137 | with increased efficiency and speed.</para> | ||
138 | <section id='test-clobberdir'> | ||
139 | <title>clobberdir</title> | ||
140 | <para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which | ||
141 | is a special script that moves files to a special location, rather than deleting | ||
142 | them. Files in this location are deleted by an <filename>rm</filename> command, | ||
143 | which is run under <filename>ionice -c 3</filename>. For example, the deletion only | ||
144 | happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor | ||
145 | runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder | ||
146 | Worker Janitor</link>.</para> | ||
147 | </section> | ||
148 | <section id='test-autobuilder-clone-cache'> | ||
149 | <title>Autobuilder Clone Cache</title> | ||
150 | <para>Cloning repositories from scratch each time they are required was slow on the | ||
151 | Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on | ||
152 | the Workers. Data is fetched from these during clones first, then "topped up" with | ||
153 | later revisions from any upstream when necesary. The cache is maintained by the | ||
154 | Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor" | ||
155 | >Autobuilder Worker Janitor</link>.</para> | ||
156 | </section> | ||
157 | <section id='test-autobuilder-worker-janitor'> | ||
158 | <title>Autobuilder Worker Janitor</title> | ||
159 | <para>This is a process running on each Worker that performs two basic operations, | ||
160 | including background file deletion at IO idle (see <link | ||
161 | linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and | ||
162 | maintainenance of a cache of cloned repositories to improve the speed the system can | ||
163 | checkout repositories.</para> | ||
164 | </section> | ||
165 | <section id='test-shared-dl-dir'> | ||
166 | <title>Shared DL_DIR</title> | ||
167 | <para>The Workers are all connected over NFS which allows DL_DIR to be shared between | ||
168 | them. This reduces network accesses from the system and allows the build to be sped | ||
169 | up. Usage of the directory within the build system is designed to be able to be | ||
170 | shared over NFS.</para> | ||
171 | </section> | ||
172 | <section id='test-shared-sstate-cache'> | ||
173 | <title>Shared SSTATE_DIR</title> | ||
174 | <para>The Workers are all connected over NFS which allows the | ||
175 | <filename>sstate</filename> directory to be shared between them. This means once | ||
176 | a Worker has built an artefact, all the others can benefit from it. Usage of the | ||
177 | directory within the directory is designed for sharing over NFS.</para> | ||
178 | </section> | ||
179 | <section id='test-resulttool'> | ||
180 | <title>Resulttool</title> | ||
181 | <para>All of the different tests run as part of the build generate output into | ||
182 | <filename>testresults.json</filename> files. This allows us to determine which | ||
183 | tests ran in a given build and their status. Additional information, such as failure | ||
184 | logs or the time taken to run the tests, may also be included.</para> | ||
185 | <para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json | ||
186 | results files. It has the ability to merge files together, display reports of the | ||
187 | test results and compare different result files.</para> | ||
188 | <para>For details, see <link linkend="" | ||
189 | >https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para> | ||
190 | </section> | ||
191 | </section> | ||
192 | <section id='test-run-config-tgt-execution'> | ||
193 | <title>run-config Target Execution</title> | ||
194 | <para>The <filename>scripts/run-config</filename> execution is where most of the work within | ||
195 | the Autobuilder happens. It runs through a number of steps; the first are general setup | ||
196 | steps that are run once and include:<orderedlist> | ||
197 | <listitem> | ||
198 | <para>Set up any <filename>buildtools-tarball</filename> if configured.</para> | ||
199 | </listitem> | ||
200 | <listitem> | ||
201 | <para>Call "buildhistory-init" if buildhistory is configured.</para> | ||
202 | </listitem> | ||
203 | </orderedlist></para> | ||
204 | <para>For each step that is configured in <filename>config.json</filename>, it will perform | ||
205 | the following:</para> | ||
206 | <para> | ||
207 | <remark>## WRITER's question: What does "logging in as stepXa" and others refer to | ||
208 | below? ##</remark> | ||
209 | <orderedlist> | ||
210 | <listitem id="test-run-config-add-layers-step"> | ||
211 | <para dir="ltr">Add any layers that are specified using the | ||
212 | <filename>bitbake-layers add-layer</filename> command (logging as | ||
213 | stepXa)</para> | ||
214 | </listitem> | ||
215 | <listitem> | ||
216 | <para dir="ltr">Call the <filename>scripts/setup-config</filename> script to | ||
217 | generate the necessary <filename>auto.conf</filename> configuration file for | ||
218 | the build</para> | ||
219 | </listitem> | ||
220 | <listitem> | ||
221 | <para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging | ||
222 | as stepXb)</para> | ||
223 | </listitem> | ||
224 | <listitem> | ||
225 | <para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command | ||
226 | (logging as stepXc)</para> | ||
227 | </listitem> | ||
228 | <listitem> | ||
229 | <para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run | ||
230 | within the BitBake build environment (logging as stepXd)</para> | ||
231 | </listitem> | ||
232 | <listitem> | ||
233 | <para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which | ||
234 | are run outside the BitBake build environment (logging as stepXd)</para> | ||
235 | </listitem> | ||
236 | <listitem> | ||
237 | <para dir="ltr">Remove any layers added in <link | ||
238 | linkend="test-run-config-add-layers-step">step 1</link> using the | ||
239 | <filename>bitbake-layers remove-layer</filename> command (logging as | ||
240 | stepXa)</para> | ||
241 | </listitem> | ||
242 | </orderedlist> | ||
243 | </para> | ||
244 | <para>Once the execution steps above complete, <filename>run-config</filename> executes a | ||
245 | set of post-build steps, including:<orderedlist> | ||
246 | <listitem> | ||
247 | <para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect | ||
248 | any output which is to be saved from the build.</para> | ||
249 | </listitem> | ||
250 | <listitem> | ||
251 | <para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any | ||
252 | test results to be saved from the build.</para> | ||
253 | </listitem> | ||
254 | <listitem> | ||
255 | <para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send | ||
256 | any error reports generated to the remote server.</para> | ||
257 | </listitem> | ||
258 | <listitem> | ||
259 | <para dir="ltr">Cleanup the build directory using <link | ||
260 | linkend="test-clobberdir"><filename>clobberdir</filename></link> if the | ||
261 | build was successful, else rename it to “build-renamed” for potential future | ||
262 | debugging.</para> | ||
263 | </listitem> | ||
264 | </orderedlist></para> | ||
265 | </section> | ||
266 | <section id='test-deploying-yp-autobuilder'> | ||
267 | <title>Deploying Yocto Autobuilder</title> | ||
268 | <para>The most up to date information about how to setup and deploy your own Autbuilder can | ||
269 | be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para> | ||
270 | <para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly | ||
271 | but it is inevitable that users will end up needing to heavily customise the | ||
272 | <filename>yocto-autobuilder-helper</filename> repository, particularly the | ||
273 | <filename>config.json</filename> file as they will want to define their own test | ||
274 | matrix.</para> | ||
275 | <para>The Autobuilder supports wo customization options: <itemizedlist> | ||
276 | <listitem> | ||
277 | <para>variable substitution</para> | ||
278 | </listitem> | ||
279 | <listitem> | ||
280 | <para>overlaying configuration files</para> | ||
281 | </listitem> | ||
282 | </itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow | ||
283 | substitution of the paths. The Helper script repository includes a | ||
284 | <filename>local-example.json</filename> file to show how you could override these | ||
285 | from a separate configuration file. Pass the following into the environment of the | ||
286 | Autobuilder:<literallayout> | ||
287 | $ ABHELPER_JSON="config.json local-example.json" | ||
288 | </literallayout>As | ||
289 | another example, you could also pass the following into the | ||
290 | environment:<literallayout> | ||
291 | $ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json" | ||
292 | </literallayout>One | ||
293 | issue users often run into is validation of the <filename>config.json</filename> files. | ||
294 | A tip for minimizing issues from invalid json files is to use a Git | ||
295 | <filename>pre-commit-hook.sh</filename> script to verify the JSON file before | ||
296 | committing it. Create a symbolic link as | ||
297 | follows:<literallayout> | ||
298 | $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | ||
299 | </literallayout></para> | ||
300 | </section> | ||
301 | |||
302 | |||
303 | |||
304 | |||
305 | |||
306 | |||
307 | |||
308 | |||
309 | </chapter> | ||
310 | <!-- | ||
311 | vim: expandtab tw=80 ts=4 | ||
312 | --> | ||