summaryrefslogtreecommitdiffstats
path: root/documentation/test-manual/test-manual-understand-autobuilder.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/test-manual/test-manual-understand-autobuilder.xml')
-rw-r--r--documentation/test-manual/test-manual-understand-autobuilder.xml314
1 files changed, 0 insertions, 314 deletions
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.xml b/documentation/test-manual/test-manual-understand-autobuilder.xml
deleted file mode 100644
index 8600367be7..0000000000
--- a/documentation/test-manual/test-manual-understand-autobuilder.xml
+++ /dev/null
@@ -1,314 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='test-manual-understand-autobuilder'>
7
8<title>Understanding the Yocto Project Autobuilder</title>
9 <section>
10 <title>Execution Flow within the Autobuilder</title>
11 <para>The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and
12 it makes sense to follow the process through the system starting there. This is best
13 visualised from the Autobuilder Console view (<link linkend=""
14 >https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para>
15 <para>Each item along the top of that view represents some "target build" and these targets
16 are all run in parallel. The 'full' build will trigger the majority of them, the "quick"
17 build will trigger some subset of them. The Autobuilder effectively runs whichever
18 configuration is defined for each of those targets on a seperate buildbot worker. To
19 understand the configuration, you need to look at the entry on
20 <filename>config.json</filename> file within the
21 <filename>yocto-autobuilder-helper</filename> repository. The targets are defined in
22 the ‘overrides' section, a quick example could be qemux86-64 which looks
23 like:<literallayout class="monospaced">
24 "qemux86-64" : {
25 "MACHINE" : "qemux86-64",
26 "TEMPLATE" : "arch-qemu",
27 "step1" : {
28 "extravars" : [
29 "IMAGE_FSTYPES_append = ' wic wic.bmap'"
30 ]
31 }
32 },
33 </literallayout>And
34 to expand that, you need the "arch-qemu" entry from the "templates" section, which looks
35 like:<literallayout class="monospaced">
36 "arch-qemu" : {
37 "BUILDINFO" : true,
38 "BUILDHISTORY" : true,
39 "step1" : {
40 "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
41 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
42 },
43 "step2" : {
44 "SDKMACHINE" : "x86_64",
45 "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
46 "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
47 },
48 "step3" : {
49 "BUILDHISTORY" : false,
50 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
51 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
52 }
53 },
54 </literallayout>Combining
55 these two entries you can see that "qemux86-64" is a three step build where the
56 <filename>bitbake BBTARGETS</filename> would be run, then <filename>bitbake
57 SANITYTARGETS</filename> for each step; all for
58 <filename>MACHINE="qemx86-64"</filename> but with differing SDKMACHINE settings. In
59 step 1 an extra variable is added to the <filename>auto.conf</filename> file to enable
60 wic image generation.</para>
61 <para>While not every detail of this is covered here, you can see how the templating
62 mechanism allows quite complex configurations to be built up yet allows duplication and
63 repetition to be kept to a minimum.</para>
64 <para>The different build targets are designed to allow for parallelisation, so different
65 machines are usually built in parallel, operations using the same machine and metadata
66 are built sequentially, with the aim of trying to optimise build efficiency as much as
67 possible.</para>
68 <para>The <filename>config.json</filename> file is processed by the scripts in the Helper
69 repository in the <filename>scripts</filename> directory. The following section details
70 how this works.</para>
71 </section>
72
73 <section id='test-autobuilder-target-exec-overview'>
74 <title>Autobuilder Target Execution Overview</title>
75
76 <para>For each given target in a build, the Autobuilder executes several steps. These are
77 configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist
78 of: <orderedlist>
79 <listitem id='test-list-tgt-exec-clobberdir'>
80 <para><emphasis>Run <filename>clobberdir</filename></emphasis></para>
81 <para>This cleans out any previous build. Old builds are left around to allow
82 easier debugging of failed builds. For additional information, see <link
83 linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para>
84 </listitem>
85 <listitem>
86 <para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para>
87 <para>This step clones the <filename>yocto-autobuilder-helper</filename> git
88 repository. This is necessary to prevent the requirement to maintain all the
89 release or project-specific code within Buildbot. The branch chosen matches
90 the release being built so we can support older releases and still make
91 changes in newer ones.</para>
92 </listitem>
93 <listitem>
94 <para><emphasis>Write layerinfo.json</emphasis></para>
95 <para>This transfers data in the Buildbot UI when the build was configured to
96 the Helper.</para>
97 </listitem>
98 <listitem>
99 <para><emphasis>Call scripts/shared-repo-unpack</emphasis></para>
100 <para>This is a call into the Helper scripts to set up a checkout of all the
101 pieces this build might need. It might clone the BitBake repository and the
102 OpenEmbedded-Core repository. It may clone the Poky repository, as well as
103 additional layers. It will use the data from the
104 <filename>layerinfo.json</filename> file to help understand the
105 configuration. It will also use a local cache of repositories to speed up
106 the clone checkouts. For additional information, see <link
107 linkend="test-autobuilder-clone-cache">Autobuilder Clone
108 Cache</link>.</para>
109 <para>This step has two possible modes of operation. If the build is part of a
110 parent build, its possible that all the repositories needed may already be
111 available, ready in a pre-prepared directory. An "a-quick" or "a-full" build
112 would prepare this before starting the other sub-target builds. This is done
113 for two reasons:<itemizedlist>
114 <listitem>
115 <para>the upstream may change during a build, for example, from a
116 forced push and this ensures we have matching content for the
117 whole build</para>
118 </listitem>
119 <listitem>
120 <para>if 15 Workers all tried to pull the same data from the same
121 repos, we can hit resource limits on upstream servers as they
122 can think they are under some kind of network attack</para>
123 </listitem>
124 </itemizedlist>This pre-prepared directory is shared among the Workers over
125 NFS. If the build is an individual build and there is no "shared" directory
126 available, it would clone from the cache and the upstreams as necessary.
127 This is considered the fallback mode.</para>
128 </listitem>
129 <listitem>
130 <para><emphasis>Call scripts/run-config</emphasis></para>
131 <para>This is another call into the Helper scripts where its expected that the
132 main functionality of this target will be executed.</para>
133 </listitem>
134 </orderedlist></para>
135 </section>
136 <section id='test-autobuilder-tech'>
137 <title>Autobuilder Technology</title>
138 <para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate
139 with increased efficiency and speed.</para>
140 <section id='test-clobberdir'>
141 <title>clobberdir</title>
142 <para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which
143 is a special script that moves files to a special location, rather than deleting
144 them. Files in this location are deleted by an <filename>rm</filename> command,
145 which is run under <filename>ionice -c 3</filename>. For example, the deletion only
146 happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor
147 runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder
148 Worker Janitor</link>.</para>
149 </section>
150 <section id='test-autobuilder-clone-cache'>
151 <title>Autobuilder Clone Cache</title>
152 <para>Cloning repositories from scratch each time they are required was slow on the
153 Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on
154 the Workers. Data is fetched from these during clones first, then "topped up" with
155 later revisions from any upstream when necesary. The cache is maintained by the
156 Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor"
157 >Autobuilder Worker Janitor</link>.</para>
158 </section>
159 <section id='test-autobuilder-worker-janitor'>
160 <title>Autobuilder Worker Janitor</title>
161 <para>This is a process running on each Worker that performs two basic operations,
162 including background file deletion at IO idle (see <link
163 linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and
164 maintainenance of a cache of cloned repositories to improve the speed the system can
165 checkout repositories.</para>
166 </section>
167 <section id='test-shared-dl-dir'>
168 <title>Shared DL_DIR</title>
169 <para>The Workers are all connected over NFS which allows DL_DIR to be shared between
170 them. This reduces network accesses from the system and allows the build to be sped
171 up. Usage of the directory within the build system is designed to be able to be
172 shared over NFS.</para>
173 </section>
174 <section id='test-shared-sstate-cache'>
175 <title>Shared SSTATE_DIR</title>
176 <para>The Workers are all connected over NFS which allows the
177 <filename>sstate</filename> directory to be shared between them. This means once
178 a Worker has built an artefact, all the others can benefit from it. Usage of the
179 directory within the directory is designed for sharing over NFS.</para>
180 </section>
181 <section id='test-resulttool'>
182 <title>Resulttool</title>
183 <para>All of the different tests run as part of the build generate output into
184 <filename>testresults.json</filename> files. This allows us to determine which
185 tests ran in a given build and their status. Additional information, such as failure
186 logs or the time taken to run the tests, may also be included.</para>
187 <para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json
188 results files. It has the ability to merge files together, display reports of the
189 test results and compare different result files.</para>
190 <para>For details, see <link linkend=""
191 >https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para>
192 </section>
193 </section>
194 <section id='test-run-config-tgt-execution'>
195 <title>run-config Target Execution</title>
196 <para>The <filename>scripts/run-config</filename> execution is where most of the work within
197 the Autobuilder happens. It runs through a number of steps; the first are general setup
198 steps that are run once and include:<orderedlist>
199 <listitem>
200 <para>Set up any <filename>buildtools-tarball</filename> if configured.</para>
201 </listitem>
202 <listitem>
203 <para>Call "buildhistory-init" if buildhistory is configured.</para>
204 </listitem>
205 </orderedlist></para>
206 <para>For each step that is configured in <filename>config.json</filename>, it will perform
207 the following:</para>
208 <para>
209 <remark>## WRITER's question: What does "logging in as stepXa" and others refer to
210 below? ##</remark>
211 <orderedlist>
212 <listitem id="test-run-config-add-layers-step">
213 <para dir="ltr">Add any layers that are specified using the
214 <filename>bitbake-layers add-layer</filename> command (logging as
215 stepXa)</para>
216 </listitem>
217 <listitem>
218 <para dir="ltr">Call the <filename>scripts/setup-config</filename> script to
219 generate the necessary <filename>auto.conf</filename> configuration file for
220 the build</para>
221 </listitem>
222 <listitem>
223 <para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging
224 as stepXb)</para>
225 </listitem>
226 <listitem>
227 <para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command
228 (logging as stepXc)</para>
229 </listitem>
230 <listitem>
231 <para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run
232 within the BitBake build environment (logging as stepXd)</para>
233 </listitem>
234 <listitem>
235 <para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which
236 are run outside the BitBake build environment (logging as stepXd)</para>
237 </listitem>
238 <listitem>
239 <para dir="ltr">Remove any layers added in <link
240 linkend="test-run-config-add-layers-step">step 1</link> using the
241 <filename>bitbake-layers remove-layer</filename> command (logging as
242 stepXa)</para>
243 </listitem>
244 </orderedlist>
245 </para>
246 <para>Once the execution steps above complete, <filename>run-config</filename> executes a
247 set of post-build steps, including:<orderedlist>
248 <listitem>
249 <para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect
250 any output which is to be saved from the build.</para>
251 </listitem>
252 <listitem>
253 <para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any
254 test results to be saved from the build.</para>
255 </listitem>
256 <listitem>
257 <para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send
258 any error reports generated to the remote server.</para>
259 </listitem>
260 <listitem>
261 <para dir="ltr">Cleanup the build directory using <link
262 linkend="test-clobberdir"><filename>clobberdir</filename></link> if the
263 build was successful, else rename it to "build-renamed" for potential future
264 debugging.</para>
265 </listitem>
266 </orderedlist></para>
267 </section>
268 <section id='test-deploying-yp-autobuilder'>
269 <title>Deploying Yocto Autobuilder</title>
270 <para>The most up to date information about how to setup and deploy your own Autbuilder can
271 be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para>
272 <para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly
273 but it is inevitable that users will end up needing to heavily customise the
274 <filename>yocto-autobuilder-helper</filename> repository, particularly the
275 <filename>config.json</filename> file as they will want to define their own test
276 matrix.</para>
277 <para>The Autobuilder supports wo customization options: <itemizedlist>
278 <listitem>
279 <para>variable substitution</para>
280 </listitem>
281 <listitem>
282 <para>overlaying configuration files</para>
283 </listitem>
284 </itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow
285 substitution of the paths. The Helper script repository includes a
286 <filename>local-example.json</filename> file to show how you could override these
287 from a separate configuration file. Pass the following into the environment of the
288 Autobuilder:<literallayout class="monospaced">
289 $ ABHELPER_JSON="config.json local-example.json"
290 </literallayout>As
291 another example, you could also pass the following into the
292 environment:<literallayout class="monospaced">
293 $ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json"
294 </literallayout>One
295 issue users often run into is validation of the <filename>config.json</filename> files.
296 A tip for minimizing issues from invalid json files is to use a Git
297 <filename>pre-commit-hook.sh</filename> script to verify the JSON file before
298 committing it. Create a symbolic link as
299 follows:<literallayout class="monospaced">
300 $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
301 </literallayout></para>
302 </section>
303
304
305
306
307
308
309
310
311</chapter>
312<!--
313vim: expandtab tw=80 ts=4
314-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-13 02:06:01 +0000