diff options
author | Nicolas Dechesne <nicolas.dechesne@linaro.org> | 2020-11-20 20:17:33 +0100 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2021-01-04 10:55:00 +0000 |
commit | fa0cb4d34b1073f215fa3c680f2316208739d53d (patch) | |
tree | ba89c1f4289fd6456af4409a6a19caf6548dfb9c /documentation/test-manual/test-manual-understand-autobuilder.xml | |
parent | a038e58f3cd82c56102444bdc5ac76c9f1550a0d (diff) | |
download | poky-fa0cb4d34b1073f215fa3c680f2316208739d53d.tar.gz |
sphinx: import docs
The Yocto Project docs was migrated from Docbook to Sphinx in YP
3.2. This 3.1 is an LTS release, and since 3.1 docs are 'close to'
the docs in 3.2, we agreed to backport sphinx docs onto 3.1.
This first patch brings all changes done in 3.2 until:
7f64574f7 README: include detailed information about sphinx
There are other changes after this commit, but they will be
selectively backported in individual patches.
This patch was generated with the following command:
git cherry-pick -n \
$(git log --reverse --oneline \
ac352ad7f95db7eeacb53c2778caa31800bd7c26..7f64574f7 \
| cut -f1 -d' ')
The following commits were applies in the dunfell docs, but not in
master, so they were first reverted (and squashed into this change). A
commit will reintroduce the content from these patches in the Sphinx
files in a followup patch.
069c27574 Documenation: Prepared for the 3.1.1 release
bd140f0f9 Documentation: Add 3.1.1 version updates missing from previous commit
17cc71a8f Documenation: Prepared for the 3.1.2 release
1a69e2c02 Documenation: Prepared for the 3.1.3 release
8910ac1c7 Documenation: Prepared for the 3.1.4 release
(From yocto-docs rev: c25fe058b88b893b0d146f3ed27320b47cdec236)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/test-manual/test-manual-understand-autobuilder.xml')
-rw-r--r-- | documentation/test-manual/test-manual-understand-autobuilder.xml | 314 |
1 files changed, 314 insertions, 0 deletions
diff --git a/documentation/test-manual/test-manual-understand-autobuilder.xml b/documentation/test-manual/test-manual-understand-autobuilder.xml new file mode 100644 index 0000000000..8600367be7 --- /dev/null +++ b/documentation/test-manual/test-manual-understand-autobuilder.xml | |||
@@ -0,0 +1,314 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | ||
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | ||
4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
5 | |||
6 | <chapter id='test-manual-understand-autobuilder'> | ||
7 | |||
8 | <title>Understanding the Yocto Project Autobuilder</title> | ||
9 | <section> | ||
10 | <title>Execution Flow within the Autobuilder</title> | ||
11 | <para>The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and | ||
12 | it makes sense to follow the process through the system starting there. This is best | ||
13 | visualised from the Autobuilder Console view (<link linkend="" | ||
14 | >https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para> | ||
15 | <para>Each item along the top of that view represents some "target build" and these targets | ||
16 | are all run in parallel. The 'full' build will trigger the majority of them, the "quick" | ||
17 | build will trigger some subset of them. The Autobuilder effectively runs whichever | ||
18 | configuration is defined for each of those targets on a seperate buildbot worker. To | ||
19 | understand the configuration, you need to look at the entry on | ||
20 | <filename>config.json</filename> file within the | ||
21 | <filename>yocto-autobuilder-helper</filename> repository. The targets are defined in | ||
22 | the ‘overrides' section, a quick example could be qemux86-64 which looks | ||
23 | like:<literallayout class="monospaced"> | ||
24 | "qemux86-64" : { | ||
25 | "MACHINE" : "qemux86-64", | ||
26 | "TEMPLATE" : "arch-qemu", | ||
27 | "step1" : { | ||
28 | "extravars" : [ | ||
29 | "IMAGE_FSTYPES_append = ' wic wic.bmap'" | ||
30 | ] | ||
31 | } | ||
32 | }, | ||
33 | </literallayout>And | ||
34 | to expand that, you need the "arch-qemu" entry from the "templates" section, which looks | ||
35 | like:<literallayout class="monospaced"> | ||
36 | "arch-qemu" : { | ||
37 | "BUILDINFO" : true, | ||
38 | "BUILDHISTORY" : true, | ||
39 | "step1" : { | ||
40 | "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", | ||
41 | "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" | ||
42 | }, | ||
43 | "step2" : { | ||
44 | "SDKMACHINE" : "x86_64", | ||
45 | "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", | ||
46 | "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" | ||
47 | }, | ||
48 | "step3" : { | ||
49 | "BUILDHISTORY" : false, | ||
50 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], | ||
51 | "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] | ||
52 | } | ||
53 | }, | ||
54 | </literallayout>Combining | ||
55 | these two entries you can see that "qemux86-64" is a three step build where the | ||
56 | <filename>bitbake BBTARGETS</filename> would be run, then <filename>bitbake | ||
57 | SANITYTARGETS</filename> for each step; all for | ||
58 | <filename>MACHINE="qemx86-64"</filename> but with differing SDKMACHINE settings. In | ||
59 | step 1 an extra variable is added to the <filename>auto.conf</filename> file to enable | ||
60 | wic image generation.</para> | ||
61 | <para>While not every detail of this is covered here, you can see how the templating | ||
62 | mechanism allows quite complex configurations to be built up yet allows duplication and | ||
63 | repetition to be kept to a minimum.</para> | ||
64 | <para>The different build targets are designed to allow for parallelisation, so different | ||
65 | machines are usually built in parallel, operations using the same machine and metadata | ||
66 | are built sequentially, with the aim of trying to optimise build efficiency as much as | ||
67 | possible.</para> | ||
68 | <para>The <filename>config.json</filename> file is processed by the scripts in the Helper | ||
69 | repository in the <filename>scripts</filename> directory. The following section details | ||
70 | how this works.</para> | ||
71 | </section> | ||
72 | |||
73 | <section id='test-autobuilder-target-exec-overview'> | ||
74 | <title>Autobuilder Target Execution Overview</title> | ||
75 | |||
76 | <para>For each given target in a build, the Autobuilder executes several steps. These are | ||
77 | configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist | ||
78 | of: <orderedlist> | ||
79 | <listitem id='test-list-tgt-exec-clobberdir'> | ||
80 | <para><emphasis>Run <filename>clobberdir</filename></emphasis></para> | ||
81 | <para>This cleans out any previous build. Old builds are left around to allow | ||
82 | easier debugging of failed builds. For additional information, see <link | ||
83 | linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para> | ||
84 | </listitem> | ||
85 | <listitem> | ||
86 | <para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para> | ||
87 | <para>This step clones the <filename>yocto-autobuilder-helper</filename> git | ||
88 | repository. This is necessary to prevent the requirement to maintain all the | ||
89 | release or project-specific code within Buildbot. The branch chosen matches | ||
90 | the release being built so we can support older releases and still make | ||
91 | changes in newer ones.</para> | ||
92 | </listitem> | ||
93 | <listitem> | ||
94 | <para><emphasis>Write layerinfo.json</emphasis></para> | ||
95 | <para>This transfers data in the Buildbot UI when the build was configured to | ||
96 | the Helper.</para> | ||
97 | </listitem> | ||
98 | <listitem> | ||
99 | <para><emphasis>Call scripts/shared-repo-unpack</emphasis></para> | ||
100 | <para>This is a call into the Helper scripts to set up a checkout of all the | ||
101 | pieces this build might need. It might clone the BitBake repository and the | ||
102 | OpenEmbedded-Core repository. It may clone the Poky repository, as well as | ||
103 | additional layers. It will use the data from the | ||
104 | <filename>layerinfo.json</filename> file to help understand the | ||
105 | configuration. It will also use a local cache of repositories to speed up | ||
106 | the clone checkouts. For additional information, see <link | ||
107 | linkend="test-autobuilder-clone-cache">Autobuilder Clone | ||
108 | Cache</link>.</para> | ||
109 | <para>This step has two possible modes of operation. If the build is part of a | ||
110 | parent build, its possible that all the repositories needed may already be | ||
111 | available, ready in a pre-prepared directory. An "a-quick" or "a-full" build | ||
112 | would prepare this before starting the other sub-target builds. This is done | ||
113 | for two reasons:<itemizedlist> | ||
114 | <listitem> | ||
115 | <para>the upstream may change during a build, for example, from a | ||
116 | forced push and this ensures we have matching content for the | ||
117 | whole build</para> | ||
118 | </listitem> | ||
119 | <listitem> | ||
120 | <para>if 15 Workers all tried to pull the same data from the same | ||
121 | repos, we can hit resource limits on upstream servers as they | ||
122 | can think they are under some kind of network attack</para> | ||
123 | </listitem> | ||
124 | </itemizedlist>This pre-prepared directory is shared among the Workers over | ||
125 | NFS. If the build is an individual build and there is no "shared" directory | ||
126 | available, it would clone from the cache and the upstreams as necessary. | ||
127 | This is considered the fallback mode.</para> | ||
128 | </listitem> | ||
129 | <listitem> | ||
130 | <para><emphasis>Call scripts/run-config</emphasis></para> | ||
131 | <para>This is another call into the Helper scripts where its expected that the | ||
132 | main functionality of this target will be executed.</para> | ||
133 | </listitem> | ||
134 | </orderedlist></para> | ||
135 | </section> | ||
136 | <section id='test-autobuilder-tech'> | ||
137 | <title>Autobuilder Technology</title> | ||
138 | <para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate | ||
139 | with increased efficiency and speed.</para> | ||
140 | <section id='test-clobberdir'> | ||
141 | <title>clobberdir</title> | ||
142 | <para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which | ||
143 | is a special script that moves files to a special location, rather than deleting | ||
144 | them. Files in this location are deleted by an <filename>rm</filename> command, | ||
145 | which is run under <filename>ionice -c 3</filename>. For example, the deletion only | ||
146 | happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor | ||
147 | runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder | ||
148 | Worker Janitor</link>.</para> | ||
149 | </section> | ||
150 | <section id='test-autobuilder-clone-cache'> | ||
151 | <title>Autobuilder Clone Cache</title> | ||
152 | <para>Cloning repositories from scratch each time they are required was slow on the | ||
153 | Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on | ||
154 | the Workers. Data is fetched from these during clones first, then "topped up" with | ||
155 | later revisions from any upstream when necesary. The cache is maintained by the | ||
156 | Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor" | ||
157 | >Autobuilder Worker Janitor</link>.</para> | ||
158 | </section> | ||
159 | <section id='test-autobuilder-worker-janitor'> | ||
160 | <title>Autobuilder Worker Janitor</title> | ||
161 | <para>This is a process running on each Worker that performs two basic operations, | ||
162 | including background file deletion at IO idle (see <link | ||
163 | linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and | ||
164 | maintainenance of a cache of cloned repositories to improve the speed the system can | ||
165 | checkout repositories.</para> | ||
166 | </section> | ||
167 | <section id='test-shared-dl-dir'> | ||
168 | <title>Shared DL_DIR</title> | ||
169 | <para>The Workers are all connected over NFS which allows DL_DIR to be shared between | ||
170 | them. This reduces network accesses from the system and allows the build to be sped | ||
171 | up. Usage of the directory within the build system is designed to be able to be | ||
172 | shared over NFS.</para> | ||
173 | </section> | ||
174 | <section id='test-shared-sstate-cache'> | ||
175 | <title>Shared SSTATE_DIR</title> | ||
176 | <para>The Workers are all connected over NFS which allows the | ||
177 | <filename>sstate</filename> directory to be shared between them. This means once | ||
178 | a Worker has built an artefact, all the others can benefit from it. Usage of the | ||
179 | directory within the directory is designed for sharing over NFS.</para> | ||
180 | </section> | ||
181 | <section id='test-resulttool'> | ||
182 | <title>Resulttool</title> | ||
183 | <para>All of the different tests run as part of the build generate output into | ||
184 | <filename>testresults.json</filename> files. This allows us to determine which | ||
185 | tests ran in a given build and their status. Additional information, such as failure | ||
186 | logs or the time taken to run the tests, may also be included.</para> | ||
187 | <para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json | ||
188 | results files. It has the ability to merge files together, display reports of the | ||
189 | test results and compare different result files.</para> | ||
190 | <para>For details, see <link linkend="" | ||
191 | >https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para> | ||
192 | </section> | ||
193 | </section> | ||
194 | <section id='test-run-config-tgt-execution'> | ||
195 | <title>run-config Target Execution</title> | ||
196 | <para>The <filename>scripts/run-config</filename> execution is where most of the work within | ||
197 | the Autobuilder happens. It runs through a number of steps; the first are general setup | ||
198 | steps that are run once and include:<orderedlist> | ||
199 | <listitem> | ||
200 | <para>Set up any <filename>buildtools-tarball</filename> if configured.</para> | ||
201 | </listitem> | ||
202 | <listitem> | ||
203 | <para>Call "buildhistory-init" if buildhistory is configured.</para> | ||
204 | </listitem> | ||
205 | </orderedlist></para> | ||
206 | <para>For each step that is configured in <filename>config.json</filename>, it will perform | ||
207 | the following:</para> | ||
208 | <para> | ||
209 | <remark>## WRITER's question: What does "logging in as stepXa" and others refer to | ||
210 | below? ##</remark> | ||
211 | <orderedlist> | ||
212 | <listitem id="test-run-config-add-layers-step"> | ||
213 | <para dir="ltr">Add any layers that are specified using the | ||
214 | <filename>bitbake-layers add-layer</filename> command (logging as | ||
215 | stepXa)</para> | ||
216 | </listitem> | ||
217 | <listitem> | ||
218 | <para dir="ltr">Call the <filename>scripts/setup-config</filename> script to | ||
219 | generate the necessary <filename>auto.conf</filename> configuration file for | ||
220 | the build</para> | ||
221 | </listitem> | ||
222 | <listitem> | ||
223 | <para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging | ||
224 | as stepXb)</para> | ||
225 | </listitem> | ||
226 | <listitem> | ||
227 | <para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command | ||
228 | (logging as stepXc)</para> | ||
229 | </listitem> | ||
230 | <listitem> | ||
231 | <para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run | ||
232 | within the BitBake build environment (logging as stepXd)</para> | ||
233 | </listitem> | ||
234 | <listitem> | ||
235 | <para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which | ||
236 | are run outside the BitBake build environment (logging as stepXd)</para> | ||
237 | </listitem> | ||
238 | <listitem> | ||
239 | <para dir="ltr">Remove any layers added in <link | ||
240 | linkend="test-run-config-add-layers-step">step 1</link> using the | ||
241 | <filename>bitbake-layers remove-layer</filename> command (logging as | ||
242 | stepXa)</para> | ||
243 | </listitem> | ||
244 | </orderedlist> | ||
245 | </para> | ||
246 | <para>Once the execution steps above complete, <filename>run-config</filename> executes a | ||
247 | set of post-build steps, including:<orderedlist> | ||
248 | <listitem> | ||
249 | <para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect | ||
250 | any output which is to be saved from the build.</para> | ||
251 | </listitem> | ||
252 | <listitem> | ||
253 | <para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any | ||
254 | test results to be saved from the build.</para> | ||
255 | </listitem> | ||
256 | <listitem> | ||
257 | <para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send | ||
258 | any error reports generated to the remote server.</para> | ||
259 | </listitem> | ||
260 | <listitem> | ||
261 | <para dir="ltr">Cleanup the build directory using <link | ||
262 | linkend="test-clobberdir"><filename>clobberdir</filename></link> if the | ||
263 | build was successful, else rename it to "build-renamed" for potential future | ||
264 | debugging.</para> | ||
265 | </listitem> | ||
266 | </orderedlist></para> | ||
267 | </section> | ||
268 | <section id='test-deploying-yp-autobuilder'> | ||
269 | <title>Deploying Yocto Autobuilder</title> | ||
270 | <para>The most up to date information about how to setup and deploy your own Autbuilder can | ||
271 | be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para> | ||
272 | <para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly | ||
273 | but it is inevitable that users will end up needing to heavily customise the | ||
274 | <filename>yocto-autobuilder-helper</filename> repository, particularly the | ||
275 | <filename>config.json</filename> file as they will want to define their own test | ||
276 | matrix.</para> | ||
277 | <para>The Autobuilder supports wo customization options: <itemizedlist> | ||
278 | <listitem> | ||
279 | <para>variable substitution</para> | ||
280 | </listitem> | ||
281 | <listitem> | ||
282 | <para>overlaying configuration files</para> | ||
283 | </listitem> | ||
284 | </itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow | ||
285 | substitution of the paths. The Helper script repository includes a | ||
286 | <filename>local-example.json</filename> file to show how you could override these | ||
287 | from a separate configuration file. Pass the following into the environment of the | ||
288 | Autobuilder:<literallayout class="monospaced"> | ||
289 | $ ABHELPER_JSON="config.json local-example.json" | ||
290 | </literallayout>As | ||
291 | another example, you could also pass the following into the | ||
292 | environment:<literallayout class="monospaced"> | ||
293 | $ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json" | ||
294 | </literallayout>One | ||
295 | issue users often run into is validation of the <filename>config.json</filename> files. | ||
296 | A tip for minimizing issues from invalid json files is to use a Git | ||
297 | <filename>pre-commit-hook.sh</filename> script to verify the JSON file before | ||
298 | committing it. Create a symbolic link as | ||
299 | follows:<literallayout class="monospaced"> | ||
300 | $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit | ||
301 | </literallayout></para> | ||
302 | </section> | ||
303 | |||
304 | |||
305 | |||
306 | |||
307 | |||
308 | |||
309 | |||
310 | |||
311 | </chapter> | ||
312 | <!-- | ||
313 | vim: expandtab tw=80 ts=4 | ||
314 | --> | ||