diff options
Diffstat (limited to 'documentation/test-manual/understand-autobuilder.rst')
-rw-r--r-- | documentation/test-manual/understand-autobuilder.rst | 80 |
1 files changed, 41 insertions, 39 deletions
diff --git a/documentation/test-manual/understand-autobuilder.rst b/documentation/test-manual/understand-autobuilder.rst index 199cc97a85..6b4fab4f0b 100644 --- a/documentation/test-manual/understand-autobuilder.rst +++ b/documentation/test-manual/understand-autobuilder.rst | |||
@@ -9,31 +9,31 @@ Execution Flow within the Autobuilder | |||
9 | 9 | ||
10 | The "a-full" and "a-quick" targets are the usual entry points into the | 10 | The "a-full" and "a-quick" targets are the usual entry points into the |
11 | Autobuilder and it makes sense to follow the process through the system | 11 | Autobuilder and it makes sense to follow the process through the system |
12 | starting there. This is best visualised from the Autobuilder Console | 12 | starting there. This is best visualized from the :yocto_ab:`Autobuilder |
13 | view (:yocto_ab:`/typhoon/#/console`). | 13 | Console view </typhoon/#/console>`. |
14 | 14 | ||
15 | Each item along the top of that view represents some "target build" and | 15 | Each item along the top of that view represents some "target build" and |
16 | these targets are all run in parallel. The 'full' build will trigger the | 16 | these targets are all run in parallel. The 'full' build will trigger the |
17 | majority of them, the "quick" build will trigger some subset of them. | 17 | majority of them, the "quick" build will trigger some subset of them. |
18 | The Autobuilder effectively runs whichever configuration is defined for | 18 | The Autobuilder effectively runs whichever configuration is defined for |
19 | each of those targets on a seperate buildbot worker. To understand the | 19 | each of those targets on a separate buildbot worker. To understand the |
20 | configuration, you need to look at the entry on ``config.json`` file | 20 | configuration, you need to look at the entry on ``config.json`` file |
21 | within the ``yocto-autobuilder-helper`` repository. The targets are | 21 | within the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>` |
22 | defined in the ‘overrides' section, a quick example could be qemux86-64 | 22 | repository. The targets are defined in the ``overrides`` section, a quick |
23 | which looks like:: | 23 | example could be ``qemux86-64`` which looks like:: |
24 | 24 | ||
25 | "qemux86-64" : { | 25 | "qemux86-64" : { |
26 | "MACHINE" : "qemux86-64", | 26 | "MACHINE" : "qemux86-64", |
27 | "TEMPLATE" : "arch-qemu", | 27 | "TEMPLATE" : "arch-qemu", |
28 | "step1" : { | 28 | "step1" : { |
29 | "extravars" : [ | 29 | "extravars" : [ |
30 | "IMAGE_FSTYPES_append = ' wic wic.bmap'" | 30 | "IMAGE_FSTYPES:append = ' wic wic.bmap'" |
31 | ] | 31 | ] |
32 | } | 32 | } |
33 | }, | 33 | }, |
34 | 34 | ||
35 | And to expand that, you need the "arch-qemu" entry from | 35 | And to expand that, you need the ``arch-qemu`` entry from |
36 | the "templates" section, which looks like:: | 36 | the ``templates`` section, which looks like:: |
37 | 37 | ||
38 | "arch-qemu" : { | 38 | "arch-qemu" : { |
39 | "BUILDINFO" : true, | 39 | "BUILDINFO" : true, |
@@ -54,20 +54,20 @@ the "templates" section, which looks like:: | |||
54 | } | 54 | } |
55 | }, | 55 | }, |
56 | 56 | ||
57 | Combining these two entries you can see that "qemux86-64" is a three step build where the | 57 | Combining these two entries you can see that ``qemux86-64`` is a three step |
58 | ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for | 58 | build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` |
59 | ``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step | 59 | for each step; all for ``MACHINE="qemux86-64"`` but with differing |
60 | 1 an extra variable is added to the ``auto.conf`` file to enable wic | 60 | :term:`SDKMACHINE` settings. In step 1, an extra variable is added to the |
61 | image generation. | 61 | ``auto.conf`` file to enable wic image generation. |
62 | 62 | ||
63 | While not every detail of this is covered here, you can see how the | 63 | While not every detail of this is covered here, you can see how the |
64 | template mechanism allows quite complex configurations to be built up | 64 | template mechanism allows quite complex configurations to be built up |
65 | yet allows duplication and repetition to be kept to a minimum. | 65 | yet allows duplication and repetition to be kept to a minimum. |
66 | 66 | ||
67 | The different build targets are designed to allow for parallelisation, | 67 | The different build targets are designed to allow for parallelization, |
68 | so different machines are usually built in parallel, operations using | 68 | so different machines are usually built in parallel, operations using |
69 | the same machine and metadata are built sequentially, with the aim of | 69 | the same machine and metadata are built sequentially, with the aim of |
70 | trying to optimise build efficiency as much as possible. | 70 | trying to optimize build efficiency as much as possible. |
71 | 71 | ||
72 | The ``config.json`` file is processed by the scripts in the Helper | 72 | The ``config.json`` file is processed by the scripts in the Helper |
73 | repository in the ``scripts`` directory. The following section details | 73 | repository in the ``scripts`` directory. The following section details |
@@ -88,9 +88,9 @@ roughly consist of: | |||
88 | 88 | ||
89 | #. *Obtain yocto-autobuilder-helper* | 89 | #. *Obtain yocto-autobuilder-helper* |
90 | 90 | ||
91 | This step clones the ``yocto-autobuilder-helper`` git repository. | 91 | This step clones the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>` |
92 | This is necessary to prevent the requirement to maintain all the | 92 | git repository. This is necessary to avoid the requirement to maintain all |
93 | release or project-specific code within Buildbot. The branch chosen | 93 | the release or project-specific code within Buildbot. The branch chosen |
94 | matches the release being built so we can support older releases and | 94 | matches the release being built so we can support older releases and |
95 | still make changes in newer ones. | 95 | still make changes in newer ones. |
96 | 96 | ||
@@ -111,7 +111,7 @@ roughly consist of: | |||
111 | :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`. | 111 | :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`. |
112 | 112 | ||
113 | This step has two possible modes of operation. If the build is part | 113 | This step has two possible modes of operation. If the build is part |
114 | of a parent build, its possible that all the repositories needed may | 114 | of a parent build, it's possible that all the repositories needed may |
115 | already be available, ready in a pre-prepared directory. An "a-quick" | 115 | already be available, ready in a pre-prepared directory. An "a-quick" |
116 | or "a-full" build would prepare this before starting the other | 116 | or "a-full" build would prepare this before starting the other |
117 | sub-target builds. This is done for two reasons: | 117 | sub-target builds. This is done for two reasons: |
@@ -130,7 +130,7 @@ roughly consist of: | |||
130 | 130 | ||
131 | #. *Call scripts/run-config* | 131 | #. *Call scripts/run-config* |
132 | 132 | ||
133 | This is another call into the Helper scripts where its expected that | 133 | This is another call into the Helper scripts where it's expected that |
134 | the main functionality of this target will be executed. | 134 | the main functionality of this target will be executed. |
135 | 135 | ||
136 | Autobuilder Technology | 136 | Autobuilder Technology |
@@ -163,16 +163,17 @@ Autobuilder Worker Janitor | |||
163 | -------------------------- | 163 | -------------------------- |
164 | 164 | ||
165 | This is a process running on each Worker that performs two basic | 165 | This is a process running on each Worker that performs two basic |
166 | operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and | 166 | operations, including background file deletion at IO idle (see |
167 | maintainenance of a cache of cloned repositories to improve the speed | 167 | "Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`) |
168 | and maintenance of a cache of cloned repositories to improve the speed | ||
168 | the system can checkout repositories. | 169 | the system can checkout repositories. |
169 | 170 | ||
170 | Shared DL_DIR | 171 | Shared DL_DIR |
171 | ------------- | 172 | ------------- |
172 | 173 | ||
173 | The Workers are all connected over NFS which allows DL_DIR to be shared | 174 | The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared |
174 | between them. This reduces network accesses from the system and allows | 175 | between them. This reduces network accesses from the system and allows |
175 | the build to be sped up. Usage of the directory within the build system | 176 | the build to be sped up. The usage of the directory within the build system |
176 | is designed to be able to be shared over NFS. | 177 | is designed to be able to be shared over NFS. |
177 | 178 | ||
178 | Shared SSTATE_DIR | 179 | Shared SSTATE_DIR |
@@ -180,8 +181,8 @@ Shared SSTATE_DIR | |||
180 | 181 | ||
181 | The Workers are all connected over NFS which allows the ``sstate`` | 182 | The Workers are all connected over NFS which allows the ``sstate`` |
182 | directory to be shared between them. This means once a Worker has built | 183 | directory to be shared between them. This means once a Worker has built |
183 | an artifact, all the others can benefit from it. Usage of the directory | 184 | an artifact, all the others can benefit from it. The usage of the directory |
184 | within the directory is designed for sharing over NFS. | 185 | within the build system is designed for sharing over NFS. |
185 | 186 | ||
186 | Resulttool | 187 | Resulttool |
187 | ---------- | 188 | ---------- |
@@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as | |||
192 | failure logs or the time taken to run the tests, may also be included. | 193 | failure logs or the time taken to run the tests, may also be included. |
193 | 194 | ||
194 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these | 195 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these |
195 | json results files. It has the ability to merge files together, display | 196 | JSON results files. It has the ability to merge files together, display |
196 | reports of the test results and compare different result files. | 197 | reports of the test results and compare different result files. |
197 | 198 | ||
198 | For details, see :yocto_wiki:`/Resulttool`. | 199 | For details, see :yocto_wiki:`/Resulttool`. |
@@ -204,9 +205,9 @@ The ``scripts/run-config`` execution is where most of the work within | |||
204 | the Autobuilder happens. It runs through a number of steps; the first | 205 | the Autobuilder happens. It runs through a number of steps; the first |
205 | are general setup steps that are run once and include: | 206 | are general setup steps that are run once and include: |
206 | 207 | ||
207 | #. Set up any ``buildtools-tarball`` if configured. | 208 | #. Set up any :term:`buildtools` tarball if configured. |
208 | 209 | ||
209 | #. Call "buildhistory-init" if buildhistory is configured. | 210 | #. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured. |
210 | 211 | ||
211 | For each step that is configured in ``config.json``, it will perform the | 212 | For each step that is configured in ``config.json``, it will perform the |
212 | following: | 213 | following: |
@@ -242,7 +243,7 @@ of post-build steps, including: | |||
242 | #. Call ``scripts/upload-error-reports`` to send any error reports | 243 | #. Call ``scripts/upload-error-reports`` to send any error reports |
243 | generated to the remote server. | 244 | generated to the remote server. |
244 | 245 | ||
245 | #. Cleanup the build directory using | 246 | #. Cleanup the :term:`Build Directory` using |
246 | :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful, | 247 | :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful, |
247 | else rename it to "build-renamed" for potential future debugging. | 248 | else rename it to "build-renamed" for potential future debugging. |
248 | 249 | ||
@@ -250,15 +251,16 @@ Deploying Yocto Autobuilder | |||
250 | =========================== | 251 | =========================== |
251 | 252 | ||
252 | The most up to date information about how to setup and deploy your own | 253 | The most up to date information about how to setup and deploy your own |
253 | Autbuilder can be found in README.md in the ``yocto-autobuilder2`` | 254 | Autobuilder can be found in :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>` |
254 | repository. | 255 | in the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>` repository. |
255 | 256 | ||
256 | We hope that people can use the ``yocto-autobuilder2`` code directly but | 257 | We hope that people can use the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>` |
257 | it is inevitable that users will end up needing to heavily customise the | 258 | code directly but it is inevitable that users will end up needing to heavily |
258 | ``yocto-autobuilder-helper`` repository, particularly the | 259 | customize the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>` |
259 | ``config.json`` file as they will want to define their own test matrix. | 260 | repository, particularly the ``config.json`` file as they will want to define |
261 | their own test matrix. | ||
260 | 262 | ||
261 | The Autobuilder supports wo customization options: | 263 | The Autobuilder supports two customization options: |
262 | 264 | ||
263 | - variable substitution | 265 | - variable substitution |
264 | 266 | ||
@@ -278,7 +280,7 @@ environment:: | |||
278 | $ ABHELPER_JSON="config.json /some/location/local.json" | 280 | $ ABHELPER_JSON="config.json /some/location/local.json" |
279 | 281 | ||
280 | One issue users often run into is validation of the ``config.json`` files. A | 282 | One issue users often run into is validation of the ``config.json`` files. A |
281 | tip for minimizing issues from invalid json files is to use a Git | 283 | tip for minimizing issues from invalid JSON files is to use a Git |
282 | ``pre-commit-hook.sh`` script to verify the JSON file before committing | 284 | ``pre-commit-hook.sh`` script to verify the JSON file before committing |
283 | it. Create a symbolic link as follows:: | 285 | it. Create a symbolic link as follows:: |
284 | 286 | ||