diff options
author | Nicolas Dechesne <nicolas.dechesne@linaro.org> | 2020-11-13 00:21:41 +0100 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2021-01-04 10:55:00 +0000 |
commit | 22384b2fe567406b3c318b2edd0d23dee0d72023 (patch) | |
tree | b19414c699324722f371d4138c305e925a43da95 /bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | |
parent | 16d477fa5d2bec49ccd83ede8249431d30e0c80c (diff) | |
download | poky-22384b2fe567406b3c318b2edd0d23dee0d72023.tar.gz |
bitbake: sphinx: import sphinx docs
The Bitbake manual was migrated to Sphinx in Yocto Project 3.2. Since
the docs between 3.2 and 3.1 are "similar", and since 3.1 is an LTS
release, we agreed to backport the documentation onto 3.1.
If we look at all docs changes in 3.1 and 3.2, we have the following:
=== Changes in 3.1
git log --oneline b94dec477a8d48ebceec91952ba290798c56c1f5..origin/1.46 -- doc/
====
324aaa7f bitbake-user-manual-metadata.xml: fix a minor error
f92e19a3 doc: More explanation to tasks that recursively depend on themselves
e4695176 doc: Clarify how task dependencies relate to RDEPENDS
25c5c79b user manual: properly tag content as <replaceable>
be367887 docs: delete reference to obsolete recipe-depends.dot
=== Changes in 3.2/master
git log --oneline b94dec477a8d48ebceec91952ba290798c56c1f5..origin/master -- doc/
====
a7c47f1e sphinx: rename Makefile.sphinx
427721d8 sphinx: remove DocBook files
d52190ea docs: static: theme_overrides.css: fix responsive design on <640px screens
9ae5cce7 docs: sphinx: report errors when dependencies are not met
ec4c481a docs: update README file after migrationg to Sphinx
c87cc35a docs: sphinx: replace special quotes with double quotes
ebdeef2c docs: ref-variables: add links to terms in glossary
29081375 bitbake-user-manual: fix bad links
a0f37789 sphinx: theme_override: Use bold for emphasis text
cbc5ca48 sphinx: theme_override: properly set font for verbatim text
08b1ae23 sphinx: remove leading '/'
99ba6fe9 sphinx: update style for important, caution and warnings
d99760cc sphinx: last manual round of fixes/improvements
4f94633a sphinx: bitbake-user-manual: insert additional blank line after title
63adcaa5 sphinx: add releases page
3e940d93 sphinx: conf: enable extlinks extension
9921c652 sphinx: index: move the boilerplate at the end of the page
4e461224 sphinx: add SPDX headers
cb19159c sphinx: Enhance the sphinx experience/nagivation with:
10a54678 sphinx: tweak html output a bit
219b2348 sphinx: Makefile.sphinx: add clean and publish targets
35fdc185 sphinx: fixes all remaining warnings
e11d2dd1 sphinx: fix links inside notes
57300955 sphinx: fixup for links
fa304c01 sphinx: override theme CSS
29af1cd2 sphinx: switch to readthedocs theme
e8359fd8 sphinx: bitbake-user-manual: use builtin sphinx glossary
6bf6c8d6 sphinx: initial sphinx support
84ccba0f sphinx: add initial build infrastructure
44b57216 bitbake-user-manual: update perforce fetcher docs
9186ca47 bitbake-user-manual: Add BBFILES_DYNAMIC
7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example
06b5cf0a bitbake-user-manual-metadata.xml: fix a minor error
c92a266c doc: More explanation to tasks that recursively depend on themselves
caf42243 doc: Clarify how task dependencies relate to RDEPENDS
647c13d4 user manual: properly tag content as <replaceable>
2effbb6e docs: delete reference to obsolete recipe-depends.dot
We can conclude the following commits exist in 3.2 and not in 3.1 (if
we filter out sphinx changes)
44b57216 bitbake-user-manual: update perforce fetcher docs
9186ca47 bitbake-user-manual: Add BBFILES_DYNAMIC
7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example
Out of these 3 changes, the following patches are for 3.2 only:
44b57216 bitbake-user-manual: update perforce fetcher docs
7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example
To backport the Sphinx docs, we then need to cherry-pick all docs
patches from 3.2/1.48 and 'undo' the two patches above.
This first patch is the first step that imports all Sphinx files, and
remove Docbook files. It was done with the following command:
git cherry-pick -n \
$(git log --reverse --oneline \
b94dec477a8d48ebceec91952ba290798c56c1f5..origin/master -- doc/ \
| cut -f1 -d' ')
(Bitbake rev: cd68f14031eb45006b44d10b348e35c69ac21ad0)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | 733 |
1 files changed, 733 insertions, 0 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst new file mode 100644 index 0000000000..d74e768f69 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | |||
@@ -0,0 +1,733 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.5 | ||
2 | |||
3 | ========= | ||
4 | Execution | ||
5 | ========= | ||
6 | |||
7 | | | ||
8 | |||
9 | The primary purpose for running BitBake is to produce some kind of | ||
10 | output such as a single installable package, a kernel, a software | ||
11 | development kit, or even a full, board-specific bootable Linux image, | ||
12 | complete with bootloader, kernel, and root filesystem. Of course, you | ||
13 | can execute the ``bitbake`` command with options that cause it to | ||
14 | execute single tasks, compile single recipe files, capture or clear | ||
15 | data, or simply return information about the execution environment. | ||
16 | |||
17 | This chapter describes BitBake's execution process from start to finish | ||
18 | when you use it to create an image. The execution process is launched | ||
19 | using the following command form: :: | ||
20 | |||
21 | $ bitbake target | ||
22 | |||
23 | For information on | ||
24 | the BitBake command and its options, see ":ref:`The BitBake Command | ||
25 | <bitbake-user-manual-command>`" section. | ||
26 | |||
27 | .. note:: | ||
28 | |||
29 | Prior to executing BitBake, you should take advantage of available | ||
30 | parallel thread execution on your build host by setting the | ||
31 | :term:`BB_NUMBER_THREADS` variable in | ||
32 | your project's ``local.conf`` configuration file. | ||
33 | |||
34 | A common method to determine this value for your build host is to run | ||
35 | the following: :: | ||
36 | |||
37 | $ grep processor /proc/cpuinfo | ||
38 | |||
39 | This command returns | ||
40 | the number of processors, which takes into account hyper-threading. | ||
41 | Thus, a quad-core build host with hyper-threading most likely shows | ||
42 | eight processors, which is the value you would then assign to | ||
43 | ``BB_NUMBER_THREADS``. | ||
44 | |||
45 | A possibly simpler solution is that some Linux distributions (e.g. | ||
46 | Debian and Ubuntu) provide the ``ncpus`` command. | ||
47 | |||
48 | Parsing the Base Configuration Metadata | ||
49 | ======================================= | ||
50 | |||
51 | The first thing BitBake does is parse base configuration metadata. Base | ||
52 | configuration metadata consists of your project's ``bblayers.conf`` file | ||
53 | to determine what layers BitBake needs to recognize, all necessary | ||
54 | ``layer.conf`` files (one from each layer), and ``bitbake.conf``. The | ||
55 | data itself is of various types: | ||
56 | |||
57 | - **Recipes:** Details about particular pieces of software. | ||
58 | |||
59 | - **Class Data:** An abstraction of common build information (e.g. how to | ||
60 | build a Linux kernel). | ||
61 | |||
62 | - **Configuration Data:** Machine-specific settings, policy decisions, | ||
63 | and so forth. Configuration data acts as the glue to bind everything | ||
64 | together. | ||
65 | |||
66 | The ``layer.conf`` files are used to construct key variables such as | ||
67 | :term:`BBPATH` and :term:`BBFILES`. | ||
68 | ``BBPATH`` is used to search for configuration and class files under the | ||
69 | ``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used | ||
70 | to locate both recipe and recipe append files (``.bb`` and | ||
71 | ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the | ||
72 | user has set the ``BBPATH`` and ``BBFILES`` directly in the environment. | ||
73 | |||
74 | Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable | ||
75 | that was just constructed. The ``bitbake.conf`` file may also include | ||
76 | other configuration files using the ``include`` or ``require`` | ||
77 | directives. | ||
78 | |||
79 | Prior to parsing configuration files, BitBake looks at certain | ||
80 | variables, including: | ||
81 | |||
82 | - :term:`BB_ENV_WHITELIST` | ||
83 | - :term:`BB_ENV_EXTRAWHITE` | ||
84 | - :term:`BB_PRESERVE_ENV` | ||
85 | - :term:`BB_ORIGENV` | ||
86 | - :term:`BITBAKE_UI` | ||
87 | |||
88 | The first four variables in this list relate to how BitBake treats shell | ||
89 | environment variables during task execution. By default, BitBake cleans | ||
90 | the environment variables and provides tight control over the shell | ||
91 | execution environment. However, through the use of these first four | ||
92 | variables, you can apply your control regarding the environment | ||
93 | variables allowed to be used by BitBake in the shell during execution of | ||
94 | tasks. See the | ||
95 | ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`" | ||
96 | section and the information about these variables in the variable | ||
97 | glossary for more information on how they work and on how to use them. | ||
98 | |||
99 | The base configuration metadata is global and therefore affects all | ||
100 | recipes and tasks that are executed. | ||
101 | |||
102 | BitBake first searches the current working directory for an optional | ||
103 | ``conf/bblayers.conf`` configuration file. This file is expected to | ||
104 | contain a :term:`BBLAYERS` variable that is a | ||
105 | space-delimited list of 'layer' directories. Recall that if BitBake | ||
106 | cannot find a ``bblayers.conf`` file, then it is assumed the user has | ||
107 | set the ``BBPATH`` and ``BBFILES`` variables directly in the | ||
108 | environment. | ||
109 | |||
110 | For each directory (layer) in this list, a ``conf/layer.conf`` file is | ||
111 | located and parsed with the :term:`LAYERDIR` variable | ||
112 | being set to the directory where the layer was found. The idea is these | ||
113 | files automatically set up :term:`BBPATH` and other | ||
114 | variables correctly for a given build directory. | ||
115 | |||
116 | BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in | ||
117 | the user-specified ``BBPATH``. That configuration file generally has | ||
118 | include directives to pull in any other metadata such as files specific | ||
119 | to the architecture, the machine, the local environment, and so forth. | ||
120 | |||
121 | Only variable definitions and include directives are allowed in BitBake | ||
122 | ``.conf`` files. Some variables directly influence BitBake's behavior. | ||
123 | These variables might have been set from the environment depending on | ||
124 | the environment variables previously mentioned or set in the | ||
125 | configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`" | ||
126 | chapter presents a full list of | ||
127 | variables. | ||
128 | |||
129 | After parsing configuration files, BitBake uses its rudimentary | ||
130 | inheritance mechanism, which is through class files, to inherit some | ||
131 | standard classes. BitBake parses a class when the inherit directive | ||
132 | responsible for getting that class is encountered. | ||
133 | |||
134 | The ``base.bbclass`` file is always included. Other classes that are | ||
135 | specified in the configuration using the | ||
136 | :term:`INHERIT` variable are also included. BitBake | ||
137 | searches for class files in a ``classes`` subdirectory under the paths | ||
138 | in ``BBPATH`` in the same way as configuration files. | ||
139 | |||
140 | A good way to get an idea of the configuration files and the class files | ||
141 | used in your execution environment is to run the following BitBake | ||
142 | command: :: | ||
143 | |||
144 | $ bitbake -e > mybb.log | ||
145 | |||
146 | Examining the top of the ``mybb.log`` | ||
147 | shows you the many configuration files and class files used in your | ||
148 | execution environment. | ||
149 | |||
150 | .. note:: | ||
151 | |||
152 | You need to be aware of how BitBake parses curly braces. If a recipe | ||
153 | uses a closing curly brace within the function and the character has | ||
154 | no leading spaces, BitBake produces a parsing error. If you use a | ||
155 | pair of curly braces in a shell function, the closing curly brace | ||
156 | must not be located at the start of the line without leading spaces. | ||
157 | |||
158 | Here is an example that causes BitBake to produce a parsing error: :: | ||
159 | |||
160 | fakeroot create_shar() { | ||
161 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh | ||
162 | usage() | ||
163 | { | ||
164 | echo "test" | ||
165 | ###### The following "}" at the start of the line causes a parsing error ###### | ||
166 | } | ||
167 | EOF | ||
168 | } | ||
169 | |||
170 | Writing the recipe this way avoids the error: | ||
171 | fakeroot create_shar() { | ||
172 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh | ||
173 | usage() | ||
174 | { | ||
175 | echo "test" | ||
176 | ###### The following "}" with a leading space at the start of the line avoids the error ###### | ||
177 | } | ||
178 | EOF | ||
179 | } | ||
180 | |||
181 | Locating and Parsing Recipes | ||
182 | ============================ | ||
183 | |||
184 | During the configuration phase, BitBake will have set | ||
185 | :term:`BBFILES`. BitBake now uses it to construct a | ||
186 | list of recipes to parse, along with any append files (``.bbappend``) to | ||
187 | apply. ``BBFILES`` is a space-separated list of available files and | ||
188 | supports wildcards. An example would be: :: | ||
189 | |||
190 | BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" | ||
191 | |||
192 | BitBake parses each | ||
193 | recipe and append file located with ``BBFILES`` and stores the values of | ||
194 | various variables into the datastore. | ||
195 | |||
196 | .. note:: | ||
197 | |||
198 | Append files are applied in the order they are encountered in BBFILES. | ||
199 | |||
200 | For each file, a fresh copy of the base configuration is made, then the | ||
201 | recipe is parsed line by line. Any inherit statements cause BitBake to | ||
202 | find and then parse class files (``.bbclass``) using | ||
203 | :term:`BBPATH` as the search path. Finally, BitBake | ||
204 | parses in order any append files found in ``BBFILES``. | ||
205 | |||
206 | One common convention is to use the recipe filename to define pieces of | ||
207 | metadata. For example, in ``bitbake.conf`` the recipe name and version | ||
208 | are used to set the variables :term:`PN` and | ||
209 | :term:`PV`: :: | ||
210 | |||
211 | PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" | ||
212 | PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" | ||
213 | |||
214 | In this example, a recipe called "something_1.2.3.bb" would set | ||
215 | ``PN`` to "something" and ``PV`` to "1.2.3". | ||
216 | |||
217 | By the time parsing is complete for a recipe, BitBake has a list of | ||
218 | tasks that the recipe defines and a set of data consisting of keys and | ||
219 | values as well as dependency information about the tasks. | ||
220 | |||
221 | BitBake does not need all of this information. It only needs a small | ||
222 | subset of the information to make decisions about the recipe. | ||
223 | Consequently, BitBake caches the values in which it is interested and | ||
224 | does not store the rest of the information. Experience has shown it is | ||
225 | faster to re-parse the metadata than to try and write it out to the disk | ||
226 | and then reload it. | ||
227 | |||
228 | Where possible, subsequent BitBake commands reuse this cache of recipe | ||
229 | information. The validity of this cache is determined by first computing | ||
230 | a checksum of the base configuration data (see | ||
231 | :term:`BB_HASHCONFIG_WHITELIST`) and | ||
232 | then checking if the checksum matches. If that checksum matches what is | ||
233 | in the cache and the recipe and class files have not changed, BitBake is | ||
234 | able to use the cache. BitBake then reloads the cached information about | ||
235 | the recipe instead of reparsing it from scratch. | ||
236 | |||
237 | Recipe file collections exist to allow the user to have multiple | ||
238 | repositories of ``.bb`` files that contain the same exact package. For | ||
239 | example, one could easily use them to make one's own local copy of an | ||
240 | upstream repository, but with custom modifications that one does not | ||
241 | want upstream. Here is an example: :: | ||
242 | |||
243 | BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb" | ||
244 | BBFILE_COLLECTIONS = "upstream local" | ||
245 | BBFILE_PATTERN_upstream = "^/stuff/openembedded/" | ||
246 | BBFILE_PATTERN_local = "^/stuff/openembedded.modified/" | ||
247 | BBFILE_PRIORITY_upstream = "5" BBFILE_PRIORITY_local = "10" | ||
248 | |||
249 | .. note:: | ||
250 | |||
251 | The layers mechanism is now the preferred method of collecting code. | ||
252 | While the collections code remains, its main use is to set layer | ||
253 | priorities and to deal with overlap (conflicts) between layers. | ||
254 | |||
255 | .. _bb-bitbake-providers: | ||
256 | |||
257 | Providers | ||
258 | ========= | ||
259 | |||
260 | Assuming BitBake has been instructed to execute a target and that all | ||
261 | the recipe files have been parsed, BitBake starts to figure out how to | ||
262 | build the target. BitBake looks through the ``PROVIDES`` list for each | ||
263 | of the recipes. A ``PROVIDES`` list is the list of names by which the | ||
264 | recipe can be known. Each recipe's ``PROVIDES`` list is created | ||
265 | implicitly through the recipe's :term:`PN` variable and | ||
266 | explicitly through the recipe's :term:`PROVIDES` | ||
267 | variable, which is optional. | ||
268 | |||
269 | When a recipe uses ``PROVIDES``, that recipe's functionality can be | ||
270 | found under an alternative name or names other than the implicit ``PN`` | ||
271 | name. As an example, suppose a recipe named ``keyboard_1.0.bb`` | ||
272 | contained the following: :: | ||
273 | |||
274 | PROVIDES += "fullkeyboard" | ||
275 | |||
276 | The ``PROVIDES`` | ||
277 | list for this recipe becomes "keyboard", which is implicit, and | ||
278 | "fullkeyboard", which is explicit. Consequently, the functionality found | ||
279 | in ``keyboard_1.0.bb`` can be found under two different names. | ||
280 | |||
281 | .. _bb-bitbake-preferences: | ||
282 | |||
283 | Preferences | ||
284 | =========== | ||
285 | |||
286 | The ``PROVIDES`` list is only part of the solution for figuring out a | ||
287 | target's recipes. Because targets might have multiple providers, BitBake | ||
288 | needs to prioritize providers by determining provider preferences. | ||
289 | |||
290 | A common example in which a target has multiple providers is | ||
291 | "virtual/kernel", which is on the ``PROVIDES`` list for each kernel | ||
292 | recipe. Each machine often selects the best kernel provider by using a | ||
293 | line similar to the following in the machine configuration file: :: | ||
294 | |||
295 | PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" | ||
296 | |||
297 | The default :term:`PREFERRED_PROVIDER` is the provider | ||
298 | with the same name as the target. BitBake iterates through each target | ||
299 | it needs to build and resolves them and their dependencies using this | ||
300 | process. | ||
301 | |||
302 | Understanding how providers are chosen is made complicated by the fact | ||
303 | that multiple versions might exist for a given provider. BitBake | ||
304 | defaults to the highest version of a provider. Version comparisons are | ||
305 | made using the same method as Debian. You can use the | ||
306 | :term:`PREFERRED_VERSION` variable to | ||
307 | specify a particular version. You can influence the order by using the | ||
308 | :term:`DEFAULT_PREFERENCE` variable. | ||
309 | |||
310 | By default, files have a preference of "0". Setting | ||
311 | ``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used | ||
312 | unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to | ||
313 | "1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides | ||
314 | any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used | ||
315 | to mark newer and more experimental recipe versions until they have | ||
316 | undergone sufficient testing to be considered stable. | ||
317 | |||
318 | When there are multiple "versions" of a given recipe, BitBake defaults | ||
319 | to selecting the most recent version, unless otherwise specified. If the | ||
320 | recipe in question has a | ||
321 | :term:`DEFAULT_PREFERENCE` set lower than | ||
322 | the other recipes (default is 0), then it will not be selected. This | ||
323 | allows the person or persons maintaining the repository of recipe files | ||
324 | to specify their preference for the default selected version. | ||
325 | Additionally, the user can specify their preferred version. | ||
326 | |||
327 | If the first recipe is named ``a_1.1.bb``, then the | ||
328 | :term:`PN` variable will be set to "a", and the | ||
329 | :term:`PV` variable will be set to 1.1. | ||
330 | |||
331 | Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by | ||
332 | default. However, if you define the following variable in a ``.conf`` | ||
333 | file that BitBake parses, you can change that preference: :: | ||
334 | |||
335 | PREFERRED_VERSION_a = "1.1" | ||
336 | |||
337 | .. note:: | ||
338 | |||
339 | It is common for a recipe to provide two versions -- a stable, | ||
340 | numbered (and preferred) version, and a version that is automatically | ||
341 | checked out from a source code repository that is considered more | ||
342 | "bleeding edge" but can be selected only explicitly. | ||
343 | |||
344 | For example, in the OpenEmbedded codebase, there is a standard, | ||
345 | versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there | ||
346 | is also a Git-based version, ``busybox_git.bb``, which explicitly | ||
347 | contains the line :: | ||
348 | |||
349 | DEFAULT_PREFERENCE = "-1" | ||
350 | |||
351 | to ensure that the | ||
352 | numbered, stable version is always preferred unless the developer | ||
353 | selects otherwise. | ||
354 | |||
355 | .. _bb-bitbake-dependencies: | ||
356 | |||
357 | Dependencies | ||
358 | ============ | ||
359 | |||
360 | Each target BitBake builds consists of multiple tasks such as ``fetch``, | ||
361 | ``unpack``, ``patch``, ``configure``, and ``compile``. For best | ||
362 | performance on multi-core systems, BitBake considers each task as an | ||
363 | independent entity with its own set of dependencies. | ||
364 | |||
365 | Dependencies are defined through several variables. You can find | ||
366 | information about variables BitBake uses in the | ||
367 | :doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a | ||
368 | basic level, it is sufficient to know that BitBake uses the | ||
369 | :term:`DEPENDS` and | ||
370 | :term:`RDEPENDS` variables when calculating | ||
371 | dependencies. | ||
372 | |||
373 | For more information on how BitBake handles dependencies, see the | ||
374 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies` | ||
375 | section. | ||
376 | |||
377 | .. _ref-bitbake-tasklist: | ||
378 | |||
379 | The Task List | ||
380 | ============= | ||
381 | |||
382 | Based on the generated list of providers and the dependency information, | ||
383 | BitBake can now calculate exactly what tasks it needs to run and in what | ||
384 | order it needs to run them. The | ||
385 | :ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks` | ||
386 | section has more information on how BitBake chooses which task to | ||
387 | execute next. | ||
388 | |||
389 | The build now starts with BitBake forking off threads up to the limit | ||
390 | set in the :term:`BB_NUMBER_THREADS` | ||
391 | variable. BitBake continues to fork threads as long as there are tasks | ||
392 | ready to run, those tasks have all their dependencies met, and the | ||
393 | thread threshold has not been exceeded. | ||
394 | |||
395 | It is worth noting that you can greatly speed up the build time by | ||
396 | properly setting the ``BB_NUMBER_THREADS`` variable. | ||
397 | |||
398 | As each task completes, a timestamp is written to the directory | ||
399 | specified by the :term:`STAMP` variable. On subsequent | ||
400 | runs, BitBake looks in the build directory within ``tmp/stamps`` and | ||
401 | does not rerun tasks that are already completed unless a timestamp is | ||
402 | found to be invalid. Currently, invalid timestamps are only considered | ||
403 | on a per recipe file basis. So, for example, if the configure stamp has | ||
404 | a timestamp greater than the compile timestamp for a given target, then | ||
405 | the compile task would rerun. Running the compile task again, however, | ||
406 | has no effect on other providers that depend on that target. | ||
407 | |||
408 | The exact format of the stamps is partly configurable. In modern | ||
409 | versions of BitBake, a hash is appended to the stamp so that if the | ||
410 | configuration changes, the stamp becomes invalid and the task is | ||
411 | automatically rerun. This hash, or signature used, is governed by the | ||
412 | signature policy that is configured (see the | ||
413 | :ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)` | ||
414 | section for information). It is also | ||
415 | possible to append extra metadata to the stamp using the | ||
416 | ``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this | ||
417 | flag to make some tasks machine-specific. | ||
418 | |||
419 | .. note:: | ||
420 | |||
421 | Some tasks are marked as "nostamp" tasks. No timestamp file is | ||
422 | created when these tasks are run. Consequently, "nostamp" tasks are | ||
423 | always rerun. | ||
424 | |||
425 | For more information on tasks, see the | ||
426 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section. | ||
427 | |||
428 | Executing Tasks | ||
429 | =============== | ||
430 | |||
431 | Tasks can be either a shell task or a Python task. For shell tasks, | ||
432 | BitBake writes a shell script to | ||
433 | ``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then | ||
434 | executes the script. The generated shell script contains all the | ||
435 | exported variables, and the shell functions with all variables expanded. | ||
436 | Output from the shell script goes to the file | ||
437 | ``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in | ||
438 | the run file and the output in the log files is a useful debugging | ||
439 | technique. | ||
440 | |||
441 | For Python tasks, BitBake executes the task internally and logs | ||
442 | information to the controlling terminal. Future versions of BitBake will | ||
443 | write the functions to files similar to the way shell tasks are handled. | ||
444 | Logging will be handled in a way similar to shell tasks as well. | ||
445 | |||
446 | The order in which BitBake runs the tasks is controlled by its task | ||
447 | scheduler. It is possible to configure the scheduler and define custom | ||
448 | implementations for specific use cases. For more information, see these | ||
449 | variables that control the behavior: | ||
450 | |||
451 | - :term:`BB_SCHEDULER` | ||
452 | |||
453 | - :term:`BB_SCHEDULERS` | ||
454 | |||
455 | It is possible to have functions run before and after a task's main | ||
456 | function. This is done using the ``[prefuncs]`` and ``[postfuncs]`` | ||
457 | flags of the task that lists the functions to run. | ||
458 | |||
459 | .. _checksums: | ||
460 | |||
461 | Checksums (Signatures) | ||
462 | ====================== | ||
463 | |||
464 | A checksum is a unique signature of a task's inputs. The signature of a | ||
465 | task can be used to determine if a task needs to be run. Because it is a | ||
466 | change in a task's inputs that triggers running the task, BitBake needs | ||
467 | to detect all the inputs to a given task. For shell tasks, this turns | ||
468 | out to be fairly easy because BitBake generates a "run" shell script for | ||
469 | each task and it is possible to create a checksum that gives you a good | ||
470 | idea of when the task's data changes. | ||
471 | |||
472 | To complicate the problem, some things should not be included in the | ||
473 | checksum. First, there is the actual specific build path of a given task | ||
474 | - the working directory. It does not matter if the working directory | ||
475 | changes because it should not affect the output for target packages. The | ||
476 | simplistic approach for excluding the working directory is to set it to | ||
477 | some fixed value and create the checksum for the "run" script. BitBake | ||
478 | goes one step better and uses the | ||
479 | :term:`BB_HASHBASE_WHITELIST` variable | ||
480 | to define a list of variables that should never be included when | ||
481 | generating the signatures. | ||
482 | |||
483 | Another problem results from the "run" scripts containing functions that | ||
484 | might or might not get called. The incremental build solution contains | ||
485 | code that figures out dependencies between shell functions. This code is | ||
486 | used to prune the "run" scripts down to the minimum set, thereby | ||
487 | alleviating this problem and making the "run" scripts much more readable | ||
488 | as a bonus. | ||
489 | |||
490 | So far we have solutions for shell scripts. What about Python tasks? The | ||
491 | same approach applies even though these tasks are more difficult. The | ||
492 | process needs to figure out what variables a Python function accesses | ||
493 | and what functions it calls. Again, the incremental build solution | ||
494 | contains code that first figures out the variable and function | ||
495 | dependencies, and then creates a checksum for the data used as the input | ||
496 | to the task. | ||
497 | |||
498 | Like the working directory case, situations exist where dependencies | ||
499 | should be ignored. For these cases, you can instruct the build process | ||
500 | to ignore a dependency by using a line like the following: :: | ||
501 | |||
502 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
503 | |||
504 | This example ensures that the | ||
505 | ``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``, | ||
506 | even if it does reference it. | ||
507 | |||
508 | Equally, there are cases where we need to add dependencies BitBake is | ||
509 | not able to find. You can accomplish this by using a line like the | ||
510 | following: :: | ||
511 | |||
512 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
513 | |||
514 | This example explicitly | ||
515 | adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. | ||
516 | |||
517 | Consider a case with in-line Python, for example, where BitBake is not | ||
518 | able to figure out dependencies. When running in debug mode (i.e. using | ||
519 | ``-DDD``), BitBake produces output when it discovers something for which | ||
520 | it cannot figure out dependencies. | ||
521 | |||
522 | Thus far, this section has limited discussion to the direct inputs into | ||
523 | a task. Information based on direct inputs is referred to as the | ||
524 | "basehash" in the code. However, there is still the question of a task's | ||
525 | indirect inputs - the things that were already built and present in the | ||
526 | build directory. The checksum (or signature) for a particular task needs | ||
527 | to add the hashes of all the tasks on which the particular task depends. | ||
528 | Choosing which dependencies to add is a policy decision. However, the | ||
529 | effect is to generate a master checksum that combines the basehash and | ||
530 | the hashes of the task's dependencies. | ||
531 | |||
532 | At the code level, there are a variety of ways both the basehash and the | ||
533 | dependent task hashes can be influenced. Within the BitBake | ||
534 | configuration file, we can give BitBake some extra information to help | ||
535 | it construct the basehash. The following statement effectively results | ||
536 | in a list of global variable dependency excludes - variables never | ||
537 | included in any checksum. This example uses variables from OpenEmbedded | ||
538 | to help illustrate the concept: :: | ||
539 | |||
540 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ | ||
541 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ | ||
542 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ | ||
543 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ | ||
544 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
545 | |||
546 | The previous example excludes the work directory, which is part of | ||
547 | ``TMPDIR``. | ||
548 | |||
549 | The rules for deciding which hashes of dependent tasks to include | ||
550 | through dependency chains are more complex and are generally | ||
551 | accomplished with a Python function. The code in | ||
552 | ``meta/lib/oe/sstatesig.py`` shows two examples of this and also | ||
553 | illustrates how you can insert your own policy into the system if so | ||
554 | desired. This file defines the two basic signature generators | ||
555 | OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there | ||
556 | is a dummy "noop" signature handler enabled in BitBake. This means that | ||
557 | behavior is unchanged from previous versions. ``OE-Core`` uses the | ||
558 | "OEBasicHash" signature handler by default through this setting in the | ||
559 | ``bitbake.conf`` file: :: | ||
560 | |||
561 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
562 | |||
563 | The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic" | ||
564 | version but adds the task hash to the stamp files. This results in any | ||
565 | metadata change that changes the task hash, automatically causing the | ||
566 | task to be run again. This removes the need to bump | ||
567 | :term:`PR` values, and changes to metadata automatically | ||
568 | ripple across the build. | ||
569 | |||
570 | It is also worth noting that the end result of these signature | ||
571 | generators is to make some dependency and hash information available to | ||
572 | the build. This information includes: | ||
573 | |||
574 | - ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the | ||
575 | recipe. | ||
576 | |||
577 | - ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each | ||
578 | dependent task. | ||
579 | |||
580 | - ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for | ||
581 | each task. | ||
582 | |||
583 | - ``BB_TASKHASH``: The hash of the currently running task. | ||
584 | |||
585 | It is worth noting that BitBake's "-S" option lets you debug BitBake's | ||
586 | processing of signatures. The options passed to -S allow different | ||
587 | debugging modes to be used, either using BitBake's own debug functions | ||
588 | or possibly those defined in the metadata/signature handler itself. The | ||
589 | simplest parameter to pass is "none", which causes a set of signature | ||
590 | information to be written out into ``STAMPS_DIR`` corresponding to the | ||
591 | targets specified. The other currently available parameter is | ||
592 | "printdiff", which causes BitBake to try to establish the closest | ||
593 | signature match it can (e.g. in the sstate cache) and then run | ||
594 | ``bitbake-diffsigs`` over the matches to determine the stamps and delta | ||
595 | where these two stamp trees diverge. | ||
596 | |||
597 | .. note:: | ||
598 | |||
599 | It is likely that future versions of BitBake will provide other | ||
600 | signature handlers triggered through additional "-S" parameters. | ||
601 | |||
602 | You can find more information on checksum metadata in the | ||
603 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene` | ||
604 | section. | ||
605 | |||
606 | Setscene | ||
607 | ======== | ||
608 | |||
609 | The setscene process enables BitBake to handle "pre-built" artifacts. | ||
610 | The ability to handle and reuse these artifacts allows BitBake the | ||
611 | luxury of not having to build something from scratch every time. | ||
612 | Instead, BitBake can use, when possible, existing build artifacts. | ||
613 | |||
614 | BitBake needs to have reliable data indicating whether or not an | ||
615 | artifact is compatible. Signatures, described in the previous section, | ||
616 | provide an ideal way of representing whether an artifact is compatible. | ||
617 | If a signature is the same, an object can be reused. | ||
618 | |||
619 | If an object can be reused, the problem then becomes how to replace a | ||
620 | given task or set of tasks with the pre-built artifact. BitBake solves | ||
621 | the problem with the "setscene" process. | ||
622 | |||
623 | When BitBake is asked to build a given target, before building anything, | ||
624 | it first asks whether cached information is available for any of the | ||
625 | targets it's building, or any of the intermediate targets. If cached | ||
626 | information is available, BitBake uses this information instead of | ||
627 | running the main tasks. | ||
628 | |||
629 | BitBake first calls the function defined by the | ||
630 | :term:`BB_HASHCHECK_FUNCTION` variable | ||
631 | with a list of tasks and corresponding hashes it wants to build. This | ||
632 | function is designed to be fast and returns a list of the tasks for | ||
633 | which it believes in can obtain artifacts. | ||
634 | |||
635 | Next, for each of the tasks that were returned as possibilities, BitBake | ||
636 | executes a setscene version of the task that the possible artifact | ||
637 | covers. Setscene versions of a task have the string "_setscene" appended | ||
638 | to the task name. So, for example, the task with the name ``xxx`` has a | ||
639 | setscene task named ``xxx_setscene``. The setscene version of the task | ||
640 | executes and provides the necessary artifacts returning either success | ||
641 | or failure. | ||
642 | |||
643 | As previously mentioned, an artifact can cover more than one task. For | ||
644 | example, it is pointless to obtain a compiler if you already have the | ||
645 | compiled binary. To handle this, BitBake calls the | ||
646 | :term:`BB_SETSCENE_DEPVALID` function for | ||
647 | each successful setscene task to know whether or not it needs to obtain | ||
648 | the dependencies of that task. | ||
649 | |||
650 | Finally, after all the setscene tasks have executed, BitBake calls the | ||
651 | function listed in | ||
652 | :term:`BB_SETSCENE_VERIFY_FUNCTION2` | ||
653 | with the list of tasks BitBake thinks has been "covered". The metadata | ||
654 | can then ensure that this list is correct and can inform BitBake that it | ||
655 | wants specific tasks to be run regardless of the setscene result. | ||
656 | |||
657 | You can find more information on setscene metadata in the | ||
658 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene` | ||
659 | section. | ||
660 | |||
661 | Logging | ||
662 | ======= | ||
663 | |||
664 | In addition to the standard command line option to control how verbose | ||
665 | builds are when execute, bitbake also supports user defined | ||
666 | configuration of the `Python | ||
667 | logging <https://docs.python.org/3/library/logging.html>`__ facilities | ||
668 | through the :term:`BB_LOGCONFIG` variable. This | ||
669 | variable defines a json or yaml `logging | ||
670 | configuration <https://docs.python.org/3/library/logging.config.html>`__ | ||
671 | that will be intelligently merged into the default configuration. The | ||
672 | logging configuration is merged using the following rules: | ||
673 | |||
674 | - The user defined configuration will completely replace the default | ||
675 | configuration if top level key ``bitbake_merge`` is set to the value | ||
676 | ``False``. In this case, all other rules are ignored. | ||
677 | |||
678 | - The user configuration must have a top level ``version`` which must | ||
679 | match the value of the default configuration. | ||
680 | |||
681 | - Any keys defined in the ``handlers``, ``formatters``, or ``filters``, | ||
682 | will be merged into the same section in the default configuration, | ||
683 | with the user specified keys taking replacing a default one if there | ||
684 | is a conflict. In practice, this means that if both the default | ||
685 | configuration and user configuration specify a handler named | ||
686 | ``myhandler``, the user defined one will replace the default. To | ||
687 | prevent the user from inadvertently replacing a default handler, | ||
688 | formatter, or filter, all of the default ones are named with a prefix | ||
689 | of "``BitBake.``" | ||
690 | |||
691 | - If a logger is defined by the user with the key ``bitbake_merge`` set | ||
692 | to ``False``, that logger will be completely replaced by user | ||
693 | configuration. In this case, no other rules will apply to that | ||
694 | logger. | ||
695 | |||
696 | - All user defined ``filter`` and ``handlers`` properties for a given | ||
697 | logger will be merged with corresponding properties from the default | ||
698 | logger. For example, if the user configuration adds a filter called | ||
699 | ``myFilter`` to the ``BitBake.SigGen``, and the default configuration | ||
700 | adds a filter called ``BitBake.defaultFilter``, both filters will be | ||
701 | applied to the logger | ||
702 | |||
703 | As an example, consider the following user logging configuration file | ||
704 | which logs all Hash Equivalence related messages of VERBOSE or higher to | ||
705 | a file called ``hashequiv.log`` :: | ||
706 | |||
707 | { | ||
708 | "version": 1, | ||
709 | "handlers": { | ||
710 | "autobuilderlog": { | ||
711 | "class": "logging.FileHandler", | ||
712 | "formatter": "logfileFormatter", | ||
713 | "level": "DEBUG", | ||
714 | "filename": "hashequiv.log", | ||
715 | "mode": "w" | ||
716 | } | ||
717 | }, | ||
718 | "formatters": { | ||
719 | "logfileFormatter": { | ||
720 | "format": "%(name)s: %(levelname)s: %(message)s" | ||
721 | } | ||
722 | }, | ||
723 | "loggers": { | ||
724 | "BitBake.SigGen.HashEquiv": { | ||
725 | "level": "VERBOSE", | ||
726 | "handlers": ["autobuilderlog"] | ||
727 | }, | ||
728 | "BitBake.RunQueue.HashEquiv": { | ||
729 | "level": "VERBOSE", | ||
730 | "handlers": ["autobuilderlog"] | ||
731 | } | ||
732 | } | ||
733 | } | ||