diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | 733 |
1 files changed, 733 insertions, 0 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst new file mode 100644 index 0000000000..d74e768f69 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | |||
@@ -0,0 +1,733 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.5 | ||
2 | |||
3 | ========= | ||
4 | Execution | ||
5 | ========= | ||
6 | |||
7 | | | ||
8 | |||
9 | The primary purpose for running BitBake is to produce some kind of | ||
10 | output such as a single installable package, a kernel, a software | ||
11 | development kit, or even a full, board-specific bootable Linux image, | ||
12 | complete with bootloader, kernel, and root filesystem. Of course, you | ||
13 | can execute the ``bitbake`` command with options that cause it to | ||
14 | execute single tasks, compile single recipe files, capture or clear | ||
15 | data, or simply return information about the execution environment. | ||
16 | |||
17 | This chapter describes BitBake's execution process from start to finish | ||
18 | when you use it to create an image. The execution process is launched | ||
19 | using the following command form: :: | ||
20 | |||
21 | $ bitbake target | ||
22 | |||
23 | For information on | ||
24 | the BitBake command and its options, see ":ref:`The BitBake Command | ||
25 | <bitbake-user-manual-command>`" section. | ||
26 | |||
27 | .. note:: | ||
28 | |||
29 | Prior to executing BitBake, you should take advantage of available | ||
30 | parallel thread execution on your build host by setting the | ||
31 | :term:`BB_NUMBER_THREADS` variable in | ||
32 | your project's ``local.conf`` configuration file. | ||
33 | |||
34 | A common method to determine this value for your build host is to run | ||
35 | the following: :: | ||
36 | |||
37 | $ grep processor /proc/cpuinfo | ||
38 | |||
39 | This command returns | ||
40 | the number of processors, which takes into account hyper-threading. | ||
41 | Thus, a quad-core build host with hyper-threading most likely shows | ||
42 | eight processors, which is the value you would then assign to | ||
43 | ``BB_NUMBER_THREADS``. | ||
44 | |||
45 | A possibly simpler solution is that some Linux distributions (e.g. | ||
46 | Debian and Ubuntu) provide the ``ncpus`` command. | ||
47 | |||
48 | Parsing the Base Configuration Metadata | ||
49 | ======================================= | ||
50 | |||
51 | The first thing BitBake does is parse base configuration metadata. Base | ||
52 | configuration metadata consists of your project's ``bblayers.conf`` file | ||
53 | to determine what layers BitBake needs to recognize, all necessary | ||
54 | ``layer.conf`` files (one from each layer), and ``bitbake.conf``. The | ||
55 | data itself is of various types: | ||
56 | |||
57 | - **Recipes:** Details about particular pieces of software. | ||
58 | |||
59 | - **Class Data:** An abstraction of common build information (e.g. how to | ||
60 | build a Linux kernel). | ||
61 | |||
62 | - **Configuration Data:** Machine-specific settings, policy decisions, | ||
63 | and so forth. Configuration data acts as the glue to bind everything | ||
64 | together. | ||
65 | |||
66 | The ``layer.conf`` files are used to construct key variables such as | ||
67 | :term:`BBPATH` and :term:`BBFILES`. | ||
68 | ``BBPATH`` is used to search for configuration and class files under the | ||
69 | ``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used | ||
70 | to locate both recipe and recipe append files (``.bb`` and | ||
71 | ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the | ||
72 | user has set the ``BBPATH`` and ``BBFILES`` directly in the environment. | ||
73 | |||
74 | Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable | ||
75 | that was just constructed. The ``bitbake.conf`` file may also include | ||
76 | other configuration files using the ``include`` or ``require`` | ||
77 | directives. | ||
78 | |||
79 | Prior to parsing configuration files, BitBake looks at certain | ||
80 | variables, including: | ||
81 | |||
82 | - :term:`BB_ENV_WHITELIST` | ||
83 | - :term:`BB_ENV_EXTRAWHITE` | ||
84 | - :term:`BB_PRESERVE_ENV` | ||
85 | - :term:`BB_ORIGENV` | ||
86 | - :term:`BITBAKE_UI` | ||
87 | |||
88 | The first four variables in this list relate to how BitBake treats shell | ||
89 | environment variables during task execution. By default, BitBake cleans | ||
90 | the environment variables and provides tight control over the shell | ||
91 | execution environment. However, through the use of these first four | ||
92 | variables, you can apply your control regarding the environment | ||
93 | variables allowed to be used by BitBake in the shell during execution of | ||
94 | tasks. See the | ||
95 | ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`" | ||
96 | section and the information about these variables in the variable | ||
97 | glossary for more information on how they work and on how to use them. | ||
98 | |||
99 | The base configuration metadata is global and therefore affects all | ||
100 | recipes and tasks that are executed. | ||
101 | |||
102 | BitBake first searches the current working directory for an optional | ||
103 | ``conf/bblayers.conf`` configuration file. This file is expected to | ||
104 | contain a :term:`BBLAYERS` variable that is a | ||
105 | space-delimited list of 'layer' directories. Recall that if BitBake | ||
106 | cannot find a ``bblayers.conf`` file, then it is assumed the user has | ||
107 | set the ``BBPATH`` and ``BBFILES`` variables directly in the | ||
108 | environment. | ||
109 | |||
110 | For each directory (layer) in this list, a ``conf/layer.conf`` file is | ||
111 | located and parsed with the :term:`LAYERDIR` variable | ||
112 | being set to the directory where the layer was found. The idea is these | ||
113 | files automatically set up :term:`BBPATH` and other | ||
114 | variables correctly for a given build directory. | ||
115 | |||
116 | BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in | ||
117 | the user-specified ``BBPATH``. That configuration file generally has | ||
118 | include directives to pull in any other metadata such as files specific | ||
119 | to the architecture, the machine, the local environment, and so forth. | ||
120 | |||
121 | Only variable definitions and include directives are allowed in BitBake | ||
122 | ``.conf`` files. Some variables directly influence BitBake's behavior. | ||
123 | These variables might have been set from the environment depending on | ||
124 | the environment variables previously mentioned or set in the | ||
125 | configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`" | ||
126 | chapter presents a full list of | ||
127 | variables. | ||
128 | |||
129 | After parsing configuration files, BitBake uses its rudimentary | ||
130 | inheritance mechanism, which is through class files, to inherit some | ||
131 | standard classes. BitBake parses a class when the inherit directive | ||
132 | responsible for getting that class is encountered. | ||
133 | |||
134 | The ``base.bbclass`` file is always included. Other classes that are | ||
135 | specified in the configuration using the | ||
136 | :term:`INHERIT` variable are also included. BitBake | ||
137 | searches for class files in a ``classes`` subdirectory under the paths | ||
138 | in ``BBPATH`` in the same way as configuration files. | ||
139 | |||
140 | A good way to get an idea of the configuration files and the class files | ||
141 | used in your execution environment is to run the following BitBake | ||
142 | command: :: | ||
143 | |||
144 | $ bitbake -e > mybb.log | ||
145 | |||
146 | Examining the top of the ``mybb.log`` | ||
147 | shows you the many configuration files and class files used in your | ||
148 | execution environment. | ||
149 | |||
150 | .. note:: | ||
151 | |||
152 | You need to be aware of how BitBake parses curly braces. If a recipe | ||
153 | uses a closing curly brace within the function and the character has | ||
154 | no leading spaces, BitBake produces a parsing error. If you use a | ||
155 | pair of curly braces in a shell function, the closing curly brace | ||
156 | must not be located at the start of the line without leading spaces. | ||
157 | |||
158 | Here is an example that causes BitBake to produce a parsing error: :: | ||
159 | |||
160 | fakeroot create_shar() { | ||
161 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh | ||
162 | usage() | ||
163 | { | ||
164 | echo "test" | ||
165 | ###### The following "}" at the start of the line causes a parsing error ###### | ||
166 | } | ||
167 | EOF | ||
168 | } | ||
169 | |||
170 | Writing the recipe this way avoids the error: | ||
171 | fakeroot create_shar() { | ||
172 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh | ||
173 | usage() | ||
174 | { | ||
175 | echo "test" | ||
176 | ###### The following "}" with a leading space at the start of the line avoids the error ###### | ||
177 | } | ||
178 | EOF | ||
179 | } | ||
180 | |||
181 | Locating and Parsing Recipes | ||
182 | ============================ | ||
183 | |||
184 | During the configuration phase, BitBake will have set | ||
185 | :term:`BBFILES`. BitBake now uses it to construct a | ||
186 | list of recipes to parse, along with any append files (``.bbappend``) to | ||
187 | apply. ``BBFILES`` is a space-separated list of available files and | ||
188 | supports wildcards. An example would be: :: | ||
189 | |||
190 | BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" | ||
191 | |||
192 | BitBake parses each | ||
193 | recipe and append file located with ``BBFILES`` and stores the values of | ||
194 | various variables into the datastore. | ||
195 | |||
196 | .. note:: | ||
197 | |||
198 | Append files are applied in the order they are encountered in BBFILES. | ||
199 | |||
200 | For each file, a fresh copy of the base configuration is made, then the | ||
201 | recipe is parsed line by line. Any inherit statements cause BitBake to | ||
202 | find and then parse class files (``.bbclass``) using | ||
203 | :term:`BBPATH` as the search path. Finally, BitBake | ||
204 | parses in order any append files found in ``BBFILES``. | ||
205 | |||
206 | One common convention is to use the recipe filename to define pieces of | ||
207 | metadata. For example, in ``bitbake.conf`` the recipe name and version | ||
208 | are used to set the variables :term:`PN` and | ||
209 | :term:`PV`: :: | ||
210 | |||
211 | PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" | ||
212 | PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" | ||
213 | |||
214 | In this example, a recipe called "something_1.2.3.bb" would set | ||
215 | ``PN`` to "something" and ``PV`` to "1.2.3". | ||
216 | |||
217 | By the time parsing is complete for a recipe, BitBake has a list of | ||
218 | tasks that the recipe defines and a set of data consisting of keys and | ||
219 | values as well as dependency information about the tasks. | ||
220 | |||
221 | BitBake does not need all of this information. It only needs a small | ||
222 | subset of the information to make decisions about the recipe. | ||
223 | Consequently, BitBake caches the values in which it is interested and | ||
224 | does not store the rest of the information. Experience has shown it is | ||
225 | faster to re-parse the metadata than to try and write it out to the disk | ||
226 | and then reload it. | ||
227 | |||
228 | Where possible, subsequent BitBake commands reuse this cache of recipe | ||
229 | information. The validity of this cache is determined by first computing | ||
230 | a checksum of the base configuration data (see | ||
231 | :term:`BB_HASHCONFIG_WHITELIST`) and | ||
232 | then checking if the checksum matches. If that checksum matches what is | ||
233 | in the cache and the recipe and class files have not changed, BitBake is | ||
234 | able to use the cache. BitBake then reloads the cached information about | ||
235 | the recipe instead of reparsing it from scratch. | ||
236 | |||
237 | Recipe file collections exist to allow the user to have multiple | ||
238 | repositories of ``.bb`` files that contain the same exact package. For | ||
239 | example, one could easily use them to make one's own local copy of an | ||
240 | upstream repository, but with custom modifications that one does not | ||
241 | want upstream. Here is an example: :: | ||
242 | |||
243 | BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb" | ||
244 | BBFILE_COLLECTIONS = "upstream local" | ||
245 | BBFILE_PATTERN_upstream = "^/stuff/openembedded/" | ||
246 | BBFILE_PATTERN_local = "^/stuff/openembedded.modified/" | ||
247 | BBFILE_PRIORITY_upstream = "5" BBFILE_PRIORITY_local = "10" | ||
248 | |||
249 | .. note:: | ||
250 | |||
251 | The layers mechanism is now the preferred method of collecting code. | ||
252 | While the collections code remains, its main use is to set layer | ||
253 | priorities and to deal with overlap (conflicts) between layers. | ||
254 | |||
255 | .. _bb-bitbake-providers: | ||
256 | |||
257 | Providers | ||
258 | ========= | ||
259 | |||
260 | Assuming BitBake has been instructed to execute a target and that all | ||
261 | the recipe files have been parsed, BitBake starts to figure out how to | ||
262 | build the target. BitBake looks through the ``PROVIDES`` list for each | ||
263 | of the recipes. A ``PROVIDES`` list is the list of names by which the | ||
264 | recipe can be known. Each recipe's ``PROVIDES`` list is created | ||
265 | implicitly through the recipe's :term:`PN` variable and | ||
266 | explicitly through the recipe's :term:`PROVIDES` | ||
267 | variable, which is optional. | ||
268 | |||
269 | When a recipe uses ``PROVIDES``, that recipe's functionality can be | ||
270 | found under an alternative name or names other than the implicit ``PN`` | ||
271 | name. As an example, suppose a recipe named ``keyboard_1.0.bb`` | ||
272 | contained the following: :: | ||
273 | |||
274 | PROVIDES += "fullkeyboard" | ||
275 | |||
276 | The ``PROVIDES`` | ||
277 | list for this recipe becomes "keyboard", which is implicit, and | ||
278 | "fullkeyboard", which is explicit. Consequently, the functionality found | ||
279 | in ``keyboard_1.0.bb`` can be found under two different names. | ||
280 | |||
281 | .. _bb-bitbake-preferences: | ||
282 | |||
283 | Preferences | ||
284 | =========== | ||
285 | |||
286 | The ``PROVIDES`` list is only part of the solution for figuring out a | ||
287 | target's recipes. Because targets might have multiple providers, BitBake | ||
288 | needs to prioritize providers by determining provider preferences. | ||
289 | |||
290 | A common example in which a target has multiple providers is | ||
291 | "virtual/kernel", which is on the ``PROVIDES`` list for each kernel | ||
292 | recipe. Each machine often selects the best kernel provider by using a | ||
293 | line similar to the following in the machine configuration file: :: | ||
294 | |||
295 | PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" | ||
296 | |||
297 | The default :term:`PREFERRED_PROVIDER` is the provider | ||
298 | with the same name as the target. BitBake iterates through each target | ||
299 | it needs to build and resolves them and their dependencies using this | ||
300 | process. | ||
301 | |||
302 | Understanding how providers are chosen is made complicated by the fact | ||
303 | that multiple versions might exist for a given provider. BitBake | ||
304 | defaults to the highest version of a provider. Version comparisons are | ||
305 | made using the same method as Debian. You can use the | ||
306 | :term:`PREFERRED_VERSION` variable to | ||
307 | specify a particular version. You can influence the order by using the | ||
308 | :term:`DEFAULT_PREFERENCE` variable. | ||
309 | |||
310 | By default, files have a preference of "0". Setting | ||
311 | ``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used | ||
312 | unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to | ||
313 | "1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides | ||
314 | any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used | ||
315 | to mark newer and more experimental recipe versions until they have | ||
316 | undergone sufficient testing to be considered stable. | ||
317 | |||
318 | When there are multiple "versions" of a given recipe, BitBake defaults | ||
319 | to selecting the most recent version, unless otherwise specified. If the | ||
320 | recipe in question has a | ||
321 | :term:`DEFAULT_PREFERENCE` set lower than | ||
322 | the other recipes (default is 0), then it will not be selected. This | ||
323 | allows the person or persons maintaining the repository of recipe files | ||
324 | to specify their preference for the default selected version. | ||
325 | Additionally, the user can specify their preferred version. | ||
326 | |||
327 | If the first recipe is named ``a_1.1.bb``, then the | ||
328 | :term:`PN` variable will be set to "a", and the | ||
329 | :term:`PV` variable will be set to 1.1. | ||
330 | |||
331 | Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by | ||
332 | default. However, if you define the following variable in a ``.conf`` | ||
333 | file that BitBake parses, you can change that preference: :: | ||
334 | |||
335 | PREFERRED_VERSION_a = "1.1" | ||
336 | |||
337 | .. note:: | ||
338 | |||
339 | It is common for a recipe to provide two versions -- a stable, | ||
340 | numbered (and preferred) version, and a version that is automatically | ||
341 | checked out from a source code repository that is considered more | ||
342 | "bleeding edge" but can be selected only explicitly. | ||
343 | |||
344 | For example, in the OpenEmbedded codebase, there is a standard, | ||
345 | versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there | ||
346 | is also a Git-based version, ``busybox_git.bb``, which explicitly | ||
347 | contains the line :: | ||
348 | |||
349 | DEFAULT_PREFERENCE = "-1" | ||
350 | |||
351 | to ensure that the | ||
352 | numbered, stable version is always preferred unless the developer | ||
353 | selects otherwise. | ||
354 | |||
355 | .. _bb-bitbake-dependencies: | ||
356 | |||
357 | Dependencies | ||
358 | ============ | ||
359 | |||
360 | Each target BitBake builds consists of multiple tasks such as ``fetch``, | ||
361 | ``unpack``, ``patch``, ``configure``, and ``compile``. For best | ||
362 | performance on multi-core systems, BitBake considers each task as an | ||
363 | independent entity with its own set of dependencies. | ||
364 | |||
365 | Dependencies are defined through several variables. You can find | ||
366 | information about variables BitBake uses in the | ||
367 | :doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a | ||
368 | basic level, it is sufficient to know that BitBake uses the | ||
369 | :term:`DEPENDS` and | ||
370 | :term:`RDEPENDS` variables when calculating | ||
371 | dependencies. | ||
372 | |||
373 | For more information on how BitBake handles dependencies, see the | ||
374 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies` | ||
375 | section. | ||
376 | |||
377 | .. _ref-bitbake-tasklist: | ||
378 | |||
379 | The Task List | ||
380 | ============= | ||
381 | |||
382 | Based on the generated list of providers and the dependency information, | ||
383 | BitBake can now calculate exactly what tasks it needs to run and in what | ||
384 | order it needs to run them. The | ||
385 | :ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks` | ||
386 | section has more information on how BitBake chooses which task to | ||
387 | execute next. | ||
388 | |||
389 | The build now starts with BitBake forking off threads up to the limit | ||
390 | set in the :term:`BB_NUMBER_THREADS` | ||
391 | variable. BitBake continues to fork threads as long as there are tasks | ||
392 | ready to run, those tasks have all their dependencies met, and the | ||
393 | thread threshold has not been exceeded. | ||
394 | |||
395 | It is worth noting that you can greatly speed up the build time by | ||
396 | properly setting the ``BB_NUMBER_THREADS`` variable. | ||
397 | |||
398 | As each task completes, a timestamp is written to the directory | ||
399 | specified by the :term:`STAMP` variable. On subsequent | ||
400 | runs, BitBake looks in the build directory within ``tmp/stamps`` and | ||
401 | does not rerun tasks that are already completed unless a timestamp is | ||
402 | found to be invalid. Currently, invalid timestamps are only considered | ||
403 | on a per recipe file basis. So, for example, if the configure stamp has | ||
404 | a timestamp greater than the compile timestamp for a given target, then | ||
405 | the compile task would rerun. Running the compile task again, however, | ||
406 | has no effect on other providers that depend on that target. | ||
407 | |||
408 | The exact format of the stamps is partly configurable. In modern | ||
409 | versions of BitBake, a hash is appended to the stamp so that if the | ||
410 | configuration changes, the stamp becomes invalid and the task is | ||
411 | automatically rerun. This hash, or signature used, is governed by the | ||
412 | signature policy that is configured (see the | ||
413 | :ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)` | ||
414 | section for information). It is also | ||
415 | possible to append extra metadata to the stamp using the | ||
416 | ``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this | ||
417 | flag to make some tasks machine-specific. | ||
418 | |||
419 | .. note:: | ||
420 | |||
421 | Some tasks are marked as "nostamp" tasks. No timestamp file is | ||
422 | created when these tasks are run. Consequently, "nostamp" tasks are | ||
423 | always rerun. | ||
424 | |||
425 | For more information on tasks, see the | ||
426 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section. | ||
427 | |||
428 | Executing Tasks | ||
429 | =============== | ||
430 | |||
431 | Tasks can be either a shell task or a Python task. For shell tasks, | ||
432 | BitBake writes a shell script to | ||
433 | ``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then | ||
434 | executes the script. The generated shell script contains all the | ||
435 | exported variables, and the shell functions with all variables expanded. | ||
436 | Output from the shell script goes to the file | ||
437 | ``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in | ||
438 | the run file and the output in the log files is a useful debugging | ||
439 | technique. | ||
440 | |||
441 | For Python tasks, BitBake executes the task internally and logs | ||
442 | information to the controlling terminal. Future versions of BitBake will | ||
443 | write the functions to files similar to the way shell tasks are handled. | ||
444 | Logging will be handled in a way similar to shell tasks as well. | ||
445 | |||
446 | The order in which BitBake runs the tasks is controlled by its task | ||
447 | scheduler. It is possible to configure the scheduler and define custom | ||
448 | implementations for specific use cases. For more information, see these | ||
449 | variables that control the behavior: | ||
450 | |||
451 | - :term:`BB_SCHEDULER` | ||
452 | |||
453 | - :term:`BB_SCHEDULERS` | ||
454 | |||
455 | It is possible to have functions run before and after a task's main | ||
456 | function. This is done using the ``[prefuncs]`` and ``[postfuncs]`` | ||
457 | flags of the task that lists the functions to run. | ||
458 | |||
459 | .. _checksums: | ||
460 | |||
461 | Checksums (Signatures) | ||
462 | ====================== | ||
463 | |||
464 | A checksum is a unique signature of a task's inputs. The signature of a | ||
465 | task can be used to determine if a task needs to be run. Because it is a | ||
466 | change in a task's inputs that triggers running the task, BitBake needs | ||
467 | to detect all the inputs to a given task. For shell tasks, this turns | ||
468 | out to be fairly easy because BitBake generates a "run" shell script for | ||
469 | each task and it is possible to create a checksum that gives you a good | ||
470 | idea of when the task's data changes. | ||
471 | |||
472 | To complicate the problem, some things should not be included in the | ||
473 | checksum. First, there is the actual specific build path of a given task | ||
474 | - the working directory. It does not matter if the working directory | ||
475 | changes because it should not affect the output for target packages. The | ||
476 | simplistic approach for excluding the working directory is to set it to | ||
477 | some fixed value and create the checksum for the "run" script. BitBake | ||
478 | goes one step better and uses the | ||
479 | :term:`BB_HASHBASE_WHITELIST` variable | ||
480 | to define a list of variables that should never be included when | ||
481 | generating the signatures. | ||
482 | |||
483 | Another problem results from the "run" scripts containing functions that | ||
484 | might or might not get called. The incremental build solution contains | ||
485 | code that figures out dependencies between shell functions. This code is | ||
486 | used to prune the "run" scripts down to the minimum set, thereby | ||
487 | alleviating this problem and making the "run" scripts much more readable | ||
488 | as a bonus. | ||
489 | |||
490 | So far we have solutions for shell scripts. What about Python tasks? The | ||
491 | same approach applies even though these tasks are more difficult. The | ||
492 | process needs to figure out what variables a Python function accesses | ||
493 | and what functions it calls. Again, the incremental build solution | ||
494 | contains code that first figures out the variable and function | ||
495 | dependencies, and then creates a checksum for the data used as the input | ||
496 | to the task. | ||
497 | |||
498 | Like the working directory case, situations exist where dependencies | ||
499 | should be ignored. For these cases, you can instruct the build process | ||
500 | to ignore a dependency by using a line like the following: :: | ||
501 | |||
502 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
503 | |||
504 | This example ensures that the | ||
505 | ``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``, | ||
506 | even if it does reference it. | ||
507 | |||
508 | Equally, there are cases where we need to add dependencies BitBake is | ||
509 | not able to find. You can accomplish this by using a line like the | ||
510 | following: :: | ||
511 | |||
512 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
513 | |||
514 | This example explicitly | ||
515 | adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. | ||
516 | |||
517 | Consider a case with in-line Python, for example, where BitBake is not | ||
518 | able to figure out dependencies. When running in debug mode (i.e. using | ||
519 | ``-DDD``), BitBake produces output when it discovers something for which | ||
520 | it cannot figure out dependencies. | ||
521 | |||
522 | Thus far, this section has limited discussion to the direct inputs into | ||
523 | a task. Information based on direct inputs is referred to as the | ||
524 | "basehash" in the code. However, there is still the question of a task's | ||
525 | indirect inputs - the things that were already built and present in the | ||
526 | build directory. The checksum (or signature) for a particular task needs | ||
527 | to add the hashes of all the tasks on which the particular task depends. | ||
528 | Choosing which dependencies to add is a policy decision. However, the | ||
529 | effect is to generate a master checksum that combines the basehash and | ||
530 | the hashes of the task's dependencies. | ||
531 | |||
532 | At the code level, there are a variety of ways both the basehash and the | ||
533 | dependent task hashes can be influenced. Within the BitBake | ||
534 | configuration file, we can give BitBake some extra information to help | ||
535 | it construct the basehash. The following statement effectively results | ||
536 | in a list of global variable dependency excludes - variables never | ||
537 | included in any checksum. This example uses variables from OpenEmbedded | ||
538 | to help illustrate the concept: :: | ||
539 | |||
540 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ | ||
541 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ | ||
542 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ | ||
543 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ | ||
544 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
545 | |||
546 | The previous example excludes the work directory, which is part of | ||
547 | ``TMPDIR``. | ||
548 | |||
549 | The rules for deciding which hashes of dependent tasks to include | ||
550 | through dependency chains are more complex and are generally | ||
551 | accomplished with a Python function. The code in | ||
552 | ``meta/lib/oe/sstatesig.py`` shows two examples of this and also | ||
553 | illustrates how you can insert your own policy into the system if so | ||
554 | desired. This file defines the two basic signature generators | ||
555 | OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there | ||
556 | is a dummy "noop" signature handler enabled in BitBake. This means that | ||
557 | behavior is unchanged from previous versions. ``OE-Core`` uses the | ||
558 | "OEBasicHash" signature handler by default through this setting in the | ||
559 | ``bitbake.conf`` file: :: | ||
560 | |||
561 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
562 | |||
563 | The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic" | ||
564 | version but adds the task hash to the stamp files. This results in any | ||
565 | metadata change that changes the task hash, automatically causing the | ||
566 | task to be run again. This removes the need to bump | ||
567 | :term:`PR` values, and changes to metadata automatically | ||
568 | ripple across the build. | ||
569 | |||
570 | It is also worth noting that the end result of these signature | ||
571 | generators is to make some dependency and hash information available to | ||
572 | the build. This information includes: | ||
573 | |||
574 | - ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the | ||
575 | recipe. | ||
576 | |||
577 | - ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each | ||
578 | dependent task. | ||
579 | |||
580 | - ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for | ||
581 | each task. | ||
582 | |||
583 | - ``BB_TASKHASH``: The hash of the currently running task. | ||
584 | |||
585 | It is worth noting that BitBake's "-S" option lets you debug BitBake's | ||
586 | processing of signatures. The options passed to -S allow different | ||
587 | debugging modes to be used, either using BitBake's own debug functions | ||
588 | or possibly those defined in the metadata/signature handler itself. The | ||
589 | simplest parameter to pass is "none", which causes a set of signature | ||
590 | information to be written out into ``STAMPS_DIR`` corresponding to the | ||
591 | targets specified. The other currently available parameter is | ||
592 | "printdiff", which causes BitBake to try to establish the closest | ||
593 | signature match it can (e.g. in the sstate cache) and then run | ||
594 | ``bitbake-diffsigs`` over the matches to determine the stamps and delta | ||
595 | where these two stamp trees diverge. | ||
596 | |||
597 | .. note:: | ||
598 | |||
599 | It is likely that future versions of BitBake will provide other | ||
600 | signature handlers triggered through additional "-S" parameters. | ||
601 | |||
602 | You can find more information on checksum metadata in the | ||
603 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene` | ||
604 | section. | ||
605 | |||
606 | Setscene | ||
607 | ======== | ||
608 | |||
609 | The setscene process enables BitBake to handle "pre-built" artifacts. | ||
610 | The ability to handle and reuse these artifacts allows BitBake the | ||
611 | luxury of not having to build something from scratch every time. | ||
612 | Instead, BitBake can use, when possible, existing build artifacts. | ||
613 | |||
614 | BitBake needs to have reliable data indicating whether or not an | ||
615 | artifact is compatible. Signatures, described in the previous section, | ||
616 | provide an ideal way of representing whether an artifact is compatible. | ||
617 | If a signature is the same, an object can be reused. | ||
618 | |||
619 | If an object can be reused, the problem then becomes how to replace a | ||
620 | given task or set of tasks with the pre-built artifact. BitBake solves | ||
621 | the problem with the "setscene" process. | ||
622 | |||
623 | When BitBake is asked to build a given target, before building anything, | ||
624 | it first asks whether cached information is available for any of the | ||
625 | targets it's building, or any of the intermediate targets. If cached | ||
626 | information is available, BitBake uses this information instead of | ||
627 | running the main tasks. | ||
628 | |||
629 | BitBake first calls the function defined by the | ||
630 | :term:`BB_HASHCHECK_FUNCTION` variable | ||
631 | with a list of tasks and corresponding hashes it wants to build. This | ||
632 | function is designed to be fast and returns a list of the tasks for | ||
633 | which it believes in can obtain artifacts. | ||
634 | |||
635 | Next, for each of the tasks that were returned as possibilities, BitBake | ||
636 | executes a setscene version of the task that the possible artifact | ||
637 | covers. Setscene versions of a task have the string "_setscene" appended | ||
638 | to the task name. So, for example, the task with the name ``xxx`` has a | ||
639 | setscene task named ``xxx_setscene``. The setscene version of the task | ||
640 | executes and provides the necessary artifacts returning either success | ||
641 | or failure. | ||
642 | |||
643 | As previously mentioned, an artifact can cover more than one task. For | ||
644 | example, it is pointless to obtain a compiler if you already have the | ||
645 | compiled binary. To handle this, BitBake calls the | ||
646 | :term:`BB_SETSCENE_DEPVALID` function for | ||
647 | each successful setscene task to know whether or not it needs to obtain | ||
648 | the dependencies of that task. | ||
649 | |||
650 | Finally, after all the setscene tasks have executed, BitBake calls the | ||
651 | function listed in | ||
652 | :term:`BB_SETSCENE_VERIFY_FUNCTION2` | ||
653 | with the list of tasks BitBake thinks has been "covered". The metadata | ||
654 | can then ensure that this list is correct and can inform BitBake that it | ||
655 | wants specific tasks to be run regardless of the setscene result. | ||
656 | |||
657 | You can find more information on setscene metadata in the | ||
658 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene` | ||
659 | section. | ||
660 | |||
661 | Logging | ||
662 | ======= | ||
663 | |||
664 | In addition to the standard command line option to control how verbose | ||
665 | builds are when execute, bitbake also supports user defined | ||
666 | configuration of the `Python | ||
667 | logging <https://docs.python.org/3/library/logging.html>`__ facilities | ||
668 | through the :term:`BB_LOGCONFIG` variable. This | ||
669 | variable defines a json or yaml `logging | ||
670 | configuration <https://docs.python.org/3/library/logging.config.html>`__ | ||
671 | that will be intelligently merged into the default configuration. The | ||
672 | logging configuration is merged using the following rules: | ||
673 | |||
674 | - The user defined configuration will completely replace the default | ||
675 | configuration if top level key ``bitbake_merge`` is set to the value | ||
676 | ``False``. In this case, all other rules are ignored. | ||
677 | |||
678 | - The user configuration must have a top level ``version`` which must | ||
679 | match the value of the default configuration. | ||
680 | |||
681 | - Any keys defined in the ``handlers``, ``formatters``, or ``filters``, | ||
682 | will be merged into the same section in the default configuration, | ||
683 | with the user specified keys taking replacing a default one if there | ||
684 | is a conflict. In practice, this means that if both the default | ||
685 | configuration and user configuration specify a handler named | ||
686 | ``myhandler``, the user defined one will replace the default. To | ||
687 | prevent the user from inadvertently replacing a default handler, | ||
688 | formatter, or filter, all of the default ones are named with a prefix | ||
689 | of "``BitBake.``" | ||
690 | |||
691 | - If a logger is defined by the user with the key ``bitbake_merge`` set | ||
692 | to ``False``, that logger will be completely replaced by user | ||
693 | configuration. In this case, no other rules will apply to that | ||
694 | logger. | ||
695 | |||
696 | - All user defined ``filter`` and ``handlers`` properties for a given | ||
697 | logger will be merged with corresponding properties from the default | ||
698 | logger. For example, if the user configuration adds a filter called | ||
699 | ``myFilter`` to the ``BitBake.SigGen``, and the default configuration | ||
700 | adds a filter called ``BitBake.defaultFilter``, both filters will be | ||
701 | applied to the logger | ||
702 | |||
703 | As an example, consider the following user logging configuration file | ||
704 | which logs all Hash Equivalence related messages of VERBOSE or higher to | ||
705 | a file called ``hashequiv.log`` :: | ||
706 | |||
707 | { | ||
708 | "version": 1, | ||
709 | "handlers": { | ||
710 | "autobuilderlog": { | ||
711 | "class": "logging.FileHandler", | ||
712 | "formatter": "logfileFormatter", | ||
713 | "level": "DEBUG", | ||
714 | "filename": "hashequiv.log", | ||
715 | "mode": "w" | ||
716 | } | ||
717 | }, | ||
718 | "formatters": { | ||
719 | "logfileFormatter": { | ||
720 | "format": "%(name)s: %(levelname)s: %(message)s" | ||
721 | } | ||
722 | }, | ||
723 | "loggers": { | ||
724 | "BitBake.SigGen.HashEquiv": { | ||
725 | "level": "VERBOSE", | ||
726 | "handlers": ["autobuilderlog"] | ||
727 | }, | ||
728 | "BitBake.RunQueue.HashEquiv": { | ||
729 | "level": "VERBOSE", | ||
730 | "handlers": ["autobuilderlog"] | ||
731 | } | ||
732 | } | ||
733 | } | ||