diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | 654 |
1 files changed, 654 insertions, 0 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst new file mode 100644 index 0000000000..9488ea9bae --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | |||
@@ -0,0 +1,654 @@ | |||
1 | ========= | ||
2 | Execution | ||
3 | ========= | ||
4 | |||
5 | The primary purpose for running BitBake is to produce some kind of | ||
6 | output such as a single installable package, a kernel, a software | ||
7 | development kit, or even a full, board-specific bootable Linux image, | ||
8 | complete with bootloader, kernel, and root filesystem. Of course, you | ||
9 | can execute the ``bitbake`` command with options that cause it to | ||
10 | execute single tasks, compile single recipe files, capture or clear | ||
11 | data, or simply return information about the execution environment. | ||
12 | |||
13 | This chapter describes BitBake's execution process from start to finish | ||
14 | when you use it to create an image. The execution process is launched | ||
15 | using the following command form: $ bitbake target For information on | ||
16 | the BitBake command and its options, see "`The BitBake | ||
17 | Command <#bitbake-user-manual-command>`__" section. | ||
18 | |||
19 | .. note:: | ||
20 | |||
21 | Prior to executing BitBake, you should take advantage of available | ||
22 | parallel thread execution on your build host by setting the | ||
23 | ```BB_NUMBER_THREADS`` <#var-bb-BB_NUMBER_THREADS>`__ variable in | ||
24 | your project's ``local.conf`` configuration file. | ||
25 | |||
26 | A common method to determine this value for your build host is to run | ||
27 | the following: $ grep processor /proc/cpuinfo This command returns | ||
28 | the number of processors, which takes into account hyper-threading. | ||
29 | Thus, a quad-core build host with hyper-threading most likely shows | ||
30 | eight processors, which is the value you would then assign to | ||
31 | ``BB_NUMBER_THREADS``. | ||
32 | |||
33 | A possibly simpler solution is that some Linux distributions (e.g. | ||
34 | Debian and Ubuntu) provide the ``ncpus`` command. | ||
35 | |||
36 | Parsing the Base Configuration Metadata | ||
37 | ======================================= | ||
38 | |||
39 | The first thing BitBake does is parse base configuration metadata. Base | ||
40 | configuration metadata consists of your project's ``bblayers.conf`` file | ||
41 | to determine what layers BitBake needs to recognize, all necessary | ||
42 | ``layer.conf`` files (one from each layer), and ``bitbake.conf``. The | ||
43 | data itself is of various types: | ||
44 | |||
45 | - *Recipes:* Details about particular pieces of software. | ||
46 | |||
47 | - *Class Data:* An abstraction of common build information (e.g. how to | ||
48 | build a Linux kernel). | ||
49 | |||
50 | - *Configuration Data:* Machine-specific settings, policy decisions, | ||
51 | and so forth. Configuration data acts as the glue to bind everything | ||
52 | together. | ||
53 | |||
54 | The ``layer.conf`` files are used to construct key variables such as | ||
55 | ```BBPATH`` <#var-bb-BBPATH>`__ and ```BBFILES`` <#var-bb-BBFILES>`__. | ||
56 | ``BBPATH`` is used to search for configuration and class files under the | ||
57 | ``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used | ||
58 | to locate both recipe and recipe append files (``.bb`` and | ||
59 | ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the | ||
60 | user has set the ``BBPATH`` and ``BBFILES`` directly in the environment. | ||
61 | |||
62 | Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable | ||
63 | that was just constructed. The ``bitbake.conf`` file may also include | ||
64 | other configuration files using the ``include`` or ``require`` | ||
65 | directives. | ||
66 | |||
67 | Prior to parsing configuration files, BitBake looks at certain | ||
68 | variables, including: | ||
69 | |||
70 | - ```BB_ENV_WHITELIST`` <#var-bb-BB_ENV_WHITELIST>`__ | ||
71 | |||
72 | - ```BB_ENV_EXTRAWHITE`` <#var-bb-BB_ENV_EXTRAWHITE>`__ | ||
73 | |||
74 | - ```BB_PRESERVE_ENV`` <#var-bb-BB_PRESERVE_ENV>`__ | ||
75 | |||
76 | - ```BB_ORIGENV`` <#var-bb-BB_ORIGENV>`__ | ||
77 | |||
78 | - ```BITBAKE_UI`` <#var-bb-BITBAKE_UI>`__ | ||
79 | |||
80 | The first four variables in this list relate to how BitBake treats shell | ||
81 | environment variables during task execution. By default, BitBake cleans | ||
82 | the environment variables and provides tight control over the shell | ||
83 | execution environment. However, through the use of these first four | ||
84 | variables, you can apply your control regarding the environment | ||
85 | variables allowed to be used by BitBake in the shell during execution of | ||
86 | tasks. See the "`Passing Information Into the Build Task | ||
87 | Environment <#passing-information-into-the-build-task-environment>`__" | ||
88 | section and the information about these variables in the variable | ||
89 | glossary for more information on how they work and on how to use them. | ||
90 | |||
91 | The base configuration metadata is global and therefore affects all | ||
92 | recipes and tasks that are executed. | ||
93 | |||
94 | BitBake first searches the current working directory for an optional | ||
95 | ``conf/bblayers.conf`` configuration file. This file is expected to | ||
96 | contain a ```BBLAYERS`` <#var-bb-BBLAYERS>`__ variable that is a | ||
97 | space-delimited list of 'layer' directories. Recall that if BitBake | ||
98 | cannot find a ``bblayers.conf`` file, then it is assumed the user has | ||
99 | set the ``BBPATH`` and ``BBFILES`` variables directly in the | ||
100 | environment. | ||
101 | |||
102 | For each directory (layer) in this list, a ``conf/layer.conf`` file is | ||
103 | located and parsed with the ```LAYERDIR`` <#var-bb-LAYERDIR>`__ variable | ||
104 | being set to the directory where the layer was found. The idea is these | ||
105 | files automatically set up ```BBPATH`` <#var-bb-BBPATH>`__ and other | ||
106 | variables correctly for a given build directory. | ||
107 | |||
108 | BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in | ||
109 | the user-specified ``BBPATH``. That configuration file generally has | ||
110 | include directives to pull in any other metadata such as files specific | ||
111 | to the architecture, the machine, the local environment, and so forth. | ||
112 | |||
113 | Only variable definitions and include directives are allowed in BitBake | ||
114 | ``.conf`` files. Some variables directly influence BitBake's behavior. | ||
115 | These variables might have been set from the environment depending on | ||
116 | the environment variables previously mentioned or set in the | ||
117 | configuration files. The "`Variables | ||
118 | Glossary <#ref-bb-variables-glos>`__" chapter presents a full list of | ||
119 | variables. | ||
120 | |||
121 | After parsing configuration files, BitBake uses its rudimentary | ||
122 | inheritance mechanism, which is through class files, to inherit some | ||
123 | standard classes. BitBake parses a class when the inherit directive | ||
124 | responsible for getting that class is encountered. | ||
125 | |||
126 | The ``base.bbclass`` file is always included. Other classes that are | ||
127 | specified in the configuration using the | ||
128 | ```INHERIT`` <#var-bb-INHERIT>`__ variable are also included. BitBake | ||
129 | searches for class files in a ``classes`` subdirectory under the paths | ||
130 | in ``BBPATH`` in the same way as configuration files. | ||
131 | |||
132 | A good way to get an idea of the configuration files and the class files | ||
133 | used in your execution environment is to run the following BitBake | ||
134 | command: $ bitbake -e > mybb.log Examining the top of the ``mybb.log`` | ||
135 | shows you the many configuration files and class files used in your | ||
136 | execution environment. | ||
137 | |||
138 | .. note:: | ||
139 | |||
140 | You need to be aware of how BitBake parses curly braces. If a recipe | ||
141 | uses a closing curly brace within the function and the character has | ||
142 | no leading spaces, BitBake produces a parsing error. If you use a | ||
143 | pair of curly braces in a shell function, the closing curly brace | ||
144 | must not be located at the start of the line without leading spaces. | ||
145 | |||
146 | Here is an example that causes BitBake to produce a parsing error: | ||
147 | fakeroot create_shar() { cat << "EOF" > | ||
148 | ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh usage() { echo "test" ###### | ||
149 | The following "}" at the start of the line causes a parsing error | ||
150 | ###### } EOF } Writing the recipe this way avoids the error: fakeroot | ||
151 | create_shar() { cat << "EOF" > | ||
152 | ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh usage() { echo "test" | ||
153 | ######The following "}" with a leading space at the start of the line | ||
154 | avoids the error ###### } EOF } | ||
155 | |||
156 | Locating and Parsing Recipes | ||
157 | ============================ | ||
158 | |||
159 | During the configuration phase, BitBake will have set | ||
160 | ```BBFILES`` <#var-bb-BBFILES>`__. BitBake now uses it to construct a | ||
161 | list of recipes to parse, along with any append files (``.bbappend``) to | ||
162 | apply. ``BBFILES`` is a space-separated list of available files and | ||
163 | supports wildcards. An example would be: BBFILES = | ||
164 | "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" BitBake parses each | ||
165 | recipe and append file located with ``BBFILES`` and stores the values of | ||
166 | various variables into the datastore. | ||
167 | |||
168 | .. note:: | ||
169 | |||
170 | Append files are applied in the order they are encountered in | ||
171 | BBFILES | ||
172 | . | ||
173 | |||
174 | For each file, a fresh copy of the base configuration is made, then the | ||
175 | recipe is parsed line by line. Any inherit statements cause BitBake to | ||
176 | find and then parse class files (``.bbclass``) using | ||
177 | ```BBPATH`` <#var-bb-BBPATH>`__ as the search path. Finally, BitBake | ||
178 | parses in order any append files found in ``BBFILES``. | ||
179 | |||
180 | One common convention is to use the recipe filename to define pieces of | ||
181 | metadata. For example, in ``bitbake.conf`` the recipe name and version | ||
182 | are used to set the variables ```PN`` <#var-bb-PN>`__ and | ||
183 | ```PV`` <#var-bb-PV>`__: PN = | ||
184 | "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or | ||
185 | 'defaultpkgname'}" PV = | ||
186 | "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or | ||
187 | '1.0'}" In this example, a recipe called "something_1.2.3.bb" would set | ||
188 | ``PN`` to "something" and ``PV`` to "1.2.3". | ||
189 | |||
190 | By the time parsing is complete for a recipe, BitBake has a list of | ||
191 | tasks that the recipe defines and a set of data consisting of keys and | ||
192 | values as well as dependency information about the tasks. | ||
193 | |||
194 | BitBake does not need all of this information. It only needs a small | ||
195 | subset of the information to make decisions about the recipe. | ||
196 | Consequently, BitBake caches the values in which it is interested and | ||
197 | does not store the rest of the information. Experience has shown it is | ||
198 | faster to re-parse the metadata than to try and write it out to the disk | ||
199 | and then reload it. | ||
200 | |||
201 | Where possible, subsequent BitBake commands reuse this cache of recipe | ||
202 | information. The validity of this cache is determined by first computing | ||
203 | a checksum of the base configuration data (see | ||
204 | ```BB_HASHCONFIG_WHITELIST`` <#var-bb-BB_HASHCONFIG_WHITELIST>`__) and | ||
205 | then checking if the checksum matches. If that checksum matches what is | ||
206 | in the cache and the recipe and class files have not changed, BitBake is | ||
207 | able to use the cache. BitBake then reloads the cached information about | ||
208 | the recipe instead of reparsing it from scratch. | ||
209 | |||
210 | Recipe file collections exist to allow the user to have multiple | ||
211 | repositories of ``.bb`` files that contain the same exact package. For | ||
212 | example, one could easily use them to make one's own local copy of an | ||
213 | upstream repository, but with custom modifications that one does not | ||
214 | want upstream. Here is an example: BBFILES = "/stuff/openembedded/*/*.bb | ||
215 | /stuff/openembedded.modified/*/*.bb" BBFILE_COLLECTIONS = "upstream | ||
216 | local" BBFILE_PATTERN_upstream = "^/stuff/openembedded/" | ||
217 | BBFILE_PATTERN_local = "^/stuff/openembedded.modified/" | ||
218 | BBFILE_PRIORITY_upstream = "5" BBFILE_PRIORITY_local = "10" | ||
219 | |||
220 | .. note:: | ||
221 | |||
222 | The layers mechanism is now the preferred method of collecting code. | ||
223 | While the collections code remains, its main use is to set layer | ||
224 | priorities and to deal with overlap (conflicts) between layers. | ||
225 | |||
226 | .. _bb-bitbake-providers: | ||
227 | |||
228 | Providers | ||
229 | ========= | ||
230 | |||
231 | Assuming BitBake has been instructed to execute a target and that all | ||
232 | the recipe files have been parsed, BitBake starts to figure out how to | ||
233 | build the target. BitBake looks through the ``PROVIDES`` list for each | ||
234 | of the recipes. A ``PROVIDES`` list is the list of names by which the | ||
235 | recipe can be known. Each recipe's ``PROVIDES`` list is created | ||
236 | implicitly through the recipe's ```PN`` <#var-bb-PN>`__ variable and | ||
237 | explicitly through the recipe's ```PROVIDES`` <#var-bb-PROVIDES>`__ | ||
238 | variable, which is optional. | ||
239 | |||
240 | When a recipe uses ``PROVIDES``, that recipe's functionality can be | ||
241 | found under an alternative name or names other than the implicit ``PN`` | ||
242 | name. As an example, suppose a recipe named ``keyboard_1.0.bb`` | ||
243 | contained the following: PROVIDES += "fullkeyboard" The ``PROVIDES`` | ||
244 | list for this recipe becomes "keyboard", which is implicit, and | ||
245 | "fullkeyboard", which is explicit. Consequently, the functionality found | ||
246 | in ``keyboard_1.0.bb`` can be found under two different names. | ||
247 | |||
248 | .. _bb-bitbake-preferences: | ||
249 | |||
250 | Preferences | ||
251 | =========== | ||
252 | |||
253 | The ``PROVIDES`` list is only part of the solution for figuring out a | ||
254 | target's recipes. Because targets might have multiple providers, BitBake | ||
255 | needs to prioritize providers by determining provider preferences. | ||
256 | |||
257 | A common example in which a target has multiple providers is | ||
258 | "virtual/kernel", which is on the ``PROVIDES`` list for each kernel | ||
259 | recipe. Each machine often selects the best kernel provider by using a | ||
260 | line similar to the following in the machine configuration file: | ||
261 | PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" The default | ||
262 | ```PREFERRED_PROVIDER`` <#var-bb-PREFERRED_PROVIDER>`__ is the provider | ||
263 | with the same name as the target. BitBake iterates through each target | ||
264 | it needs to build and resolves them and their dependencies using this | ||
265 | process. | ||
266 | |||
267 | Understanding how providers are chosen is made complicated by the fact | ||
268 | that multiple versions might exist for a given provider. BitBake | ||
269 | defaults to the highest version of a provider. Version comparisons are | ||
270 | made using the same method as Debian. You can use the | ||
271 | ```PREFERRED_VERSION`` <#var-bb-PREFERRED_VERSION>`__ variable to | ||
272 | specify a particular version. You can influence the order by using the | ||
273 | ```DEFAULT_PREFERENCE`` <#var-bb-DEFAULT_PREFERENCE>`__ variable. | ||
274 | |||
275 | By default, files have a preference of "0". Setting | ||
276 | ``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used | ||
277 | unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to | ||
278 | "1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides | ||
279 | any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used | ||
280 | to mark newer and more experimental recipe versions until they have | ||
281 | undergone sufficient testing to be considered stable. | ||
282 | |||
283 | When there are multiple “versions” of a given recipe, BitBake defaults | ||
284 | to selecting the most recent version, unless otherwise specified. If the | ||
285 | recipe in question has a | ||
286 | ```DEFAULT_PREFERENCE`` <#var-bb-DEFAULT_PREFERENCE>`__ set lower than | ||
287 | the other recipes (default is 0), then it will not be selected. This | ||
288 | allows the person or persons maintaining the repository of recipe files | ||
289 | to specify their preference for the default selected version. | ||
290 | Additionally, the user can specify their preferred version. | ||
291 | |||
292 | If the first recipe is named ``a_1.1.bb``, then the | ||
293 | ```PN`` <#var-bb-PN>`__ variable will be set to “a”, and the | ||
294 | ```PV`` <#var-bb-PV>`__ variable will be set to 1.1. | ||
295 | |||
296 | Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by | ||
297 | default. However, if you define the following variable in a ``.conf`` | ||
298 | file that BitBake parses, you can change that preference: | ||
299 | PREFERRED_VERSION_a = "1.1" | ||
300 | |||
301 | .. note:: | ||
302 | |||
303 | It is common for a recipe to provide two versions -- a stable, | ||
304 | numbered (and preferred) version, and a version that is automatically | ||
305 | checked out from a source code repository that is considered more | ||
306 | "bleeding edge" but can be selected only explicitly. | ||
307 | |||
308 | For example, in the OpenEmbedded codebase, there is a standard, | ||
309 | versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there | ||
310 | is also a Git-based version, ``busybox_git.bb``, which explicitly | ||
311 | contains the line DEFAULT_PREFERENCE = "-1" to ensure that the | ||
312 | numbered, stable version is always preferred unless the developer | ||
313 | selects otherwise. | ||
314 | |||
315 | .. _bb-bitbake-dependencies: | ||
316 | |||
317 | Dependencies | ||
318 | ============ | ||
319 | |||
320 | Each target BitBake builds consists of multiple tasks such as ``fetch``, | ||
321 | ``unpack``, ``patch``, ``configure``, and ``compile``. For best | ||
322 | performance on multi-core systems, BitBake considers each task as an | ||
323 | independent entity with its own set of dependencies. | ||
324 | |||
325 | Dependencies are defined through several variables. You can find | ||
326 | information about variables BitBake uses in the `Variables | ||
327 | Glossary <#ref-bb-variables-glos>`__ near the end of this manual. At a | ||
328 | basic level, it is sufficient to know that BitBake uses the | ||
329 | ```DEPENDS`` <#var-bb-DEPENDS>`__ and | ||
330 | ```RDEPENDS`` <#var-bb-RDEPENDS>`__ variables when calculating | ||
331 | dependencies. | ||
332 | |||
333 | For more information on how BitBake handles dependencies, see the | ||
334 | "`Dependencies <#dependencies>`__" section. | ||
335 | |||
336 | .. _ref-bitbake-tasklist: | ||
337 | |||
338 | The Task List | ||
339 | ============= | ||
340 | |||
341 | Based on the generated list of providers and the dependency information, | ||
342 | BitBake can now calculate exactly what tasks it needs to run and in what | ||
343 | order it needs to run them. The "`Executing Tasks <#executing-tasks>`__" | ||
344 | section has more information on how BitBake chooses which task to | ||
345 | execute next. | ||
346 | |||
347 | The build now starts with BitBake forking off threads up to the limit | ||
348 | set in the ```BB_NUMBER_THREADS`` <#var-bb-BB_NUMBER_THREADS>`__ | ||
349 | variable. BitBake continues to fork threads as long as there are tasks | ||
350 | ready to run, those tasks have all their dependencies met, and the | ||
351 | thread threshold has not been exceeded. | ||
352 | |||
353 | It is worth noting that you can greatly speed up the build time by | ||
354 | properly setting the ``BB_NUMBER_THREADS`` variable. | ||
355 | |||
356 | As each task completes, a timestamp is written to the directory | ||
357 | specified by the ```STAMP`` <#var-bb-STAMP>`__ variable. On subsequent | ||
358 | runs, BitBake looks in the build directory within ``tmp/stamps`` and | ||
359 | does not rerun tasks that are already completed unless a timestamp is | ||
360 | found to be invalid. Currently, invalid timestamps are only considered | ||
361 | on a per recipe file basis. So, for example, if the configure stamp has | ||
362 | a timestamp greater than the compile timestamp for a given target, then | ||
363 | the compile task would rerun. Running the compile task again, however, | ||
364 | has no effect on other providers that depend on that target. | ||
365 | |||
366 | The exact format of the stamps is partly configurable. In modern | ||
367 | versions of BitBake, a hash is appended to the stamp so that if the | ||
368 | configuration changes, the stamp becomes invalid and the task is | ||
369 | automatically rerun. This hash, or signature used, is governed by the | ||
370 | signature policy that is configured (see the "`Checksums | ||
371 | (Signatures) <#checksums>`__" section for information). It is also | ||
372 | possible to append extra metadata to the stamp using the | ||
373 | ``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this | ||
374 | flag to make some tasks machine-specific. | ||
375 | |||
376 | .. note:: | ||
377 | |||
378 | Some tasks are marked as "nostamp" tasks. No timestamp file is | ||
379 | created when these tasks are run. Consequently, "nostamp" tasks are | ||
380 | always rerun. | ||
381 | |||
382 | For more information on tasks, see the "`Tasks <#tasks>`__" section. | ||
383 | |||
384 | Executing Tasks | ||
385 | =============== | ||
386 | |||
387 | Tasks can be either a shell task or a Python task. For shell tasks, | ||
388 | BitBake writes a shell script to | ||
389 | ``${``\ ```T`` <#var-bb-T>`__\ ``}/run.do_taskname.pid`` and then | ||
390 | executes the script. The generated shell script contains all the | ||
391 | exported variables, and the shell functions with all variables expanded. | ||
392 | Output from the shell script goes to the file | ||
393 | ``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in | ||
394 | the run file and the output in the log files is a useful debugging | ||
395 | technique. | ||
396 | |||
397 | For Python tasks, BitBake executes the task internally and logs | ||
398 | information to the controlling terminal. Future versions of BitBake will | ||
399 | write the functions to files similar to the way shell tasks are handled. | ||
400 | Logging will be handled in a way similar to shell tasks as well. | ||
401 | |||
402 | The order in which BitBake runs the tasks is controlled by its task | ||
403 | scheduler. It is possible to configure the scheduler and define custom | ||
404 | implementations for specific use cases. For more information, see these | ||
405 | variables that control the behavior: | ||
406 | |||
407 | - ```BB_SCHEDULER`` <#var-bb-BB_SCHEDULER>`__ | ||
408 | |||
409 | - ```BB_SCHEDULERS`` <#var-bb-BB_SCHEDULERS>`__ | ||
410 | |||
411 | It is possible to have functions run before and after a task's main | ||
412 | function. This is done using the ``[prefuncs]`` and ``[postfuncs]`` | ||
413 | flags of the task that lists the functions to run. | ||
414 | |||
415 | .. _checksums: | ||
416 | |||
417 | Checksums (Signatures) | ||
418 | ====================== | ||
419 | |||
420 | A checksum is a unique signature of a task's inputs. The signature of a | ||
421 | task can be used to determine if a task needs to be run. Because it is a | ||
422 | change in a task's inputs that triggers running the task, BitBake needs | ||
423 | to detect all the inputs to a given task. For shell tasks, this turns | ||
424 | out to be fairly easy because BitBake generates a "run" shell script for | ||
425 | each task and it is possible to create a checksum that gives you a good | ||
426 | idea of when the task's data changes. | ||
427 | |||
428 | To complicate the problem, some things should not be included in the | ||
429 | checksum. First, there is the actual specific build path of a given task | ||
430 | - the working directory. It does not matter if the working directory | ||
431 | changes because it should not affect the output for target packages. The | ||
432 | simplistic approach for excluding the working directory is to set it to | ||
433 | some fixed value and create the checksum for the "run" script. BitBake | ||
434 | goes one step better and uses the | ||
435 | ```BB_HASHBASE_WHITELIST`` <#var-bb-BB_HASHBASE_WHITELIST>`__ variable | ||
436 | to define a list of variables that should never be included when | ||
437 | generating the signatures. | ||
438 | |||
439 | Another problem results from the "run" scripts containing functions that | ||
440 | might or might not get called. The incremental build solution contains | ||
441 | code that figures out dependencies between shell functions. This code is | ||
442 | used to prune the "run" scripts down to the minimum set, thereby | ||
443 | alleviating this problem and making the "run" scripts much more readable | ||
444 | as a bonus. | ||
445 | |||
446 | So far we have solutions for shell scripts. What about Python tasks? The | ||
447 | same approach applies even though these tasks are more difficult. The | ||
448 | process needs to figure out what variables a Python function accesses | ||
449 | and what functions it calls. Again, the incremental build solution | ||
450 | contains code that first figures out the variable and function | ||
451 | dependencies, and then creates a checksum for the data used as the input | ||
452 | to the task. | ||
453 | |||
454 | Like the working directory case, situations exist where dependencies | ||
455 | should be ignored. For these cases, you can instruct the build process | ||
456 | to ignore a dependency by using a line like the following: | ||
457 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" This example ensures that the | ||
458 | ``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``, | ||
459 | even if it does reference it. | ||
460 | |||
461 | Equally, there are cases where we need to add dependencies BitBake is | ||
462 | not able to find. You can accomplish this by using a line like the | ||
463 | following: PACKAGE_ARCHS[vardeps] = "MACHINE" This example explicitly | ||
464 | adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. | ||
465 | |||
466 | Consider a case with in-line Python, for example, where BitBake is not | ||
467 | able to figure out dependencies. When running in debug mode (i.e. using | ||
468 | ``-DDD``), BitBake produces output when it discovers something for which | ||
469 | it cannot figure out dependencies. | ||
470 | |||
471 | Thus far, this section has limited discussion to the direct inputs into | ||
472 | a task. Information based on direct inputs is referred to as the | ||
473 | "basehash" in the code. However, there is still the question of a task's | ||
474 | indirect inputs - the things that were already built and present in the | ||
475 | build directory. The checksum (or signature) for a particular task needs | ||
476 | to add the hashes of all the tasks on which the particular task depends. | ||
477 | Choosing which dependencies to add is a policy decision. However, the | ||
478 | effect is to generate a master checksum that combines the basehash and | ||
479 | the hashes of the task's dependencies. | ||
480 | |||
481 | At the code level, there are a variety of ways both the basehash and the | ||
482 | dependent task hashes can be influenced. Within the BitBake | ||
483 | configuration file, we can give BitBake some extra information to help | ||
484 | it construct the basehash. The following statement effectively results | ||
485 | in a list of global variable dependency excludes - variables never | ||
486 | included in any checksum. This example uses variables from OpenEmbedded | ||
487 | to help illustrate the concept: BB_HASHBASE_WHITELIST ?= "TMPDIR FILE | ||
488 | PATH PWD BB_TASKHASH BBPATH DL_DIR \\ SSTATE_DIR THISDIR FILESEXTRAPATHS | ||
489 | FILE_DIRNAME HOME LOGNAME SHELL \\ USER FILESPATH STAGING_DIR_HOST | ||
490 | STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ PRSERV_DUMPDIR | ||
491 | PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ CCACHE_DIR | ||
492 | EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" The | ||
493 | previous example excludes the work directory, which is part of | ||
494 | ``TMPDIR``. | ||
495 | |||
496 | The rules for deciding which hashes of dependent tasks to include | ||
497 | through dependency chains are more complex and are generally | ||
498 | accomplished with a Python function. The code in | ||
499 | ``meta/lib/oe/sstatesig.py`` shows two examples of this and also | ||
500 | illustrates how you can insert your own policy into the system if so | ||
501 | desired. This file defines the two basic signature generators | ||
502 | OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there | ||
503 | is a dummy "noop" signature handler enabled in BitBake. This means that | ||
504 | behavior is unchanged from previous versions. ``OE-Core`` uses the | ||
505 | "OEBasicHash" signature handler by default through this setting in the | ||
506 | ``bitbake.conf`` file: BB_SIGNATURE_HANDLER ?= "OEBasicHash" The | ||
507 | "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic" | ||
508 | version but adds the task hash to the stamp files. This results in any | ||
509 | metadata change that changes the task hash, automatically causing the | ||
510 | task to be run again. This removes the need to bump | ||
511 | ```PR`` <#var-bb-PR>`__ values, and changes to metadata automatically | ||
512 | ripple across the build. | ||
513 | |||
514 | It is also worth noting that the end result of these signature | ||
515 | generators is to make some dependency and hash information available to | ||
516 | the build. This information includes: | ||
517 | |||
518 | - ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the | ||
519 | recipe. | ||
520 | |||
521 | - ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each | ||
522 | dependent task. | ||
523 | |||
524 | - ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for | ||
525 | each task. | ||
526 | |||
527 | - ``BB_TASKHASH``: The hash of the currently running task. | ||
528 | |||
529 | It is worth noting that BitBake's "-S" option lets you debug BitBake's | ||
530 | processing of signatures. The options passed to -S allow different | ||
531 | debugging modes to be used, either using BitBake's own debug functions | ||
532 | or possibly those defined in the metadata/signature handler itself. The | ||
533 | simplest parameter to pass is "none", which causes a set of signature | ||
534 | information to be written out into ``STAMPS_DIR`` corresponding to the | ||
535 | targets specified. The other currently available parameter is | ||
536 | "printdiff", which causes BitBake to try to establish the closest | ||
537 | signature match it can (e.g. in the sstate cache) and then run | ||
538 | ``bitbake-diffsigs`` over the matches to determine the stamps and delta | ||
539 | where these two stamp trees diverge. | ||
540 | |||
541 | .. note:: | ||
542 | |||
543 | It is likely that future versions of BitBake will provide other | ||
544 | signature handlers triggered through additional "-S" parameters. | ||
545 | |||
546 | You can find more information on checksum metadata in the "`Task | ||
547 | Checksums and Setscene <#task-checksums-and-setscene>`__" section. | ||
548 | |||
549 | Setscene | ||
550 | ======== | ||
551 | |||
552 | The setscene process enables BitBake to handle "pre-built" artifacts. | ||
553 | The ability to handle and reuse these artifacts allows BitBake the | ||
554 | luxury of not having to build something from scratch every time. | ||
555 | Instead, BitBake can use, when possible, existing build artifacts. | ||
556 | |||
557 | BitBake needs to have reliable data indicating whether or not an | ||
558 | artifact is compatible. Signatures, described in the previous section, | ||
559 | provide an ideal way of representing whether an artifact is compatible. | ||
560 | If a signature is the same, an object can be reused. | ||
561 | |||
562 | If an object can be reused, the problem then becomes how to replace a | ||
563 | given task or set of tasks with the pre-built artifact. BitBake solves | ||
564 | the problem with the "setscene" process. | ||
565 | |||
566 | When BitBake is asked to build a given target, before building anything, | ||
567 | it first asks whether cached information is available for any of the | ||
568 | targets it's building, or any of the intermediate targets. If cached | ||
569 | information is available, BitBake uses this information instead of | ||
570 | running the main tasks. | ||
571 | |||
572 | BitBake first calls the function defined by the | ||
573 | ```BB_HASHCHECK_FUNCTION`` <#var-bb-BB_HASHCHECK_FUNCTION>`__ variable | ||
574 | with a list of tasks and corresponding hashes it wants to build. This | ||
575 | function is designed to be fast and returns a list of the tasks for | ||
576 | which it believes in can obtain artifacts. | ||
577 | |||
578 | Next, for each of the tasks that were returned as possibilities, BitBake | ||
579 | executes a setscene version of the task that the possible artifact | ||
580 | covers. Setscene versions of a task have the string "_setscene" appended | ||
581 | to the task name. So, for example, the task with the name ``xxx`` has a | ||
582 | setscene task named ``xxx_setscene``. The setscene version of the task | ||
583 | executes and provides the necessary artifacts returning either success | ||
584 | or failure. | ||
585 | |||
586 | As previously mentioned, an artifact can cover more than one task. For | ||
587 | example, it is pointless to obtain a compiler if you already have the | ||
588 | compiled binary. To handle this, BitBake calls the | ||
589 | ```BB_SETSCENE_DEPVALID`` <#var-bb-BB_SETSCENE_DEPVALID>`__ function for | ||
590 | each successful setscene task to know whether or not it needs to obtain | ||
591 | the dependencies of that task. | ||
592 | |||
593 | Finally, after all the setscene tasks have executed, BitBake calls the | ||
594 | function listed in | ||
595 | ```BB_SETSCENE_VERIFY_FUNCTION2`` <#var-bb-BB_SETSCENE_VERIFY_FUNCTION2>`__ | ||
596 | with the list of tasks BitBake thinks has been "covered". The metadata | ||
597 | can then ensure that this list is correct and can inform BitBake that it | ||
598 | wants specific tasks to be run regardless of the setscene result. | ||
599 | |||
600 | You can find more information on setscene metadata in the "`Task | ||
601 | Checksums and Setscene <#task-checksums-and-setscene>`__" section. | ||
602 | |||
603 | Logging | ||
604 | ======= | ||
605 | |||
606 | In addition to the standard command line option to control how verbose | ||
607 | builds are when execute, bitbake also supports user defined | ||
608 | configuration of the `Python | ||
609 | logging <https://docs.python.org/3/library/logging.html>`__ facilities | ||
610 | through the ```BB_LOGCONFIG`` <#var-bb-BB_LOGCONFIG>`__ variable. This | ||
611 | variable defines a json or yaml `logging | ||
612 | configuration <https://docs.python.org/3/library/logging.config.html>`__ | ||
613 | that will be intelligently merged into the default configuration. The | ||
614 | logging configuration is merged using the following rules: | ||
615 | |||
616 | - The user defined configuration will completely replace the default | ||
617 | configuration if top level key ``bitbake_merge`` is set to the value | ||
618 | ``False``. In this case, all other rules are ignored. | ||
619 | |||
620 | - The user configuration must have a top level ``version`` which must | ||
621 | match the value of the default configuration. | ||
622 | |||
623 | - Any keys defined in the ``handlers``, ``formatters``, or ``filters``, | ||
624 | will be merged into the same section in the default configuration, | ||
625 | with the user specified keys taking replacing a default one if there | ||
626 | is a conflict. In practice, this means that if both the default | ||
627 | configuration and user configuration specify a handler named | ||
628 | ``myhandler``, the user defined one will replace the default. To | ||
629 | prevent the user from inadvertently replacing a default handler, | ||
630 | formatter, or filter, all of the default ones are named with a prefix | ||
631 | of "``BitBake.``" | ||
632 | |||
633 | - If a logger is defined by the user with the key ``bitbake_merge`` set | ||
634 | to ``False``, that logger will be completely replaced by user | ||
635 | configuration. In this case, no other rules will apply to that | ||
636 | logger. | ||
637 | |||
638 | - All user defined ``filter`` and ``handlers`` properties for a given | ||
639 | logger will be merged with corresponding properties from the default | ||
640 | logger. For example, if the user configuration adds a filter called | ||
641 | ``myFilter`` to the ``BitBake.SigGen``, and the default configuration | ||
642 | adds a filter called ``BitBake.defaultFilter``, both filters will be | ||
643 | applied to the logger | ||
644 | |||
645 | As an example, consider the following user logging configuration file | ||
646 | which logs all Hash Equivalence related messages of VERBOSE or higher to | ||
647 | a file called ``hashequiv.log`` { "version": 1, "handlers": { | ||
648 | "autobuilderlog": { "class": "logging.FileHandler", "formatter": | ||
649 | "logfileFormatter", "level": "DEBUG", "filename": "hashequiv.log", | ||
650 | "mode": "w" } }, "formatters": { "logfileFormatter": { "format": | ||
651 | "%(name)s: %(levelname)s: %(message)s" } }, "loggers": { | ||
652 | "BitBake.SigGen.HashEquiv": { "level": "VERBOSE", "handlers": | ||
653 | ["autobuilderlog"] }, "BitBake.RunQueue.HashEquiv": { "level": | ||
654 | "VERBOSE", "handlers": ["autobuilderlog"] } } } | ||