summaryrefslogtreecommitdiffstats
path: root/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
diff options
context:
space:
mode:
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml')
-rw-r--r--bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml1029
1 files changed, 0 insertions, 1029 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
deleted file mode 100644
index 04c5a26b99..0000000000
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
+++ /dev/null
@@ -1,1029 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
3
4<chapter id="bitbake-user-manual-execution">
5 <title>Execution</title>
6
7 <para>
8 The primary purpose for running BitBake is to produce some kind
9 of output such as a single installable package, a kernel, a software
10 development kit, or even a full, board-specific bootable Linux image,
11 complete with bootloader, kernel, and root filesystem.
12 Of course, you can execute the <filename>bitbake</filename>
13 command with options that cause it to execute single tasks,
14 compile single recipe files, capture or clear data, or simply
15 return information about the execution environment.
16 </para>
17
18 <para>
19 This chapter describes BitBake's execution process from start
20 to finish when you use it to create an image.
21 The execution process is launched using the following command
22 form:
23 <literallayout class='monospaced'>
24 $ bitbake <replaceable>target</replaceable>
25 </literallayout>
26 For information on the BitBake command and its options,
27 see
28 "<link linkend='bitbake-user-manual-command'>The BitBake Command</link>"
29 section.
30 <note>
31 <para>
32 Prior to executing BitBake, you should take advantage of available
33 parallel thread execution on your build host by setting the
34 <link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
35 variable in your project's <filename>local.conf</filename>
36 configuration file.
37 </para>
38
39 <para>
40 A common method to determine this value for your build host is to run
41 the following:
42 <literallayout class='monospaced'>
43 $ grep processor /proc/cpuinfo
44 </literallayout>
45 This command returns the number of processors, which takes into
46 account hyper-threading.
47 Thus, a quad-core build host with hyper-threading most likely
48 shows eight processors, which is the value you would then assign to
49 <filename>BB_NUMBER_THREADS</filename>.
50 </para>
51
52 <para>
53 A possibly simpler solution is that some Linux distributions
54 (e.g. Debian and Ubuntu) provide the <filename>ncpus</filename> command.
55 </para>
56 </note>
57 </para>
58
59 <section id='parsing-the-base-configuration-metadata'>
60 <title>Parsing the Base Configuration Metadata</title>
61
62 <para>
63 The first thing BitBake does is parse base configuration
64 metadata.
65 Base configuration metadata consists of your project's
66 <filename>bblayers.conf</filename> file to determine what
67 layers BitBake needs to recognize, all necessary
68 <filename>layer.conf</filename> files (one from each layer),
69 and <filename>bitbake.conf</filename>.
70 The data itself is of various types:
71 <itemizedlist>
72 <listitem><para><emphasis>Recipes:</emphasis>
73 Details about particular pieces of software.
74 </para></listitem>
75 <listitem><para><emphasis>Class Data:</emphasis>
76 An abstraction of common build information
77 (e.g. how to build a Linux kernel).
78 </para></listitem>
79 <listitem><para><emphasis>Configuration Data:</emphasis>
80 Machine-specific settings, policy decisions,
81 and so forth.
82 Configuration data acts as the glue to bind everything
83 together.</para></listitem>
84 </itemizedlist>
85 </para>
86
87 <para>
88 The <filename>layer.conf</filename> files are used to
89 construct key variables such as
90 <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
91 and
92 <link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
93 <filename>BBPATH</filename> is used to search for
94 configuration and class files under the
95 <filename>conf</filename> and <filename>classes</filename>
96 directories, respectively.
97 <filename>BBFILES</filename> is used to locate both recipe
98 and recipe append files
99 (<filename>.bb</filename> and <filename>.bbappend</filename>).
100 If there is no <filename>bblayers.conf</filename> file,
101 it is assumed the user has set the <filename>BBPATH</filename>
102 and <filename>BBFILES</filename> directly in the environment.
103 </para>
104
105 <para>
106 Next, the <filename>bitbake.conf</filename> file is located
107 using the <filename>BBPATH</filename> variable that was
108 just constructed.
109 The <filename>bitbake.conf</filename> file may also include other
110 configuration files using the
111 <filename>include</filename> or
112 <filename>require</filename> directives.
113 </para>
114
115 <para>
116 Prior to parsing configuration files, BitBake looks
117 at certain variables, including:
118 <itemizedlist>
119 <listitem><para>
120 <link linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
121 </para></listitem>
122 <listitem><para>
123 <link linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
124 </para></listitem>
125 <listitem><para>
126 <link linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
127 </para></listitem>
128 <listitem><para>
129 <link linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
130 </para></listitem>
131 <listitem><para>
132 <link linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
133 </para></listitem>
134 </itemizedlist>
135 The first four variables in this list relate to how BitBake treats shell
136 environment variables during task execution.
137 By default, BitBake cleans the environment variables and provides tight
138 control over the shell execution environment.
139 However, through the use of these first four variables, you can
140 apply your control regarding the
141 environment variables allowed to be used by BitBake in the shell
142 during execution of tasks.
143 See the
144 "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>"
145 section and the information about these variables in the
146 variable glossary for more information on how they work and
147 on how to use them.
148 </para>
149
150 <para>
151 The base configuration metadata is global
152 and therefore affects all recipes and tasks that are executed.
153 </para>
154
155 <para>
156 BitBake first searches the current working directory for an
157 optional <filename>conf/bblayers.conf</filename> configuration file.
158 This file is expected to contain a
159 <link linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
160 variable that is a space-delimited list of 'layer' directories.
161 Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
162 file, then it is assumed the user has set the <filename>BBPATH</filename>
163 and <filename>BBFILES</filename> variables directly in the environment.
164 </para>
165
166 <para>
167 For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
168 file is located and parsed with the
169 <link linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
170 variable being set to the directory where the layer was found.
171 The idea is these files automatically set up
172 <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
173 and other variables correctly for a given build directory.
174 </para>
175
176 <para>
177 BitBake then expects to find the <filename>conf/bitbake.conf</filename>
178 file somewhere in the user-specified <filename>BBPATH</filename>.
179 That configuration file generally has include directives to pull
180 in any other metadata such as files specific to the architecture,
181 the machine, the local environment, and so forth.
182 </para>
183
184 <para>
185 Only variable definitions and include directives are allowed
186 in BitBake <filename>.conf</filename> files.
187 Some variables directly influence BitBake's behavior.
188 These variables might have been set from the environment
189 depending on the environment variables previously
190 mentioned or set in the configuration files.
191 The
192 "<link linkend='ref-bb-variables-glos'>Variables Glossary</link>"
193 chapter presents a full list of variables.
194 </para>
195
196 <para>
197 After parsing configuration files, BitBake uses its rudimentary
198 inheritance mechanism, which is through class files, to inherit
199 some standard classes.
200 BitBake parses a class when the inherit directive responsible
201 for getting that class is encountered.
202 </para>
203
204 <para>
205 The <filename>base.bbclass</filename> file is always included.
206 Other classes that are specified in the configuration using the
207 <link linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
208 variable are also included.
209 BitBake searches for class files in a
210 <filename>classes</filename> subdirectory under
211 the paths in <filename>BBPATH</filename> in the same way as
212 configuration files.
213 </para>
214
215 <para>
216 A good way to get an idea of the configuration files and
217 the class files used in your execution environment is to
218 run the following BitBake command:
219 <literallayout class='monospaced'>
220 $ bitbake -e > mybb.log
221 </literallayout>
222 Examining the top of the <filename>mybb.log</filename>
223 shows you the many configuration files and class files
224 used in your execution environment.
225 </para>
226
227 <note>
228 <para>
229 You need to be aware of how BitBake parses curly braces.
230 If a recipe uses a closing curly brace within the function and
231 the character has no leading spaces, BitBake produces a parsing
232 error.
233 If you use a pair of curly braces in a shell function, the
234 closing curly brace must not be located at the start of the line
235 without leading spaces.
236 </para>
237
238 <para>
239 Here is an example that causes BitBake to produce a parsing
240 error:
241 <literallayout class='monospaced'>
242 fakeroot create_shar() {
243 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
244 usage()
245 {
246 echo "test"
247 ###### The following "}" at the start of the line causes a parsing error ######
248 }
249 EOF
250 }
251 </literallayout>
252 Writing the recipe this way avoids the error:
253 <literallayout class='monospaced'>
254 fakeroot create_shar() {
255 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
256 usage()
257 {
258 echo "test"
259 ######The following "}" with a leading space at the start of the line avoids the error ######
260 }
261 EOF
262 }
263 </literallayout>
264 </para>
265 </note>
266 </section>
267
268 <section id='locating-and-parsing-recipes'>
269 <title>Locating and Parsing Recipes</title>
270
271 <para>
272 During the configuration phase, BitBake will have set
273 <link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
274 BitBake now uses it to construct a list of recipes to parse,
275 along with any append files (<filename>.bbappend</filename>)
276 to apply.
277 <filename>BBFILES</filename> is a space-separated list of
278 available files and supports wildcards.
279 An example would be:
280 <literallayout class='monospaced'>
281 BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
282 </literallayout>
283 BitBake parses each recipe and append file located
284 with <filename>BBFILES</filename> and stores the values of
285 various variables into the datastore.
286 <note>
287 Append files are applied in the order they are encountered in
288 <filename>BBFILES</filename>.
289 </note>
290 For each file, a fresh copy of the base configuration is
291 made, then the recipe is parsed line by line.
292 Any inherit statements cause BitBake to find and
293 then parse class files (<filename>.bbclass</filename>)
294 using
295 <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
296 as the search path.
297 Finally, BitBake parses in order any append files found in
298 <filename>BBFILES</filename>.
299 </para>
300
301 <para>
302 One common convention is to use the recipe filename to define
303 pieces of metadata.
304 For example, in <filename>bitbake.conf</filename> the recipe
305 name and version are used to set the variables
306 <link linkend='var-bb-PN'><filename>PN</filename></link> and
307 <link linkend='var-bb-PV'><filename>PV</filename></link>:
308 <literallayout class='monospaced'>
309 PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
310 PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
311 </literallayout>
312 In this example, a recipe called "something_1.2.3.bb" would set
313 <filename>PN</filename> to "something" and
314 <filename>PV</filename> to "1.2.3".
315 </para>
316
317 <para>
318 By the time parsing is complete for a recipe, BitBake
319 has a list of tasks that the recipe defines and a set of
320 data consisting of keys and values as well as
321 dependency information about the tasks.
322 </para>
323
324 <para>
325 BitBake does not need all of this information.
326 It only needs a small subset of the information to make
327 decisions about the recipe.
328 Consequently, BitBake caches the values in which it is
329 interested and does not store the rest of the information.
330 Experience has shown it is faster to re-parse the metadata than to
331 try and write it out to the disk and then reload it.
332 </para>
333
334 <para>
335 Where possible, subsequent BitBake commands reuse this cache of
336 recipe information.
337 The validity of this cache is determined by first computing a
338 checksum of the base configuration data (see
339 <link linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
340 and then checking if the checksum matches.
341 If that checksum matches what is in the cache and the recipe
342 and class files have not changed, BitBake is able to use
343 the cache.
344 BitBake then reloads the cached information about the recipe
345 instead of reparsing it from scratch.
346 </para>
347
348 <para>
349 Recipe file collections exist to allow the user to
350 have multiple repositories of
351 <filename>.bb</filename> files that contain the same
352 exact package.
353 For example, one could easily use them to make one's
354 own local copy of an upstream repository, but with
355 custom modifications that one does not want upstream.
356 Here is an example:
357 <literallayout class='monospaced'>
358 BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
359 BBFILE_COLLECTIONS = "upstream local"
360 BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
361 BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
362 BBFILE_PRIORITY_upstream = "5"
363 BBFILE_PRIORITY_local = "10"
364 </literallayout>
365 <note>
366 The layers mechanism is now the preferred method of collecting
367 code.
368 While the collections code remains, its main use is to set layer
369 priorities and to deal with overlap (conflicts) between layers.
370 </note>
371 </para>
372 </section>
373
374 <section id='bb-bitbake-providers'>
375 <title>Providers</title>
376
377 <para>
378 Assuming BitBake has been instructed to execute a target
379 and that all the recipe files have been parsed, BitBake
380 starts to figure out how to build the target.
381 BitBake looks through the <filename>PROVIDES</filename> list
382 for each of the recipes.
383 A <filename>PROVIDES</filename> list is the list of names by which
384 the recipe can be known.
385 Each recipe's <filename>PROVIDES</filename> list is created
386 implicitly through the recipe's
387 <link linkend='var-bb-PN'><filename>PN</filename></link> variable
388 and explicitly through the recipe's
389 <link linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
390 variable, which is optional.
391 </para>
392
393 <para>
394 When a recipe uses <filename>PROVIDES</filename>, that recipe's
395 functionality can be found under an alternative name or names other
396 than the implicit <filename>PN</filename> name.
397 As an example, suppose a recipe named <filename>keyboard_1.0.bb</filename>
398 contained the following:
399 <literallayout class='monospaced'>
400 PROVIDES += "fullkeyboard"
401 </literallayout>
402 The <filename>PROVIDES</filename> list for this recipe becomes
403 "keyboard", which is implicit, and "fullkeyboard", which is explicit.
404 Consequently, the functionality found in
405 <filename>keyboard_1.0.bb</filename> can be found under two
406 different names.
407 </para>
408 </section>
409
410 <section id='bb-bitbake-preferences'>
411 <title>Preferences</title>
412
413 <para>
414 The <filename>PROVIDES</filename> list is only part of the solution
415 for figuring out a target's recipes.
416 Because targets might have multiple providers, BitBake needs
417 to prioritize providers by determining provider preferences.
418 </para>
419
420 <para>
421 A common example in which a target has multiple providers
422 is "virtual/kernel", which is on the
423 <filename>PROVIDES</filename> list for each kernel recipe.
424 Each machine often selects the best kernel provider by using a
425 line similar to the following in the machine configuration file:
426 <literallayout class='monospaced'>
427 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
428 </literallayout>
429 The default
430 <link linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
431 is the provider with the same name as the target.
432 BitBake iterates through each target it needs to build and
433 resolves them and their dependencies using this process.
434 </para>
435
436 <para>
437 Understanding how providers are chosen is made complicated by the fact
438 that multiple versions might exist for a given provider.
439 BitBake defaults to the highest version of a provider.
440 Version comparisons are made using the same method as Debian.
441 You can use the
442 <link linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
443 variable to specify a particular version.
444 You can influence the order by using the
445 <link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
446 variable.
447 </para>
448
449 <para>
450 By default, files have a preference of "0".
451 Setting <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
452 recipe unlikely to be used unless it is explicitly referenced.
453 Setting <filename>DEFAULT_PREFERENCE</filename> to "1" makes it
454 likely the recipe is used.
455 <filename>PREFERRED_VERSION</filename> overrides any
456 <filename>DEFAULT_PREFERENCE</filename> setting.
457 <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer
458 and more experimental recipe versions until they have undergone
459 sufficient testing to be considered stable.
460 </para>
461
462 <para>
463 When there are multiple “versions” of a given recipe,
464 BitBake defaults to selecting the most recent
465 version, unless otherwise specified.
466 If the recipe in question has a
467 <link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
468 set lower than the other recipes (default is 0), then
469 it will not be selected.
470 This allows the person or persons maintaining
471 the repository of recipe files to specify
472 their preference for the default selected version.
473 Additionally, the user can specify their preferred version.
474 </para>
475
476 <para>
477 If the first recipe is named <filename>a_1.1.bb</filename>, then the
478 <link linkend='var-bb-PN'><filename>PN</filename></link> variable
479 will be set to “a”, and the
480 <link linkend='var-bb-PV'><filename>PV</filename></link>
481 variable will be set to 1.1.
482 </para>
483
484 <para>
485 Thus, if a recipe named <filename>a_1.2.bb</filename> exists, BitBake
486 will choose 1.2 by default.
487 However, if you define the following variable in a
488 <filename>.conf</filename> file that BitBake parses, you
489 can change that preference:
490 <literallayout class='monospaced'>
491 PREFERRED_VERSION_a = "1.1"
492 </literallayout>
493 </para>
494
495 <note>
496 <para>
497 It is common for a recipe to provide two versions -- a stable,
498 numbered (and preferred) version, and a version that is
499 automatically checked out from a source code repository that
500 is considered more "bleeding edge" but can be selected only
501 explicitly.
502 </para>
503
504 <para>
505 For example, in the OpenEmbedded codebase, there is a standard,
506 versioned recipe file for BusyBox,
507 <filename>busybox_1.22.1.bb</filename>,
508 but there is also a Git-based version,
509 <filename>busybox_git.bb</filename>, which explicitly contains the line
510 <literallayout class='monospaced'>
511 DEFAULT_PREFERENCE = "-1"
512 </literallayout>
513 to ensure that the numbered, stable version is always preferred
514 unless the developer selects otherwise.
515 </para>
516 </note>
517 </section>
518
519 <section id='bb-bitbake-dependencies'>
520 <title>Dependencies</title>
521
522 <para>
523 Each target BitBake builds consists of multiple tasks such as
524 <filename>fetch</filename>, <filename>unpack</filename>,
525 <filename>patch</filename>, <filename>configure</filename>,
526 and <filename>compile</filename>.
527 For best performance on multi-core systems, BitBake considers each
528 task as an independent
529 entity with its own set of dependencies.
530 </para>
531
532 <para>
533 Dependencies are defined through several variables.
534 You can find information about variables BitBake uses in
535 the <link linkend='ref-bb-variables-glos'>Variables Glossary</link>
536 near the end of this manual.
537 At a basic level, it is sufficient to know that BitBake uses the
538 <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
539 <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
540 calculating dependencies.
541 </para>
542
543 <para>
544 For more information on how BitBake handles dependencies, see the
545 "<link linkend='dependencies'>Dependencies</link>" section.
546 </para>
547 </section>
548
549 <section id='ref-bitbake-tasklist'>
550 <title>The Task List</title>
551
552 <para>
553 Based on the generated list of providers and the dependency information,
554 BitBake can now calculate exactly what tasks it needs to run and in what
555 order it needs to run them.
556 The
557 "<link linkend='executing-tasks'>Executing Tasks</link>" section has more
558 information on how BitBake chooses which task to execute next.
559 </para>
560
561 <para>
562 The build now starts with BitBake forking off threads up to the limit set in the
563 <link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
564 variable.
565 BitBake continues to fork threads as long as there are tasks ready to run,
566 those tasks have all their dependencies met, and the thread threshold has not been
567 exceeded.
568 </para>
569
570 <para>
571 It is worth noting that you can greatly speed up the build time by properly setting
572 the <filename>BB_NUMBER_THREADS</filename> variable.
573 </para>
574
575 <para>
576 As each task completes, a timestamp is written to the directory specified by the
577 <link linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable.
578 On subsequent runs, BitBake looks in the build directory within
579 <filename>tmp/stamps</filename> and does not rerun
580 tasks that are already completed unless a timestamp is found to be invalid.
581 Currently, invalid timestamps are only considered on a per
582 recipe file basis.
583 So, for example, if the configure stamp has a timestamp greater than the
584 compile timestamp for a given target, then the compile task would rerun.
585 Running the compile task again, however, has no effect on other providers
586 that depend on that target.
587 </para>
588
589 <para>
590 The exact format of the stamps is partly configurable.
591 In modern versions of BitBake, a hash is appended to the
592 stamp so that if the configuration changes, the stamp becomes
593 invalid and the task is automatically rerun.
594 This hash, or signature used, is governed by the signature policy
595 that is configured (see the
596 "<link linkend='checksums'>Checksums (Signatures)</link>"
597 section for information).
598 It is also possible to append extra metadata to the stamp using
599 the <filename>[stamp-extra-info]</filename> task flag.
600 For example, OpenEmbedded uses this flag to make some tasks machine-specific.
601 </para>
602
603 <note>
604 Some tasks are marked as "nostamp" tasks.
605 No timestamp file is created when these tasks are run.
606 Consequently, "nostamp" tasks are always rerun.
607 </note>
608
609 <para>
610 For more information on tasks, see the
611 "<link linkend='tasks'>Tasks</link>" section.
612 </para>
613 </section>
614
615 <section id='executing-tasks'>
616 <title>Executing Tasks</title>
617
618 <para>
619 Tasks can be either a shell task or a Python task.
620 For shell tasks, BitBake writes a shell script to
621 <filename>${</filename><link linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.<replaceable>pid</replaceable></filename>
622 and then executes the script.
623 The generated shell script contains all the exported variables,
624 and the shell functions with all variables expanded.
625 Output from the shell script goes to the file
626 <filename>${T}/log.do_taskname.<replaceable>pid</replaceable></filename>.
627 Looking at the expanded shell functions in the run file and
628 the output in the log files is a useful debugging technique.
629 </para>
630
631 <para>
632 For Python tasks, BitBake executes the task internally and logs
633 information to the controlling terminal.
634 Future versions of BitBake will write the functions to files
635 similar to the way shell tasks are handled.
636 Logging will be handled in a way similar to shell tasks as well.
637 </para>
638
639 <para>
640 The order in which BitBake runs the tasks is controlled by its
641 task scheduler.
642 It is possible to configure the scheduler and define custom
643 implementations for specific use cases.
644 For more information, see these variables that control the
645 behavior:
646 <itemizedlist>
647 <listitem><para>
648 <link linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
649 </para></listitem>
650 <listitem><para>
651 <link linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
652 </para></listitem>
653 </itemizedlist>
654 It is possible to have functions run before and after a task's main
655 function.
656 This is done using the <filename>[prefuncs]</filename>
657 and <filename>[postfuncs]</filename> flags of the task
658 that lists the functions to run.
659 </para>
660 </section>
661
662 <section id='checksums'>
663 <title>Checksums (Signatures)</title>
664
665 <para>
666 A checksum is a unique signature of a task's inputs.
667 The signature of a task can be used to determine if a task
668 needs to be run.
669 Because it is a change in a task's inputs that triggers running
670 the task, BitBake needs to detect all the inputs to a given task.
671 For shell tasks, this turns out to be fairly easy because
672 BitBake generates a "run" shell script for each task and
673 it is possible to create a checksum that gives you a good idea of when
674 the task's data changes.
675 </para>
676
677 <para>
678 To complicate the problem, some things should not be included in
679 the checksum.
680 First, there is the actual specific build path of a given task -
681 the working directory.
682 It does not matter if the working directory changes because it should not
683 affect the output for target packages.
684 The simplistic approach for excluding the working directory is to set
685 it to some fixed value and create the checksum for the "run" script.
686 BitBake goes one step better and uses the
687 <link linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
688 variable to define a list of variables that should never be included
689 when generating the signatures.
690 </para>
691
692 <para>
693 Another problem results from the "run" scripts containing functions that
694 might or might not get called.
695 The incremental build solution contains code that figures out dependencies
696 between shell functions.
697 This code is used to prune the "run" scripts down to the minimum set,
698 thereby alleviating this problem and making the "run" scripts much more
699 readable as a bonus.
700 </para>
701
702 <para>
703 So far we have solutions for shell scripts.
704 What about Python tasks?
705 The same approach applies even though these tasks are more difficult.
706 The process needs to figure out what variables a Python function accesses
707 and what functions it calls.
708 Again, the incremental build solution contains code that first figures out
709 the variable and function dependencies, and then creates a checksum for the data
710 used as the input to the task.
711 </para>
712
713 <para>
714 Like the working directory case, situations exist where dependencies
715 should be ignored.
716 For these cases, you can instruct the build process to ignore a dependency
717 by using a line like the following:
718 <literallayout class='monospaced'>
719 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
720 </literallayout>
721 This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
722 depend on the value of <filename>MACHINE</filename>, even if it does reference it.
723 </para>
724
725 <para>
726 Equally, there are cases where we need to add dependencies BitBake
727 is not able to find.
728 You can accomplish this by using a line like the following:
729 <literallayout class='monospaced'>
730 PACKAGE_ARCHS[vardeps] = "MACHINE"
731 </literallayout>
732 This example explicitly adds the <filename>MACHINE</filename> variable as a
733 dependency for <filename>PACKAGE_ARCHS</filename>.
734 </para>
735
736 <para>
737 Consider a case with in-line Python, for example, where BitBake is not
738 able to figure out dependencies.
739 When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
740 produces output when it discovers something for which it cannot figure out
741 dependencies.
742 </para>
743
744 <para>
745 Thus far, this section has limited discussion to the direct inputs into a task.
746 Information based on direct inputs is referred to as the "basehash" in the
747 code.
748 However, there is still the question of a task's indirect inputs - the
749 things that were already built and present in the build directory.
750 The checksum (or signature) for a particular task needs to add the hashes
751 of all the tasks on which the particular task depends.
752 Choosing which dependencies to add is a policy decision.
753 However, the effect is to generate a master checksum that combines the basehash
754 and the hashes of the task's dependencies.
755 </para>
756
757 <para>
758 At the code level, there are a variety of ways both the basehash and the
759 dependent task hashes can be influenced.
760 Within the BitBake configuration file, we can give BitBake some extra information
761 to help it construct the basehash.
762 The following statement effectively results in a list of global variable
763 dependency excludes - variables never included in any checksum.
764 This example uses variables from OpenEmbedded to help illustrate
765 the concept:
766 <literallayout class='monospaced'>
767 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
768 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \
769 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
770 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
771 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
772 </literallayout>
773 The previous example excludes the work directory, which is part of
774 <filename>TMPDIR</filename>.
775 </para>
776
777 <para>
778 The rules for deciding which hashes of dependent tasks to include through
779 dependency chains are more complex and are generally accomplished with a
780 Python function.
781 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
782 of this and also illustrates how you can insert your own policy into the system
783 if so desired.
784 This file defines the two basic signature generators OpenEmbedded-Core
785 uses: "OEBasic" and "OEBasicHash".
786 By default, there is a dummy "noop" signature handler enabled in BitBake.
787 This means that behavior is unchanged from previous versions.
788 <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
789 through this setting in the <filename>bitbake.conf</filename> file:
790 <literallayout class='monospaced'>
791 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
792 </literallayout>
793 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
794 "OEBasic" version but adds the task hash to the stamp files.
795 This results in any metadata change that changes the task hash, automatically
796 causing the task to be run again.
797 This removes the need to bump
798 <link linkend='var-bb-PR'><filename>PR</filename></link>
799 values, and changes to metadata automatically ripple across the build.
800 </para>
801
802 <para>
803 It is also worth noting that the end result of these signature generators is to
804 make some dependency and hash information available to the build.
805 This information includes:
806 <itemizedlist>
807 <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
808 The base hashes for each task in the recipe.
809 </para></listitem>
810 <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
811 The base hashes for each dependent task.
812 </para></listitem>
813 <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
814 The task dependencies for each task.
815 </para></listitem>
816 <listitem><para><filename>BB_TASKHASH</filename>:
817 The hash of the currently running task.
818 </para></listitem>
819 </itemizedlist>
820 </para>
821
822 <para>
823 It is worth noting that BitBake's "-S" option lets you
824 debug BitBake's processing of signatures.
825 The options passed to -S allow different debugging modes
826 to be used, either using BitBake's own debug functions
827 or possibly those defined in the metadata/signature handler
828 itself.
829 The simplest parameter to pass is "none", which causes a
830 set of signature information to be written out into
831 <filename>STAMPS_DIR</filename>
832 corresponding to the targets specified.
833 The other currently available parameter is "printdiff",
834 which causes BitBake to try to establish the closest
835 signature match it can (e.g. in the sstate cache) and then
836 run <filename>bitbake-diffsigs</filename> over the matches
837 to determine the stamps and delta where these two
838 stamp trees diverge.
839 <note>
840 It is likely that future versions of BitBake will
841 provide other signature handlers triggered through
842 additional "-S" parameters.
843 </note>
844 </para>
845
846 <para>
847 You can find more information on checksum metadata in the
848 "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
849 section.
850 </para>
851 </section>
852
853 <section id='setscene'>
854 <title>Setscene</title>
855
856 <para>
857 The setscene process enables BitBake to handle "pre-built" artifacts.
858 The ability to handle and reuse these artifacts allows BitBake
859 the luxury of not having to build something from scratch every time.
860 Instead, BitBake can use, when possible, existing build artifacts.
861 </para>
862
863 <para>
864 BitBake needs to have reliable data indicating whether or not an
865 artifact is compatible.
866 Signatures, described in the previous section, provide an ideal
867 way of representing whether an artifact is compatible.
868 If a signature is the same, an object can be reused.
869 </para>
870
871 <para>
872 If an object can be reused, the problem then becomes how to
873 replace a given task or set of tasks with the pre-built artifact.
874 BitBake solves the problem with the "setscene" process.
875 </para>
876
877 <para>
878 When BitBake is asked to build a given target, before building anything,
879 it first asks whether cached information is available for any of the
880 targets it's building, or any of the intermediate targets.
881 If cached information is available, BitBake uses this information instead of
882 running the main tasks.
883 </para>
884
885 <para>
886 BitBake first calls the function defined by the
887 <link linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
888 variable with a list of tasks and corresponding
889 hashes it wants to build.
890 This function is designed to be fast and returns a list
891 of the tasks for which it believes in can obtain artifacts.
892 </para>
893
894 <para>
895 Next, for each of the tasks that were returned as possibilities,
896 BitBake executes a setscene version of the task that the possible
897 artifact covers.
898 Setscene versions of a task have the string "_setscene" appended to the
899 task name.
900 So, for example, the task with the name <filename>xxx</filename> has
901 a setscene task named <filename>xxx_setscene</filename>.
902 The setscene version of the task executes and provides the necessary
903 artifacts returning either success or failure.
904 </para>
905
906 <para>
907 As previously mentioned, an artifact can cover more than one task.
908 For example, it is pointless to obtain a compiler if you
909 already have the compiled binary.
910 To handle this, BitBake calls the
911 <link linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
912 function for each successful setscene task to know whether or not it needs
913 to obtain the dependencies of that task.
914 </para>
915
916 <para>
917 Finally, after all the setscene tasks have executed, BitBake calls the
918 function listed in
919 <link linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
920 with the list of tasks BitBake thinks has been "covered".
921 The metadata can then ensure that this list is correct and can
922 inform BitBake that it wants specific tasks to be run regardless
923 of the setscene result.
924 </para>
925
926 <para>
927 You can find more information on setscene metadata in the
928 "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
929 section.
930 </para>
931 </section>
932
933 <section id="logging">
934 <title>Logging</title>
935 <para>
936 In addition to the standard command line option to control how
937 verbose builds are when execute, bitbake also supports user defined
938 configuration of the
939 <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
940 facilities through the
941 <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
942 variable. This variable defines a json or yaml
943 <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
944 that will be intelligently merged into the default configuration.
945 The logging configuration is merged using the following rules:
946 <itemizedlist>
947 <listitem><para>
948 The user defined configuration will completely replace the default
949 configuration if top level key
950 <filename>bitbake_merge</filename> is set to the value
951 <filename>False</filename>. In this case, all other rules
952 are ignored.
953 </para></listitem>
954 <listitem><para>
955 The user configuration must have a top level
956 <filename>version</filename> which must match the value of
957 the default configuration.
958 </para></listitem>
959 <listitem><para>
960 Any keys defined in the <filename>handlers</filename>,
961 <filename>formatters</filename>, or <filename>filters</filename>,
962 will be merged into the same section in the default
963 configuration, with the user specified keys taking
964 replacing a default one if there is a conflict. In
965 practice, this means that if both the default configuration
966 and user configuration specify a handler named
967 <filename>myhandler</filename>, the user defined one will
968 replace the default. To prevent the user from inadvertently
969 replacing a default handler, formatter, or filter, all of
970 the default ones are named with a prefix of
971 "<filename>BitBake.</filename>"
972 </para></listitem>
973 <listitem><para>
974 If a logger is defined by the user with the key
975 <filename>bitbake_merge</filename> set to
976 <filename>False</filename>, that logger will be completely
977 replaced by user configuration. In this case, no other
978 rules will apply to that logger.
979 </para></listitem>
980 <listitem><para>
981 All user defined <filename>filter</filename> and
982 <filename>handlers</filename> properties for a given logger
983 will be merged with corresponding properties from the
984 default logger. For example, if the user configuration adds
985 a filter called <filename>myFilter</filename> to the
986 <filename>BitBake.SigGen</filename>, and the default
987 configuration adds a filter called
988 <filename>BitBake.defaultFilter</filename>, both filters
989 will be applied to the logger
990 </para></listitem>
991 </itemizedlist>
992 </para>
993
994 <para>
995 As an example, consider the following user logging configuration
996 file which logs all Hash Equivalence related messages of VERBOSE or
997 higher to a file called <filename>hashequiv.log</filename>
998 <literallayout class='monospaced'>
999 {
1000 "version": 1,
1001 "handlers": {
1002 "autobuilderlog": {
1003 "class": "logging.FileHandler",
1004 "formatter": "logfileFormatter",
1005 "level": "DEBUG",
1006 "filename": "hashequiv.log",
1007 "mode": "w"
1008 }
1009 },
1010 "formatters": {
1011 "logfileFormatter": {
1012 "format": "%(name)s: %(levelname)s: %(message)s"
1013 }
1014 },
1015 "loggers": {
1016 "BitBake.SigGen.HashEquiv": {
1017 "level": "VERBOSE",
1018 "handlers": ["autobuilderlog"]
1019 },
1020 "BitBake.RunQueue.HashEquiv": {
1021 "level": "VERBOSE",
1022 "handlers": ["autobuilderlog"]
1023 }
1024 }
1025 }
1026 </literallayout>
1027 </para>
1028 </section>
1029</chapter>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-01-15 05:52:53 +0000