summaryrefslogtreecommitdiffstats
path: root/bitbake/doc/user-manual/user-manual-execution.xml
diff options
context:
space:
mode:
authorRichard Purdie <richard.purdie@linuxfoundation.org>2014-04-11 17:38:18 +0100
committerRichard Purdie <richard.purdie@linuxfoundation.org>2014-04-21 23:03:18 +0100
commit392de9d5685e17aed868eb5659032103748825f4 (patch)
treedf0ea22b9719dbddb374aeb185ec9760eccd74bb /bitbake/doc/user-manual/user-manual-execution.xml
parent982aa36251a4a855102575b8a590929787c645fd (diff)
downloadpoky-392de9d5685e17aed868eb5659032103748825f4.tar.gz
bitbake: doc: Rename user-manual -> bitbake-user-manual
This manual gets combined with other manuals and in that context, it helps a lot if its seen as the Bitbake User Manual. Renames are a pain but this is worthwhile so that other docs can correctly be combined with this one. This also clarifies things like google search results which is helpful. (Bitbake rev: 452a62ae0c2793e281d6769fd3e45500a74898d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'bitbake/doc/user-manual/user-manual-execution.xml')
-rw-r--r--bitbake/doc/user-manual/user-manual-execution.xml850
1 files changed, 0 insertions, 850 deletions
diff --git a/bitbake/doc/user-manual/user-manual-execution.xml b/bitbake/doc/user-manual/user-manual-execution.xml
deleted file mode 100644
index 1cb7fc63ce..0000000000
--- a/bitbake/doc/user-manual/user-manual-execution.xml
+++ /dev/null
@@ -1,850 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
3
4<chapter id="user-manual-execution">
5 <title>Execution</title>
6
7 <para>
8 The primary purpose for running BitBake is to produce some kind
9 of output such as an image, a kernel, or a software development
10 kit.
11 Of course, you can execute the <filename>bitbake</filename>
12 command with options that cause it to execute single tasks,
13 compile single recipe files, capture or clear data, or simply
14 return information about the execution environment.
15 </para>
16
17 <para>
18 This chapter describes BitBake's execution process from start
19 to finish when you use it to create an image.
20 The execution process is launched using the following command
21 form:
22 <literallayout class='monospaced'>
23 $ bitbake &lt;target&gt;
24 </literallayout>
25 For information on the BitBake command and its options,
26 see
27 "<link linkend='user-manual-command'>The BitBake Command</link>"
28 section.
29 </para>
30
31 <note>
32 Prior to executing BitBake, you should take advantage of parallel
33 thread execution by setting the
34 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
35 variable in your <filename>local.conf</filename>
36 configuration file.
37 </note>
38
39 <section id='parsing-the-base-configuration-metadata'>
40 <title>Parsing the Base Configuration Metadata</title>
41
42 <para>
43 The first thing BitBake does is parse base configuration
44 metadata.
45 Base configuration metadata consists of the
46 <filename>bblayers.conf</filename> file to determine what
47 layers BitBake needs to recognize, all necessary
48 <filename>layer.conf</filename> files (one from each layer),
49 and <filename>bitbake.conf</filename>.
50 The data itself is of various types:
51 <itemizedlist>
52 <listitem><para><emphasis>Recipes:</emphasis>
53 Details about particular pieces of software.
54 </para></listitem>
55 <listitem><para><emphasis>Class Data:</emphasis>
56 An abstraction of common build information
57 (e.g. how to build a Linux kernel).
58 </para></listitem>
59 <listitem><para><emphasis>Configuration Data:</emphasis>
60 Machine-specific settings, policy decisions,
61 and so forth.
62 Configuration data acts as the glue to bind everything
63 together.</para></listitem>
64 </itemizedlist>
65 </para>
66
67 <para>
68 The <filename>layer.conf</filename> files are used to
69 construct key variables such as
70 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
71 and
72 <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
73 <filename>BBPATH</filename> is used to search for
74 configuration and class files under
75 <filename>conf/</filename> and <filename>class/</filename>
76 directories, respectively.
77 <filename>BBFILES</filename> is used to find recipe files
78 (<filename>.bb</filename> and <filename>.bbappend</filename>).
79 If there is no <filename>bblayers.conf</filename> file,
80 it is assumed the user has set the <filename>BBPATH</filename>
81 and <filename>BBFILES</filename> directly in the environment.
82 </para>
83
84 <para>
85 Next, the <filename>bitbake.conf</filename> file is searched
86 using the <filename>BBPATH</filename> variable that was
87 just constructed.
88 The <filename>bitbake.conf</filename> file may also include other
89 configuration files using the
90 <filename>include</filename> or
91 <filename>require</filename> directives.
92 </para>
93
94 <para>
95 Prior to parsing configuration files, Bitbake looks
96 at certain variables, including:
97 <itemizedlist>
98 <listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem>
99 <listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem>
100 <listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem>
101 <listitem><para>
102 <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
103 </para></listitem>
104 </itemizedlist>
105 You can find information on how to pass environment variables into the BitBake
106 execution environment in the
107 "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>" section.
108 </para>
109
110 <para>
111 The base configuration metadata is global
112 and therefore affects all recipes and tasks that are executed.
113 </para>
114
115 <para>
116 BitBake first searches the current working directory for an
117 optional <filename>conf/bblayers.conf</filename> configuration file.
118 This file is expected to contain a
119 <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
120 variable that is a space delimited list of 'layer' directories.
121 Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
122 file then it is assumed the user has set the <filename>BBPATH</filename>
123 and <filename>BBFILES</filename> directly in the environment.
124 </para>
125
126 <para>
127 For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
128 file is searched for and parsed with the
129 <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
130 variable being set to the directory where the layer was found.
131 The idea is these files automatically setup
132 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
133 and other variables correctly for a given build directory.
134 </para>
135
136 <para>
137 BitBake then expects to find the <filename>conf/bitbake.conf</filename>
138 file somewhere in the user-specified <filename>BBPATH</filename>.
139 That configuration file generally has include directives to pull
140 in any other metadata such as files specific to the architecture,
141 the machine, the local environment, and so forth.
142 </para>
143
144 <para>
145 Only variable definitions and include directives are allowed
146 in <filename>.conf</filename> files.
147 Some variables directly influence BitBake's behavior.
148 These variables might have been set from the environment
149 depending on the environment variables previously
150 mentioned or set in the configuration files.
151 The
152 "<link linkend='ref-variables-glos'>Variables Glossary</link>"
153 chapter presents a full list of variables.
154 </para>
155
156 <para>
157 After parsing configuration files, BitBake uses its rudimentary
158 inheritance mechanism, which is through class files, to inherit
159 some standard classes.
160 BitBake parses a class when the inherit directive responsible
161 for getting that class is encountered.
162 </para>
163
164 <para>
165 The <filename>base.bbclass</filename> file is always included.
166 Other classes that are specified in the configuration using the
167 <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
168 variable are also included.
169 BitBake searches for class files in a "classes" subdirectory under
170 the paths in <filename>BBPATH</filename> in the same way as
171 configuration files.
172 </para>
173
174 <para>
175 A good way to get an idea of the configuration files and
176 the class files used in your execution environment is to
177 run the following BitBake command:
178 <literallayout class='monospaced'>
179 $ bitbake -e > mybb.log
180 </literallayout>
181 Examining the top of the <filename>mybb.log</filename>
182 shows you the many configuration files and class files
183 used in your execution environment.
184 </para>
185
186 <note>
187 <para>
188 You need to be aware of how BitBake parses curly braces.
189 If a recipe uses a closing curly brace within the function and
190 the character has no leading spaces, BitBake produces a parsing
191 error.
192 If you use a pair of curly brace in a shell function, the
193 closing curly brace must not be located at the start of the line
194 without leading spaces.
195 </para>
196
197 <para>
198 Here is an example that causes BitBake to produce a parsing
199 error:
200 <literallayout class='monospaced'>
201 fakeroot create_shar() {
202 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
203 usage()
204 {
205 echo "test"
206 ###### The following "}" at the start of the line causes a parsing error ######
207 }
208 EOF
209 }
210 </literallayout>
211 Writing the recipe this way avoids the error:
212 <literallayout class='monospaced'>
213 fakeroot create_shar() {
214 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
215 usage()
216 {
217 echo "test"
218 ######The following "}" with a leading space at the start of the line avoids the error ######
219 }
220 EOF
221 }
222 </literallayout>
223 </para>
224 </note>
225 </section>
226
227 <section id='locating-and-parsing-recipes'>
228 <title>Locating and Parsing Recipes</title>
229
230 <para>
231 During the configuration phase, BitBake will have set
232 <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
233 BitBake now uses it to construct a list of recipes to parse,
234 along with any append files (<filename>.bbappend</filename>)
235 to apply.
236 <filename>BBFILES</filename> is a space-separated list of
237 available files and supports wildcards.
238 An example would be:
239 <literallayout class='monospaced'>
240 BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
241 </literallayout>
242 BitBake parses each recipe and append file located
243 with <filename>BBFILES</filename> and stores the values of
244 various variables into the datastore.
245 <note>
246 Append files are applied in the order they are encountered in
247 <filename>BBFILES</filename>.
248 </note>
249 For each file, a fresh copy of the base configuration is
250 made, then the recipe is parsed line by line.
251 Any inherit statements cause BitBake to find and
252 then parse class files (<filename>.bbclass</filename>)
253 using
254 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
255 as the search path.
256 Finally, BitBake parses in order any append files found in
257 <filename>BBFILES</filename>.
258 </para>
259
260 <para>
261 One common convention is to use the recipe filename to define
262 pieces of metadata.
263 For example, in <filename>bitbake.conf</filename> the recipe
264 name and version set
265 <link linkend='var-PN'><filename>PN</filename></link> and
266 <link linkend='var-PV'><filename>PV</filename></link>:
267 <literallayout class='monospaced'>
268 PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}"
269 PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or 'defaultpkgname'}"
270 </literallayout>
271 In this example, a recipe called "something_1.2.3.bb" sets
272 <filename>PN</filename> to "something" and
273 <filename>PV</filename> to "1.2.3".
274 </para>
275
276 <para>
277 By the time parsing is complete for a recipe, BitBake
278 has a list of tasks that the recipe defines and a set of
279 data consisting of keys and values as well as
280 dependency information about the tasks.
281 </para>
282
283 <para>
284 BitBake does not need all of this information.
285 It only needs a small subset of the information to make
286 decisions about the recipe.
287 Consequently, BitBake caches the values in which it is
288 interested and does not store the rest of the information.
289 Experience has shown it is faster to re-parse the metadata than to
290 try and write it out to the disk and then reload it.
291 </para>
292
293 <para>
294 Where possible, subsequent BitBake commands reuse this cache of
295 recipe information.
296 The validity of this cache is determined by first computing a
297 checksum of the base configuration data (see
298 <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
299 and then checking if the checksum matches.
300 If that checksum matches what is in the cache and the recipe
301 and class files have not changed, Bitbake is able to use
302 the cache.
303 BitBake then reloads the cached information about the recipe
304 instead of reparsing it from scratch.
305 </para>
306
307 <para>
308 Recipe file collections exist to allow the user to
309 have multiple repositories of
310 <filename>.bb</filename> files that contain the same
311 exact package.
312 For example, one could easily use them to make one's
313 own local copy of an upstream repository, but with
314 custom modifications that one does not want upstream.
315 Here is an example:
316 <literallayout class='monospaced'>
317 BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
318 BBFILE_COLLECTIONS = "upstream local"
319 BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
320 BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
321 BBFILE_PRIORITY_upstream = "5"
322 BBFILE_PRIORITY_local = "10"
323 </literallayout>
324 <note>
325 The layers mechanism is now the preferred method of collecting
326 code.
327 While the collections code remains, its main use is to set layer
328 priorities and to deal with overlap (conflicts) between layers.
329 </note>
330 </para>
331 </section>
332
333 <section id='bb-bitbake-providers'>
334 <title>Preferences and Providers</title>
335
336 <para>
337 Assuming BitBake has been instructed to execute a target
338 and that all the recipe files have been parsed, BitBake
339 starts to figure out how to build the target.
340 BitBake starts by looking through the
341 <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
342 set in recipe files.
343 The default <filename>PROVIDES</filename> for a recipe is its name
344 (<link linkend='var-PN'><filename>PN</filename></link>),
345 however, a recipe can provide multiple things.
346 </para>
347
348 <para>
349 As an example of adding an extra provider, suppose a recipe named
350 <filename>foo_1.0.bb</filename> contained the following:
351 <literallayout class='monospaced'>
352 PROVIDES += "virtual/bar_1.0"
353 </literallayout>
354 The recipe now provides both "foo_1.0" and "virtual/bar_1.0".
355 The "virtual/" namespace is often used to denote cases where
356 multiple providers are expected with the user choosing between
357 them.
358 Kernels and toolchain components are common cases of this in
359 OpenEmbedded.
360 </para>
361
362 <para>
363 Sometimes a target might have multiple providers.
364 A common example is "virtual/kernel", which is provided by each
365 kernel recipe.
366 Each machine often selects the best kernel provider by using a
367 line similar to the following in the machine configuration file:
368 <literallayout class='monospaced'>
369 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
370 </literallayout>
371 The default
372 <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
373 is the provider with the same name as the target.
374 Bitbake iterates through each target it needs to build and
375 resolves them and their dependencies using this process.
376 </para>
377
378 <para>
379 Understanding how providers are chosen is made complicated by the fact
380 that multiple versions might exist.
381 BitBake defaults to the highest version of a provider.
382 Version comparisons are made using the same method as Debian.
383 You can use the
384 <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
385 variable to specify a particular version.
386 You can influence the order by using the
387 <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
388 variable.
389 By default, files have a preference of "0".
390 Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
391 recipe unlikely to be used unless it is explicitly referenced.
392 Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the recipe is used.
393 <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
394 <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental recipe
395 versions until they have undergone sufficient testing to be considered stable.
396 </para>
397
398 <para>
399 When there are multiple “versions” of a given recipe,
400 BitBake defaults to selecting the most recent
401 version, unless otherwise specified.
402 If the recipe in question has a
403 <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
404 set lower than
405 the other recipes (default is 0), then it will not be
406 selected.
407 This allows the person or persons maintaining
408 the repository of recipe files to specify
409 their preference for the default selected version.
410 In addition, the user can specify their preferred version.
411 </para>
412
413 <para>
414 If the first recipe is named <filename>a_1.1.bb</filename>,
415 then the
416 <link linkend='var-PN'><filename>PN</filename></link> variable
417 will be set to “a”, and the
418 <link linkend='var-PV'><filename>PV</filename></link>
419 variable will be set to 1.1.
420 </para>
421
422 <para>
423 If we then have a recipe named <filename>a_1.2.bb</filename>, BitBake
424 will choose 1.2 by default.
425 However, if we define the following variable in a
426 <filename>.conf</filename> file that BitBake parses, we
427 can change that.
428 <literallayout class='monospaced'>
429 PREFERRED_VERSION_a = "1.1"
430 </literallayout>
431 </para>
432
433 <para>
434 In summary, BitBake has created a list of providers, which is prioritized, for each target.
435 </para>
436 </section>
437
438 <section id='bb-bitbake-dependencies'>
439 <title>Dependencies</title>
440
441 <para>
442 Each target BitBake builds consists of multiple tasks such as
443 <filename>fetch</filename>, <filename>unpack</filename>,
444 <filename>patch</filename>, <filename>configure</filename>,
445 and <filename>compile</filename>.
446 For best performance on multi-core systems, BitBake considers each
447 task as an independent
448 entity with its own set of dependencies.
449 </para>
450
451 <para>
452 Dependencies are defined through several variables.
453 You can find information about variables BitBake uses in
454 the <link linkend='ref-variables-glos'>Variables Glossary</link>
455 near the end of this manual.
456 At a basic level, it is sufficient to know that BitBake uses the
457 <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
458 <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
459 calculating dependencies.
460 </para>
461
462 <para>
463 For more information on how BitBake handles dependencies, see the
464 "<link linkend='dependencies'>Dependencies</link>" section.
465 </para>
466 </section>
467
468 <section id='ref-bitbake-tasklist'>
469 <title>The Task List</title>
470
471 <para>
472 Based on the generated list of providers and the dependency information,
473 BitBake can now calculate exactly what tasks it needs to run and in what
474 order it needs to run them.
475 The
476 "<link linkend='executing-tasks'>Executing Tasks</link>" section has more
477 information on how BitBake chooses which task to execute next.
478 </para>
479
480 <para>
481 The build now starts with BitBake forking off threads up to the limit set in the
482 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
483 variable.
484 BitBake continues to fork threads as long as there are tasks ready to run,
485 those tasks have all their dependencies met, and the thread threshold has not been
486 exceeded.
487 </para>
488
489 <para>
490 It is worth noting that you can greatly speed up the build time by properly setting
491 the <filename>BB_NUMBER_THREADS</filename> variable.
492 </para>
493
494 <para>
495 As each task completes, a timestamp is written to the directory specified by the
496 <link linkend='var-STAMP'><filename>STAMP</filename></link> variable.
497 On subsequent runs, BitBake looks in the build directory within
498 <filename>tmp/stamps</filename>and does not rerun
499 tasks that are already completed unless a timestamp is found to be invalid.
500 Currently, invalid timestamps are only considered on a per
501 recipe file basis.
502 So, for example, if the configure stamp has a timestamp greater than the
503 compile timestamp for a given target, then the compile task would rerun.
504 Running the compile task again, however, has no effect on other providers
505 that depend on that target.
506 </para>
507
508 <para>
509 The exact format of the stamps is partly configurable.
510 In modern versions of BitBake, a hash is appended to the
511 stamp so that if the configuration changes, the stamp becomes
512 invalid and the task is automatically rerun.
513 This hash, or signature used, is governed by the signature policy
514 that is configured (see the
515 "<link linkend='checksums'>Checksums (Signatures)</link>"
516 section for information).
517 It is also possible to append extra metadata to the stamp using
518 the "stamp-extra-info" task flag.
519 For example, OpenEmbedded uses this flag to make some tasks machine-specific.
520 </para>
521
522 <note>
523 Some tasks are marked as "nostamp" tasks.
524 No timestamp file is created when these tasks are run.
525 Consequently, "nostamp" tasks are always rerun.
526 </note>
527
528 <para>
529 For more information on tasks, see the
530 "<link linkend='tasks'>Tasks</link>" section.
531 </para>
532 </section>
533
534 <section id='executing-tasks'>
535 <title>Executing Tasks</title>
536
537 <para>
538 Tasks can either be a shell task or a Python task.
539 For shell tasks, BitBake writes a shell script to
540 <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
541 and then executes the script.
542 The generated shell script contains all the exported variables,
543 and the shell functions with all variables expanded.
544 Output from the shell script goes to the file
545 <filename>${T}/log.do_taskname.pid</filename>.
546 Looking at the expanded shell functions in the run file and
547 the output in the log files is a useful debugging technique.
548 </para>
549
550 <para>
551 For Python tasks, BitBake executes the task internally and logs
552 information to the controlling terminal.
553 Future versions of BitBake will write the functions to files
554 similar to the way shell tasks are handled.
555 Logging will be handled in a way similar to shell tasks as well.
556 </para>
557
558 <para>
559 The order in which BitBake runs the tasks is controlled by its
560 task scheduler.
561 It is possible to configure the scheduler and define custom
562 implementations for specific use cases.
563 For more information, see these variables that control the
564 behavior:
565 <itemizedlist>
566 <listitem><para>
567 <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
568 </para></listitem>
569 <listitem><para>
570 <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
571 </para></listitem>
572 </itemizedlist>
573 It is possible to have functions run before and after a task's main
574 function.
575 This is done using the "prefuncs" and "postfuncs" flags of the task
576 that lists the functions to run.
577 </para>
578 </section>
579
580 <section id='checksums'>
581 <title>Checksums (Signatures)</title>
582
583 <para>
584 A checksum is a unique signature of a task's inputs.
585 The signature of a task can be used to determine if a task
586 needs to be run.
587 Because it is a change in a task's inputs that triggers running
588 the task, BitBake needs to detect all the inputs to a given task.
589 For shell tasks, this turns out to be fairly easy because
590 BitBake generates a "run" shell script for each task and
591 it is possible to create a checksum that gives you a good idea of when
592 the task's data changes.
593 </para>
594
595 <para>
596 To complicate the problem, some things should not be included in
597 the checksum.
598 First, there is the actual specific build path of a given task -
599 the working directory.
600 It does not matter if the working directory changes because it should not
601 affect the output for target packages.
602 The simplistic approach for excluding the working directory is to set
603 it to some fixed value and create the checksum for the "run" script.
604 BitBake goes one step better and uses the
605 <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
606 variable to define a list of variables that should never be included
607 when generating the signatures.
608 </para>
609
610 <para>
611 Another problem results from the "run" scripts containing functions that
612 might or might not get called.
613 The incremental build solution contains code that figures out dependencies
614 between shell functions.
615 This code is used to prune the "run" scripts down to the minimum set,
616 thereby alleviating this problem and making the "run" scripts much more
617 readable as a bonus.
618 </para>
619
620 <para>
621 So far we have solutions for shell scripts.
622 What about Python tasks?
623 The same approach applies even though these tasks are more difficult.
624 The process needs to figure out what variables a Python function accesses
625 and what functions it calls.
626 Again, the incremental build solution contains code that first figures out
627 the variable and function dependencies, and then creates a checksum for the data
628 used as the input to the task.
629 </para>
630
631 <para>
632 Like the working directory case, situations exist where dependencies
633 should be ignored.
634 For these cases, you can instruct the build process to ignore a dependency
635 by using a line like the following:
636 <literallayout class='monospaced'>
637 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
638 </literallayout>
639 This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
640 depend on the value of <filename>MACHINE</filename>, even if it does reference it.
641 </para>
642
643 <para>
644 Equally, there are cases where we need to add dependencies BitBake
645 is not able to find.
646 You can accomplish this by using a line like the following:
647 <literallayout class='monospaced'>
648 PACKAGE_ARCHS[vardeps] = "MACHINE"
649 </literallayout>
650 This example explicitly adds the <filename>MACHINE</filename> variable as a
651 dependency for <filename>PACKAGE_ARCHS</filename>.
652 </para>
653
654 <para>
655 Consider a case with in-line Python, for example, where BitBake is not
656 able to figure out dependencies.
657 When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
658 produces output when it discovers something for which it cannot figure out
659 dependencies.
660 </para>
661
662 <para>
663 Thus far, this section has limited discussion to the direct inputs into a task.
664 Information based on direct inputs is referred to as the "basehash" in the
665 code.
666 However, there is still the question of a task's indirect inputs - the
667 things that were already built and present in the build directory.
668 The checksum (or signature) for a particular task needs to add the hashes
669 of all the tasks on which the particular task depends.
670 Choosing which dependencies to add is a policy decision.
671 However, the effect is to generate a master checksum that combines the basehash
672 and the hashes of the task's dependencies.
673 </para>
674
675 <para>
676 At the code level, there are a variety of ways both the basehash and the
677 dependent task hashes can be influenced.
678 Within the BitBake configuration file, we can give BitBake some extra information
679 to help it construct the basehash.
680 The following statement effectively results in a list of global variable
681 dependency excludes - variables never included in any checksum.
682 This example uses variables from OpenEmbedded to help illustrate
683 the concept:
684 <literallayout class='monospaced'>
685 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
686 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
687 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
688 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
689 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
690 </literallayout>
691 The previous example excludes the work directory, which is part of
692 <filename>TMPDIR</filename>.
693 </para>
694
695 <para>
696 The rules for deciding which hashes of dependent tasks to include through
697 dependency chains are more complex and are generally accomplished with a
698 Python function.
699 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
700 of this and also illustrates how you can insert your own policy into the system
701 if so desired.
702 This file defines the two basic signature generators OpenEmbedded Core
703 uses: "OEBasic" and "OEBasicHash".
704 By default, there is a dummy "noop" signature handler enabled in BitBake.
705 This means that behavior is unchanged from previous versions.
706 <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
707 through this setting in the <filename>bitbake.conf</filename> file:
708 <literallayout class='monospaced'>
709 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
710 </literallayout>
711 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
712 "OEBasic" version but adds the task hash to the stamp files.
713 This results in any metadata change that changes the task hash, automatically
714 causing the task to be run again.
715 This removes the need to bump
716 <link linkend='var-PR'><filename>PR</filename></link>
717 values, and changes to metadata automatically ripple across the build.
718 </para>
719
720 <para>
721 It is also worth noting that the end result of these signature generators is to
722 make some dependency and hash information available to the build.
723 This information includes:
724 <itemizedlist>
725 <listitem><para><filename>BB_BASEHASH_task-&lt;taskname&gt;</filename>:
726 The base hashes for each task in the recipe.
727 </para></listitem>
728 <listitem><para><filename>BB_BASEHASH_&lt;filename:taskname&gt;</filename>:
729 The base hashes for each dependent task.
730 </para></listitem>
731 <listitem><para><filename>BBHASHDEPS_&lt;filename:taskname&gt;</filename>:
732 The task dependencies for each task.
733 </para></listitem>
734 <listitem><para><filename>BB_TASKHASH</filename>:
735 The hash of the currently running task.
736 </para></listitem>
737 </itemizedlist>
738 </para>
739
740 <para>
741 It is worth noting that BitBake's "-S" option lets you
742 debug Bitbake's processing of signatures.
743 The options passed to -S allow different debugging modes
744 to be used, either using BitBake's own debug functions
745 or possibly those defined in the metadata/signature handler
746 itself.
747 The simplest parameter to pass is "none", which causes a
748 set of signature information to be written out into
749 <filename>STAMP_DIR</filename>
750 corresponding to the targets specified.
751 The other currently available parameter is "printdiff",
752 which causes BitBake to try to establish the closest
753 signature match it can (e.g. in the sstate cache) and then
754 run <filename>bitbake-diffsigs</filename> over the matches
755 to determine the stamps and delta where these two
756 stamp trees diverge.
757 <note>
758 It is likely that future versions of BitBake with
759 provide other signature handlers triggered through
760 additional "-S" paramters.
761 </note>
762 </para>
763
764 <para>
765 You can find more information on checksum metadata in the
766 "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
767 section.
768 </para>
769 </section>
770
771 <section id='setscene'>
772 <title>Setscene</title>
773
774 <para>
775 The setscene process enables BitBake to handle "pre-built" artifacts.
776 The ability to handle and reuse these artifacts allows BitBake
777 the luxury of not having to build something from scratch every time.
778 Instead, BitBake can use, when possible, existing build artifacts.
779 </para>
780
781 <para>
782 BitBake needs to have reliable data indicating whether or not an
783 artifact is compatible.
784 Signatures, described in the previous section, provide an ideal
785 way of representing whether an artifact is compatible.
786 If a signature is the same, an object can be reused.
787 </para>
788
789 <para>
790 If an object can be reused, the problem then becomes how to
791 replace a given task or set of tasks with the pre-built artifact.
792 BitBake solves the problem with the "setscene" process.
793 </para>
794
795 <para>
796 When BitBake is asked to build a given target, before building anything,
797 it first asks whether cached information is available for any of the
798 targets it's building, or any of the intermediate targets.
799 If cached information is available, BitBake uses this information instead of
800 running the main tasks.
801 </para>
802
803 <para>
804 BitBake first calls the function defined by the
805 <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
806 variable with a list of tasks and corresponding
807 hashes it wants to build.
808 This function is designed to be fast and returns a list
809 of the tasks for which it believes in can obtain artifacts.
810 </para>
811
812 <para>
813 Next, for each of the tasks that were returned as possibilities,
814 BitBake executes a setscene version of the task that the possible
815 artifact covers.
816 Setscene versions of a task have the string "_setscene" appended to the
817 task name.
818 So, for example, the task with the name <filename>xxx</filename> has
819 a setscene task named <filename>xxx_setscene</filename>.
820 The setscene version of the task executes and provides the necessary
821 artifacts returning either success or failure.
822 </para>
823
824 <para>
825 As previously mentioned, an artifact can cover more than one task.
826 For example, it is pointless to obtain a compiler if you
827 already have the compiled binary.
828 To handle this, BitBake calls the
829 <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
830 function for each successful setscene task to know whether or not it needs
831 to obtain the dependencies of that task.
832 </para>
833
834 <para>
835 Finally, after all the setscene tasks have executed, BitBake calls the
836 function listed in
837 <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION'><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></link>
838 with the list of tasks BitBake thinks has been "covered".
839 The metadata can then ensure that this list is correct and can
840 inform BitBake that it wants specific tasks to be run regardless
841 of the setscene result.
842 </para>
843
844 <para>
845 You can find more information on setscene metadata in the
846 "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
847 section.
848 </para>
849 </section>
850</chapter>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-10-12 12:19:14 +0000