diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml | 850 |
1 files changed, 850 insertions, 0 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml new file mode 100644 index 0000000000..8514f23f25 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml | |||
@@ -0,0 +1,850 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> | ||
3 | |||
4 | <chapter id="bitbake-user-manual-execution"> | ||
5 | <title>Execution</title> | ||
6 | |||
7 | <para> | ||
8 | The primary purpose for running BitBake is to produce some kind | ||
9 | of output such as an image, a kernel, or a software development | ||
10 | kit. | ||
11 | Of course, you can execute the <filename>bitbake</filename> | ||
12 | command with options that cause it to execute single tasks, | ||
13 | compile single recipe files, capture or clear data, or simply | ||
14 | return information about the execution environment. | ||
15 | </para> | ||
16 | |||
17 | <para> | ||
18 | This chapter describes BitBake's execution process from start | ||
19 | to finish when you use it to create an image. | ||
20 | The execution process is launched using the following command | ||
21 | form: | ||
22 | <literallayout class='monospaced'> | ||
23 | $ bitbake <target> | ||
24 | </literallayout> | ||
25 | For information on the BitBake command and its options, | ||
26 | see | ||
27 | "<link linkend='bitbake-user-manual-command'>The BitBake Command</link>" | ||
28 | section. | ||
29 | </para> | ||
30 | |||
31 | <note> | ||
32 | Prior to executing BitBake, you should take advantage of parallel | ||
33 | thread execution by setting the | ||
34 | <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link> | ||
35 | variable in your <filename>local.conf</filename> | ||
36 | configuration file. | ||
37 | </note> | ||
38 | |||
39 | <section id='parsing-the-base-configuration-metadata'> | ||
40 | <title>Parsing the Base Configuration Metadata</title> | ||
41 | |||
42 | <para> | ||
43 | The first thing BitBake does is parse base configuration | ||
44 | metadata. | ||
45 | Base configuration metadata consists of the | ||
46 | <filename>bblayers.conf</filename> file to determine what | ||
47 | layers BitBake needs to recognize, all necessary | ||
48 | <filename>layer.conf</filename> files (one from each layer), | ||
49 | and <filename>bitbake.conf</filename>. | ||
50 | The data itself is of various types: | ||
51 | <itemizedlist> | ||
52 | <listitem><para><emphasis>Recipes:</emphasis> | ||
53 | Details about particular pieces of software. | ||
54 | </para></listitem> | ||
55 | <listitem><para><emphasis>Class Data:</emphasis> | ||
56 | An abstraction of common build information | ||
57 | (e.g. how to build a Linux kernel). | ||
58 | </para></listitem> | ||
59 | <listitem><para><emphasis>Configuration Data:</emphasis> | ||
60 | Machine-specific settings, policy decisions, | ||
61 | and so forth. | ||
62 | Configuration data acts as the glue to bind everything | ||
63 | together.</para></listitem> | ||
64 | </itemizedlist> | ||
65 | </para> | ||
66 | |||
67 | <para> | ||
68 | The <filename>layer.conf</filename> files are used to | ||
69 | construct key variables such as | ||
70 | <link linkend='var-BBPATH'><filename>BBPATH</filename></link> | ||
71 | and | ||
72 | <link linkend='var-BBFILES'><filename>BBFILES</filename></link>. | ||
73 | <filename>BBPATH</filename> is used to search for | ||
74 | configuration and class files under | ||
75 | <filename>conf/</filename> and <filename>class/</filename> | ||
76 | directories, respectively. | ||
77 | <filename>BBFILES</filename> is used to find recipe files | ||
78 | (<filename>.bb</filename> and <filename>.bbappend</filename>). | ||
79 | If there is no <filename>bblayers.conf</filename> file, | ||
80 | it is assumed the user has set the <filename>BBPATH</filename> | ||
81 | and <filename>BBFILES</filename> directly in the environment. | ||
82 | </para> | ||
83 | |||
84 | <para> | ||
85 | Next, the <filename>bitbake.conf</filename> file is searched | ||
86 | using the <filename>BBPATH</filename> variable that was | ||
87 | just constructed. | ||
88 | The <filename>bitbake.conf</filename> file may also include other | ||
89 | configuration files using the | ||
90 | <filename>include</filename> or | ||
91 | <filename>require</filename> directives. | ||
92 | </para> | ||
93 | |||
94 | <para> | ||
95 | Prior to parsing configuration files, Bitbake looks | ||
96 | at certain variables, including: | ||
97 | <itemizedlist> | ||
98 | <listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem> | ||
99 | <listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem> | ||
100 | <listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem> | ||
101 | <listitem><para> | ||
102 | <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link> | ||
103 | </para></listitem> | ||
104 | </itemizedlist> | ||
105 | You can find information on how to pass environment variables into the BitBake | ||
106 | execution environment in the | ||
107 | "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>" section. | ||
108 | </para> | ||
109 | |||
110 | <para> | ||
111 | The base configuration metadata is global | ||
112 | and therefore affects all recipes and tasks that are executed. | ||
113 | </para> | ||
114 | |||
115 | <para> | ||
116 | BitBake first searches the current working directory for an | ||
117 | optional <filename>conf/bblayers.conf</filename> configuration file. | ||
118 | This file is expected to contain a | ||
119 | <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link> | ||
120 | variable that is a space delimited list of 'layer' directories. | ||
121 | Recall that if BitBake cannot find a <filename>bblayers.conf</filename> | ||
122 | file then it is assumed the user has set the <filename>BBPATH</filename> | ||
123 | and <filename>BBFILES</filename> directly in the environment. | ||
124 | </para> | ||
125 | |||
126 | <para> | ||
127 | For each directory (layer) in this list, a <filename>conf/layer.conf</filename> | ||
128 | file is searched for and parsed with the | ||
129 | <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link> | ||
130 | variable being set to the directory where the layer was found. | ||
131 | The idea is these files automatically setup | ||
132 | <link linkend='var-BBPATH'><filename>BBPATH</filename></link> | ||
133 | and other variables correctly for a given build directory. | ||
134 | </para> | ||
135 | |||
136 | <para> | ||
137 | BitBake then expects to find the <filename>conf/bitbake.conf</filename> | ||
138 | file somewhere in the user-specified <filename>BBPATH</filename>. | ||
139 | That configuration file generally has include directives to pull | ||
140 | in any other metadata such as files specific to the architecture, | ||
141 | the machine, the local environment, and so forth. | ||
142 | </para> | ||
143 | |||
144 | <para> | ||
145 | Only variable definitions and include directives are allowed | ||
146 | in <filename>.conf</filename> files. | ||
147 | Some variables directly influence BitBake's behavior. | ||
148 | These variables might have been set from the environment | ||
149 | depending on the environment variables previously | ||
150 | mentioned or set in the configuration files. | ||
151 | The | ||
152 | "<link linkend='ref-variables-glos'>Variables Glossary</link>" | ||
153 | chapter presents a full list of variables. | ||
154 | </para> | ||
155 | |||
156 | <para> | ||
157 | After parsing configuration files, BitBake uses its rudimentary | ||
158 | inheritance mechanism, which is through class files, to inherit | ||
159 | some standard classes. | ||
160 | BitBake parses a class when the inherit directive responsible | ||
161 | for getting that class is encountered. | ||
162 | </para> | ||
163 | |||
164 | <para> | ||
165 | The <filename>base.bbclass</filename> file is always included. | ||
166 | Other classes that are specified in the configuration using the | ||
167 | <link linkend='var-INHERIT'><filename>INHERIT</filename></link> | ||
168 | variable are also included. | ||
169 | BitBake searches for class files in a "classes" subdirectory under | ||
170 | the paths in <filename>BBPATH</filename> in the same way as | ||
171 | configuration files. | ||
172 | </para> | ||
173 | |||
174 | <para> | ||
175 | A good way to get an idea of the configuration files and | ||
176 | the class files used in your execution environment is to | ||
177 | run the following BitBake command: | ||
178 | <literallayout class='monospaced'> | ||
179 | $ bitbake -e > mybb.log | ||
180 | </literallayout> | ||
181 | Examining the top of the <filename>mybb.log</filename> | ||
182 | shows you the many configuration files and class files | ||
183 | used in your execution environment. | ||
184 | </para> | ||
185 | |||
186 | <note> | ||
187 | <para> | ||
188 | You need to be aware of how BitBake parses curly braces. | ||
189 | If a recipe uses a closing curly brace within the function and | ||
190 | the character has no leading spaces, BitBake produces a parsing | ||
191 | error. | ||
192 | If you use a pair of curly brace in a shell function, the | ||
193 | closing curly brace must not be located at the start of the line | ||
194 | without leading spaces. | ||
195 | </para> | ||
196 | |||
197 | <para> | ||
198 | Here is an example that causes BitBake to produce a parsing | ||
199 | error: | ||
200 | <literallayout class='monospaced'> | ||
201 | fakeroot create_shar() { | ||
202 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh | ||
203 | usage() | ||
204 | { | ||
205 | echo "test" | ||
206 | ###### The following "}" at the start of the line causes a parsing error ###### | ||
207 | } | ||
208 | EOF | ||
209 | } | ||
210 | </literallayout> | ||
211 | Writing the recipe this way avoids the error: | ||
212 | <literallayout class='monospaced'> | ||
213 | fakeroot create_shar() { | ||
214 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh | ||
215 | usage() | ||
216 | { | ||
217 | echo "test" | ||
218 | ######The following "}" with a leading space at the start of the line avoids the error ###### | ||
219 | } | ||
220 | EOF | ||
221 | } | ||
222 | </literallayout> | ||
223 | </para> | ||
224 | </note> | ||
225 | </section> | ||
226 | |||
227 | <section id='locating-and-parsing-recipes'> | ||
228 | <title>Locating and Parsing Recipes</title> | ||
229 | |||
230 | <para> | ||
231 | During the configuration phase, BitBake will have set | ||
232 | <link linkend='var-BBFILES'><filename>BBFILES</filename></link>. | ||
233 | BitBake now uses it to construct a list of recipes to parse, | ||
234 | along with any append files (<filename>.bbappend</filename>) | ||
235 | to apply. | ||
236 | <filename>BBFILES</filename> is a space-separated list of | ||
237 | available files and supports wildcards. | ||
238 | An example would be: | ||
239 | <literallayout class='monospaced'> | ||
240 | BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" | ||
241 | </literallayout> | ||
242 | BitBake parses each recipe and append file located | ||
243 | with <filename>BBFILES</filename> and stores the values of | ||
244 | various variables into the datastore. | ||
245 | <note> | ||
246 | Append files are applied in the order they are encountered in | ||
247 | <filename>BBFILES</filename>. | ||
248 | </note> | ||
249 | For each file, a fresh copy of the base configuration is | ||
250 | made, then the recipe is parsed line by line. | ||
251 | Any inherit statements cause BitBake to find and | ||
252 | then parse class files (<filename>.bbclass</filename>) | ||
253 | using | ||
254 | <link linkend='var-BBPATH'><filename>BBPATH</filename></link> | ||
255 | as the search path. | ||
256 | Finally, BitBake parses in order any append files found in | ||
257 | <filename>BBFILES</filename>. | ||
258 | </para> | ||
259 | |||
260 | <para> | ||
261 | One common convention is to use the recipe filename to define | ||
262 | pieces of metadata. | ||
263 | For example, in <filename>bitbake.conf</filename> the recipe | ||
264 | name and version set | ||
265 | <link linkend='var-PN'><filename>PN</filename></link> and | ||
266 | <link linkend='var-PV'><filename>PV</filename></link>: | ||
267 | <literallayout class='monospaced'> | ||
268 | PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}" | ||
269 | PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or 'defaultpkgname'}" | ||
270 | </literallayout> | ||
271 | In this example, a recipe called "something_1.2.3.bb" sets | ||
272 | <filename>PN</filename> to "something" and | ||
273 | <filename>PV</filename> to "1.2.3". | ||
274 | </para> | ||
275 | |||
276 | <para> | ||
277 | By the time parsing is complete for a recipe, BitBake | ||
278 | has a list of tasks that the recipe defines and a set of | ||
279 | data consisting of keys and values as well as | ||
280 | dependency information about the tasks. | ||
281 | </para> | ||
282 | |||
283 | <para> | ||
284 | BitBake does not need all of this information. | ||
285 | It only needs a small subset of the information to make | ||
286 | decisions about the recipe. | ||
287 | Consequently, BitBake caches the values in which it is | ||
288 | interested and does not store the rest of the information. | ||
289 | Experience has shown it is faster to re-parse the metadata than to | ||
290 | try and write it out to the disk and then reload it. | ||
291 | </para> | ||
292 | |||
293 | <para> | ||
294 | Where possible, subsequent BitBake commands reuse this cache of | ||
295 | recipe information. | ||
296 | The validity of this cache is determined by first computing a | ||
297 | checksum of the base configuration data (see | ||
298 | <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>) | ||
299 | and then checking if the checksum matches. | ||
300 | If that checksum matches what is in the cache and the recipe | ||
301 | and class files have not changed, Bitbake is able to use | ||
302 | the cache. | ||
303 | BitBake then reloads the cached information about the recipe | ||
304 | instead of reparsing it from scratch. | ||
305 | </para> | ||
306 | |||
307 | <para> | ||
308 | Recipe file collections exist to allow the user to | ||
309 | have multiple repositories of | ||
310 | <filename>.bb</filename> files that contain the same | ||
311 | exact package. | ||
312 | For example, one could easily use them to make one's | ||
313 | own local copy of an upstream repository, but with | ||
314 | custom modifications that one does not want upstream. | ||
315 | Here is an example: | ||
316 | <literallayout class='monospaced'> | ||
317 | BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb" | ||
318 | BBFILE_COLLECTIONS = "upstream local" | ||
319 | BBFILE_PATTERN_upstream = "^/stuff/openembedded/" | ||
320 | BBFILE_PATTERN_local = "^/stuff/openembedded.modified/" | ||
321 | BBFILE_PRIORITY_upstream = "5" | ||
322 | BBFILE_PRIORITY_local = "10" | ||
323 | </literallayout> | ||
324 | <note> | ||
325 | The layers mechanism is now the preferred method of collecting | ||
326 | code. | ||
327 | While the collections code remains, its main use is to set layer | ||
328 | priorities and to deal with overlap (conflicts) between layers. | ||
329 | </note> | ||
330 | </para> | ||
331 | </section> | ||
332 | |||
333 | <section id='bb-bitbake-providers'> | ||
334 | <title>Preferences and Providers</title> | ||
335 | |||
336 | <para> | ||
337 | Assuming BitBake has been instructed to execute a target | ||
338 | and that all the recipe files have been parsed, BitBake | ||
339 | starts to figure out how to build the target. | ||
340 | BitBake starts by looking through the | ||
341 | <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link> | ||
342 | set in recipe files. | ||
343 | The default <filename>PROVIDES</filename> for a recipe is its name | ||
344 | (<link linkend='var-PN'><filename>PN</filename></link>), | ||
345 | however, a recipe can provide multiple things. | ||
346 | </para> | ||
347 | |||
348 | <para> | ||
349 | As an example of adding an extra provider, suppose a recipe named | ||
350 | <filename>foo_1.0.bb</filename> contained the following: | ||
351 | <literallayout class='monospaced'> | ||
352 | PROVIDES += "virtual/bar_1.0" | ||
353 | </literallayout> | ||
354 | The recipe now provides both "foo_1.0" and "virtual/bar_1.0". | ||
355 | The "virtual/" namespace is often used to denote cases where | ||
356 | multiple providers are expected with the user choosing between | ||
357 | them. | ||
358 | Kernels and toolchain components are common cases of this in | ||
359 | OpenEmbedded. | ||
360 | </para> | ||
361 | |||
362 | <para> | ||
363 | Sometimes a target might have multiple providers. | ||
364 | A common example is "virtual/kernel", which is provided by each | ||
365 | kernel recipe. | ||
366 | Each machine often selects the best kernel provider by using a | ||
367 | line similar to the following in the machine configuration file: | ||
368 | <literallayout class='monospaced'> | ||
369 | PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" | ||
370 | </literallayout> | ||
371 | The default | ||
372 | <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link> | ||
373 | is the provider with the same name as the target. | ||
374 | Bitbake iterates through each target it needs to build and | ||
375 | resolves them and their dependencies using this process. | ||
376 | </para> | ||
377 | |||
378 | <para> | ||
379 | Understanding how providers are chosen is made complicated by the fact | ||
380 | that multiple versions might exist. | ||
381 | BitBake defaults to the highest version of a provider. | ||
382 | Version comparisons are made using the same method as Debian. | ||
383 | You can use the | ||
384 | <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link> | ||
385 | variable to specify a particular version. | ||
386 | You can influence the order by using the | ||
387 | <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link> | ||
388 | variable. | ||
389 | By default, files have a preference of "0". | ||
390 | Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the | ||
391 | recipe unlikely to be used unless it is explicitly referenced. | ||
392 | Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the recipe is used. | ||
393 | <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting. | ||
394 | <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental recipe | ||
395 | versions until they have undergone sufficient testing to be considered stable. | ||
396 | </para> | ||
397 | |||
398 | <para> | ||
399 | When there are multiple “versions” of a given recipe, | ||
400 | BitBake defaults to selecting the most recent | ||
401 | version, unless otherwise specified. | ||
402 | If the recipe in question has a | ||
403 | <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link> | ||
404 | set lower than | ||
405 | the other recipes (default is 0), then it will not be | ||
406 | selected. | ||
407 | This allows the person or persons maintaining | ||
408 | the repository of recipe files to specify | ||
409 | their preference for the default selected version. | ||
410 | In addition, the user can specify their preferred version. | ||
411 | </para> | ||
412 | |||
413 | <para> | ||
414 | If the first recipe is named <filename>a_1.1.bb</filename>, | ||
415 | then the | ||
416 | <link linkend='var-PN'><filename>PN</filename></link> variable | ||
417 | will be set to “a”, and the | ||
418 | <link linkend='var-PV'><filename>PV</filename></link> | ||
419 | variable will be set to 1.1. | ||
420 | </para> | ||
421 | |||
422 | <para> | ||
423 | If we then have a recipe named <filename>a_1.2.bb</filename>, BitBake | ||
424 | will choose 1.2 by default. | ||
425 | However, if we define the following variable in a | ||
426 | <filename>.conf</filename> file that BitBake parses, we | ||
427 | can change that. | ||
428 | <literallayout class='monospaced'> | ||
429 | PREFERRED_VERSION_a = "1.1" | ||
430 | </literallayout> | ||
431 | </para> | ||
432 | |||
433 | <para> | ||
434 | In summary, BitBake has created a list of providers, which is prioritized, for each target. | ||
435 | </para> | ||
436 | </section> | ||
437 | |||
438 | <section id='bb-bitbake-dependencies'> | ||
439 | <title>Dependencies</title> | ||
440 | |||
441 | <para> | ||
442 | Each target BitBake builds consists of multiple tasks such as | ||
443 | <filename>fetch</filename>, <filename>unpack</filename>, | ||
444 | <filename>patch</filename>, <filename>configure</filename>, | ||
445 | and <filename>compile</filename>. | ||
446 | For best performance on multi-core systems, BitBake considers each | ||
447 | task as an independent | ||
448 | entity with its own set of dependencies. | ||
449 | </para> | ||
450 | |||
451 | <para> | ||
452 | Dependencies are defined through several variables. | ||
453 | You can find information about variables BitBake uses in | ||
454 | the <link linkend='ref-variables-glos'>Variables Glossary</link> | ||
455 | near the end of this manual. | ||
456 | At a basic level, it is sufficient to know that BitBake uses the | ||
457 | <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and | ||
458 | <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when | ||
459 | calculating dependencies. | ||
460 | </para> | ||
461 | |||
462 | <para> | ||
463 | For more information on how BitBake handles dependencies, see the | ||
464 | "<link linkend='dependencies'>Dependencies</link>" section. | ||
465 | </para> | ||
466 | </section> | ||
467 | |||
468 | <section id='ref-bitbake-tasklist'> | ||
469 | <title>The Task List</title> | ||
470 | |||
471 | <para> | ||
472 | Based on the generated list of providers and the dependency information, | ||
473 | BitBake can now calculate exactly what tasks it needs to run and in what | ||
474 | order it needs to run them. | ||
475 | The | ||
476 | "<link linkend='executing-tasks'>Executing Tasks</link>" section has more | ||
477 | information on how BitBake chooses which task to execute next. | ||
478 | </para> | ||
479 | |||
480 | <para> | ||
481 | The build now starts with BitBake forking off threads up to the limit set in the | ||
482 | <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link> | ||
483 | variable. | ||
484 | BitBake continues to fork threads as long as there are tasks ready to run, | ||
485 | those tasks have all their dependencies met, and the thread threshold has not been | ||
486 | exceeded. | ||
487 | </para> | ||
488 | |||
489 | <para> | ||
490 | It is worth noting that you can greatly speed up the build time by properly setting | ||
491 | the <filename>BB_NUMBER_THREADS</filename> variable. | ||
492 | </para> | ||
493 | |||
494 | <para> | ||
495 | As each task completes, a timestamp is written to the directory specified by the | ||
496 | <link linkend='var-STAMP'><filename>STAMP</filename></link> variable. | ||
497 | On subsequent runs, BitBake looks in the build directory within | ||
498 | <filename>tmp/stamps</filename>and does not rerun | ||
499 | tasks that are already completed unless a timestamp is found to be invalid. | ||
500 | Currently, invalid timestamps are only considered on a per | ||
501 | recipe file basis. | ||
502 | So, for example, if the configure stamp has a timestamp greater than the | ||
503 | compile timestamp for a given target, then the compile task would rerun. | ||
504 | Running the compile task again, however, has no effect on other providers | ||
505 | that depend on that target. | ||
506 | </para> | ||
507 | |||
508 | <para> | ||
509 | The exact format of the stamps is partly configurable. | ||
510 | In modern versions of BitBake, a hash is appended to the | ||
511 | stamp so that if the configuration changes, the stamp becomes | ||
512 | invalid and the task is automatically rerun. | ||
513 | This hash, or signature used, is governed by the signature policy | ||
514 | that is configured (see the | ||
515 | "<link linkend='checksums'>Checksums (Signatures)</link>" | ||
516 | section for information). | ||
517 | It is also possible to append extra metadata to the stamp using | ||
518 | the "stamp-extra-info" task flag. | ||
519 | For example, OpenEmbedded uses this flag to make some tasks machine-specific. | ||
520 | </para> | ||
521 | |||
522 | <note> | ||
523 | Some tasks are marked as "nostamp" tasks. | ||
524 | No timestamp file is created when these tasks are run. | ||
525 | Consequently, "nostamp" tasks are always rerun. | ||
526 | </note> | ||
527 | |||
528 | <para> | ||
529 | For more information on tasks, see the | ||
530 | "<link linkend='tasks'>Tasks</link>" section. | ||
531 | </para> | ||
532 | </section> | ||
533 | |||
534 | <section id='executing-tasks'> | ||
535 | <title>Executing Tasks</title> | ||
536 | |||
537 | <para> | ||
538 | Tasks can either be a shell task or a Python task. | ||
539 | For shell tasks, BitBake writes a shell script to | ||
540 | <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename> | ||
541 | and then executes the script. | ||
542 | The generated shell script contains all the exported variables, | ||
543 | and the shell functions with all variables expanded. | ||
544 | Output from the shell script goes to the file | ||
545 | <filename>${T}/log.do_taskname.pid</filename>. | ||
546 | Looking at the expanded shell functions in the run file and | ||
547 | the output in the log files is a useful debugging technique. | ||
548 | </para> | ||
549 | |||
550 | <para> | ||
551 | For Python tasks, BitBake executes the task internally and logs | ||
552 | information to the controlling terminal. | ||
553 | Future versions of BitBake will write the functions to files | ||
554 | similar to the way shell tasks are handled. | ||
555 | Logging will be handled in a way similar to shell tasks as well. | ||
556 | </para> | ||
557 | |||
558 | <para> | ||
559 | The order in which BitBake runs the tasks is controlled by its | ||
560 | task scheduler. | ||
561 | It is possible to configure the scheduler and define custom | ||
562 | implementations for specific use cases. | ||
563 | For more information, see these variables that control the | ||
564 | behavior: | ||
565 | <itemizedlist> | ||
566 | <listitem><para> | ||
567 | <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link> | ||
568 | </para></listitem> | ||
569 | <listitem><para> | ||
570 | <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link> | ||
571 | </para></listitem> | ||
572 | </itemizedlist> | ||
573 | It is possible to have functions run before and after a task's main | ||
574 | function. | ||
575 | This is done using the "prefuncs" and "postfuncs" flags of the task | ||
576 | that lists the functions to run. | ||
577 | </para> | ||
578 | </section> | ||
579 | |||
580 | <section id='checksums'> | ||
581 | <title>Checksums (Signatures)</title> | ||
582 | |||
583 | <para> | ||
584 | A checksum is a unique signature of a task's inputs. | ||
585 | The signature of a task can be used to determine if a task | ||
586 | needs to be run. | ||
587 | Because it is a change in a task's inputs that triggers running | ||
588 | the task, BitBake needs to detect all the inputs to a given task. | ||
589 | For shell tasks, this turns out to be fairly easy because | ||
590 | BitBake generates a "run" shell script for each task and | ||
591 | it is possible to create a checksum that gives you a good idea of when | ||
592 | the task's data changes. | ||
593 | </para> | ||
594 | |||
595 | <para> | ||
596 | To complicate the problem, some things should not be included in | ||
597 | the checksum. | ||
598 | First, there is the actual specific build path of a given task - | ||
599 | the working directory. | ||
600 | It does not matter if the working directory changes because it should not | ||
601 | affect the output for target packages. | ||
602 | The simplistic approach for excluding the working directory is to set | ||
603 | it to some fixed value and create the checksum for the "run" script. | ||
604 | BitBake goes one step better and uses the | ||
605 | <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link> | ||
606 | variable to define a list of variables that should never be included | ||
607 | when generating the signatures. | ||
608 | </para> | ||
609 | |||
610 | <para> | ||
611 | Another problem results from the "run" scripts containing functions that | ||
612 | might or might not get called. | ||
613 | The incremental build solution contains code that figures out dependencies | ||
614 | between shell functions. | ||
615 | This code is used to prune the "run" scripts down to the minimum set, | ||
616 | thereby alleviating this problem and making the "run" scripts much more | ||
617 | readable as a bonus. | ||
618 | </para> | ||
619 | |||
620 | <para> | ||
621 | So far we have solutions for shell scripts. | ||
622 | What about Python tasks? | ||
623 | The same approach applies even though these tasks are more difficult. | ||
624 | The process needs to figure out what variables a Python function accesses | ||
625 | and what functions it calls. | ||
626 | Again, the incremental build solution contains code that first figures out | ||
627 | the variable and function dependencies, and then creates a checksum for the data | ||
628 | used as the input to the task. | ||
629 | </para> | ||
630 | |||
631 | <para> | ||
632 | Like the working directory case, situations exist where dependencies | ||
633 | should be ignored. | ||
634 | For these cases, you can instruct the build process to ignore a dependency | ||
635 | by using a line like the following: | ||
636 | <literallayout class='monospaced'> | ||
637 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
638 | </literallayout> | ||
639 | This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not | ||
640 | depend on the value of <filename>MACHINE</filename>, even if it does reference it. | ||
641 | </para> | ||
642 | |||
643 | <para> | ||
644 | Equally, there are cases where we need to add dependencies BitBake | ||
645 | is not able to find. | ||
646 | You can accomplish this by using a line like the following: | ||
647 | <literallayout class='monospaced'> | ||
648 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
649 | </literallayout> | ||
650 | This example explicitly adds the <filename>MACHINE</filename> variable as a | ||
651 | dependency for <filename>PACKAGE_ARCHS</filename>. | ||
652 | </para> | ||
653 | |||
654 | <para> | ||
655 | Consider a case with in-line Python, for example, where BitBake is not | ||
656 | able to figure out dependencies. | ||
657 | When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake | ||
658 | produces output when it discovers something for which it cannot figure out | ||
659 | dependencies. | ||
660 | </para> | ||
661 | |||
662 | <para> | ||
663 | Thus far, this section has limited discussion to the direct inputs into a task. | ||
664 | Information based on direct inputs is referred to as the "basehash" in the | ||
665 | code. | ||
666 | However, there is still the question of a task's indirect inputs - the | ||
667 | things that were already built and present in the build directory. | ||
668 | The checksum (or signature) for a particular task needs to add the hashes | ||
669 | of all the tasks on which the particular task depends. | ||
670 | Choosing which dependencies to add is a policy decision. | ||
671 | However, the effect is to generate a master checksum that combines the basehash | ||
672 | and the hashes of the task's dependencies. | ||
673 | </para> | ||
674 | |||
675 | <para> | ||
676 | At the code level, there are a variety of ways both the basehash and the | ||
677 | dependent task hashes can be influenced. | ||
678 | Within the BitBake configuration file, we can give BitBake some extra information | ||
679 | to help it construct the basehash. | ||
680 | The following statement effectively results in a list of global variable | ||
681 | dependency excludes - variables never included in any checksum. | ||
682 | This example uses variables from OpenEmbedded to help illustrate | ||
683 | the concept: | ||
684 | <literallayout class='monospaced'> | ||
685 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ | ||
686 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ | ||
687 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ | ||
688 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ | ||
689 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
690 | </literallayout> | ||
691 | The previous example excludes the work directory, which is part of | ||
692 | <filename>TMPDIR</filename>. | ||
693 | </para> | ||
694 | |||
695 | <para> | ||
696 | The rules for deciding which hashes of dependent tasks to include through | ||
697 | dependency chains are more complex and are generally accomplished with a | ||
698 | Python function. | ||
699 | The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples | ||
700 | of this and also illustrates how you can insert your own policy into the system | ||
701 | if so desired. | ||
702 | This file defines the two basic signature generators OpenEmbedded Core | ||
703 | uses: "OEBasic" and "OEBasicHash". | ||
704 | By default, there is a dummy "noop" signature handler enabled in BitBake. | ||
705 | This means that behavior is unchanged from previous versions. | ||
706 | <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default | ||
707 | through this setting in the <filename>bitbake.conf</filename> file: | ||
708 | <literallayout class='monospaced'> | ||
709 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
710 | </literallayout> | ||
711 | The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the | ||
712 | "OEBasic" version but adds the task hash to the stamp files. | ||
713 | This results in any metadata change that changes the task hash, automatically | ||
714 | causing the task to be run again. | ||
715 | This removes the need to bump | ||
716 | <link linkend='var-PR'><filename>PR</filename></link> | ||
717 | values, and changes to metadata automatically ripple across the build. | ||
718 | </para> | ||
719 | |||
720 | <para> | ||
721 | It is also worth noting that the end result of these signature generators is to | ||
722 | make some dependency and hash information available to the build. | ||
723 | This information includes: | ||
724 | <itemizedlist> | ||
725 | <listitem><para><filename>BB_BASEHASH_task-<taskname></filename>: | ||
726 | The base hashes for each task in the recipe. | ||
727 | </para></listitem> | ||
728 | <listitem><para><filename>BB_BASEHASH_<filename:taskname></filename>: | ||
729 | The base hashes for each dependent task. | ||
730 | </para></listitem> | ||
731 | <listitem><para><filename>BBHASHDEPS_<filename:taskname></filename>: | ||
732 | The task dependencies for each task. | ||
733 | </para></listitem> | ||
734 | <listitem><para><filename>BB_TASKHASH</filename>: | ||
735 | The hash of the currently running task. | ||
736 | </para></listitem> | ||
737 | </itemizedlist> | ||
738 | </para> | ||
739 | |||
740 | <para> | ||
741 | It is worth noting that BitBake's "-S" option lets you | ||
742 | debug Bitbake's processing of signatures. | ||
743 | The options passed to -S allow different debugging modes | ||
744 | to be used, either using BitBake's own debug functions | ||
745 | or possibly those defined in the metadata/signature handler | ||
746 | itself. | ||
747 | The simplest parameter to pass is "none", which causes a | ||
748 | set of signature information to be written out into | ||
749 | <filename>STAMP_DIR</filename> | ||
750 | corresponding to the targets specified. | ||
751 | The other currently available parameter is "printdiff", | ||
752 | which causes BitBake to try to establish the closest | ||
753 | signature match it can (e.g. in the sstate cache) and then | ||
754 | run <filename>bitbake-diffsigs</filename> over the matches | ||
755 | to determine the stamps and delta where these two | ||
756 | stamp trees diverge. | ||
757 | <note> | ||
758 | It is likely that future versions of BitBake with | ||
759 | provide other signature handlers triggered through | ||
760 | additional "-S" paramters. | ||
761 | </note> | ||
762 | </para> | ||
763 | |||
764 | <para> | ||
765 | You can find more information on checksum metadata in the | ||
766 | "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>" | ||
767 | section. | ||
768 | </para> | ||
769 | </section> | ||
770 | |||
771 | <section id='setscene'> | ||
772 | <title>Setscene</title> | ||
773 | |||
774 | <para> | ||
775 | The setscene process enables BitBake to handle "pre-built" artifacts. | ||
776 | The ability to handle and reuse these artifacts allows BitBake | ||
777 | the luxury of not having to build something from scratch every time. | ||
778 | Instead, BitBake can use, when possible, existing build artifacts. | ||
779 | </para> | ||
780 | |||
781 | <para> | ||
782 | BitBake needs to have reliable data indicating whether or not an | ||
783 | artifact is compatible. | ||
784 | Signatures, described in the previous section, provide an ideal | ||
785 | way of representing whether an artifact is compatible. | ||
786 | If a signature is the same, an object can be reused. | ||
787 | </para> | ||
788 | |||
789 | <para> | ||
790 | If an object can be reused, the problem then becomes how to | ||
791 | replace a given task or set of tasks with the pre-built artifact. | ||
792 | BitBake solves the problem with the "setscene" process. | ||
793 | </para> | ||
794 | |||
795 | <para> | ||
796 | When BitBake is asked to build a given target, before building anything, | ||
797 | it first asks whether cached information is available for any of the | ||
798 | targets it's building, or any of the intermediate targets. | ||
799 | If cached information is available, BitBake uses this information instead of | ||
800 | running the main tasks. | ||
801 | </para> | ||
802 | |||
803 | <para> | ||
804 | BitBake first calls the function defined by the | ||
805 | <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link> | ||
806 | variable with a list of tasks and corresponding | ||
807 | hashes it wants to build. | ||
808 | This function is designed to be fast and returns a list | ||
809 | of the tasks for which it believes in can obtain artifacts. | ||
810 | </para> | ||
811 | |||
812 | <para> | ||
813 | Next, for each of the tasks that were returned as possibilities, | ||
814 | BitBake executes a setscene version of the task that the possible | ||
815 | artifact covers. | ||
816 | Setscene versions of a task have the string "_setscene" appended to the | ||
817 | task name. | ||
818 | So, for example, the task with the name <filename>xxx</filename> has | ||
819 | a setscene task named <filename>xxx_setscene</filename>. | ||
820 | The setscene version of the task executes and provides the necessary | ||
821 | artifacts returning either success or failure. | ||
822 | </para> | ||
823 | |||
824 | <para> | ||
825 | As previously mentioned, an artifact can cover more than one task. | ||
826 | For example, it is pointless to obtain a compiler if you | ||
827 | already have the compiled binary. | ||
828 | To handle this, BitBake calls the | ||
829 | <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link> | ||
830 | function for each successful setscene task to know whether or not it needs | ||
831 | to obtain the dependencies of that task. | ||
832 | </para> | ||
833 | |||
834 | <para> | ||
835 | Finally, after all the setscene tasks have executed, BitBake calls the | ||
836 | function listed in | ||
837 | <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION'><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></link> | ||
838 | with the list of tasks BitBake thinks has been "covered". | ||
839 | The metadata can then ensure that this list is correct and can | ||
840 | inform BitBake that it wants specific tasks to be run regardless | ||
841 | of the setscene result. | ||
842 | </para> | ||
843 | |||
844 | <para> | ||
845 | You can find more information on setscene metadata in the | ||
846 | "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>" | ||
847 | section. | ||
848 | </para> | ||
849 | </section> | ||
850 | </chapter> | ||