summaryrefslogtreecommitdiffstats
path: root/bitbake/doc
diff options
context:
space:
mode:
Diffstat (limited to 'bitbake/doc')
-rw-r--r--bitbake/doc/user-manual/user-manual-execution.xml726
1 files changed, 364 insertions, 362 deletions
diff --git a/bitbake/doc/user-manual/user-manual-execution.xml b/bitbake/doc/user-manual/user-manual-execution.xml
index aa345de698..e9f19be6de 100644
--- a/bitbake/doc/user-manual/user-manual-execution.xml
+++ b/bitbake/doc/user-manual/user-manual-execution.xml
@@ -5,8 +5,9 @@
5 <title>Execution</title> 5 <title>Execution</title>
6 6
7 <para> 7 <para>
8 The primary purpose for running BitBake is to produce an 8 The primary purpose for running BitBake is to produce some kind
9 image, which can be a kernel or a software development kit (SDK). 9 of output such as an image, a kernel, or a software development
10 kit.
10 Of course, you can execute the <filename>bitbake</filename> 11 Of course, you can execute the <filename>bitbake</filename>
11 command with options that cause it to execute single tasks, 12 command with options that cause it to execute single tasks,
12 compile single recipe files, capture or clear data, or simply 13 compile single recipe files, capture or clear data, or simply
@@ -38,24 +39,56 @@
38 layers BitBake needs to recognize, all necessary 39 layers BitBake needs to recognize, all necessary
39 <filename>layer.conf</filename> files (one from each layer), 40 <filename>layer.conf</filename> files (one from each layer),
40 and <filename>bitbake.conf</filename>. 41 and <filename>bitbake.conf</filename>.
42 The data itself is of various types:
43 <itemizedlist>
44 <listitem><para><emphasis>Recipes:</emphasis>
45 Details about particular pieces of software.
46 </para></listitem>
47 <listitem><para><emphasis>Class Data:</emphasis>
48 An abstraction of common build information
49 (e.g. how to build a Linux kernel).
50 </para></listitem>
51 <listitem><para><emphasis>Configuration Data:</emphasis>
52 Machine-specific settings, policy decisions,
53 and so forth.
54 Configuration data acts as the glue to bind everything
55 together.</para></listitem>
56 </itemizedlist>
41 </para> 57 </para>
42 58
43 <para> 59 <para>
44 The <filename>bitbake.conf</filename> file resides in the 60 The <filename>layer.conf</filename> files are used to
45 <filename>conf</filename> directory, which must be listed in 61 construct key variables such as
46 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>. 62 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
47 This configuratoin file lists and includes other configuration 63 and
48 files from the <filename>conf</filename> directory below the 64 <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
49 directories listed in <filename>BBPATH</filename>. 65 <filename>BBPATH</filename> is used to search for
50 In general, the most important of these included 66 configuration and class files under
51 configuration files from a user's perspective 67 <filename>conf/</filename> and <filename>class/</filename>
52 is <filename>local.conf</filename>, which contains the user's 68 directories, respectively.
53 customized settings for the build environment. 69 <filename>BBFILES</filename> is used to find recipe files
54 </para> 70 (<filename>.bb</filename> and <filename>.bbappend</filename>).
55 71 If there is no <filename>bblayers.conf</filename> file,
56 <para> 72 it is assumed the user has set the <filename>BBPATH</filename>
57 Other notable configuration files are the distribution configuration 73 and <filename>BBFILES</filename> directly in the environment.
58 file and the machine configuration file. 74 </para>
75
76 <para>
77 Next, the <filename>bitbake.conf</filename> file is searched
78 using the <filename>BBPATH</filename> variable that was
79 just constructed.
80 The <filename>bitbake.conf</filename> file usually indicates
81 all the other key include files to parse.
82 The usual convention is to have machine, distro, site, and local
83 configurations.
84 This means a user provides their own customizations
85 through a <filename>local.conf</filename> file.
86 </para>
87
88 <para>
89 As mentioned in the previous paragraph, two of the other notable
90 configuration files are the distro and machine configuration
91 files.
59 These configuration files are normally identified by 92 These configuration files are normally identified by
60 variables unique to the build systems using BitBake. 93 variables unique to the build systems using BitBake.
61 For example, the Yocto Project uses the 94 For example, the Yocto Project uses the
@@ -87,6 +120,9 @@
87 This file is expected to contain a 120 This file is expected to contain a
88 <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link> 121 <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
89 variable that is a space delimited list of 'layer' directories. 122 variable that is a space delimited list of 'layer' directories.
123 Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
124 file then it is assumed the user has set the <filename>BBPATH</filename>
125 and <filename>BBFILES</filename> directly in the environment.
90 </para> 126 </para>
91 127
92 <para> 128 <para>
@@ -194,460 +230,426 @@
194 </para> 230 </para>
195 </section> 231 </section>
196 232
197 <section id='locating-recipes'> 233 <section id='locating-and-parsing-recipes'>
198 <title>Locating Recipes</title> 234 <title>Locating and Parsing Recipes</title>
199
200 <para>
201 The <filename>BBFILES</filename> variable is how BitBake
202 locates files.
203 This variable is a space-separated list of files
204 that are available, supports wildcards, and is set shortly
205 after the parsing phase of BitBake's execution.
206 </para>
207 235
208 <para> 236 <para>
237 During the configuration phase, BitBake will have
238 set
239 <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
240 BitBake now uses it to construct a list of recipes to parse,
241 along with any append files (<filename>.bbappend</filename>)
242 to apply.
243 <filename>BBFILES</filename> is a space-separated list of
244 available files and supports wildcards.
245 An example would be:
209 <literallayout class='monospaced'> 246 <literallayout class='monospaced'>
210 BBFILES = "/path/to/bbfiles/*.bb" 247 BBFILES = "/path/to/bbfiles/*.bb"
211 </literallayout> 248 </literallayout>
212 With regard to dependencies, it expects the 249 BitBake parses each recipe and append file located
213 <filename>.bb</filename> to define a 250 with <filename>BBFILES</filename> and stores the values of
214 <filename>DEPENDS</filename> variable, which contains a 251 various variables into the datastore.
215 space separated list of “package names”, which themselves 252 For each file, a fresh copy of the base configuration is
216 are the <filename>PN</filename> variable. The 253 made, then the recipe is parsed line by line.
217 <filename>PN</filename> variable is, in general, 254 Any inherit statements cause BitBake to find and
218 set to a component of the <filename>.bb</filename> 255 then parse class files (<filename>.bbclass</filename>)
219 filename by default. 256 using
257 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
258 as the search path.
259 Finally, BitBake parses in order any append files found in
260 <filename>BBFILES</filename>.
220 </para> 261 </para>
221 </section>
222
223 <section id='parsing-recipes'>
224 <title>Parsing Recipes</title>
225 262
226 <para> 263 <para>
227 After classes are included, the variable 264 One common convention is to use the recipe filename to define
228 <filename>BBFILES</filename> is set, usually in 265 pieces of metadata.
229 <filename>local.conf</filename>, and defines the list of 266 For example, in <filename>bitbake.conf</filename> the recipe
230 places to search for recipe and append files. 267 name and version set
231 Adding extra content to <filename>BBFILES</filename> is best 268 <link linkend='var-PN'><filename>PN</filename></link> and
232 achieved through the use of BitBake layers. 269 <link linkend='var-PV'><filename>PV</filename></link>:
270 <literallayout class='monospaced'>
271 PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}"
272 PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or 'defaultpkgname'}"
273 </literallayout>
274 In this example, a recipe called "something_1.2.3.bb" sets
275 <filename>PN</filename> to "something" and
276 <filename>PV</filename> to "1.2.3".
233 </para> 277 </para>
234 278
235 <para> 279 <para>
236 BitBake parses each recipe and append file located with 280 By the time parsing is complete for a recipe, BitBake
237 <filename>BBFILES</filename> and stores the values of various 281 has a list of tasks that the recipe defines and a set of
238 variables into the datastore. 282 data consisting of keys and values.
239 In summary, for each recipe and append file pairing, the configuration
240 plus the base class of variables are set, followed by the data in the
241 recipe file itself, followed by any inherit commands
242 that the recipe file might contain.
243 </para> 283 </para>
244 284
245 <para> 285 <para>
246 Part of parsing a recipe is making sure that all the recipes 286 BitBake does not need all this information.
247 that the recipe being parsed depends on are understood. 287 It only needs a small subset of the information to make
248 These other recipes could be located in other layers or in 288 decisions about the recipe.
249 a specific layer version. 289 Consequently, BitBake caches the values in which it is
250 These two variables in a recipe can help with with these 290 interested.
251 cases:
252 <itemizedlist>
253 <listitem><para>
254 <link linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
255 </para></listitem>
256 <listitem><para>
257 <link linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link>
258 </para></listitem>
259 </itemizedlist>
260 </para> 291 </para>
261 292
262 <para> 293 <para>
263 Because parsing recipe and append files is a time consuming 294 Subsequent BitBake commands then parse the base
264 process, a cache, referred to as the "setscene" 295 configuration and compute a checksum of that data.
265 is kept to speed up subsequent parsing. 296 If that checksum matches what is in the cache, the
266 The setscene is invalid if the timestamps of a recipe changes, 297 recipe and class files have not changed.
267 any of the include files change, configuration files change, 298 In this case, BitBake reloads the cached information
268 or class files on which the recipe file depends change. 299 about the recipe instead of reparsing it from scratch.
269 </para> 300 </para>
270 </section> 301 </section>
271 302
272 <section id='executing-tasks'> 303 <section id='bb-bitbake-providers'>
273 <title>Executing Tasks</title> 304 <title>Preferences and Providers</title>
274 305
275 <para> 306 <para>
276 Tasks can either be a shell task or a Python task. 307 Assuming BitBake has been instructed to execute a target and
277 For shell tasks, BitBake writes a shell script to 308 that all the recipe files have been parsed, BitBake starts to
278 <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> 309 build the target and look for providers of that target.
279 and then executes the script. 310 Once a provider is selected, BitBake resolves all the dependencies for
280 The generated shell script contains all the exported variables, 311 the target.
281 and the shell functions with all variables expanded. 312 As an example, suppose the target is
282 Output from the shell script goes to the file 313 <filename>core-image-sato</filename>.
283 <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>. 314 In this case, it would lead to
284 Looking at the expanded shell functions in the run file and 315 <filename>packagegroup-core-x11-sato</filename>,
285 the output in the log files is a useful debugging technique. 316 which in turn leads to recipes like <filename>matchbox-terminal</filename>,
317 <filename>pcmanfm</filename> and <filename>gthumb</filename>.
318 These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
286 </para> 319 </para>
287 320
288 <para> 321 <para>
289 For Python tasks, BitBake executes the task internally and logs 322 Sometimes a target might have multiple providers.
290 information to the controlling terminal. 323 A common example is "virtual/kernel", which is provided by each kernel package.
291 Future versions of BitBake will write the functions to files 324 Each machine often selects the best kernel provider by using a line similar to the
292 similar to the way shell tasks are handled. 325 following in the machine configuration file:
293 Logging will be handled in a way similar to shell tasks as well.
294 </para> 326 </para>
295 327
328 <literallayout class='monospaced'>
329 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
330 </literallayout>
331
296 <para> 332 <para>
297 Variables specific to scheduling functionality exist: 333 The default
298 <itemizedlist> 334 <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
299 <listitem><para> 335 is the provider with the same name as the target.
300 <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
301 </para></listitem>
302 <listitem><para>
303 <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
304 </para></listitem>
305 </itemizedlist>
306 </para> 336 </para>
307 </section>
308 337
309 <section id='source-fetching-dev-environment'> 338 <para>
310 <title>Source Fetching</title> 339 Understanding how providers are chosen is made complicated by the fact
340 that multiple versions might exist.
341 BitBake defaults to the highest version of a provider.
342 Version comparisons are made using the same method as Debian.
343 You can use the
344 <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
345 variable to specify a particular version (usually in the distro configuration).
346 You can influence the order by using the
347 <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
348 variable.
349 By default, files have a preference of "0".
350 Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
351 package unlikely to be used unless it is explicitly referenced.
352 Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
353 <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
354 <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
355 versions until they have undergone sufficient testing to be considered stable.
356 </para>
311 357
312 <para> 358 <para>
313 The first stages of building a recipe are to fetch and unpack 359 In summary, BitBake has created a list of providers, which is prioritized, for each target.
314 the source code:
315 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
316 </para> 360 </para>
361 </section>
362
363 <section id='bb-bitbake-dependencies'>
364 <title>Dependencies</title>
317 365
318 <para> 366 <para>
319 The <filename>do_fetch</filename> and 367 Each target BitBake builds consists of multiple tasks such as
320 <filename>do_unpack</filename> tasks fetch the source files 368 <filename>fetch</filename>, <filename>unpack</filename>,
321 and unpack them into the work directory. 369 <filename>patch</filename>, <filename>configure</filename>,
322 By default, everything is accomplished in the 370 and <filename>compile</filename>.
323 build directory, 371 For best performance on multi-core systems, BitBake considers each
324 which has a defined structure. 372 task as an independent
373 entity with its own set of dependencies.
325 </para> 374 </para>
326 375
327 <para> 376 <para>
328 Unpacked source files are pointed to by a variable. 377 Dependencies are defined through several variables.
329 For example, in the Yocto Project and OpenEmbedded build systems, 378 You can find information about variables BitBake uses in
330 the <filename>S</filename> variable points to these source files. 379 the <link linkend='ref-variables-glos'>Variables Glossary</link>
331 Each recipe has an area in the Build Directory where the 380 near the end of this manual.
332 unpacked source code resides. 381 At a basic level, it is sufficient to know that BitBake uses the
333 The name of that directory for any given recipe is defined from 382 <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
334 several different variables. 383 <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
335 You can see the variables that define these directories 384 calculating dependencies.
336 by looking at the figure that shows the structure and variables
337 used in the Yocto Project:
338 <itemizedlist>
339 <listitem><para><filename>TMPDIR</filename>
340 </para></listitem>
341 <listitem><para><filename>PACKAGE_ARCH</filename>
342 </para></listitem>
343 <listitem><para><filename>TARGET_OS</filename>
344 </para></listitem>
345 <listitem><para><link linkend='var-PN'><filename>PN</filename></link>
346 </para></listitem>
347 <listitem><para><link linkend='var-PV'><filename>PV</filename></link>
348 </para></listitem>
349 <listitem><para><link linkend='var-PR'><filename>PR</filename></link>
350 </para></listitem>
351 <listitem><para><filename>WORKDIR</filename>
352 </para></listitem>
353 <listitem><para><filename>S</filename>
354 </para></listitem>
355 </itemizedlist>
356 </para> 385 </para>
357 386
358 <para> 387 <para>
359 Briefly, the <filename>S</filename> directory contains the 388 For more information on how BitBake handles dependencies, see the
360 unpacked source files for a recipe. 389 "<link linkend='dependencies'>Dependencies</link>" section.
361 The <filename>WORKDIR</filename> directory is where all the
362 building goes on for a given recipe.
363 </para> 390 </para>
364 </section> 391 </section>
365 392
366 <section id='patching-dev-environment'> 393 <section id='ref-bitbake-tasklist'>
367 <title>Patching</title> 394 <title>The Task List</title>
368 395
369 <para> 396 <para>
370 Once source code is fetched and unpacked, BitBake locates 397 Based on the generated list of providers and the dependency information,
371 patch files and applies them to the source files: 398 BitBake can now calculate exactly what tasks it needs to run and in what
372 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" /> 399 order it needs to run them.
400 The build now starts with BitBake forking off threads up to the limit set in the
401 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
402 variable.
403 BitBake continues to fork threads as long as there are tasks ready to run,
404 those tasks have all their dependencies met, and the thread threshold has not been
405 exceeded.
373 </para> 406 </para>
374 407
375 <para> 408 <para>
376 The <filename>do_patch</filename> task processes recipes by 409 It is worth noting that you can greatly speed up the build time by properly setting
377 using the 410 the <filename>BB_NUMBER_THREADS</filename> variable.
378 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
379 variable to locate applicable patch files, which by default
380 are <filename>*.patch</filename> or
381 <filename>*.diff</filename> files, or any file if
382 "apply=yes" is specified for the file in
383 <filename>SRC_URI</filename>.
384 </para> 411 </para>
385 412
386 <para> 413 <para>
387 BitBake finds and applies multiple patches for a single recipe 414 As each task completes, a timestamp is written to the directory specified by the
388 in the order in which it finds the patches. 415 <link linkend='var-STAMP'><filename>STAMP</filename></link> variable.
389 Patches are applied to the recipe's source files located in the 416 On subsequent runs, BitBake looks in the build directory within
390 <filename>S</filename> directory. 417 <filename>tmp/stamps</filename>and does not rerun
418 tasks that are already completed unless a timestamp is found to be invalid.
419 Currently, invalid timestamps are only considered on a per
420 recipe file basis.
421 So, for example, if the configure stamp has a timestamp greater than the
422 compile timestamp for a given target, then the compile task would rerun.
423 Running the compile task again, however, has no effect on other providers
424 that depend on that target.
425 This behavior could change or become configurable in future versions of BitBake.
391 </para> 426 </para>
392 427
428 <note>
429 Some tasks are marked as "nostamp" tasks.
430 No timestamp file is created when these tasks are run.
431 Consequently, "nostamp" tasks are always rerun.
432 </note>
433
393 <para> 434 <para>
394 For more information on how the source directories are 435 For more information on tasks, see the
395 created, see the 436 "<link linkend='tasks'>Tasks</link>" section.
396 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
397 section.
398 </para> 437 </para>
399 </section> 438 </section>
400 439
401 <section id='configuration-and-compilation-dev-environment'> 440 <section id='executing-tasks'>
402 <title>Configuration and Compilation</title> 441 <title>Executing Tasks</title>
403
404 <para>
405 After source code is patched, BitBake executes tasks that
406 configure and compile the source code:
407 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
408 </para>
409 442
410 <para> 443 <para>
411 This step in the build process consists of three tasks: 444 Tasks can either be a shell task or a Python task.
412 <itemizedlist> 445 For shell tasks, BitBake writes a shell script to
413 <listitem><para><emphasis><filename>do_configure</filename>:</emphasis> 446 <filename>${WORKDIR}/temp/run.do_taskname.pid</filename>
414 This task configures the source by enabling and 447 and then executes the script.
415 disabling any build-time and configuration options for 448 The generated shell script contains all the exported variables,
416 the software being built. 449 and the shell functions with all variables expanded.
417 Configurations can come from the recipe itself as well 450 Output from the shell script goes to the file
418 as from an inherited class. 451 <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
419 Additionally, the software itself might configure itself 452 Looking at the expanded shell functions in the run file and
420 depending on the target for which it is being built. 453 the output in the log files is a useful debugging technique.
421 </para>
422
423 <para>The configurations handled by the
424 <filename>do_configure</filename> task are specific
425 to source code configuration for the source code
426 being built by the recipe.</para>
427
428 <para>If you are using the Autotools class
429 (<filename>autotools.bbclass</filename>),
430 you can add additional configuration options by using
431 the <filename>EXTRA_OECONF</filename>
432 variable.
433 For information on how this variable works within
434 that class, see the
435 <filename>meta/classes/autotools.bbclass</filename> file.
436 </para></listitem>
437 <listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
438 Once a configuration task has been satisfied, BitBake
439 compiles the source using the
440 <filename>do_compile</filename> task.
441 Compilation occurs in the directory pointed to by the
442 <link linkend='var-B'><filename>B</filename></link>
443 variable.
444 Realize that the <filename>B</filename> directory is, by
445 default, the same as the
446 <filename>S</filename>
447 directory.</para></listitem>
448 <listitem><para><emphasis><filename>do_install</filename>:</emphasis>
449 Once compilation is done, BitBake executes the
450 <filename>do_install</filename> task.
451 This task copies files from the <filename>B</filename>
452 directory and places them in a holding area pointed to
453 by the <filename>D</filename> variable.</para></listitem>
454 </itemizedlist>
455 </para> 454 </para>
456 </section>
457
458 <section id='package-splitting-dev-environment'>
459 <title>Package Splitting</title>
460 455
461 <para> 456 <para>
462 After source code is configured and compiled, the 457 For Python tasks, BitBake executes the task internally and logs
463 OpenEmbedded build system analyzes 458 information to the controlling terminal.
464 the results and splits the output into packages: 459 Future versions of BitBake will write the functions to files
465 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" /> 460 similar to the way shell tasks are handled.
461 Logging will be handled in a way similar to shell tasks as well.
466 </para> 462 </para>
467 463
468 <para> 464 <para>
469 The <filename>do_package</filename> and 465 Variables specific to scheduling functionality exist:
470 <filename>do_packagedata</filename> tasks combine to analyze
471 the files found in the <filename>D</filename> directory
472 and split them into subsets based on available packages and
473 files.
474 The analyzing process involves the following as well as other
475 items: splitting out debugging symbols,
476 looking at shared library dependencies between packages,
477 and looking at package relationships.
478 The <filename>do_packagedata</filename> task creates package
479 metadata based on the analysis such that the
480 OpenEmbedded build system can generate the final packages.
481 Working, staged, and intermediate results of the analysis
482 and package splitting process use these areas:
483 <itemizedlist> 466 <itemizedlist>
484 <listitem><para><filename>PKGD</filename> 467 <listitem><para>
485 </para></listitem> 468 <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
486 <listitem><para><filename>PKGDATA_DIR</filename>
487 </para></listitem>
488 <listitem><para><filename>PKGDESTWORK</filename>
489 </para></listitem> 469 </para></listitem>
490 <listitem><para><filename>PKGDEST</filename> 470 <listitem><para>
471 <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
491 </para></listitem> 472 </para></listitem>
492 </itemizedlist> 473 </itemizedlist>
493 The <filename>FILES</filename>
494 variable defines the files that go into each package in
495 <link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>.
496 If you want details on how this is accomplished in the Yocto Project
497 for example, you can look at the <filename>package.bbclass</filename>
498 file in a Yocto tree.
499 </para>
500
501 <para>
502 Depending on the type of packages being created (RPM, DEB, or
503 IPK), the <filename>do_package_write_*</filename> task
504 creates the actual packages and places them in the
505 Package Feed area, which is
506 <filename>${TMPDIR}/deploy</filename>.
507 <note>
508 Support for creating feeds directly from the
509 <filename>deploy/*</filename> directories does not exist.
510 Creating such feeds usually requires some kind of feed
511 maintenance mechanism that would upload the new packages
512 into an official package feed (e.g. the
513 Ångström distribution).
514 This functionality is highly distribution-specific
515 and thus is not provided out of the box.
516 </note>
517 </para> 474 </para>
518 </section> 475 </section>
519 476
520 <section id='image-generation-dev-environment'> 477 <section id='setscene'>
521 <title>Image Generation</title> 478 <title>Setscene</title>
522 479
523 <para> 480 <para>
524 Once packages are split and stored in the Package Feeds area, 481 This section needs to get the concept of the setscene across.
525 the OpenEmbedded build system uses BitBake to generate the 482 The reader needs to know what it is and what it is used for during
526 root filesystem image: 483 the build process.
527 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
528 </para> 484 </para>
529 485
530 <para> 486 <para>
531 The image generation process consists of several stages and 487 You can find more information on setscene metadata in the
532 depends on many variables. 488 "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
533 The <filename>do_rootfs</filename> task uses these key variables 489 section.
534 to help create the list of packages to actually install:
535 <itemizedlist>
536 <listitem><para><filename>IMAGE_INSTALL</filename>:
537 Lists out the base set of packages to install from
538 the Package Feeds area.</para></listitem>
539 <listitem><para><filename>PACKAGE_EXCLUDE</filename>:
540 Specifies packages that should not be installed.
541 </para></listitem>
542 <listitem><para><filename>IMAGE_FEATURES</filename>:
543 Specifies features to include in the image.
544 Most of these features map to additional packages for
545 installation.</para></listitem>
546 <listitem><para><filename>PACKAGE_CLASSES</filename>:
547 Specifies the package backend to use and consequently
548 helps determine where to locate packages within the
549 Package Feeds area.</para></listitem>
550 <listitem><para><filename>IMAGE_LINGUAS</filename>:
551 Determines the language(s) for which additional
552 language support packages are installed.
553 </para></listitem>
554 </itemizedlist>
555 </para> 490 </para>
491 </section>
492
493 <section id='checksums'>
494 <title>Checksums (Signatures)</title>
556 495
557 <para> 496 <para>
558 Package installation is under control of the package manager 497 A checksum is a unique signature of a task's inputs.
559 (e.g. smart/rpm, opkg, or apt/dpkg) regardless of whether or 498 The setscene code uses a checksum to determine if a task needs
560 not package management is enabled for the target. 499 to be run.
561 At the end of the process, if package management is not 500 Because it is a change in a task's inputs that triggers running
562 enabled for the target, the package manager's data files 501 the task, the process needs to detect all the inputs to a given task.
563 are deleted from the root filesystem. 502 For shell tasks, this turns out to be fairly easy because
503 BitBake generates a "run" shell script for each task and
504 it is possible to create a checksum that gives you a good idea of when
505 the task's data changes.
564 </para> 506 </para>
565 507
566 <para> 508 <para>
567 During image generation, the build system attempts to run 509 To complicate the problem, some things should not be included in
568 all post-installation scripts. 510 the checksum.
569 Any that fail to run on the build host are run on the 511 First, there is the actual specific build path of a given task -
570 target when the target system is first booted. 512 the working directory.
571 If you are using a 513 It does not matter if the working directory changes because it should not
572 read-only root filesystem, 514 affect the output for target packages.
573 all the post installation scripts must succeed during the 515 The simplistic approach for excluding the working directory is to set
574 package installation phase since the root filesystem cannot be 516 it to some fixed value and create the checksum for the "run" script.
575 written into.
576 </para> 517 </para>
577 518
578 <para> 519 <para>
579 During Optimization, optimizing processes are run across 520 Another problem results from the "run" scripts containing functions that
580 the image. 521 might or might not get called.
581 These processes include <filename>mklibs</filename> and 522 The incremental build solution contains code that figures out dependencies
582 <filename>prelink</filename>. 523 between shell functions.
583 The <filename>mklibs</filename> process optimizes the size 524 This code is used to prune the "run" scripts down to the minimum set,
584 of the libraries. 525 thereby alleviating this problem and making the "run" scripts much more
585 A <filename>prelink</filename> process optimizes the dynamic 526 readable as a bonus.
586 linking of shared libraries to reduce start up time of
587 executables.
588 </para> 527 </para>
589 528
590 <para> 529 <para>
591 Part of the image generation process includes compressing the 530 So far we have solutions for shell scripts.
592 root filesystem image. 531 What about Python tasks?
593 Compression is accomplished through several optimization 532 The same approach applies even though these tasks are more difficult.
594 routines designed to reduce the overall size of the image. 533 The process needs to figure out what variables a Python function accesses
534 and what functions it calls.
535 Again, the incremental build solution contains code that first figures out
536 the variable and function dependencies, and then creates a checksum for the data
537 used as the input to the task.
595 </para> 538 </para>
596 539
597 <para> 540 <para>
598 After the root filesystem has been constructed, the image 541 Like the working directory case, situations exist where dependencies
599 generation process turns everything into an image file or 542 should be ignored.
600 a set of image files. 543 For these cases, you can instruct the build process to ignore a dependency
601 The formats used for the root filesystem depend on the 544 by using a line like the following:
602 <filename>IMAGE_FSTYPES</filename> variable. 545 <literallayout class='monospaced'>
546 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
547 </literallayout>
548 This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
549 depend on the value of <filename>MACHINE</filename>, even if it does reference it.
603 </para> 550 </para>
604 551
605 <note> 552 <para>
606 The entire image generation process is run under Pseudo. 553 Equally, there are cases where we need to add dependencies BitBake
607 Running under Pseudo ensures that the files in the root 554 is not able to find.
608 filesystem have correct ownership. 555 You can accomplish this by using a line like the following:
609 </note> 556 <literallayout class='monospaced'>
610 </section> 557 PACKAGE_ARCHS[vardeps] = "MACHINE"
611 558 </literallayout>
612 <section id='sdk-generation-dev-environment'> 559 This example explicitly adds the <filename>MACHINE</filename> variable as a
613 <title>SDK Generation</title> 560 dependency for <filename>PACKAGE_ARCHS</filename>.
561 </para>
614 562
615 <para> 563 <para>
616 The OpenEmbedded build system uses BitBake to generate the 564 Consider a case with in-line Python, for example, where BitBake is not
617 Software Development Kit (SDK) installer script: 565 able to figure out dependencies.
618 <imagedata fileref="figures/sdk-generation.png" align="center" width="6in" depth="7in" /> 566 When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
567 produces output when it discovers something for which it cannot figure out
568 dependencies.
619 </para> 569 </para>
620 570
621 <para> 571 <para>
622 Like image generation, the SDK script process consists of 572 Thus far, this section has limited discussion to the direct inputs into a task.
623 several stages and depends on many variables. 573 Information based on direct inputs is referred to as the "basehash" in the
624 The <filename>do_populate_sdk</filename> task uses these 574 code.
625 key variables to help create the list of packages to actually 575 However, there is still the question of a task's indirect inputs - the
626 install. 576 things that were already built and present in the build directory.
577 The checksum (or signature) for a particular task needs to add the hashes
578 of all the tasks on which the particular task depends.
579 Choosing which dependencies to add is a policy decision.
580 However, the effect is to generate a master checksum that combines the basehash
581 and the hashes of the task's dependencies.
627 </para> 582 </para>
628 583
629 <para> 584 <para>
630 The <filename>do_populate_sdk</filename> task handles two 585 At the code level, there are a variety of ways both the basehash and the
631 parts: a target part and a host part. 586 dependent task hashes can be influenced.
632 The target part is the part built for the target hardware and 587 Within the BitBake configuration file, we can give BitBake some extra information
633 includes libraries and headers. 588 to help it construct the basehash.
634 The host part is the part of the SDK that runs on the 589 The following statement effectively results in a list of global variable
635 <filename>SDKMACHINE</filename>. 590 dependency excludes - variables never included in any checksum.
591 This example uses variables from OpenEmbedded to help illustrate
592 the concept:
593 <literallayout class='monospaced'>
594 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
595 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
596 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
597 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
598 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
599 </literallayout>
600 The previous example excludes the work directory, which is part of
601 <filename>TMPDIR</filename>.
602 </para>
603
604 <para>
605 The rules for deciding which hashes of dependent tasks to include through
606 dependency chains are more complex and are generally accomplished with a
607 Python function.
608 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
609 of this and also illustrates how you can insert your own policy into the system
610 if so desired.
611 This file defines the two basic signature generators OpenEmbedded Core
612 uses: "OEBasic" and "OEBasicHash".
613 By default, there is a dummy "noop" signature handler enabled in BitBake.
614 This means that behavior is unchanged from previous versions.
615 <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
616 through this setting in the <filename>bitbake.conf</filename> file:
617 <literallayout class='monospaced'>
618 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
619 </literallayout>
620 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
621 "OEBasic" version but adds the task hash to the stamp files.
622 This results in any metadata change that changes the task hash, automatically
623 causing the task to be run again.
624 This removes the need to bump
625 <link linkend='var-PR'><filename>PR</filename></link>
626 values, and changes to metadata automatically ripple across the build.
636 </para> 627 </para>
637 628
638 <para> 629 <para>
639 Once both parts are constructed, the 630 It is also worth noting that the end result of these signature generators is to
640 <filename>do_populate_sdk</filename> task performs some cleanup 631 make some dependency and hash information available to the build.
641 on both parts. 632 This information includes:
642 After the cleanup, the task creates a cross-development 633 <itemizedlist>
643 environment setup script and any configuration files that 634 <listitem><para><filename>BB_BASEHASH_task-&lt;taskname&gt;</filename>:
644 might be needed. 635 The base hashes for each task in the recipe.
636 </para></listitem>
637 <listitem><para><filename>BB_BASEHASH_&lt;filename:taskname&gt;</filename>:
638 The base hashes for each dependent task.
639 </para></listitem>
640 <listitem><para><filename>BBHASHDEPS_&lt;filename:taskname&gt;</filename>:
641 The task dependencies for each task.
642 </para></listitem>
643 <listitem><para><filename>BB_TASKHASH</filename>:
644 The hash of the currently running task.
645 </para></listitem>
646 </itemizedlist>
645 </para> 647 </para>
646 648
647 <para> 649 <para>
648 The final output of the task is the Cross-development 650 You can find more information on checksum metadata in the
649 toolchain installation script (<filename>.sh</filename> file), 651 "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
650 which includes the environment setup script. 652 section.
651 </para> 653 </para>
652 </section> 654 </section>
653</chapter> 655</chapter>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-15 13:56:28 +0000