summaryrefslogtreecommitdiffstats
path: root/documentation/overview-manual/overview-manual-concepts.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/overview-manual/overview-manual-concepts.xml')
-rw-r--r--documentation/overview-manual/overview-manual-concepts.xml3235
1 files changed, 0 insertions, 3235 deletions
diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml
deleted file mode 100644
index 58b64bd269..0000000000
--- a/documentation/overview-manual/overview-manual-concepts.xml
+++ /dev/null
@@ -1,3235 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id=' overview-manual-concepts'>
7<title>Yocto Project Concepts</title>
8
9 <para>
10 This chapter provides explanations for Yocto Project concepts that
11 go beyond the surface of "how-to" information and reference (or
12 look-up) material.
13 Concepts such as components, the
14 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
15 workflow, cross-development toolchains, shared state cache, and so
16 forth are explained.
17 </para>
18
19 <section id='yocto-project-components'>
20 <title>Yocto Project Components</title>
21
22 <para>
23 The
24 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
25 task executor together with various types of configuration files
26 form the
27 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>.
28 This section overviews these components by describing their use and
29 how they interact.
30 </para>
31
32 <para>
33 BitBake handles the parsing and execution of the data files.
34 The data itself is of various types:
35 <itemizedlist>
36 <listitem><para>
37 <emphasis>Recipes:</emphasis>
38 Provides details about particular pieces of software.
39 </para></listitem>
40 <listitem><para>
41 <emphasis>Class Data:</emphasis>
42 Abstracts common build information (e.g. how to build a
43 Linux kernel).
44 </para></listitem>
45 <listitem><para>
46 <emphasis>Configuration Data:</emphasis>
47 Defines machine-specific settings, policy decisions, and
48 so forth.
49 Configuration data acts as the glue to bind everything
50 together.
51 </para></listitem>
52 </itemizedlist>
53 </para>
54
55 <para>
56 BitBake knows how to combine multiple data sources together and
57 refers to each data source as a layer.
58 For information on layers, see the
59 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
60 section of the Yocto Project Development Tasks Manual.
61 </para>
62
63 <para>
64 Following are some brief details on these core components.
65 For additional information on how these components interact during
66 a build, see the
67 "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>"
68 section.
69 </para>
70
71 <section id='usingpoky-components-bitbake'>
72 <title>BitBake</title>
73
74 <para>
75 BitBake is the tool at the heart of the
76 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
77 and is responsible for parsing the
78 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>,
79 generating a list of tasks from it, and then executing those
80 tasks.
81 </para>
82
83 <para>
84 This section briefly introduces BitBake.
85 If you want more information on BitBake, see the
86 <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>.
87 </para>
88
89 <para>
90 To see a list of the options BitBake supports, use either of
91 the following commands:
92 <literallayout class='monospaced'>
93 $ bitbake -h
94 $ bitbake --help
95 </literallayout>
96 </para>
97
98 <para>
99 The most common usage for BitBake is
100 <filename>bitbake <replaceable>packagename</replaceable></filename>,
101 where <filename>packagename</filename> is the name of the
102 package you want to build (referred to as the "target").
103 The target often equates to the first part of a recipe's
104 filename (e.g. "foo" for a recipe named
105 <filename>foo_1.3.0-r0.bb</filename>).
106 So, to process the
107 <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you
108 might type the following:
109 <literallayout class='monospaced'>
110 $ bitbake matchbox-desktop
111 </literallayout>
112 Several different versions of
113 <filename>matchbox-desktop</filename> might exist.
114 BitBake chooses the one selected by the distribution
115 configuration.
116 You can get more details about how BitBake chooses between
117 different target versions and providers in the
118 "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>"
119 section of the BitBake User Manual.
120 </para>
121
122 <para>
123 BitBake also tries to execute any dependent tasks first.
124 So for example, before building
125 <filename>matchbox-desktop</filename>, BitBake would build a
126 cross compiler and <filename>glibc</filename> if they had not
127 already been built.
128 </para>
129
130 <para>
131 A useful BitBake option to consider is the
132 <filename>-k</filename> or <filename>--continue</filename>
133 option.
134 This option instructs BitBake to try and continue processing
135 the job as long as possible even after encountering an error.
136 When an error occurs, the target that failed and those that
137 depend on it cannot be remade.
138 However, when you use this option other dependencies can
139 still be processed.
140 </para>
141 </section>
142
143 <section id='overview-components-recipes'>
144 <title>Recipes</title>
145
146 <para>
147 Files that have the <filename>.bb</filename> suffix are
148 "recipes" files.
149 In general, a recipe contains information about a single piece
150 of software.
151 This information includes the location from which to download
152 the unaltered source, any source patches to be applied to that
153 source (if needed), which special configuration options to
154 apply, how to compile the source files, and how to package the
155 compiled output.
156 </para>
157
158 <para>
159 The term "package" is sometimes used to refer to recipes.
160 However, since the word "package" is used for the packaged
161 output from the OpenEmbedded build system (i.e.
162 <filename>.ipk</filename> or <filename>.deb</filename> files),
163 this document avoids using the term "package" when referring
164 to recipes.
165 </para>
166 </section>
167
168 <section id='overview-components-classes'>
169 <title>Classes</title>
170
171 <para>
172 Class files (<filename>.bbclass</filename>) contain information
173 that is useful to share between recipes files.
174 An example is the
175 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
176 class, which contains common settings for any application that
177 Autotools uses.
178 The
179 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
180 chapter in the Yocto Project Reference Manual provides
181 details about classes and how to use them.
182 </para>
183 </section>
184
185 <section id='overview-components-configurations'>
186 <title>Configurations</title>
187
188 <para>
189 The configuration files (<filename>.conf</filename>) define
190 various configuration variables that govern the OpenEmbedded
191 build process.
192 These files fall into several areas that define machine
193 configuration options, distribution configuration options,
194 compiler tuning options, general common configuration options,
195 and user configuration options in
196 <filename>conf/local.conf</filename>, which is found in the
197 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
198 </para>
199 </section>
200 </section>
201
202 <section id='overview-layers'>
203 <title>Layers</title>
204
205 <para>
206 Layers are repositories that contain related metadata (i.e.
207 sets of instructions) that tell the OpenEmbedded build system how
208 to build a target.
209 Yocto Project's
210 <link linkend='the-yocto-project-layer-model'>layer model</link>
211 facilitates collaboration, sharing, customization, and reuse
212 within the Yocto Project development environment.
213 Layers logically separate information for your project.
214 For example, you can use a layer to hold all the configurations
215 for a particular piece of hardware.
216 Isolating hardware-specific configurations allows you to share
217 other metadata by using a different layer where that metadata
218 might be common across several pieces of hardware.
219 </para>
220
221 <para>
222 Many layers exist that work in the Yocto Project development
223 environment.
224 The
225 <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project Curated Layer Index</ulink>
226 and
227 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>
228 both contain layers from which you can use or leverage.
229 </para>
230
231 <para>
232 By convention, layers in the Yocto Project follow a specific form.
233 Conforming to a known structure allows BitBake to make assumptions
234 during builds on where to find types of metadata.
235 You can find procedures and learn about tools (i.e.
236 <filename>bitbake-layers</filename>) for creating layers suitable
237 for the Yocto Project in the
238 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
239 section of the Yocto Project Development Tasks Manual.
240 </para>
241 </section>
242
243 <section id="openembedded-build-system-build-concepts">
244 <title>OpenEmbedded Build System Concepts</title>
245
246 <para>
247 This section takes a more detailed look inside the build
248 process used by the
249 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
250 which is the build system specific to the Yocto Project.
251 At the heart of the build system is BitBake, the task executor.
252 </para>
253
254 <para>
255 The following diagram represents the high-level workflow of a
256 build.
257 The remainder of this section expands on the fundamental input,
258 output, process, and metadata logical blocks that make up the
259 workflow.
260 </para>
261
262 <para id='general-workflow-figure'>
263 <imagedata fileref="figures/YP-flow-diagram.png" format="PNG" align='center' width="8in"/>
264 </para>
265
266 <para>
267 In general, the build's workflow consists of several functional
268 areas:
269 <itemizedlist>
270 <listitem><para>
271 <emphasis>User Configuration:</emphasis>
272 metadata you can use to control the build process.
273 </para></listitem>
274 <listitem><para>
275 <emphasis>Metadata Layers:</emphasis>
276 Various layers that provide software, machine, and
277 distro metadata.
278 </para></listitem>
279 <listitem><para>
280 <emphasis>Source Files:</emphasis>
281 Upstream releases, local projects, and SCMs.
282 </para></listitem>
283 <listitem><para>
284 <emphasis>Build System:</emphasis>
285 Processes under the control of
286 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>.
287 This block expands on how BitBake fetches source, applies
288 patches, completes compilation, analyzes output for package
289 generation, creates and tests packages, generates images,
290 and generates cross-development tools.
291 </para></listitem>
292 <listitem><para>
293 <emphasis>Package Feeds:</emphasis>
294 Directories containing output packages (RPM, DEB or IPK),
295 which are subsequently used in the construction of an
296 image or Software Development Kit (SDK), produced by the
297 build system.
298 These feeds can also be copied and shared using a web
299 server or other means to facilitate extending or updating
300 existing images on devices at runtime if runtime package
301 management is enabled.
302 </para></listitem>
303 <listitem><para>
304 <emphasis>Images:</emphasis>
305 Images produced by the workflow.
306 </para></listitem>
307 <listitem><para>
308 <emphasis>Application Development SDK:</emphasis>
309 Cross-development tools that are produced along with
310 an image or separately with BitBake.
311 </para></listitem>
312 </itemizedlist>
313 </para>
314
315 <section id="user-configuration">
316 <title>User Configuration</title>
317
318 <para>
319 User configuration helps define the build.
320 Through user configuration, you can tell BitBake the
321 target architecture for which you are building the image,
322 where to store downloaded source, and other build properties.
323 </para>
324
325 <para>
326 The following figure shows an expanded representation of the
327 "User Configuration" box of the
328 <link linkend='general-workflow-figure'>general workflow figure</link>:
329 </para>
330
331 <para>
332 <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" />
333 </para>
334
335 <para>
336 BitBake needs some basic configuration files in order to
337 complete a build.
338 These files are <filename>*.conf</filename> files.
339 The minimally necessary ones reside as example files in the
340 <filename>build/conf</filename> directory of the
341 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
342 For simplicity, this section refers to the Source Directory as
343 the "Poky Directory."
344 </para>
345
346 <para>
347 When you clone the
348 <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>
349 Git repository or you download and unpack a Yocto Project
350 release, you can set up the Source Directory to be named
351 anything you want.
352 For this discussion, the cloned repository uses the default
353 name <filename>poky</filename>.
354 <note>
355 The Poky repository is primarily an aggregation of existing
356 repositories.
357 It is not a canonical upstream source.
358 </note>
359 </para>
360
361 <para>
362 The <filename>meta-poky</filename> layer inside Poky contains
363 a <filename>conf</filename> directory that has example
364 configuration files.
365 These example files are used as a basis for creating actual
366 configuration files when you source
367 <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>,
368 which is the build environment script.
369 </para>
370
371 <para>
372 Sourcing the build environment script creates a
373 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
374 if one does not already exist.
375 BitBake uses the Build Directory for all its work during
376 builds.
377 The Build Directory has a <filename>conf</filename> directory
378 that contains default versions of your
379 <filename>local.conf</filename> and
380 <filename>bblayers.conf</filename> configuration files.
381 These default configuration files are created only if versions
382 do not already exist in the Build Directory at the time you
383 source the build environment setup script.
384 </para>
385
386 <para>
387 Because the Poky repository is fundamentally an aggregation of
388 existing repositories, some users might be familiar with
389 running the <filename>&OE_INIT_FILE;</filename> script
390 in the context of separate
391 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>
392 and BitBake repositories rather than a single Poky repository.
393 This discussion assumes the script is executed from
394 within a cloned or unpacked version of Poky.
395 </para>
396
397 <para>
398 Depending on where the script is sourced, different
399 sub-scripts are called to set up the Build Directory
400 (Yocto or OpenEmbedded).
401 Specifically, the script
402 <filename>scripts/oe-setup-builddir</filename> inside the
403 poky directory sets up the Build Directory and seeds the
404 directory (if necessary) with configuration files appropriate
405 for the Yocto Project development environment.
406 <note>
407 The <filename>scripts/oe-setup-builddir</filename> script
408 uses the <filename>$TEMPLATECONF</filename> variable to
409 determine which sample configuration files to locate.
410 </note>
411 </para>
412
413 <para>
414 The <filename>local.conf</filename> file provides many
415 basic variables that define a build environment.
416 Here is a list of a few.
417 To see the default configurations in a
418 <filename>local.conf</filename> file created by the build
419 environment script, see the
420 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample'><filename>local.conf.sample</filename></ulink>
421 in the <filename>meta-poky</filename> layer:
422 <itemizedlist>
423 <listitem><para>
424 <emphasis>Target Machine Selection:</emphasis>
425 Controlled by the
426 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>
427 variable.
428 </para></listitem>
429 <listitem><para>
430 <emphasis>Download Directory:</emphasis>
431 Controlled by the
432 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
433 variable.
434 </para></listitem>
435 <listitem><para>
436 <emphasis>Shared State Directory:</emphasis>
437 Controlled by the
438 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
439 variable.
440 </para></listitem>
441 <listitem><para>
442 <emphasis>Build Output:</emphasis>
443 Controlled by the
444 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
445 variable.
446 </para></listitem>
447 <listitem><para>
448 <emphasis>Distribution Policy:</emphasis>
449 Controlled by the
450 <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink>
451 variable.
452 </para></listitem>
453 <listitem><para>
454 <emphasis>Packaging Format:</emphasis>
455 Controlled by the
456 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
457 variable.
458 </para></listitem>
459 <listitem><para>
460 <emphasis>SDK Target Architecture:</emphasis>
461 Controlled by the
462 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>
463 variable.
464 </para></listitem>
465 <listitem><para>
466 <emphasis>Extra Image Packages:</emphasis>
467 Controlled by the
468 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink>
469 variable.
470 </para></listitem>
471 </itemizedlist>
472 <note>
473 Configurations set in the
474 <filename>conf/local.conf</filename> file can also be set
475 in the <filename>conf/site.conf</filename> and
476 <filename>conf/auto.conf</filename> configuration files.
477 </note>
478 </para>
479
480 <para>
481 The <filename>bblayers.conf</filename> file tells BitBake what
482 layers you want considered during the build.
483 By default, the layers listed in this file include layers
484 minimally needed by the build system.
485 However, you must manually add any custom layers you have
486 created.
487 You can find more information on working with the
488 <filename>bblayers.conf</filename> file in the
489 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
490 section in the Yocto Project Development Tasks Manual.
491 </para>
492
493 <para>
494 The files <filename>site.conf</filename> and
495 <filename>auto.conf</filename> are not created by the
496 environment initialization script.
497 If you want the <filename>site.conf</filename> file, you
498 need to create that yourself.
499 The <filename>auto.conf</filename> file is typically created by
500 an autobuilder:
501 <itemizedlist>
502 <listitem><para>
503 <emphasis><filename>site.conf</filename>:</emphasis>
504 You can use the <filename>conf/site.conf</filename>
505 configuration file to configure multiple
506 build directories.
507 For example, suppose you had several build environments
508 and they shared some common features.
509 You can set these default build properties here.
510 A good example is perhaps the packaging format to use
511 through the
512 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
513 variable.</para>
514
515 <para>One useful scenario for using the
516 <filename>conf/site.conf</filename> file is to extend
517 your
518 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
519 variable to include the path to a
520 <filename>conf/site.conf</filename>.
521 Then, when BitBake looks for Metadata using
522 <filename>BBPATH</filename>, it finds the
523 <filename>conf/site.conf</filename> file and applies
524 your common configurations found in the file.
525 To override configurations in a particular build
526 directory, alter the similar configurations within
527 that build directory's
528 <filename>conf/local.conf</filename> file.
529 </para></listitem>
530 <listitem><para>
531 <emphasis><filename>auto.conf</filename>:</emphasis>
532 The file is usually created and written to by
533 an autobuilder.
534 The settings put into the file are typically the
535 same as you would find in the
536 <filename>conf/local.conf</filename> or the
537 <filename>conf/site.conf</filename> files.
538 </para></listitem>
539 </itemizedlist>
540 </para>
541
542 <para>
543 You can edit all configuration files to further define
544 any particular build environment.
545 This process is represented by the "User Configuration Edits"
546 box in the figure.
547 </para>
548
549 <para>
550 When you launch your build with the
551 <filename>bitbake <replaceable>target</replaceable></filename>
552 command, BitBake sorts out the configurations to ultimately
553 define your build environment.
554 It is important to understand that the
555 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
556 reads the configuration files in a specific order:
557 <filename>site.conf</filename>, <filename>auto.conf</filename>,
558 and <filename>local.conf</filename>.
559 And, the build system applies the normal assignment statement
560 rules as described in the
561 "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>"
562 chapter of the BitBake User Manual.
563 Because the files are parsed in a specific order, variable
564 assignments for the same variable could be affected.
565 For example, if the <filename>auto.conf</filename> file and
566 the <filename>local.conf</filename> set
567 <replaceable>variable1</replaceable> to different values,
568 because the build system parses <filename>local.conf</filename>
569 after <filename>auto.conf</filename>,
570 <replaceable>variable1</replaceable> is assigned the value from
571 the <filename>local.conf</filename> file.
572 </para>
573 </section>
574
575 <section id="metadata-machine-configuration-and-policy-configuration">
576 <title>Metadata, Machine Configuration, and Policy Configuration</title>
577
578 <para>
579 The previous section described the user configurations that
580 define BitBake's global behavior.
581 This section takes a closer look at the layers the build system
582 uses to further control the build.
583 These layers provide Metadata for the software, machine, and
584 policies.
585 </para>
586
587 <para>
588 In general, three types of layer input exists.
589 You can see them below the "User Configuration" box in the
590 <link linkend='general-workflow-figure'>general workflow figure</link>:
591 <itemizedlist>
592 <listitem><para>
593 <emphasis>Metadata (<filename>.bb</filename> + Patches):</emphasis>
594 Software layers containing user-supplied recipe files,
595 patches, and append files.
596 A good example of a software layer might be the
597 <ulink url='https://github.com/meta-qt5/meta-qt5'><filename>meta-qt5</filename></ulink>
598 layer from the
599 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>.
600 This layer is for version 5.0 of the popular
601 <ulink url='https://wiki.qt.io/About_Qt'>Qt</ulink>
602 cross-platform application development framework for
603 desktop, embedded and mobile.
604 </para></listitem>
605 <listitem><para>
606 <emphasis>Machine BSP Configuration:</emphasis>
607 Board Support Package (BSP) layers (i.e. "BSP Layer"
608 in the following figure) providing machine-specific
609 configurations.
610 This type of information is specific to a particular
611 target architecture.
612 A good example of a BSP layer from the
613 <link linkend='gs-reference-distribution-poky'>Poky Reference Distribution</link>
614 is the
615 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink>
616 layer.
617 </para></listitem>
618 <listitem><para>
619 <emphasis>Policy Configuration:</emphasis>
620 Distribution Layers (i.e. "Distro Layer" in the
621 following figure) providing top-level or general
622 policies for the images or SDKs being built for a
623 particular distribution.
624 For example, in the Poky Reference Distribution the
625 distro layer is the
626 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink>
627 layer.
628 Within the distro layer is a
629 <filename>conf/distro</filename> directory that
630 contains distro configuration files (e.g.
631 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink>
632 that contain many policy configurations for the
633 Poky distribution.
634 </para></listitem>
635 </itemizedlist>
636 </para>
637
638 <para>
639 The following figure shows an expanded representation of
640 these three layers from the
641 <link linkend='general-workflow-figure'>general workflow figure</link>:
642 </para>
643
644 <para>
645 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="8in" />
646 </para>
647
648 <para>
649 In general, all layers have a similar structure.
650 They all contain a licensing file
651 (e.g. <filename>COPYING.MIT</filename>) if the layer is to be
652 distributed, a <filename>README</filename> file as good
653 practice and especially if the layer is to be distributed, a
654 configuration directory, and recipe directories.
655 You can learn about the general structure for layers used with
656 the Yocto Project in the
657 "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-your-own-layer'>Creating Your Own Layer</ulink>"
658 section in the Yocto Project Development Tasks Manual.
659 For a general discussion on layers and the many layers from
660 which you can draw, see the
661 "<link linkend='overview-layers'>Layers</link>" and
662 "<link linkend='the-yocto-project-layer-model'>The Yocto Project Layer Model</link>"
663 sections both earlier in this manual.
664 </para>
665
666 <para>
667 If you explored the previous links, you discovered some
668 areas where many layers that work with the Yocto Project
669 exist.
670 The
671 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
672 also shows layers categorized under "Yocto Metadata Layers."
673 <note>
674 Layers exist in the Yocto Project Source Repositories that
675 cannot be found in the OpenEmbedded Layer Index.
676 These layers are either deprecated or experimental
677 in nature.
678 </note>
679 </para>
680
681 <para>
682 BitBake uses the <filename>conf/bblayers.conf</filename> file,
683 which is part of the user configuration, to find what layers it
684 should be using as part of the build.
685 </para>
686
687 <section id="distro-layer">
688 <title>Distro Layer</title>
689
690 <para>
691 The distribution layer provides policy configurations for
692 your distribution.
693 Best practices dictate that you isolate these types of
694 configurations into their own layer.
695 Settings you provide in
696 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
697 similar settings that BitBake finds in your
698 <filename>conf/local.conf</filename> file in the Build
699 Directory.
700 </para>
701
702 <para>
703 The following list provides some explanation and references
704 for what you typically find in the distribution layer:
705 <itemizedlist>
706 <listitem><para>
707 <emphasis>classes:</emphasis>
708 Class files (<filename>.bbclass</filename>) hold
709 common functionality that can be shared among
710 recipes in the distribution.
711 When your recipes inherit a class, they take on the
712 settings and functions for that class.
713 You can read more about class files in the
714 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
715 chapter of the Yocto Reference Manual.
716 </para></listitem>
717 <listitem><para>
718 <emphasis>conf:</emphasis>
719 This area holds configuration files for the
720 layer (<filename>conf/layer.conf</filename>),
721 the distribution
722 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
723 and any distribution-wide include files.
724 </para></listitem>
725 <listitem><para>
726 <emphasis>recipes-*:</emphasis>
727 Recipes and append files that affect common
728 functionality across the distribution.
729 This area could include recipes and append files
730 to add distribution-specific configuration,
731 initialization scripts, custom image recipes,
732 and so forth.
733 Examples of <filename>recipes-*</filename>
734 directories are <filename>recipes-core</filename>
735 and <filename>recipes-extra</filename>.
736 Hierarchy and contents within a
737 <filename>recipes-*</filename> directory can vary.
738 Generally, these directories contain recipe files
739 (<filename>*.bb</filename>), recipe append files
740 (<filename>*.bbappend</filename>), directories
741 that are distro-specific for configuration files,
742 and so forth.
743 </para></listitem>
744 </itemizedlist>
745 </para>
746 </section>
747
748 <section id="bsp-layer">
749 <title>BSP Layer</title>
750
751 <para>
752 The BSP Layer provides machine configurations that
753 target specific hardware.
754 Everything in this layer is specific to the machine for
755 which you are building the image or the SDK.
756 A common structure or form is defined for BSP layers.
757 You can learn more about this structure in the
758 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
759 <note>
760 In order for a BSP layer to be considered compliant
761 with the Yocto Project, it must meet some structural
762 requirements.
763 </note>
764 </para>
765
766 <para>
767 The BSP Layer's configuration directory contains
768 configuration files for the machine
769 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>)
770 and, of course, the layer
771 (<filename>conf/layer.conf</filename>).
772 </para>
773
774 <para>
775 The remainder of the layer is dedicated to specific recipes
776 by function: <filename>recipes-bsp</filename>,
777 <filename>recipes-core</filename>,
778 <filename>recipes-graphics</filename>,
779 <filename>recipes-kernel</filename>, and so forth.
780 Metadata can exist for multiple formfactors, graphics
781 support systems, and so forth.
782 <note>
783 While the figure shows several
784 <filename>recipes-*</filename> directories, not all
785 these directories appear in all BSP layers.
786 </note>
787 </para>
788 </section>
789
790 <section id="software-layer">
791 <title>Software Layer</title>
792
793 <para>
794 The software layer provides the Metadata for additional
795 software packages used during the build.
796 This layer does not include Metadata that is specific to
797 the distribution or the machine, which are found in their
798 respective layers.
799 </para>
800
801 <para>
802 This layer contains any recipes, append files, and
803 patches, that your project needs.
804 </para>
805 </section>
806 </section>
807
808 <section id="sources-dev-environment">
809 <title>Sources</title>
810
811 <para>
812 In order for the OpenEmbedded build system to create an
813 image or any target, it must be able to access source files.
814 The
815 <link linkend='general-workflow-figure'>general workflow figure</link>
816 represents source files using the "Upstream Project Releases",
817 "Local Projects", and "SCMs (optional)" boxes.
818 The figure represents mirrors, which also play a role in
819 locating source files, with the "Source Materials" box.
820 </para>
821
822 <para>
823 The method by which source files are ultimately organized is
824 a function of the project.
825 For example, for released software, projects tend to use
826 tarballs or other archived files that can capture the
827 state of a release guaranteeing that it is statically
828 represented.
829 On the other hand, for a project that is more dynamic or
830 experimental in nature, a project might keep source files in a
831 repository controlled by a Source Control Manager (SCM) such as
832 Git.
833 Pulling source from a repository allows you to control
834 the point in the repository (the revision) from which you
835 want to build software.
836 Finally, a combination of the two might exist, which would
837 give the consumer a choice when deciding where to get
838 source files.
839 </para>
840
841 <para>
842 BitBake uses the
843 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
844 variable to point to source files regardless of their location.
845 Each recipe must have a <filename>SRC_URI</filename> variable
846 that points to the source.
847 </para>
848
849 <para>
850 Another area that plays a significant role in where source
851 files come from is pointed to by the
852 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
853 variable.
854 This area is a cache that can hold previously downloaded
855 source.
856 You can also instruct the OpenEmbedded build system to create
857 tarballs from Git repositories, which is not the default
858 behavior, and store them in the <filename>DL_DIR</filename>
859 by using the
860 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
861 variable.
862 </para>
863
864 <para>
865 Judicious use of a <filename>DL_DIR</filename> directory can
866 save the build system a trip across the Internet when looking
867 for files.
868 A good method for using a download directory is to have
869 <filename>DL_DIR</filename> point to an area outside of your
870 Build Directory.
871 Doing so allows you to safely delete the Build Directory
872 if needed without fear of removing any downloaded source file.
873 </para>
874
875 <para>
876 The remainder of this section provides a deeper look into the
877 source files and the mirrors.
878 Here is a more detailed look at the source file area of the
879 <link linkend='general-workflow-figure'>general workflow figure</link>:
880 </para>
881
882 <para>
883 <imagedata fileref="figures/source-input.png" width="6in" depth="6in" align="center" />
884 </para>
885
886 <section id='upstream-project-releases'>
887 <title>Upstream Project Releases</title>
888
889 <para>
890 Upstream project releases exist anywhere in the form of an
891 archived file (e.g. tarball or zip file).
892 These files correspond to individual recipes.
893 For example, the figure uses specific releases each for
894 BusyBox, Qt, and Dbus.
895 An archive file can be for any released product that can be
896 built using a recipe.
897 </para>
898 </section>
899
900 <section id='local-projects'>
901 <title>Local Projects</title>
902
903 <para>
904 Local projects are custom bits of software the user
905 provides.
906 These bits reside somewhere local to a project - perhaps
907 a directory into which the user checks in items (e.g.
908 a local directory containing a development source tree
909 used by the group).
910 </para>
911
912 <para>
913 The canonical method through which to include a local
914 project is to use the
915 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
916 class to include that local project.
917 You use either the <filename>local.conf</filename> or a
918 recipe's append file to override or set the
919 recipe to point to the local directory on your disk to pull
920 in the whole source tree.
921 </para>
922 </section>
923
924 <section id='scms'>
925 <title>Source Control Managers (Optional)</title>
926
927 <para>
928 Another place from which the build system can get source
929 files is with
930 <ulink url='&YOCTO_DOCS_BB_URL;#bb-fetchers'>fetchers</ulink>
931 employing various Source Control Managers (SCMs) such as
932 Git or Subversion.
933 In such cases, a repository is cloned or checked out.
934 The
935 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
936 task inside BitBake uses
937 the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
938 variable and the argument's prefix to determine the correct
939 fetcher module.
940 <note>
941 For information on how to have the OpenEmbedded build
942 system generate tarballs for Git repositories and place
943 them in the
944 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
945 directory, see the
946 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
947 variable in the Yocto Project Reference Manual.
948 </note>
949 </para>
950
951 <para>
952 When fetching a repository, BitBake uses the
953 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
954 variable to determine the specific revision from which to
955 build.
956 </para>
957 </section>
958
959 <section id='source-mirrors'>
960 <title>Source Mirror(s)</title>
961
962 <para>
963 Two kinds of mirrors exist: pre-mirrors and regular
964 mirrors.
965 The
966 <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink>
967 and
968 <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink>
969 variables point to these, respectively.
970 BitBake checks pre-mirrors before looking upstream for any
971 source files.
972 Pre-mirrors are appropriate when you have a shared
973 directory that is not a directory defined by the
974 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
975 variable.
976 A Pre-mirror typically points to a shared directory that is
977 local to your organization.
978 </para>
979
980 <para>
981 Regular mirrors can be any site across the Internet
982 that is used as an alternative location for source
983 code should the primary site not be functioning for
984 some reason or another.
985 </para>
986 </section>
987 </section>
988
989 <section id="package-feeds-dev-environment">
990 <title>Package Feeds</title>
991
992 <para>
993 When the OpenEmbedded build system generates an image or an
994 SDK, it gets the packages from a package feed area located
995 in the
996 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
997 The
998 <link linkend='general-workflow-figure'>general workflow figure</link>
999 shows this package feeds area in the upper-right corner.
1000 </para>
1001
1002 <para>
1003 This section looks a little closer into the package feeds
1004 area used by the build system.
1005 Here is a more detailed look at the area:
1006 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
1007 </para>
1008
1009 <para>
1010 Package feeds are an intermediary step in the build process.
1011 The OpenEmbedded build system provides classes to generate
1012 different package types, and you specify which classes to
1013 enable through the
1014 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
1015 variable.
1016 Before placing the packages into package feeds,
1017 the build process validates them with generated output quality
1018 assurance checks through the
1019 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink>
1020 class.
1021 </para>
1022
1023 <para>
1024 The package feed area resides in the Build Directory.
1025 The directory the build system uses to temporarily store
1026 packages is determined by a combination of variables and the
1027 particular package manager in use.
1028 See the "Package Feeds" box in the illustration and note the
1029 information to the right of that area.
1030 In particular, the following defines where package files are
1031 kept:
1032 <itemizedlist>
1033 <listitem><para>
1034 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1035 Defined as <filename>tmp/deploy</filename> in the Build
1036 Directory.
1037 </para></listitem>
1038 <listitem><para>
1039 <filename>DEPLOY_DIR_*</filename>:
1040 Depending on the package manager used, the package type
1041 sub-folder.
1042 Given RPM, IPK, or DEB packaging and tarball creation,
1043 the
1044 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>,
1045 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>,
1046 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>,
1047 or
1048 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>,
1049 variables are used, respectively.
1050 </para></listitem>
1051 <listitem><para>
1052 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1053 Defines architecture-specific sub-folders.
1054 For example, packages could exist for the i586 or
1055 qemux86 architectures.
1056 </para></listitem>
1057 </itemizedlist>
1058 </para>
1059
1060 <para>
1061 BitBake uses the
1062 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1063 tasks to generate packages and place them into the package
1064 holding area (e.g. <filename>do_package_write_ipk</filename>
1065 for IPK packages).
1066 See the
1067 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>",
1068 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>",
1069 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>",
1070 and
1071 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>"
1072 sections in the Yocto Project Reference Manual
1073 for additional information.
1074 As an example, consider a scenario where an IPK packaging
1075 manager is being used and package architecture support for
1076 both i586 and qemux86 exist.
1077 Packages for the i586 architecture are placed in
1078 <filename>build/tmp/deploy/ipk/i586</filename>, while packages
1079 for the qemux86 architecture are placed in
1080 <filename>build/tmp/deploy/ipk/qemux86</filename>.
1081 </para>
1082 </section>
1083
1084 <section id='bitbake-dev-environment'>
1085 <title>BitBake</title>
1086
1087 <para>
1088 The OpenEmbedded build system uses
1089 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
1090 to produce images and Software Development Kits (SDKs).
1091 You can see from the
1092 <link linkend='general-workflow-figure'>general workflow figure</link>,
1093 the BitBake area consists of several functional areas.
1094 This section takes a closer look at each of those areas.
1095 <note>
1096 Separate documentation exists for the BitBake tool.
1097 See the
1098 <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>
1099 for reference material on BitBake.
1100 </note>
1101 </para>
1102
1103 <section id='source-fetching-dev-environment'>
1104 <title>Source Fetching</title>
1105
1106 <para>
1107 The first stages of building a recipe are to fetch and
1108 unpack the source code:
1109 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
1110 </para>
1111
1112 <para>
1113 The
1114 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
1115 and
1116 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1117 tasks fetch the source files and unpack them into the
1118 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
1119 <note>
1120 For every local file (e.g. <filename>file://</filename>)
1121 that is part of a recipe's
1122 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1123 statement, the OpenEmbedded build system takes a
1124 checksum of the file for the recipe and inserts the
1125 checksum into the signature for the
1126 <filename>do_fetch</filename> task.
1127 If any local file has been modified, the
1128 <filename>do_fetch</filename> task and all tasks that
1129 depend on it are re-executed.
1130 </note>
1131 By default, everything is accomplished in the Build
1132 Directory, which has a defined structure.
1133 For additional general information on the Build Directory,
1134 see the
1135 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>"
1136 section in the Yocto Project Reference Manual.
1137 </para>
1138
1139 <para>
1140 Each recipe has an area in the Build Directory where the
1141 unpacked source code resides.
1142 The
1143 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1144 variable points to this area for a recipe's unpacked source
1145 code.
1146 The name of that directory for any given recipe is defined
1147 from several different variables.
1148 The preceding figure and the following list describe
1149 the Build Directory's hierarchy:
1150 <itemizedlist>
1151 <listitem><para>
1152 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>:
1153 The base directory where the OpenEmbedded build
1154 system performs all its work during the build.
1155 The default base directory is the
1156 <filename>tmp</filename> directory.
1157 </para></listitem>
1158 <listitem><para>
1159 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1160 The architecture of the built package or packages.
1161 Depending on the eventual destination of the
1162 package or packages (i.e. machine architecture,
1163 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>,
1164 SDK, or specific machine),
1165 <filename>PACKAGE_ARCH</filename> varies.
1166 See the variable's description for details.
1167 </para></listitem>
1168 <listitem><para>
1169 <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>:
1170 The operating system of the target device.
1171 A typical value would be "linux" (e.g.
1172 "qemux86-poky-linux").
1173 </para></listitem>
1174 <listitem><para>
1175 <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>:
1176 The name of the recipe used to build the package.
1177 This variable can have multiple meanings.
1178 However, when used in the context of input files,
1179 <filename>PN</filename> represents the the name
1180 of the recipe.
1181 </para></listitem>
1182 <listitem><para>
1183 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>:
1184 The location where the OpenEmbedded build system
1185 builds a recipe (i.e. does the work to create the
1186 package).
1187 <itemizedlist>
1188 <listitem><para>
1189 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1190 The version of the recipe used to build the
1191 package.
1192 </para></listitem>
1193 <listitem><para>
1194 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>:
1195 The revision of the recipe used to build the
1196 package.
1197 </para></listitem>
1198 </itemizedlist>
1199 </para></listitem>
1200 <listitem><para>
1201 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>:
1202 Contains the unpacked source files for a given
1203 recipe.
1204 <itemizedlist>
1205 <listitem><para>
1206 <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>:
1207 The name of the recipe used to build the
1208 package.
1209 The <filename>BPN</filename> variable is
1210 a version of the <filename>PN</filename>
1211 variable but with common prefixes and
1212 suffixes removed.
1213 </para></listitem>
1214 <listitem><para>
1215 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1216 The version of the recipe used to build the
1217 package.
1218 </para></listitem>
1219 </itemizedlist>
1220 </para></listitem>
1221 </itemizedlist>
1222 <note>
1223 In the previous figure, notice that two sample
1224 hierarchies exist: one based on package architecture (i.e.
1225 <filename>PACKAGE_ARCH</filename>) and one based on a
1226 machine (i.e. <filename>MACHINE</filename>).
1227 The underlying structures are identical.
1228 The differentiator being what the OpenEmbedded build
1229 system is using as a build target (e.g. general
1230 architecture, a build host, an SDK, or a specific
1231 machine).
1232 </note>
1233 </para>
1234 </section>
1235
1236 <section id='patching-dev-environment'>
1237 <title>Patching</title>
1238
1239 <para>
1240 Once source code is fetched and unpacked, BitBake locates
1241 patch files and applies them to the source files:
1242 <imagedata fileref="figures/patching.png" align="center" width="7in" depth="6in" />
1243 </para>
1244
1245 <para>
1246 The
1247 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1248 task uses a recipe's
1249 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1250 statements and the
1251 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink>
1252 variable to locate applicable patch files.
1253 </para>
1254
1255 <para>
1256 Default processing for patch files assumes the files have
1257 either <filename>*.patch</filename> or
1258 <filename>*.diff</filename> file types.
1259 You can use <filename>SRC_URI</filename> parameters to
1260 change the way the build system recognizes patch files.
1261 See the
1262 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1263 task for more information.
1264 </para>
1265
1266 <para>
1267 BitBake finds and applies multiple patches for a single
1268 recipe in the order in which it locates the patches.
1269 The <filename>FILESPATH</filename> variable defines the
1270 default set of directories that the build system uses to
1271 search for patch files.
1272 Once found, patches are applied to the recipe's source
1273 files, which are located in the
1274 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1275 directory.
1276 </para>
1277
1278 <para>
1279 For more information on how the source directories are
1280 created, see the
1281 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
1282 section.
1283 For more information on how to create patches and how the
1284 build system processes patches, see the
1285 "<ulink url='&YOCTO_DOCS_DEV_URL;#new-recipe-patching-code'>Patching Code</ulink>"
1286 section in the Yocto Project Development Tasks Manual.
1287 You can also see the
1288 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-modify-to-modify-the-source-of-an-existing-component'>Use <filename>devtool modify</filename> to Modify the Source of an Existing Component</ulink>"
1289 section in the Yocto Project Application Development and
1290 the Extensible Software Development Kit (SDK) manual and
1291 the
1292 "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</ulink>"
1293 section in the Yocto Project Linux Kernel Development
1294 Manual.
1295 </para>
1296 </section>
1297
1298 <section id='configuration-compilation-and-staging-dev-environment'>
1299 <title>Configuration, Compilation, and Staging</title>
1300
1301 <para>
1302 After source code is patched, BitBake executes tasks that
1303 configure and compile the source code.
1304 Once compilation occurs, the files are copied to a holding
1305 area (staged) in preparation for packaging:
1306 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
1307 </para>
1308
1309 <para>
1310 This step in the build process consists of the following
1311 tasks:
1312 <itemizedlist>
1313 <listitem><para>
1314 <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>:
1315 This task sets up the two sysroots in
1316 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>
1317 (i.e. <filename>recipe-sysroot</filename> and
1318 <filename>recipe-sysroot-native</filename>) so that
1319 during the packaging phase the sysroots can contain
1320 the contents of the
1321 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1322 tasks of the recipes on which the recipe
1323 containing the tasks depends.
1324 A sysroot exists for both the target and for the
1325 native binaries, which run on the host system.
1326 </para></listitem>
1327 <listitem><para>
1328 <emphasis><filename>do_configure</filename></emphasis>:
1329 This task configures the source by enabling and
1330 disabling any build-time and configuration options
1331 for the software being built.
1332 Configurations can come from the recipe itself as
1333 well as from an inherited class.
1334 Additionally, the software itself might configure
1335 itself depending on the target for which it is
1336 being built.</para>
1337
1338 <para>The configurations handled by the
1339 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink>
1340 task are specific to configurations for the source
1341 code being built by the recipe.</para>
1342
1343 <para>If you are using the
1344 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1345 class, you can add additional configuration options
1346 by using the
1347 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
1348 or
1349 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
1350 variables.
1351 For information on how this variable works within
1352 that class, see the
1353 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1354 class
1355 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass'>here</ulink>.
1356 </para></listitem>
1357 <listitem><para>
1358 <emphasis><filename>do_compile</filename></emphasis>:
1359 Once a configuration task has been satisfied,
1360 BitBake compiles the source using the
1361 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink>
1362 task.
1363 Compilation occurs in the directory pointed to by
1364 the
1365 <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink>
1366 variable.
1367 Realize that the <filename>B</filename> directory
1368 is, by default, the same as the
1369 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1370 directory.
1371 </para></listitem>
1372 <listitem><para>
1373 <emphasis><filename>do_install</filename></emphasis>:
1374 After compilation completes, BitBake executes the
1375 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1376 task.
1377 This task copies files from the
1378 <filename>B</filename> directory and places them
1379 in a holding area pointed to by the
1380 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1381 variable.
1382 Packaging occurs later using files from this
1383 holding directory.
1384 </para></listitem>
1385 </itemizedlist>
1386 </para>
1387 </section>
1388
1389 <section id='package-splitting-dev-environment'>
1390 <title>Package Splitting</title>
1391
1392 <para>
1393 After source code is configured, compiled, and staged, the
1394 build system analyzes the results and splits the output
1395 into packages:
1396 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
1397 </para>
1398
1399 <para>
1400 The
1401 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
1402 and
1403 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
1404 tasks combine to analyze the files found in the
1405 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1406 directory and split them into subsets based on available
1407 packages and files.
1408 Analysis involves the following as well as other items:
1409 splitting out debugging symbols, looking at shared library
1410 dependencies between packages, and looking at package
1411 relationships.
1412 </para>
1413
1414 <para>
1415 The <filename>do_packagedata</filename> task creates
1416 package metadata based on the analysis such that the
1417 build system can generate the final packages.
1418 The
1419 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1420 task stages (copies) a subset of the files installed by
1421 the
1422 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1423 task into the appropriate sysroot.
1424 Working, staged, and intermediate results of the analysis
1425 and package splitting process use several areas:
1426 <itemizedlist>
1427 <listitem><para>
1428 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>:
1429 The destination directory
1430 (i.e. <filename>package</filename>) for packages
1431 before they are split into individual packages.
1432 </para></listitem>
1433 <listitem><para>
1434 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>:
1435 A temporary work area (i.e.
1436 <filename>pkgdata</filename>) used by the
1437 <filename>do_package</filename> task to save
1438 package metadata.
1439 </para></listitem>
1440 <listitem><para>
1441 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>:
1442 The parent directory (i.e.
1443 <filename>packages-split</filename>) for packages
1444 after they have been split.
1445 </para></listitem>
1446 <listitem><para>
1447 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>:
1448 A shared, global-state directory that holds
1449 packaging metadata generated during the packaging
1450 process.
1451 The packaging process copies metadata from
1452 <filename>PKGDESTWORK</filename> to the
1453 <filename>PKGDATA_DIR</filename> area where it
1454 becomes globally available.
1455 </para></listitem>
1456 <listitem><para>
1457 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></ulink>:
1458 The path for the sysroot for the system on which
1459 a component is built to run (i.e.
1460 <filename>recipe-sysroot</filename>).
1461 </para></listitem>
1462 <listitem><para>
1463 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_NATIVE'><filename>STAGING_DIR_NATIVE</filename></ulink>:
1464 The path for the sysroot used when building
1465 components for the build host (i.e.
1466 <filename>recipe-sysroot-native</filename>).
1467 </para></listitem>
1468 <listitem><para>
1469 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_TARGET'><filename>STAGING_DIR_TARGET</filename></ulink>:
1470 The path for the sysroot used when a component that
1471 is built to execute on a system and it generates
1472 code for yet another machine (e.g. cross-canadian
1473 recipes).
1474 </para></listitem>
1475 </itemizedlist>
1476 The
1477 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>
1478 variable defines the files that go into each package in
1479 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>.
1480 If you want details on how this is accomplished, you can
1481 look at
1482 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass'><filename>package.bbclass</filename></ulink>.
1483 </para>
1484
1485 <para>
1486 Depending on the type of packages being created (RPM, DEB,
1487 or IPK), the
1488 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1489 task creates the actual packages and places them in the
1490 Package Feed area, which is
1491 <filename>${TMPDIR}/deploy</filename>.
1492 You can see the
1493 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
1494 section for more detail on that part of the build process.
1495 <note>
1496 Support for creating feeds directly from the
1497 <filename>deploy/*</filename> directories does not
1498 exist.
1499 Creating such feeds usually requires some kind of feed
1500 maintenance mechanism that would upload the new
1501 packages into an official package feed (e.g. the
1502 Ångström distribution).
1503 This functionality is highly distribution-specific
1504 and thus is not provided out of the box.
1505 </note>
1506 </para>
1507 </section>
1508
1509 <section id='image-generation-dev-environment'>
1510 <title>Image Generation</title>
1511
1512 <para>
1513 Once packages are split and stored in the Package Feeds
1514 area, the build system uses BitBake to generate the root
1515 filesystem image:
1516 <imagedata fileref="figures/image-generation.png" align="center" width="7.5in" depth="7.5in" />
1517 </para>
1518
1519 <para>
1520 The image generation process consists of several stages and
1521 depends on several tasks and variables.
1522 The
1523 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>
1524 task creates the root filesystem (file and directory
1525 structure) for an image.
1526 This task uses several key variables to help create the
1527 list of packages to actually install:
1528 <itemizedlist>
1529 <listitem><para>
1530 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>:
1531 Lists out the base set of packages from which to
1532 install from the Package Feeds area.
1533 </para></listitem>
1534 <listitem><para>
1535 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>:
1536 Specifies packages that should not be installed
1537 into the image.
1538 </para></listitem>
1539 <listitem><para>
1540 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>:
1541 Specifies features to include in the image.
1542 Most of these features map to additional packages
1543 for installation.
1544 </para></listitem>
1545 <listitem><para>
1546 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>:
1547 Specifies the package backend (e.g. RPM, DEB, or
1548 IPK) to use and consequently helps determine where
1549 to locate packages within the Package Feeds area.
1550 </para></listitem>
1551 <listitem><para>
1552 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>:
1553 Determines the language(s) for which additional
1554 language support packages are installed.
1555 </para></listitem>
1556 <listitem><para>
1557 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>:
1558 The final list of packages passed to the package
1559 manager for installation into the image.
1560 </para></listitem>
1561 </itemizedlist>
1562 </para>
1563
1564 <para>
1565 With
1566 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink>
1567 pointing to the location of the filesystem under
1568 construction and the <filename>PACKAGE_INSTALL</filename>
1569 variable providing the final list of packages to install,
1570 the root file system is created.
1571 </para>
1572
1573 <para>
1574 Package installation is under control of the package
1575 manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of
1576 whether or not package management is enabled for the
1577 target.
1578 At the end of the process, if package management is not
1579 enabled for the target, the package manager's data files
1580 are deleted from the root filesystem.
1581 As part of the final stage of package installation,
1582 post installation scripts that are part of the packages
1583 are run.
1584 Any scripts that fail to run on the build host are run on
1585 the target when the target system is first booted.
1586 If you are using a
1587 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
1588 all the post installation scripts must succeed on the
1589 build host during the package installation phase since the
1590 root filesystem on the target is read-only.
1591 </para>
1592
1593 <para>
1594 The final stages of the <filename>do_rootfs</filename> task
1595 handle post processing.
1596 Post processing includes creation of a manifest file and
1597 optimizations.
1598 </para>
1599
1600 <para>
1601 The manifest file (<filename>.manifest</filename>) resides
1602 in the same directory as the root filesystem image.
1603 This file lists out, line-by-line, the installed packages.
1604 The manifest file is useful for the
1605 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink>
1606 class, for example, to determine whether or not to run
1607 specific tests.
1608 See the
1609 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink>
1610 variable for additional information.
1611 </para>
1612
1613 <para>
1614 Optimizing processes that are run across the image include
1615 <filename>mklibs</filename>, <filename>prelink</filename>,
1616 and any other post-processing commands as defined by the
1617 <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink>
1618 variable.
1619 The <filename>mklibs</filename> process optimizes the size
1620 of the libraries, while the <filename>prelink</filename>
1621 process optimizes the dynamic linking of shared libraries
1622 to reduce start up time of executables.
1623 </para>
1624
1625 <para>
1626 After the root filesystem is built, processing begins on
1627 the image through the
1628 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink>
1629 task.
1630 The build system runs any pre-processing commands as
1631 defined by the
1632 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink>
1633 variable.
1634 This variable specifies a list of functions to call before
1635 the build system creates the final image output files.
1636 </para>
1637
1638 <para>
1639 The build system dynamically creates
1640 <filename>do_image_*</filename> tasks as needed, based
1641 on the image types specified in the
1642 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
1643 variable.
1644 The process turns everything into an image file or a set of
1645 image files and can compress the root filesystem image to
1646 reduce the overall size of the image.
1647 The formats used for the root filesystem depend on the
1648 <filename>IMAGE_FSTYPES</filename> variable.
1649 Compression depends on whether the formats support
1650 compression.
1651 </para>
1652
1653 <para>
1654 As an example, a dynamically created task when creating a
1655 particular image <replaceable>type</replaceable> would
1656 take the following form:
1657 <literallayout class='monospaced'>
1658 do_image_<replaceable>type</replaceable>
1659 </literallayout>
1660 So, if the <replaceable>type</replaceable> as specified by
1661 the <filename>IMAGE_FSTYPES</filename> were
1662 <filename>ext4</filename>, the dynamically generated task
1663 would be as follows:
1664 <literallayout class='monospaced'>
1665 do_image_ext4
1666 </literallayout>
1667 </para>
1668
1669 <para>
1670 The final task involved in image creation is the
1671 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink>
1672 task.
1673 This task completes the image by applying any image
1674 post processing as defined through the
1675 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink>
1676 variable.
1677 The variable specifies a list of functions to call once the
1678 build system has created the final image output files.
1679 <note>
1680 The entire image generation process is run under
1681 <link linkend='fakeroot-and-pseudo'>Pseudo</link>.
1682 Running under Pseudo ensures that the files in the
1683 root filesystem have correct ownership.
1684 </note>
1685 </para>
1686 </section>
1687
1688 <section id='sdk-generation-dev-environment'>
1689 <title>SDK Generation</title>
1690
1691 <para>
1692 The OpenEmbedded build system uses BitBake to generate the
1693 Software Development Kit (SDK) installer scripts for both
1694 the standard SDK and the extensible SDK (eSDK):
1695 </para>
1696
1697 <para>
1698 <imagedata fileref="figures/sdk-generation.png" width="9in" align="center" />
1699 <note>
1700 For more information on the cross-development toolchain
1701 generation, see the
1702 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1703 section.
1704 For information on advantages gained when building a
1705 cross-development toolchain using the
1706 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1707 task, see the
1708 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
1709 section in the Yocto Project Application Development
1710 and the Extensible Software Development Kit (eSDK)
1711 manual.
1712 </note>
1713 </para>
1714
1715 <para>
1716 Like image generation, the SDK script process consists of
1717 several stages and depends on many variables.
1718 The
1719 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1720 and
1721 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk_ext'><filename>do_populate_sdk_ext</filename></ulink>
1722 tasks use these key variables to help create the list of
1723 packages to actually install.
1724 For information on the variables listed in the figure,
1725 see the
1726 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1727 section.
1728 </para>
1729
1730 <para>
1731 The <filename>do_populate_sdk</filename> task helps create
1732 the standard SDK and handles two parts: a target part and a
1733 host part.
1734 The target part is the part built for the target hardware
1735 and includes libraries and headers.
1736 The host part is the part of the SDK that runs on the
1737 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>.
1738 </para>
1739
1740 <para>
1741 The <filename>do_populate_sdk_ext</filename> task helps
1742 create the extensible SDK and handles host and target parts
1743 differently than its counter part does for the standard SDK.
1744 For the extensible SDK, the task encapsulates the build
1745 system, which includes everything needed (host and target)
1746 for the SDK.
1747 </para>
1748
1749 <para>
1750 Regardless of the type of SDK being constructed, the
1751 tasks perform some cleanup after which a cross-development
1752 environment setup script and any needed configuration files
1753 are created.
1754 The final output is the Cross-development
1755 toolchain installation script (<filename>.sh</filename>
1756 file), which includes the environment setup script.
1757 </para>
1758 </section>
1759
1760 <section id='stamp-files-and-the-rerunning-of-tasks'>
1761 <title>Stamp Files and the Rerunning of Tasks</title>
1762
1763 <para>
1764 For each task that completes successfully, BitBake writes a
1765 stamp file into the
1766 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink>
1767 directory.
1768 The beginning of the stamp file's filename is determined
1769 by the
1770 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
1771 variable, and the end of the name consists of the task's
1772 name and current
1773 <link linkend='overview-checksums'>input checksum</link>.
1774 <note>
1775 This naming scheme assumes that
1776 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
1777 is "OEBasicHash", which is almost always the case in
1778 current OpenEmbedded.
1779 </note>
1780 To determine if a task needs to be rerun, BitBake checks
1781 if a stamp file with a matching input checksum exists
1782 for the task.
1783 If such a stamp file exists, the task's output is
1784 assumed to exist and still be valid.
1785 If the file does not exist, the task is rerun.
1786 <note>
1787 <para>The stamp mechanism is more general than the
1788 shared state (sstate) cache mechanism described in the
1789 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
1790 section.
1791 BitBake avoids rerunning any task that has a valid
1792 stamp file, not just tasks that can be accelerated
1793 through the sstate cache.</para>
1794
1795 <para>However, you should realize that stamp files only
1796 serve as a marker that some work has been done and that
1797 these files do not record task output.
1798 The actual task output would usually be somewhere in
1799 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
1800 (e.g. in some recipe's
1801 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.)
1802 What the sstate cache mechanism adds is a way to cache
1803 task output that can then be shared between build
1804 machines.</para>
1805 </note>
1806 Since <filename>STAMPS_DIR</filename> is usually a
1807 subdirectory of <filename>TMPDIR</filename>, removing
1808 <filename>TMPDIR</filename> will also remove
1809 <filename>STAMPS_DIR</filename>, which means tasks will
1810 properly be rerun to repopulate
1811 <filename>TMPDIR</filename>.
1812 </para>
1813
1814 <para>
1815 If you want some task to always be considered "out of
1816 date", you can mark it with the
1817 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
1818 varflag.
1819 If some other task depends on such a task, then that
1820 task will also always be considered out of date, which
1821 might not be what you want.
1822 </para>
1823
1824 <para>
1825 For details on how to view information about a task's
1826 signature, see the
1827 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
1828 section in the Yocto Project Development Tasks Manual.
1829 </para>
1830 </section>
1831
1832 <section id='setscene-tasks-and-shared-state'>
1833 <title>Setscene Tasks and Shared State</title>
1834
1835 <para>
1836 The description of tasks so far assumes that BitBake needs
1837 to build everything and no available prebuilt objects
1838 exist.
1839 BitBake does support skipping tasks if prebuilt objects are
1840 available.
1841 These objects are usually made available in the form of a
1842 shared state (sstate) cache.
1843 <note>
1844 For information on variables affecting sstate, see the
1845 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
1846 and
1847 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
1848 variables.
1849 </note>
1850 </para>
1851
1852 <para>
1853 The idea of a setscene task (i.e
1854 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
1855 is a version of the task where
1856 instead of building something, BitBake can skip to the end
1857 result and simply place a set of files into specific
1858 locations as needed.
1859 In some cases, it makes sense to have a setscene task
1860 variant (e.g. generating package files in the
1861 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1862 task).
1863 In other cases, it does not make sense (e.g. a
1864 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1865 task or a
1866 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1867 task) since the work involved would be equal to or greater
1868 than the underlying task.
1869 </para>
1870
1871 <para>
1872 In the build system, the common tasks that have setscene
1873 variants are
1874 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>,
1875 <filename>do_package_write_*</filename>,
1876 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>,
1877 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>,
1878 and
1879 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>.
1880 Notice that these tasks represent most of the tasks whose
1881 output is an end result.
1882 </para>
1883
1884 <para>
1885 The build system has knowledge of the relationship between
1886 these tasks and other preceding tasks.
1887 For example, if BitBake runs
1888 <filename>do_populate_sysroot_setscene</filename> for
1889 something, it does not make sense to run any of the
1890 <filename>do_fetch</filename>,
1891 <filename>do_unpack</filename>,
1892 <filename>do_patch</filename>,
1893 <filename>do_configure</filename>,
1894 <filename>do_compile</filename>, and
1895 <filename>do_install</filename> tasks.
1896 However, if <filename>do_package</filename> needs to be
1897 run, BitBake needs to run those other tasks.
1898 </para>
1899
1900 <para>
1901 It becomes more complicated if everything can come
1902 from an sstate cache because some objects are simply
1903 not required at all.
1904 For example, you do not need a compiler or native tools,
1905 such as quilt, if nothing exists to compile or patch.
1906 If the <filename>do_package_write_*</filename> packages
1907 are available from sstate, BitBake does not need the
1908 <filename>do_package</filename> task data.
1909 </para>
1910
1911 <para>
1912 To handle all these complexities, BitBake runs in two
1913 phases.
1914 The first is the "setscene" stage.
1915 During this stage, BitBake first checks the sstate cache
1916 for any targets it is planning to build.
1917 BitBake does a fast check to see if the object exists
1918 rather than a complete download.
1919 If nothing exists, the second phase, which is the setscene
1920 stage, completes and the main build proceeds.
1921 </para>
1922
1923 <para>
1924 If objects are found in the sstate cache, the build system
1925 works backwards from the end targets specified by the user.
1926 For example, if an image is being built, the build system
1927 first looks for the packages needed for that image and the
1928 tools needed to construct an image.
1929 If those are available, the compiler is not needed.
1930 Thus, the compiler is not even downloaded.
1931 If something was found to be unavailable, or the
1932 download or setscene task fails, the build system then
1933 tries to install dependencies, such as the compiler, from
1934 the cache.
1935 </para>
1936
1937 <para>
1938 The availability of objects in the sstate cache is
1939 handled by the function specified by the
1940 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
1941 variable and returns a list of available objects.
1942 The function specified by the
1943 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
1944 variable is the function that determines whether a given
1945 dependency needs to be followed, and whether for any given
1946 relationship the function needs to be passed.
1947 The function returns a True or False value.
1948 </para>
1949 </section>
1950 </section>
1951
1952 <section id='images-dev-environment'>
1953 <title>Images</title>
1954
1955 <para>
1956 The images produced by the build system are compressed forms
1957 of the root filesystem and are ready to boot on a target
1958 device.
1959 You can see from the
1960 <link linkend='general-workflow-figure'>general workflow figure</link>
1961 that BitBake output, in part, consists of images.
1962 This section takes a closer look at this output:
1963 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1964 </para>
1965
1966 <note>
1967 For a list of example images that the Yocto Project provides,
1968 see the
1969 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>"
1970 chapter in the Yocto Project Reference Manual.
1971 </note>
1972
1973 <para>
1974 The build process writes images out to the
1975 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1976 inside the
1977 <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
1978 folder as shown in the figure.
1979 This folder contains any files expected to be loaded on the
1980 target device.
1981 The
1982 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>
1983 variable points to the <filename>deploy</filename> directory,
1984 while the
1985 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink>
1986 variable points to the appropriate directory containing images
1987 for the current configuration.
1988 <itemizedlist>
1989 <listitem><para>
1990 <replaceable>kernel-image</replaceable>:
1991 A kernel binary file.
1992 The
1993 <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink>
1994 variable determines the naming scheme for the
1995 kernel image file.
1996 Depending on this variable, the file could begin with
1997 a variety of naming strings.
1998 The
1999 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2000 directory can contain multiple image files for the
2001 machine.
2002 </para></listitem>
2003 <listitem><para>
2004 <replaceable>root-filesystem-image</replaceable>:
2005 Root filesystems for the target device (e.g.
2006 <filename>*.ext3</filename> or
2007 <filename>*.bz2</filename> files).
2008 The
2009 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
2010 variable determines the root filesystem image type.
2011 The
2012 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2013 directory can contain multiple root filesystems for the
2014 machine.
2015 </para></listitem>
2016 <listitem><para>
2017 <replaceable>kernel-modules</replaceable>:
2018 Tarballs that contain all the modules built for the
2019 kernel.
2020 Kernel module tarballs exist for legacy purposes and
2021 can be suppressed by setting the
2022 <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink>
2023 variable to "0".
2024 The
2025 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2026 directory can contain multiple kernel module tarballs
2027 for the machine.
2028 </para></listitem>
2029 <listitem><para>
2030 <replaceable>bootloaders</replaceable>:
2031 If applicable to the target machine, bootloaders
2032 supporting the image.
2033 The <filename>deploy/images/</filename><replaceable>machine</replaceable>
2034 directory can contain multiple bootloaders for the
2035 machine.
2036 </para></listitem>
2037 <listitem><para>
2038 <replaceable>symlinks</replaceable>:
2039 The
2040 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2041 folder contains a symbolic link that points to the
2042 most recently built file for each machine.
2043 These links might be useful for external scripts that
2044 need to obtain the latest version of each file.
2045 </para></listitem>
2046 </itemizedlist>
2047 </para>
2048 </section>
2049
2050 <section id='sdk-dev-environment'>
2051 <title>Application Development SDK</title>
2052
2053 <para>
2054 In the
2055 <link linkend='general-workflow-figure'>general workflow figure</link>,
2056 the output labeled "Application Development SDK" represents an
2057 SDK.
2058 The SDK generation process differs depending on whether you
2059 build an extensible SDK (e.g.
2060 <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>)
2061 or a standard SDK (e.g.
2062 <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>).
2063 This section takes a closer look at this output:
2064 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
2065 </para>
2066
2067 <para>
2068 The specific form of this output is a set of files that
2069 includes a self-extracting SDK installer
2070 (<filename>*.sh</filename>), host and target manifest files,
2071 and files used for SDK testing.
2072 When the SDK installer file is run, it installs the SDK.
2073 The SDK consists of a cross-development toolchain, a set of
2074 libraries and headers, and an SDK environment setup script.
2075 Running this installer essentially sets up your
2076 cross-development environment.
2077 You can think of the cross-toolchain as the "host"
2078 part because it runs on the SDK machine.
2079 You can think of the libraries and headers as the "target"
2080 part because they are built for the target hardware.
2081 The environment setup script is added so that you can
2082 initialize the environment before using the tools.
2083 </para>
2084
2085 <note><title>Notes</title>
2086 <itemizedlist>
2087 <listitem><para>
2088 The Yocto Project supports several methods by which
2089 you can set up this cross-development environment.
2090 These methods include downloading pre-built SDK
2091 installers or building and installing your own SDK
2092 installer.
2093 </para></listitem>
2094 <listitem><para>
2095 For background information on cross-development
2096 toolchains in the Yocto Project development
2097 environment, see the
2098 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
2099 section.
2100 </para></listitem>
2101 <listitem><para>
2102 For information on setting up a cross-development
2103 environment, see the
2104 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2105 manual.
2106 </para></listitem>
2107 </itemizedlist>
2108 </note>
2109
2110 <para>
2111 All the output files for an SDK are written to the
2112 <filename>deploy/sdk</filename> folder inside the
2113 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
2114 as shown in the previous figure.
2115 Depending on the type of SDK, several variables exist that help
2116 configure these files.
2117 The following list shows the variables associated with an
2118 extensible SDK:
2119 <itemizedlist>
2120 <listitem><para>
2121 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2122 Points to the <filename>deploy</filename> directory.
2123 </para></listitem>
2124 <listitem><para>
2125 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>:
2126 Controls whether or not shared state artifacts are
2127 copied into the extensible SDK.
2128 By default, all required shared state artifacts are
2129 copied into the SDK.
2130 </para></listitem>
2131 <listitem><para>
2132 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>:
2133 Specifies whether or not packagedata is included in the
2134 extensible SDK for all recipes in the "world" target.
2135 </para></listitem>
2136 <listitem><para>
2137 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>:
2138 Specifies whether or not the toolchain is included
2139 when building the extensible SDK.
2140 </para></listitem>
2141 <listitem><para>
2142 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>:
2143 A list of variables allowed through from the build
2144 system configuration into the extensible SDK
2145 configuration.
2146 </para></listitem>
2147 <listitem><para>
2148 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>:
2149 A list of variables not allowed through from the build
2150 system configuration into the extensible SDK
2151 configuration.
2152 </para></listitem>
2153 <listitem><para>
2154 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>:
2155 A list of classes to remove from the
2156 <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink>
2157 value globally within the extensible SDK configuration.
2158 </para></listitem>
2159 </itemizedlist>
2160 This next list, shows the variables associated with a standard
2161 SDK:
2162 <itemizedlist>
2163 <listitem><para>
2164 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2165 Points to the <filename>deploy</filename> directory.
2166 </para></listitem>
2167 <listitem><para>
2168 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>:
2169 Specifies the architecture of the machine on which the
2170 cross-development tools are run to create packages for
2171 the target hardware.
2172 </para></listitem>
2173 <listitem><para>
2174 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>:
2175 Lists the features to include in the "target" part
2176 of the SDK.
2177 </para></listitem>
2178 <listitem><para>
2179 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>:
2180 Lists packages that make up the host part of the SDK
2181 (i.e. the part that runs on the
2182 <filename>SDKMACHINE</filename>).
2183 When you use
2184 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
2185 to create the SDK, a set of default packages apply.
2186 This variable allows you to add more packages.
2187 </para></listitem>
2188 <listitem><para>
2189 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>:
2190 Lists packages that make up the target part of the SDK
2191 (i.e. the part built for the target hardware).
2192 </para></listitem>
2193 <listitem><para>
2194 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>:
2195 Defines the default SDK installation path offered by
2196 the installation script.
2197 </para></listitem>
2198 <listitem><para>
2199 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_HOST_MANIFEST'><filename>SDK_HOST_MANIFEST</filename></ulink>:
2200 Lists all the installed packages that make up the host
2201 part of the SDK.
2202 This variable also plays a minor role for extensible
2203 SDK development as well.
2204 However, it is mainly used for the standard SDK.
2205 </para></listitem>
2206 <listitem><para>
2207 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_TARGET_MANIFEST'><filename>SDK_TARGET_MANIFEST</filename></ulink>:
2208 Lists all the installed packages that make up the
2209 target part of the SDK.
2210 This variable also plays a minor role for extensible
2211 SDK development as well.
2212 However, it is mainly used for the standard SDK.
2213 </para></listitem>
2214 </itemizedlist>
2215 </para>
2216 </section>
2217 </section>
2218
2219 <section id="cross-development-toolchain-generation">
2220 <title>Cross-Development Toolchain Generation</title>
2221
2222 <para>
2223 The Yocto Project does most of the work for you when it comes to
2224 creating
2225 <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>.
2226 This section provides some technical background on how
2227 cross-development toolchains are created and used.
2228 For more information on toolchains, you can also see the
2229 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2230 manual.
2231 </para>
2232
2233 <para>
2234 In the Yocto Project development environment, cross-development
2235 toolchains are used to build images and applications that run
2236 on the target hardware.
2237 With just a few commands, the OpenEmbedded build system creates
2238 these necessary toolchains for you.
2239 </para>
2240
2241 <para>
2242 The following figure shows a high-level build environment regarding
2243 toolchain construction and use.
2244 </para>
2245
2246 <para>
2247 <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" />
2248 </para>
2249
2250 <para>
2251 Most of the work occurs on the Build Host.
2252 This is the machine used to build images and generally work within
2253 the the Yocto Project environment.
2254 When you run
2255 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2256 to create an image, the OpenEmbedded build system
2257 uses the host <filename>gcc</filename> compiler to bootstrap a
2258 cross-compiler named <filename>gcc-cross</filename>.
2259 The <filename>gcc-cross</filename> compiler is what BitBake uses to
2260 compile source files when creating the target image.
2261 You can think of <filename>gcc-cross</filename> simply as an
2262 automatically generated cross-compiler that is used internally
2263 within BitBake only.
2264 <note>
2265 The extensible SDK does not use
2266 <filename>gcc-cross-canadian</filename> since this SDK
2267 ships a copy of the OpenEmbedded build system and the sysroot
2268 within it contains <filename>gcc-cross</filename>.
2269 </note>
2270 </para>
2271
2272 <para>
2273 The chain of events that occurs when <filename>gcc-cross</filename> is
2274 bootstrapped is as follows:
2275 <literallayout class='monospaced'>
2276 gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime
2277 </literallayout>
2278 <itemizedlist>
2279 <listitem><para>
2280 <filename>gcc</filename>:
2281 The build host's GNU Compiler Collection (GCC).
2282 </para></listitem>
2283 <listitem><para>
2284 <filename>binutils-cross</filename>:
2285 The bare minimum binary utilities needed in order to run
2286 the <filename>gcc-cross-initial</filename> phase of the
2287 bootstrap operation.
2288 </para></listitem>
2289 <listitem><para>
2290 <filename>gcc-cross-initial</filename>:
2291 An early stage of the bootstrap process for creating
2292 the cross-compiler.
2293 This stage builds enough of the <filename>gcc-cross</filename>,
2294 the C library, and other pieces needed to finish building the
2295 final cross-compiler in later stages.
2296 This tool is a "native" package (i.e. it is designed to run on
2297 the build host).
2298 </para></listitem>
2299 <listitem><para>
2300 <filename>linux-libc-headers</filename>:
2301 Headers needed for the cross-compiler.
2302 </para></listitem>
2303 <listitem><para>
2304 <filename>glibc-initial</filename>:
2305 An initial version of the Embedded GNU C Library
2306 (GLIBC) needed to bootstrap <filename>glibc</filename>.
2307 </para></listitem>
2308 <listitem><para>
2309 <filename>glibc</filename>:
2310 The GNU C Library.
2311 </para></listitem>
2312 <listitem><para>
2313 <filename>gcc-cross</filename>:
2314 The final stage of the bootstrap process for the
2315 cross-compiler.
2316 This stage results in the actual cross-compiler that
2317 BitBake uses when it builds an image for a targeted
2318 device.
2319 <note>
2320 If you are replacing this cross compiler toolchain
2321 with a custom version, you must replace
2322 <filename>gcc-cross</filename>.
2323 </note>
2324 This tool is also a "native" package (i.e. it is
2325 designed to run on the build host).
2326 </para></listitem>
2327 <listitem><para>
2328 <filename>gcc-runtime</filename>:
2329 Runtime libraries resulting from the toolchain bootstrapping
2330 process.
2331 This tool produces a binary that consists of the
2332 runtime libraries need for the targeted device.
2333 </para></listitem>
2334 </itemizedlist>
2335 </para>
2336
2337 <para>
2338 You can use the OpenEmbedded build system to build an installer for
2339 the relocatable SDK used to develop applications.
2340 When you run the installer, it installs the toolchain, which
2341 contains the development tools (e.g.,
2342 <filename>gcc-cross-canadian</filename>,
2343 <filename>binutils-cross-canadian</filename>, and other
2344 <filename>nativesdk-*</filename> tools),
2345 which are tools native to the SDK (i.e. native to
2346 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>),
2347 you need to cross-compile and test your software.
2348 The figure shows the commands you use to easily build out this
2349 toolchain.
2350 This cross-development toolchain is built to execute on the
2351 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2352 which might or might not be the same
2353 machine as the Build Host.
2354 <note>
2355 If your target architecture is supported by the Yocto Project,
2356 you can take advantage of pre-built images that ship with the
2357 Yocto Project and already contain cross-development toolchain
2358 installers.
2359 </note>
2360 </para>
2361
2362 <para>
2363 Here is the bootstrap process for the relocatable toolchain:
2364 <literallayout class='monospaced'>
2365 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers ->
2366 glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
2367 </literallayout>
2368 <itemizedlist>
2369 <listitem><para>
2370 <filename>gcc</filename>:
2371 The build host's GNU Compiler Collection (GCC).
2372 </para></listitem>
2373 <listitem><para>
2374 <filename>binutils-crosssdk</filename>:
2375 The bare minimum binary utilities needed in order to run
2376 the <filename>gcc-crosssdk-initial</filename> phase of the
2377 bootstrap operation.
2378 </para></listitem>
2379 <listitem><para>
2380 <filename>gcc-crosssdk-initial</filename>:
2381 An early stage of the bootstrap process for creating
2382 the cross-compiler.
2383 This stage builds enough of the
2384 <filename>gcc-crosssdk</filename> and supporting pieces so that
2385 the final stage of the bootstrap process can produce the
2386 finished cross-compiler.
2387 This tool is a "native" binary that runs on the build host.
2388 </para></listitem>
2389 <listitem><para>
2390 <filename>linux-libc-headers</filename>:
2391 Headers needed for the cross-compiler.
2392 </para></listitem>
2393 <listitem><para>
2394 <filename>glibc-initial</filename>:
2395 An initial version of the Embedded GLIBC needed to bootstrap
2396 <filename>nativesdk-glibc</filename>.
2397 </para></listitem>
2398 <listitem><para>
2399 <filename>nativesdk-glibc</filename>:
2400 The Embedded GLIBC needed to bootstrap the
2401 <filename>gcc-crosssdk</filename>.
2402 </para></listitem>
2403 <listitem><para>
2404 <filename>gcc-crosssdk</filename>:
2405 The final stage of the bootstrap process for the
2406 relocatable cross-compiler.
2407 The <filename>gcc-crosssdk</filename> is a transitory
2408 compiler and never leaves the build host.
2409 Its purpose is to help in the bootstrap process to create
2410 the eventual <filename>gcc-cross-canadian</filename>
2411 compiler, which is relocatable.
2412 This tool is also a "native" package (i.e. it is
2413 designed to run on the build host).
2414 </para></listitem>
2415 <listitem><para>
2416 <filename>gcc-cross-canadian</filename>:
2417 The final relocatable cross-compiler.
2418 When run on the
2419 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2420 this tool
2421 produces executable code that runs on the target device.
2422 Only one cross-canadian compiler is produced per architecture
2423 since they can be targeted at different processor optimizations
2424 using configurations passed to the compiler through the
2425 compile commands.
2426 This circumvents the need for multiple compilers and thus
2427 reduces the size of the toolchains.
2428 </para></listitem>
2429 </itemizedlist>
2430 </para>
2431
2432 <note>
2433 For information on advantages gained when building a
2434 cross-development toolchain installer, see the
2435 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
2436 appendix in the Yocto Project Application Development and the
2437 Extensible Software Development Kit (eSDK) manual.
2438 </note>
2439 </section>
2440
2441 <section id="shared-state-cache">
2442 <title>Shared State Cache</title>
2443
2444 <para>
2445 By design, the OpenEmbedded build system builds everything from
2446 scratch unless
2447 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2448 can determine that parts do not need to be rebuilt.
2449 Fundamentally, building from scratch is attractive as it means all
2450 parts are built fresh and no possibility of stale data exists that
2451 can cause problems.
2452 When developers hit problems, they typically default back to
2453 building from scratch so they have a know state from the
2454 start.
2455 </para>
2456
2457 <para>
2458 Building an image from scratch is both an advantage and a
2459 disadvantage to the process.
2460 As mentioned in the previous paragraph, building from scratch
2461 ensures that everything is current and starts from a known state.
2462 However, building from scratch also takes much longer as it
2463 generally means rebuilding things that do not necessarily need
2464 to be rebuilt.
2465 </para>
2466
2467 <para>
2468 The Yocto Project implements shared state code that supports
2469 incremental builds.
2470 The implementation of the shared state code answers the following
2471 questions that were fundamental roadblocks within the OpenEmbedded
2472 incremental build support system:
2473 <itemizedlist>
2474 <listitem><para>
2475 What pieces of the system have changed and what pieces have
2476 not changed?
2477 </para></listitem>
2478 <listitem><para>
2479 How are changed pieces of software removed and replaced?
2480 </para></listitem>
2481 <listitem><para>
2482 How are pre-built components that do not need to be rebuilt
2483 from scratch used when they are available?
2484 </para></listitem>
2485 </itemizedlist>
2486 </para>
2487
2488 <para>
2489 For the first question, the build system detects changes in the
2490 "inputs" to a given task by creating a checksum (or signature) of
2491 the task's inputs.
2492 If the checksum changes, the system assumes the inputs have changed
2493 and the task needs to be rerun.
2494 For the second question, the shared state (sstate) code tracks
2495 which tasks add which output to the build process.
2496 This means the output from a given task can be removed, upgraded
2497 or otherwise manipulated.
2498 The third question is partly addressed by the solution for the
2499 second question assuming the build system can fetch the sstate
2500 objects from remote locations and install them if they are deemed
2501 to be valid.
2502 <note><title>Notes</title>
2503 <itemizedlist>
2504 <listitem><para>
2505 The build system does not maintain
2506 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2507 information as part of the shared state packages.
2508 Consequently, considerations exist that affect
2509 maintaining shared state feeds.
2510 For information on how the build system works with
2511 packages and can track incrementing
2512 <filename>PR</filename> information, see the
2513 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
2514 section in the Yocto Project Development Tasks Manual.
2515 </para></listitem>
2516 <listitem><para>
2517 The code in the build system that supports incremental
2518 builds is not simple code.
2519 For techniques that help you work around issues related
2520 to shared state code, see the
2521 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'>Viewing Metadata Used to Create the Input Signature of a Shared State Task</ulink>"
2522 and
2523 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-invalidating-shared-state-to-force-a-task-to-run'>Invalidating Shared State to Force a Task to Run</ulink>"
2524 sections both in the Yocto Project Development Tasks
2525 Manual.
2526 </para></listitem>
2527 </itemizedlist>
2528 </note>
2529 </para>
2530
2531 <para>
2532 The rest of this section goes into detail about the overall
2533 incremental build architecture, the checksums (signatures), and
2534 shared state.
2535 </para>
2536
2537 <section id='concepts-overall-architecture'>
2538 <title>Overall Architecture</title>
2539
2540 <para>
2541 When determining what parts of the system need to be built,
2542 BitBake works on a per-task basis rather than a per-recipe
2543 basis.
2544 You might wonder why using a per-task basis is preferred over
2545 a per-recipe basis.
2546 To help explain, consider having the IPK packaging backend
2547 enabled and then switching to DEB.
2548 In this case, the
2549 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2550 and
2551 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2552 task outputs are still valid.
2553 However, with a per-recipe approach, the build would not
2554 include the <filename>.deb</filename> files.
2555 Consequently, you would have to invalidate the whole build and
2556 rerun it.
2557 Rerunning everything is not the best solution.
2558 Also, in this case, the core must be "taught" much about
2559 specific tasks.
2560 This methodology does not scale well and does not allow users
2561 to easily add new tasks in layers or as external recipes
2562 without touching the packaged-staging core.
2563 </para>
2564 </section>
2565
2566 <section id='overview-checksums'>
2567 <title>Checksums (Signatures)</title>
2568
2569 <para>
2570 The shared state code uses a checksum, which is a unique
2571 signature of a task's inputs, to determine if a task needs to
2572 be run again.
2573 Because it is a change in a task's inputs that triggers a
2574 rerun, the process needs to detect all the inputs to a given
2575 task.
2576 For shell tasks, this turns out to be fairly easy because
2577 the build process generates a "run" shell script for each task
2578 and it is possible to create a checksum that gives you a good
2579 idea of when the task's data changes.
2580 </para>
2581
2582 <para>
2583 To complicate the problem, there are things that should not be
2584 included in the checksum.
2585 First, there is the actual specific build path of a given
2586 task - the
2587 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2588 It does not matter if the work directory changes because it
2589 should not affect the output for target packages.
2590 Also, the build process has the objective of making native
2591 or cross packages relocatable.
2592 <note>
2593 Both native and cross packages run on the
2594 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>.
2595 However, cross packages generate output for the target
2596 architecture.
2597 </note>
2598 The checksum therefore needs to exclude
2599 <filename>WORKDIR</filename>.
2600 The simplistic approach for excluding the work directory is to
2601 set <filename>WORKDIR</filename> to some fixed value and
2602 create the checksum for the "run" script.
2603 </para>
2604
2605 <para>
2606 Another problem results from the "run" scripts containing
2607 functions that might or might not get called.
2608 The incremental build solution contains code that figures out
2609 dependencies between shell functions.
2610 This code is used to prune the "run" scripts down to the
2611 minimum set, thereby alleviating this problem and making the
2612 "run" scripts much more readable as a bonus.
2613 </para>
2614
2615 <para>
2616 So far, solutions for shell scripts exist.
2617 What about Python tasks?
2618 The same approach applies even though these tasks are more
2619 difficult.
2620 The process needs to figure out what variables a Python
2621 function accesses and what functions it calls.
2622 Again, the incremental build solution contains code that first
2623 figures out the variable and function dependencies, and then
2624 creates a checksum for the data used as the input to the task.
2625 </para>
2626
2627 <para>
2628 Like the <filename>WORKDIR</filename> case, situations exist
2629 where dependencies should be ignored.
2630 For these situations, you can instruct the build process to
2631 ignore a dependency by using a line like the following:
2632 <literallayout class='monospaced'>
2633 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
2634 </literallayout>
2635 This example ensures that the
2636 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
2637 variable does not depend on the value of
2638 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
2639 even if it does reference it.
2640 </para>
2641
2642 <para>
2643 Equally, there are cases where you need to add dependencies
2644 BitBake is not able to find.
2645 You can accomplish this by using a line like the following:
2646 <literallayout class='monospaced'>
2647 PACKAGE_ARCHS[vardeps] = "MACHINE"
2648 </literallayout>
2649 This example explicitly adds the <filename>MACHINE</filename>
2650 variable as a dependency for
2651 <filename>PACKAGE_ARCHS</filename>.
2652 </para>
2653
2654 <para>
2655 As an example, consider a case with in-line Python where
2656 BitBake is not able to figure out dependencies.
2657 When running in debug mode (i.e. using
2658 <filename>-DDD</filename>), BitBake produces output when it
2659 discovers something for which it cannot figure out dependencies.
2660 The Yocto Project team has currently not managed to cover
2661 those dependencies in detail and is aware of the need to fix
2662 this situation.
2663 </para>
2664
2665 <para>
2666 Thus far, this section has limited discussion to the direct
2667 inputs into a task.
2668 Information based on direct inputs is referred to as the
2669 "basehash" in the code.
2670 However, the question of a task's indirect inputs still
2671 exits - items already built and present in the
2672 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
2673 The checksum (or signature) for a particular task needs to add
2674 the hashes of all the tasks on which the particular task
2675 depends.
2676 Choosing which dependencies to add is a policy decision.
2677 However, the effect is to generate a master checksum that
2678 combines the basehash and the hashes of the task's
2679 dependencies.
2680 </para>
2681
2682 <para>
2683 At the code level, a variety of ways exist by which both the
2684 basehash and the dependent task hashes can be influenced.
2685 Within the BitBake configuration file, you can give BitBake
2686 some extra information to help it construct the basehash.
2687 The following statement effectively results in a list of
2688 global variable dependency excludes (i.e. variables never
2689 included in any checksum):
2690 <literallayout class='monospaced'>
2691 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
2692 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
2693 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
2694 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
2695 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
2696 </literallayout>
2697 The previous example excludes
2698 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
2699 since that variable is actually constructed as a path within
2700 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
2701 which is on the whitelist.
2702 </para>
2703
2704 <para>
2705 The rules for deciding which hashes of dependent tasks to
2706 include through dependency chains are more complex and are
2707 generally accomplished with a Python function.
2708 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
2709 two examples of this and also illustrates how you can insert
2710 your own policy into the system if so desired.
2711 This file defines the two basic signature generators
2712 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
2713 uses: "OEBasic" and "OEBasicHash".
2714 By default, a dummy "noop" signature handler is enabled
2715 in BitBake.
2716 This means that behavior is unchanged from previous versions.
2717 OE-Core uses the "OEBasicHash" signature handler by default
2718 through this setting in the <filename>bitbake.conf</filename>
2719 file:
2720 <literallayout class='monospaced'>
2721 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
2722 </literallayout>
2723 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
2724 is the same as the "OEBasic" version but adds the task hash to
2725 the
2726 <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp files</link>.
2727 This results in any metadata change that changes the task hash,
2728 automatically causing the task to be run again.
2729 This removes the need to bump
2730 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2731 values, and changes to metadata automatically ripple across
2732 the build.
2733 </para>
2734
2735 <para>
2736 It is also worth noting that the end result of these
2737 signature generators is to make some dependency and hash
2738 information available to the build.
2739 This information includes:
2740 <itemizedlist>
2741 <listitem><para>
2742 <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
2743 The base hashes for each task in the recipe.
2744 </para></listitem>
2745 <listitem><para>
2746 <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2747 The base hashes for each dependent task.
2748 </para></listitem>
2749 <listitem><para>
2750 <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2751 The task dependencies for each task.
2752 </para></listitem>
2753 <listitem><para>
2754 <filename>BB_TASKHASH</filename>:
2755 The hash of the currently running task.
2756 </para></listitem>
2757 </itemizedlist>
2758 </para>
2759 </section>
2760
2761 <section id='shared-state'>
2762 <title>Shared State</title>
2763
2764 <para>
2765 Checksums and dependencies, as discussed in the previous
2766 section, solve half the problem of supporting a shared state.
2767 The other half of the problem is being able to use checksum
2768 information during the build and being able to reuse or rebuild
2769 specific components.
2770 </para>
2771
2772 <para>
2773 The
2774 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2775 class is a relatively generic implementation of how to
2776 "capture" a snapshot of a given task.
2777 The idea is that the build process does not care about the
2778 source of a task's output.
2779 Output could be freshly built or it could be downloaded and
2780 unpacked from somewhere.
2781 In other words, the build process does not need to worry about
2782 its origin.
2783 </para>
2784
2785 <para>
2786 Two types of output exist.
2787 One type is just about creating a directory in
2788 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2789 A good example is the output of either
2790 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2791 or
2792 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
2793 The other type of output occurs when a set of data is merged
2794 into a shared directory tree such as the sysroot.
2795 </para>
2796
2797 <para>
2798 The Yocto Project team has tried to keep the details of the
2799 implementation hidden in <filename>sstate</filename> class.
2800 From a user's perspective, adding shared state wrapping to a
2801 task is as simple as this
2802 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2803 example taken from the
2804 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
2805 class:
2806 <literallayout class='monospaced'>
2807 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
2808 SSTATETASKS += "do_deploy"
2809 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
2810 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
2811
2812 python do_deploy_setscene () {
2813 sstate_setscene(d)
2814 }
2815 addtask do_deploy_setscene
2816 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
2817 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
2818 </literallayout>
2819 The following list explains the previous example:
2820 <itemizedlist>
2821 <listitem><para>
2822 Adding "do_deploy" to <filename>SSTATETASKS</filename>
2823 adds some required sstate-related processing, which is
2824 implemented in the
2825 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2826 class, to before and after the
2827 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2828 task.
2829 </para></listitem>
2830 <listitem><para>
2831 The
2832 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
2833 declares that <filename>do_deploy</filename> places its
2834 output in <filename>${DEPLOYDIR}</filename> when run
2835 normally (i.e. when not using the sstate cache).
2836 This output becomes the input to the shared state cache.
2837 </para></listitem>
2838 <listitem><para>
2839 The
2840 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
2841 line causes the contents of the shared state cache to be
2842 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
2843 <note>
2844 If <filename>do_deploy</filename> is not already in
2845 the shared state cache or if its input checksum
2846 (signature) has changed from when the output was
2847 cached, the task runs to populate the shared
2848 state cache, after which the contents of the shared
2849 state cache is copied to
2850 <filename>${DEPLOY_DIR_IMAGE}</filename>.
2851 If <filename>do_deploy</filename> is in the shared
2852 state cache and its signature indicates that the
2853 cached output is still valid (i.e. if no
2854 relevant task inputs have changed), then the
2855 contents of the shared state cache copies
2856 directly to
2857 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
2858 <filename>do_deploy_setscene</filename> task
2859 instead, skipping the
2860 <filename>do_deploy</filename> task.
2861 </note>
2862 </para></listitem>
2863 <listitem><para>
2864 The following task definition is glue logic needed to
2865 make the previous settings effective:
2866 <literallayout class='monospaced'>
2867 python do_deploy_setscene () {
2868 sstate_setscene(d)
2869 }
2870 addtask do_deploy_setscene
2871 </literallayout>
2872 <filename>sstate_setscene()</filename> takes the flags
2873 above as input and accelerates the
2874 <filename>do_deploy</filename> task through the
2875 shared state cache if possible.
2876 If the task was accelerated,
2877 <filename>sstate_setscene()</filename> returns True.
2878 Otherwise, it returns False, and the normal
2879 <filename>do_deploy</filename> task runs.
2880 For more information, see the
2881 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
2882 section in the BitBake User Manual.
2883 </para></listitem>
2884 <listitem><para>
2885 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
2886 line creates <filename>${DEPLOYDIR}</filename> and
2887 <filename>${B}</filename> before the
2888 <filename>do_deploy</filename> task runs, and also sets
2889 the current working directory of
2890 <filename>do_deploy</filename> to
2891 <filename>${B}</filename>.
2892 For more information, see the
2893 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
2894 section in the BitBake User Manual.
2895 <note>
2896 In cases where
2897 <filename>sstate-inputdirs</filename> and
2898 <filename>sstate-outputdirs</filename> would be the
2899 same, you can use
2900 <filename>sstate-plaindirs</filename>.
2901 For example, to preserve the
2902 <filename>${PKGD}</filename> and
2903 <filename>${PKGDEST}</filename> output from the
2904 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2905 task, use the following:
2906 <literallayout class='monospaced'>
2907 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
2908 </literallayout>
2909 </note>
2910 </para></listitem>
2911 <listitem><para>
2912 The <filename>do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"</filename>
2913 line appends extra metadata to the
2914 <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp file</link>.
2915 In this case, the metadata makes the task specific
2916 to a machine's architecture.
2917 See
2918 "<ulink url='&YOCTO_DOCS_BB_URL;#ref-bitbake-tasklist'>The Task List</ulink>"
2919 section in the BitBake User Manual for more
2920 information on the <filename>stamp-extra-info</filename>
2921 flag.
2922 </para></listitem>
2923 <listitem><para>
2924 <filename>sstate-inputdirs</filename> and
2925 <filename>sstate-outputdirs</filename> can also be used
2926 with multiple directories.
2927 For example, the following declares
2928 <filename>PKGDESTWORK</filename> and
2929 <filename>SHLIBWORK</filename> as shared state
2930 input directories, which populates the shared state
2931 cache, and <filename>PKGDATA_DIR</filename> and
2932 <filename>SHLIBSDIR</filename> as the corresponding
2933 shared state output directories:
2934 <literallayout class='monospaced'>
2935 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
2936 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
2937 </literallayout>
2938 </para></listitem>
2939 <listitem><para>
2940 These methods also include the ability to take a
2941 lockfile when manipulating shared state directory
2942 structures, for cases where file additions or removals
2943 are sensitive:
2944 <literallayout class='monospaced'>
2945 do_package[sstate-lockfile] = "${PACKAGELOCK}"
2946 </literallayout>
2947 </para></listitem>
2948 </itemizedlist>
2949 </para>
2950
2951 <para>
2952 Behind the scenes, the shared state code works by looking in
2953 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
2954 and
2955 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
2956 for shared state files.
2957 Here is an example:
2958 <literallayout class='monospaced'>
2959 SSTATE_MIRRORS ?= "\
2960 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
2961 file://.* file:///some/local/dir/sstate/PATH"
2962 </literallayout>
2963 <note>
2964 The shared state directory
2965 (<filename>SSTATE_DIR</filename>) is organized into
2966 two-character subdirectories, where the subdirectory
2967 names are based on the first two characters of the hash.
2968 If the shared state directory structure for a mirror has the
2969 same structure as <filename>SSTATE_DIR</filename>, you must
2970 specify "PATH" as part of the URI to enable the build system
2971 to map to the appropriate subdirectory.
2972 </note>
2973 </para>
2974
2975 <para>
2976 The shared state package validity can be detected just by
2977 looking at the filename since the filename contains the task
2978 checksum (or signature) as described earlier in this section.
2979 If a valid shared state package is found, the build process
2980 downloads it and uses it to accelerate the task.
2981 </para>
2982
2983 <para>
2984 The build processes use the <filename>*_setscene</filename>
2985 tasks for the task acceleration phase.
2986 BitBake goes through this phase before the main execution
2987 code and tries to accelerate any tasks for which it can find
2988 shared state packages.
2989 If a shared state package for a task is available, the
2990 shared state package is used.
2991 This means the task and any tasks on which it is dependent
2992 are not executed.
2993 </para>
2994
2995 <para>
2996 As a real world example, the aim is when building an IPK-based
2997 image, only the
2998 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
2999 tasks would have their shared state packages fetched and
3000 extracted.
3001 Since the sysroot is not used, it would never get extracted.
3002 This is another reason why a task-based approach is preferred
3003 over a recipe-based approach, which would have to install the
3004 output from every task.
3005 </para>
3006 </section>
3007 </section>
3008
3009 <section id='automatically-added-runtime-dependencies'>
3010 <title>Automatically Added Runtime Dependencies</title>
3011
3012 <para>
3013 The OpenEmbedded build system automatically adds common types of
3014 runtime dependencies between packages, which means that you do not
3015 need to explicitly declare the packages using
3016 <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>.
3017 Three automatic mechanisms exist (<filename>shlibdeps</filename>,
3018 <filename>pcdeps</filename>, and <filename>depchains</filename>)
3019 that handle shared libraries, package configuration (pkg-config)
3020 modules, and <filename>-dev</filename> and
3021 <filename>-dbg</filename> packages, respectively.
3022 For other types of runtime dependencies, you must manually declare
3023 the dependencies.
3024 <itemizedlist>
3025 <listitem><para>
3026 <filename>shlibdeps</filename>:
3027 During the
3028 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
3029 task of each recipe, all shared libraries installed by the
3030 recipe are located.
3031 For each shared library, the package that contains the
3032 shared library is registered as providing the shared
3033 library.
3034 More specifically, the package is registered as providing
3035 the
3036 <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>
3037 of the library.
3038 The resulting shared-library-to-package mapping
3039 is saved globally in
3040 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>
3041 by the
3042 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
3043 task.</para>
3044
3045 <para>Simultaneously, all executables and shared libraries
3046 installed by the recipe are inspected to see what shared
3047 libraries they link against.
3048 For each shared library dependency that is found,
3049 <filename>PKGDATA_DIR</filename> is queried to
3050 see if some package (likely from a different recipe)
3051 contains the shared library.
3052 If such a package is found, a runtime dependency is added
3053 from the package that depends on the shared library to the
3054 package that contains the library.</para>
3055
3056 <para>The automatically added runtime dependency also
3057 includes a version restriction.
3058 This version restriction specifies that at least the
3059 current version of the package that provides the shared
3060 library must be used, as if
3061 "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"
3062 had been added to <filename>RDEPENDS</filename>.
3063 This forces an upgrade of the package containing the shared
3064 library when installing the package that depends on the
3065 library, if needed.</para>
3066
3067 <para>If you want to avoid a package being registered as
3068 providing a particular shared library (e.g. because the library
3069 is for internal use only), then add the library to
3070 <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink>
3071 inside the package's recipe.
3072 </para></listitem>
3073 <listitem><para>
3074 <filename>pcdeps</filename>:
3075 During the <filename>do_package</filename> task of each
3076 recipe, all pkg-config modules
3077 (<filename>*.pc</filename> files) installed by the recipe
3078 are located.
3079 For each module, the package that contains the module is
3080 registered as providing the module.
3081 The resulting module-to-package mapping is saved globally in
3082 <filename>PKGDATA_DIR</filename> by the
3083 <filename>do_packagedata</filename> task.</para>
3084
3085 <para>Simultaneously, all pkg-config modules installed by
3086 the recipe are inspected to see what other pkg-config
3087 modules they depend on.
3088 A module is seen as depending on another module if it
3089 contains a "Requires:" line that specifies the other module.
3090 For each module dependency,
3091 <filename>PKGDATA_DIR</filename> is queried to see if some
3092 package contains the module.
3093 If such a package is found, a runtime dependency is added
3094 from the package that depends on the module to the package
3095 that contains the module.
3096 <note>
3097 The <filename>pcdeps</filename> mechanism most often
3098 infers dependencies between <filename>-dev</filename>
3099 packages.
3100 </note>
3101 </para></listitem>
3102 <listitem><para>
3103 <filename>depchains</filename>:
3104 If a package <filename>foo</filename> depends on a package
3105 <filename>bar</filename>, then <filename>foo-dev</filename>
3106 and <filename>foo-dbg</filename> are also made to depend on
3107 <filename>bar-dev</filename> and
3108 <filename>bar-dbg</filename>, respectively.
3109 Taking the <filename>-dev</filename> packages as an
3110 example, the <filename>bar-dev</filename> package might
3111 provide headers and shared library symlinks needed by
3112 <filename>foo-dev</filename>, which shows the need
3113 for a dependency between the packages.</para>
3114
3115 <para>The dependencies added by
3116 <filename>depchains</filename> are in the form of
3117 <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>.
3118 <note>
3119 By default, <filename>foo-dev</filename> also has an
3120 <filename>RDEPENDS</filename>-style dependency on
3121 <filename>foo</filename>, because the default value of
3122 <filename>RDEPENDS_${PN}-dev</filename> (set in
3123 <filename>bitbake.conf</filename>) includes
3124 "${PN}".
3125 </note></para>
3126
3127 <para>To ensure that the dependency chain is never broken,
3128 <filename>-dev</filename> and <filename>-dbg</filename>
3129 packages are always generated by default, even if the
3130 packages turn out to be empty.
3131 See the
3132 <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink>
3133 variable for more information.
3134 </para></listitem>
3135 </itemizedlist>
3136 </para>
3137
3138 <para>
3139 The <filename>do_package</filename> task depends on the
3140 <filename>do_packagedata</filename> task of each recipe in
3141 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>
3142 through use of a
3143 <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>
3144 declaration, which guarantees that the required
3145 shared-library/module-to-package mapping information will be available
3146 when needed as long as <filename>DEPENDS</filename> has been
3147 correctly set.
3148 </para>
3149 </section>
3150
3151 <section id='fakeroot-and-pseudo'>
3152 <title>Fakeroot and Pseudo</title>
3153
3154 <para>
3155 Some tasks are easier to implement when allowed to perform certain
3156 operations that are normally reserved for the root user (e.g.
3157 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>,
3158 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>,
3159 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>,
3160 and
3161 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>).
3162 For example, the <filename>do_install</filename> task benefits
3163 from being able to set the UID and GID of installed files to
3164 arbitrary values.
3165 </para>
3166
3167 <para>
3168 One approach to allowing tasks to perform root-only operations
3169 would be to require
3170 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
3171 to run as root.
3172 However, this method is cumbersome and has security issues.
3173 The approach that is actually used is to run tasks that benefit
3174 from root privileges in a "fake" root environment.
3175 Within this environment, the task and its child processes believe
3176 that they are running as the root user, and see an internally
3177 consistent view of the filesystem.
3178 As long as generating the final output (e.g. a package or an image)
3179 does not require root privileges, the fact that some earlier
3180 steps ran in a fake root environment does not cause problems.
3181 </para>
3182
3183 <para>
3184 The capability to run tasks in a fake root environment is known as
3185 "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>",
3186 which is derived from the BitBake keyword/variable
3187 flag that requests a fake root environment for a task.
3188 </para>
3189
3190 <para>
3191 In the
3192 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
3193 the program that implements fakeroot is known as
3194 <ulink url='https://www.yoctoproject.org/software-item/pseudo/'>Pseudo</ulink>.
3195 Pseudo overrides system calls by using the environment variable
3196 <filename>LD_PRELOAD</filename>, which results in the illusion
3197 of running as root.
3198 To keep track of "fake" file ownership and permissions resulting
3199 from operations that require root permissions, Pseudo uses
3200 an SQLite 3 database.
3201 This database is stored in
3202 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename>
3203 for individual recipes.
3204 Storing the database in a file as opposed to in memory
3205 gives persistence between tasks and builds, which is not
3206 accomplished using fakeroot.
3207 <note><title>Caution</title>
3208 If you add your own task that manipulates the same files or
3209 directories as a fakeroot task, then that task also needs to
3210 run under fakeroot.
3211 Otherwise, the task cannot run root-only operations, and
3212 cannot see the fake file ownership and permissions set by the
3213 other task.
3214 You need to also add a dependency on
3215 <filename>virtual/fakeroot-native:do_populate_sysroot</filename>,
3216 giving the following:
3217 <literallayout class='monospaced'>
3218 fakeroot do_mytask () {
3219 ...
3220 }
3221 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
3222 </literallayout>
3223 </note>
3224 For more information, see the
3225 <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>
3226 variables in the BitBake User Manual.
3227 You can also reference the
3228 "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>"
3229 article for background information on Fakeroot and Pseudo.
3230 </para>
3231 </section>
3232</chapter>
3233<!--
3234vim: expandtab tw=80 ts=4
3235-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-04-23 01:32:25 +0000