summaryrefslogtreecommitdiffstats
path: root/documentation/overview-manual/overview-manual-concepts.xml
diff options
context:
space:
mode:
authorScott Rifenbark <srifenbark@gmail.com>2018-04-12 10:52:45 -0700
committerRichard Purdie <richard.purdie@linuxfoundation.org>2018-05-24 17:16:12 +0100
commitc555881ac8e62e1bc1ddd9e0337e0c70a560b3b6 (patch)
tree80925d3cd31d9fe7e0b6692cd9ea39fb3e947146 /documentation/overview-manual/overview-manual-concepts.xml
parent4fd8b39b243f1b0d49971c233a5db2b09f1a2048 (diff)
downloadpoky-c555881ac8e62e1bc1ddd9e0337e0c70a560b3b6.tar.gz
poky.ent: Added YOCTO_DOCS_OM_URL entity
The variabe for the "getting-started" manual goes away and is replaced by this one for the new "overview-manual." (From yocto-docs rev: 45fc9beac6db4c40c3660fc9e54cc11e9c1f96c4) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/overview-manual/overview-manual-concepts.xml')
-rw-r--r--documentation/overview-manual/overview-manual-concepts.xml3612
1 files changed, 3612 insertions, 0 deletions
diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml
new file mode 100644
index 0000000000..123cf15d19
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-concepts.xml
@@ -0,0 +1,3612 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id=' overview-manual-concepts'>
6<title>Yocto Project Concepts</title>
7
8 <section id='yocto-project-components'>
9 <title>Yocto Project Components</title>
10
11 <para>
12 The
13 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
14 task executor together with various types of configuration files
15 form the OpenEmbedded-Core.
16 This section overviews these components by describing their use and
17 how they interact.
18 </para>
19
20 <para>
21 BitBake handles the parsing and execution of the data files.
22 The data itself is of various types:
23 <itemizedlist>
24 <listitem><para>
25 <emphasis>Recipes:</emphasis>
26 Provides details about particular pieces of software.
27 </para></listitem>
28 <listitem><para>
29 <emphasis>Class Data:</emphasis>
30 Abstracts common build information (e.g. how to build a
31 Linux kernel).
32 </para></listitem>
33 <listitem><para>
34 <emphasis>Configuration Data:</emphasis>
35 Defines machine-specific settings, policy decisions, and
36 so forth.
37 Configuration data acts as the glue to bind everything
38 together.
39 </para></listitem>
40 </itemizedlist>
41 </para>
42
43 <para>
44 BitBake knows how to combine multiple data sources together and
45 refers to each data source as a layer.
46 For information on layers, see the
47 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
48 section of the Yocto Project Development Tasks Manual.
49 </para>
50
51 <para>
52 Following are some brief details on these core components.
53 For additional information on how these components interact during
54 a build, see the
55 "<link linkend='development-concepts'>Development Concepts</link>"
56 section.
57 </para>
58
59 <section id='usingpoky-components-bitbake'>
60 <title>BitBake</title>
61
62 <para>
63 BitBake is the tool at the heart of the
64 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
65 and is responsible for parsing the
66 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>,
67 generating a list of tasks from it, and then executing those
68 tasks.
69 </para>
70
71 <para>
72 This section briefly introduces BitBake.
73 If you want more information on BitBake, see the
74 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>.
75 </para>
76
77 <para>
78 To see a list of the options BitBake supports, use either of
79 the following commands:
80 <literallayout class='monospaced'>
81 $ bitbake -h
82 $ bitbake --help
83 </literallayout>
84 </para>
85
86 <para>
87 The most common usage for BitBake is
88 <filename>bitbake <replaceable>packagename</replaceable></filename>,
89 where <filename>packagename</filename> is the name of the
90 package you want to build (referred to as the "target").
91 The target often equates to the first part of a recipe's
92 filename (e.g. "foo" for a recipe named
93 <filename>foo_1.3.0-r0.bb</filename>).
94 So, to process the
95 <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you
96 might type the following:
97 <literallayout class='monospaced'>
98 $ bitbake matchbox-desktop
99 </literallayout>
100 Several different versions of
101 <filename>matchbox-desktop</filename> might exist.
102 BitBake chooses the one selected by the distribution
103 configuration.
104 You can get more details about how BitBake chooses between
105 different target versions and providers in the
106 "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>"
107 section of the BitBake User Manual.
108 </para>
109
110 <para>
111 BitBake also tries to execute any dependent tasks first.
112 So for example, before building
113 <filename>matchbox-desktop</filename>, BitBake would build a
114 cross compiler and <filename>glibc</filename> if they had not
115 already been built.
116 </para>
117
118 <para>
119 A useful BitBake option to consider is the
120 <filename>-k</filename> or <filename>--continue</filename>
121 option.
122 This option instructs BitBake to try and continue processing
123 the job as long as possible even after encountering an error.
124 When an error occurs, the target that failed and those that
125 depend on it cannot be remade.
126 However, when you use this option other dependencies can
127 still be processed.
128 </para>
129 </section>
130
131 <section id='concepts-components-metadata'>
132 <title>Metadata (Recipes)</title>
133
134 <para>
135 Files that have the <filename>.bb</filename> suffix are
136 "recipes" files.
137 In general, a recipe contains information about a single piece
138 of software.
139 This information includes the location from which to download
140 the unaltered source, any source patches to be applied to that
141 source (if needed), which special configuration options to
142 apply, how to compile the source files, and how to package the
143 compiled output.
144 </para>
145
146 <para>
147 The term "package" is sometimes used to refer to recipes.
148 However, since the word "package" is used for the packaged
149 output from the OpenEmbedded build system (i.e.
150 <filename>.ipk</filename> or <filename>.deb</filename> files),
151 this document avoids using the term "package" when referring
152 to recipes.
153 </para>
154 </section>
155
156 <section id='concepts-components-classes'>
157 <title>Classes</title>
158
159 <para>
160 Class files (<filename>.bbclass</filename>) contain information
161 that is useful to share between Metadata files.
162 An example is the
163 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
164 class, which contains common settings for any application that
165 Autotools uses.
166 The
167 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
168 chapter in the Yocto Project Reference Manual provides
169 details about classes and how to use them.
170 </para>
171 </section>
172
173 <section id='concepts-components-configuration'>
174 <title>Configuration</title>
175
176 <para>
177 The configuration files (<filename>.conf</filename>) define
178 various configuration variables that govern the OpenEmbedded
179 build process.
180 These files fall into several areas that define machine
181 configuration options, distribution configuration options,
182 compiler tuning options, general common configuration options,
183 and user configuration options in
184 <filename>local.conf</filename>, which is found in the
185 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
186 </para>
187 </section>
188 </section>
189
190 <section id='cm-layers'>
191 <title>Layers</title>
192
193 <para>
194 Layers are repositories that contain related sets of instructions
195 that tell the OpenEmbedded build system what to do.
196 You use different layers to logically separate information in your
197 build.
198 You can collaborate, share, and reuse layers.
199 The Layer Model simultaneously supports collaboration and
200 customization.
201 </para>
202
203 <para>
204 For more introductory information on the Yocto Project's layer
205 model, see the
206 "<ulink url='&YOCTO_DOCS_GS_URL;#the-yocto-project-layer-model'>The Yocto Project Layer Model</ulink>"
207 section in the Yocto Project Overview and Concepts Manual.
208 For procedures on how to create layers, see the
209 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
210 section in the Yocto Project Development Tasks Manual.
211 </para>
212 </section>
213
214 <section id="development-concepts">
215 <title>Development Concepts</title>
216
217 <para>
218 This section takes a more detailed look inside the build
219 process used by the OpenEmbedded build system.
220 The following diagram represents the build at a high level.
221 The remainder of this section expands on the fundamental input,
222 output, process, and
223 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
224 blocks that make up the build process.
225 </para>
226
227 <para id='general-yocto-environment-figure'>
228 <imagedata fileref="figures/yocto-environment-ref.png" align="center" width="8in" depth="4.25in" />
229 </para>
230
231 <para>
232 In general, the build process consists of several functional areas:
233 <itemizedlist>
234 <listitem><para>
235 <emphasis>User Configuration:</emphasis>
236 Metadata you can use to control the build process.
237 </para></listitem>
238 <listitem><para>
239 <emphasis>Metadata Layers:</emphasis>
240 Various layers that provide software, machine, and
241 distro Metadata.
242 </para></listitem>
243 <listitem><para>
244 <emphasis>Source Files:</emphasis>
245 Upstream releases, local projects, and SCMs.
246 </para></listitem>
247 <listitem><para>
248 <emphasis>Build System:</emphasis>
249 Processes under the control of
250 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>.
251 This block expands on how BitBake fetches source, applies
252 patches, completes compilation, analyzes output for package
253 generation, creates and tests packages, generates images,
254 and generates cross-development tools.
255 </para></listitem>
256 <listitem><para>
257 <emphasis>Package Feeds:</emphasis>
258 Directories containing output packages (RPM, DEB or IPK),
259 which are subsequently used in the construction of an
260 image or SDK, produced by the build system.
261 These feeds can also be copied and shared using a web
262 server or other means to facilitate extending or updating
263 existing images on devices at runtime if runtime package
264 management is enabled.
265 </para></listitem>
266 <listitem><para>
267 <emphasis>Images:</emphasis>
268 Images produced by the development process.
269 </para></listitem>
270 <listitem><para>
271 <emphasis>Application Development SDK:</emphasis>
272 Cross-development tools that are produced along with
273 an image or separately with BitBake.
274 </para></listitem>
275 </itemizedlist>
276 </para>
277
278 <section id="user-configuration">
279 <title>User Configuration</title>
280
281 <para>
282 User configuration helps define the build.
283 Through user configuration, you can tell BitBake the
284 target architecture for which you are building the image,
285 where to store downloaded source, and other build properties.
286 </para>
287
288 <para>
289 The following figure shows an expanded representation of the
290 "User Configuration" box of the
291 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>:
292 </para>
293
294 <para>
295 <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" />
296 </para>
297
298 <para>
299 BitBake needs some basic configuration files in order to
300 complete a build.
301 These files are <filename>*.conf</filename> files.
302 The minimally necessary ones reside as example files in the
303 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
304 For simplicity, this section refers to the Source Directory as
305 the "Poky Directory."
306 </para>
307
308 <para>
309 When you clone the <filename>poky</filename> Git repository
310 or you download and unpack a Yocto Project release, you
311 can set up the Source Directory to be named anything you want.
312 For this discussion, the cloned repository uses the default
313 name <filename>poky</filename>.
314 <note>
315 The
316 <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>
317 repository is primarily an aggregation of existing
318 repositories.
319 It is not a canonical upstream source.
320 </note>
321 </para>
322
323 <para>
324 The <filename>meta-poky</filename> layer inside Poky contains
325 a <filename>conf</filename> directory that has example
326 configuration files.
327 These example files are used as a basis for creating actual
328 configuration files when you source the build environment
329 script
330 (i.e.
331 <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>).
332 </para>
333
334 <para>
335 Sourcing the build environment script creates a
336 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
337 if one does not already exist.
338 BitBake uses the Build Directory for all its work during
339 builds.
340 The Build Directory has a <filename>conf</filename> directory
341 that contains default versions of your
342 <filename>local.conf</filename> and
343 <filename>bblayers.conf</filename> configuration files.
344 These default configuration files are created only if versions
345 do not already exist in the Build Directory at the time you
346 source the build environment setup script.
347 </para>
348
349 <para>
350 Because the Poky repository is fundamentally an aggregation of
351 existing repositories, some users might be familiar with
352 running the <filename>&OE_INIT_FILE;</filename> script
353 in the context of separate
354 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>
355 and BitBake repositories rather than a single Poky repository.
356 This discussion assumes the script is executed from
357 within a cloned or unpacked version of Poky.
358 </para>
359
360 <para>
361 Depending on where the script is sourced, different
362 sub-scripts are called to set up the Build Directory
363 (Yocto or OpenEmbedded).
364 Specifically, the script
365 <filename>scripts/oe-setup-builddir</filename> inside the
366 poky directory sets up the Build Directory and seeds the
367 directory (if necessary) with configuration files appropriate
368 for the Yocto Project development environment.
369 <note>
370 The <filename>scripts/oe-setup-builddir</filename> script
371 uses the <filename>$TEMPLATECONF</filename> variable to
372 determine which sample configuration files to locate.
373 </note>
374 </para>
375
376 <para>
377 The <filename>local.conf</filename> file provides many
378 basic variables that define a build environment.
379 Here is a list of a few.
380 To see the default configurations in a
381 <filename>local.conf</filename> file created by the build
382 environment script, see the
383 <filename>local.conf.sample</filename> in the
384 <filename>meta-poky</filename> layer:
385 <itemizedlist>
386 <listitem><para>
387 <emphasis>Parallelism Options:</emphasis>
388 Controlled by the
389 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></ulink>,
390 <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink>,
391 and
392 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></ulink>
393 variables.
394 </para></listitem>
395 <listitem><para>
396 <emphasis>Target Machine Selection:</emphasis>
397 Controlled by the
398 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>
399 variable.
400 </para></listitem>
401 <listitem><para>
402 <emphasis>Download Directory:</emphasis>
403 Controlled by the
404 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
405 variable.
406 </para></listitem>
407 <listitem><para>
408 <emphasis>Shared State Directory:</emphasis>
409 Controlled by the
410 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
411 variable.
412 </para></listitem>
413 <listitem><para>
414 <emphasis>Build Output:</emphasis>
415 Controlled by the
416 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
417 variable.
418 </para></listitem>
419 </itemizedlist>
420 <note>
421 Configurations set in the
422 <filename>conf/local.conf</filename> file can also be set
423 in the <filename>conf/site.conf</filename> and
424 <filename>conf/auto.conf</filename> configuration files.
425 </note>
426 </para>
427
428 <para>
429 The <filename>bblayers.conf</filename> file tells BitBake what
430 layers you want considered during the build.
431 By default, the layers listed in this file include layers
432 minimally needed by the build system.
433 However, you must manually add any custom layers you have
434 created.
435 You can find more information on working with the
436 <filename>bblayers.conf</filename> file in the
437 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
438 section in the Yocto Project Development Tasks Manual.
439 </para>
440
441 <para>
442 The files <filename>site.conf</filename> and
443 <filename>auto.conf</filename> are not created by the
444 environment initialization script.
445 If you want the <filename>site.conf</filename> file, you
446 need to create that yourself.
447 The <filename>auto.conf</filename> file is typically created by
448 an autobuilder:
449 <itemizedlist>
450 <listitem><para>
451 <emphasis><filename>site.conf</filename>:</emphasis>
452 You can use the <filename>conf/site.conf</filename>
453 configuration file to configure multiple
454 build directories.
455 For example, suppose you had several build environments
456 and they shared some common features.
457 You can set these default build properties here.
458 A good example is perhaps the packaging format to use
459 through the
460 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
461 variable.</para>
462
463 <para>One useful scenario for using the
464 <filename>conf/site.conf</filename> file is to extend
465 your
466 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
467 variable to include the path to a
468 <filename>conf/site.conf</filename>.
469 Then, when BitBake looks for Metadata using
470 <filename>BBPATH</filename>, it finds the
471 <filename>conf/site.conf</filename> file and applies
472 your common configurations found in the file.
473 To override configurations in a particular build
474 directory, alter the similar configurations within
475 that build directory's
476 <filename>conf/local.conf</filename> file.
477 </para></listitem>
478 <listitem><para>
479 <emphasis><filename>auto.conf</filename>:</emphasis>
480 The file is usually created and written to by
481 an autobuilder.
482 The settings put into the file are typically the
483 same as you would find in the
484 <filename>conf/local.conf</filename> or the
485 <filename>conf/site.conf</filename> files.
486 </para></listitem>
487 </itemizedlist>
488 </para>
489
490 <para>
491 You can edit all configuration files to further define
492 any particular build environment.
493 This process is represented by the "User Configuration Edits"
494 box in the figure.
495 </para>
496
497 <para>
498 When you launch your build with the
499 <filename>bitbake <replaceable>target</replaceable></filename>
500 command, BitBake sorts out the configurations to ultimately
501 define your build environment.
502 It is important to understand that the
503 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
504 reads the configuration files in a specific order:
505 <filename>site.conf</filename>, <filename>auto.conf</filename>,
506 and <filename>local.conf</filename>.
507 And, the build system applies the normal assignment statement
508 rules.
509 Because the files are parsed in a specific order, variable
510 assignments for the same variable could be affected.
511 For example, if the <filename>auto.conf</filename> file and
512 the <filename>local.conf</filename> set
513 <replaceable>variable1</replaceable> to different values,
514 because the build system parses <filename>local.conf</filename>
515 after <filename>auto.conf</filename>,
516 <replaceable>variable1</replaceable> is assigned the value from
517 the <filename>local.conf</filename> file.
518 </para>
519 </section>
520
521 <section id="metadata-machine-configuration-and-policy-configuration">
522 <title>Metadata, Machine Configuration, and Policy Configuration</title>
523
524 <para>
525 The previous section described the user configurations that
526 define BitBake's global behavior.
527 This section takes a closer look at the layers the build system
528 uses to further control the build.
529 These layers provide Metadata for the software, machine, and
530 policy.
531 </para>
532
533 <para>
534 In general, three types of layer input exist:
535 <itemizedlist>
536 <listitem><para>
537 <emphasis>Policy Configuration:</emphasis>
538 Distribution Layers provide top-level or general
539 policies for the image or SDK being built.
540 For example, this layer would dictate whether BitBake
541 produces RPM or IPK packages.
542 </para></listitem>
543 <listitem><para>
544 <emphasis>Machine Configuration:</emphasis>
545 Board Support Package (BSP) layers provide machine
546 configurations.
547 This type of information is specific to a particular
548 target architecture.
549 </para></listitem>
550 <listitem><para>
551 <emphasis>Metadata:</emphasis>
552 Software layers contain user-supplied recipe files,
553 patches, and append files.
554 </para></listitem>
555 </itemizedlist>
556 </para>
557
558 <para>
559 The following figure shows an expanded representation of the
560 Metadata, Machine Configuration, and Policy Configuration input
561 (layers) boxes of the
562 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>:
563 </para>
564
565 <para>
566 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="7.5in" />
567 </para>
568
569 <para>
570 In general, all layers have a similar structure.
571 They all contain a licensing file
572 (e.g. <filename>COPYING</filename>) if the layer is to be
573 distributed, a <filename>README</filename> file as good
574 practice and especially if the layer is to be distributed, a
575 configuration directory, and recipe directories.
576 </para>
577
578 <para>
579 The Yocto Project has many layers that can be used.
580 You can see a web-interface listing of them on the
581 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
582 page.
583 The layers appear at the bottom categorized under
584 "Yocto Metadata Layers."
585 These layers are fundamentally a subset of the
586 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Metadata Index</ulink>,
587 which lists all layers provided by the OpenEmbedded community.
588 <note>
589 Layers exist in the Yocto Project Source Repositories that
590 cannot be found in the OpenEmbedded Metadata Index.
591 These layers are either deprecated or experimental
592 in nature.
593 </note>
594 </para>
595
596 <para>
597 BitBake uses the <filename>conf/bblayers.conf</filename> file,
598 which is part of the user configuration, to find what layers it
599 should be using as part of the build.
600 </para>
601
602 <para>
603 For more information on layers, see the
604 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
605 section in the Yocto Project Development Tasks Manual.
606 </para>
607
608 <section id="distro-layer">
609 <title>Distro Layer</title>
610
611 <para>
612 The distribution layer provides policy configurations for
613 your distribution.
614 Best practices dictate that you isolate these types of
615 configurations into their own layer.
616 Settings you provide in
617 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
618 similar settings that BitBake finds in your
619 <filename>conf/local.conf</filename> file in the Build
620 Directory.
621 </para>
622
623 <para>
624 The following list provides some explanation and references
625 for what you typically find in the distribution layer:
626 <itemizedlist>
627 <listitem><para>
628 <emphasis>classes:</emphasis>
629 Class files (<filename>.bbclass</filename>) hold
630 common functionality that can be shared among
631 recipes in the distribution.
632 When your recipes inherit a class, they take on the
633 settings and functions for that class.
634 You can read more about class files in the
635 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
636 chapter of the Yocto Reference Manual.
637 </para></listitem>
638 <listitem><para>
639 <emphasis>conf:</emphasis>
640 This area holds configuration files for the
641 layer (<filename>conf/layer.conf</filename>),
642 the distribution
643 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
644 and any distribution-wide include files.
645 </para></listitem>
646 <listitem><para>
647 <emphasis>recipes-*:</emphasis>
648 Recipes and append files that affect common
649 functionality across the distribution.
650 This area could include recipes and append files
651 to add distribution-specific configuration,
652 initialization scripts, custom image recipes,
653 and so forth.
654 </para></listitem>
655 </itemizedlist>
656 </para>
657 </section>
658
659 <section id="bsp-layer">
660 <title>BSP Layer</title>
661
662 <para>
663 The BSP Layer provides machine configurations.
664 Everything in this layer is specific to the machine for
665 which you are building the image or the SDK.
666 A common structure or form is defined for BSP layers.
667 You can learn more about this structure in the
668 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
669 <note>
670 In order for a BSP layer to be considered compliant
671 with the Yocto Project, it must meet some structural
672 requirements.
673 </note>
674 </para>
675
676 <para>
677 The BSP Layer's configuration directory contains
678 configuration files for the machine
679 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>)
680 and, of course, the layer
681 (<filename>conf/layer.conf</filename>).
682 </para>
683
684 <para>
685 The remainder of the layer is dedicated to specific recipes
686 by function: <filename>recipes-bsp</filename>,
687 <filename>recipes-core</filename>,
688 <filename>recipes-graphics</filename>, and
689 <filename>recipes-kernel</filename>.
690 Metadata can exist for multiple formfactors, graphics
691 support systems, and so forth.
692 <note>
693 While the figure shows several
694 <filename>recipes-*</filename> directories, not all
695 these directories appear in all BSP layers.
696 </note>
697 </para>
698 </section>
699
700 <section id="software-layer">
701 <title>Software Layer</title>
702
703 <para>
704 The software layer provides the Metadata for additional
705 software packages used during the build.
706 This layer does not include Metadata that is specific to
707 the distribution or the machine, which are found in their
708 respective layers.
709 </para>
710
711 <para>
712 This layer contains any new recipes that your project
713 needs in the form of recipe files.
714 </para>
715 </section>
716 </section>
717
718 <section id="sources-dev-environment">
719 <title>Sources</title>
720
721 <para>
722 In order for the OpenEmbedded build system to create an
723 image or any target, it must be able to access source files.
724 The
725 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>
726 represents source files using the "Upstream Project Releases",
727 "Local Projects", and "SCMs (optional)" boxes.
728 The figure represents mirrors, which also play a role in
729 locating source files, with the "Source Mirror(s)" box.
730 </para>
731
732 <para>
733 The method by which source files are ultimately organized is
734 a function of the project.
735 For example, for released software, projects tend to use
736 tarballs or other archived files that can capture the
737 state of a release guaranteeing that it is statically
738 represented.
739 On the other hand, for a project that is more dynamic or
740 experimental in nature, a project might keep source files in a
741 repository controlled by a Source Control Manager (SCM) such as
742 Git.
743 Pulling source from a repository allows you to control
744 the point in the repository (the revision) from which you
745 want to build software.
746 Finally, a combination of the two might exist, which would
747 give the consumer a choice when deciding where to get
748 source files.
749 </para>
750
751 <para>
752 BitBake uses the
753 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
754 variable to point to source files regardless of their location.
755 Each recipe must have a <filename>SRC_URI</filename> variable
756 that points to the source.
757 </para>
758
759 <para>
760 Another area that plays a significant role in where source
761 files come from is pointed to by the
762 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
763 variable.
764 This area is a cache that can hold previously downloaded
765 source.
766 You can also instruct the OpenEmbedded build system to create
767 tarballs from Git repositories, which is not the default
768 behavior, and store them in the <filename>DL_DIR</filename>
769 by using the
770 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
771 variable.
772 </para>
773
774 <para>
775 Judicious use of a <filename>DL_DIR</filename> directory can
776 save the build system a trip across the Internet when looking
777 for files.
778 A good method for using a download directory is to have
779 <filename>DL_DIR</filename> point to an area outside of your
780 Build Directory.
781 Doing so allows you to safely delete the Build Directory
782 if needed without fear of removing any downloaded source file.
783 </para>
784
785 <para>
786 The remainder of this section provides a deeper look into the
787 source files and the mirrors.
788 Here is a more detailed look at the source file area of the
789 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>:
790 <imagedata fileref="figures/source-input.png" align="center" width="7in" depth="7.5in" />
791 </para>
792
793 <section id='upstream-project-releases'>
794 <title>Upstream Project Releases</title>
795
796 <para>
797 Upstream project releases exist anywhere in the form of an
798 archived file (e.g. tarball or zip file).
799 These files correspond to individual recipes.
800 For example, the figure uses specific releases each for
801 BusyBox, Qt, and Dbus.
802 An archive file can be for any released product that can be
803 built using a recipe.
804 </para>
805 </section>
806
807 <section id='local-projects'>
808 <title>Local Projects</title>
809
810 <para>
811 Local projects are custom bits of software the user
812 provides.
813 These bits reside somewhere local to a project - perhaps
814 a directory into which the user checks in items (e.g.
815 a local directory containing a development source tree
816 used by the group).
817 </para>
818
819 <para>
820 The canonical method through which to include a local
821 project is to use the
822 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
823 class to include that local project.
824 You use either the <filename>local.conf</filename> or a
825 recipe's append file to override or set the
826 recipe to point to the local directory on your disk to pull
827 in the whole source tree.
828 </para>
829
830 <para>
831 For information on how to use the
832 <filename>externalsrc</filename> class, see the
833 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></ulink>"
834 section.
835 </para>
836 </section>
837
838 <section id='scms'>
839 <title>Source Control Managers (Optional)</title>
840
841 <para>
842 Another place the build system can get source files from is
843 through an SCM such as Git or Subversion.
844 In this case, a repository is cloned or checked out.
845 The
846 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
847 task inside BitBake uses
848 the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
849 variable and the argument's prefix to determine the correct
850 fetcher module.
851 <note>
852 For information on how to have the OpenEmbedded build
853 system generate tarballs for Git repositories and place
854 them in the
855 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
856 directory, see the
857 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
858 variable.
859 </note>
860 </para>
861
862 <para>
863 When fetching a repository, BitBake uses the
864 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
865 variable to determine the specific revision from which to
866 build.
867 </para>
868 </section>
869
870 <section id='source-mirrors'>
871 <title>Source Mirror(s)</title>
872
873 <para>
874 Two kinds of mirrors exist: pre-mirrors and regular
875 mirrors.
876 The
877 <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink>
878 and
879 <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink>
880 variables point to these, respectively.
881 BitBake checks pre-mirrors before looking upstream for any
882 source files.
883 Pre-mirrors are appropriate when you have a shared
884 directory that is not a directory defined by the
885 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
886 variable.
887 A Pre-mirror typically points to a shared directory that is
888 local to your organization.
889 </para>
890
891 <para>
892 Regular mirrors can be any site across the Internet
893 that is used as an alternative location for source
894 code should the primary site not be functioning for
895 some reason or another.
896 </para>
897 </section>
898 </section>
899
900 <section id="package-feeds-dev-environment">
901 <title>Package Feeds</title>
902
903 <para>
904 When the OpenEmbedded build system generates an image or an
905 SDK, it gets the packages from a package feed area located
906 in the
907 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
908 The
909 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>
910 shows this package feeds area in the upper-right corner.
911 </para>
912
913 <para>
914 This section looks a little closer into the package feeds
915 area used by the build system.
916 Here is a more detailed look at the area:
917 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
918 </para>
919
920 <para>
921 Package feeds are an intermediary step in the build process.
922 The OpenEmbedded build system provides classes to generate
923 different package types, and you specify which classes to
924 enable through the
925 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
926 variable.
927 Before placing the packages into package feeds,
928 the build process validates them with generated output quality
929 assurance checks through the
930 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink>
931 class.
932 </para>
933
934 <para>
935 The package feed area resides in the Build Directory.
936 The directory the build system uses to temporarily store
937 packages is determined by a combination of variables and the
938 particular package manager in use.
939 See the "Package Feeds" box in the illustration and note the
940 information to the right of that area.
941 In particular, the following defines where package files are
942 kept:
943 <itemizedlist>
944 <listitem><para>
945 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
946 Defined as <filename>tmp/deploy</filename> in the Build
947 Directory.
948 </para></listitem>
949 <listitem><para>
950 <filename>DEPLOY_DIR_*</filename>:
951 Depending on the package manager used, the package type
952 sub-folder.
953 Given RPM, IPK, or DEB packaging and tarball creation,
954 the
955 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>,
956 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>,
957 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>,
958 or
959 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>,
960 variables are used, respectively.
961 </para></listitem>
962 <listitem><para>
963 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
964 Defines architecture-specific sub-folders.
965 For example, packages could exist for the i586 or
966 qemux86 architectures.
967 </para></listitem>
968 </itemizedlist>
969 </para>
970
971 <para>
972 BitBake uses the
973 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
974 tasks to generate packages and place them into the package
975 holding area (e.g. <filename>do_package_write_ipk</filename>
976 for IPK packages).
977 See the
978 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>",
979 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>",
980 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>",
981 and
982 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>"
983 sections in the Yocto Project Reference Manual
984 for additional information.
985 As an example, consider a scenario where an IPK packaging
986 manager is being used and package architecture support for
987 both i586 and qemux86 exist.
988 Packages for the i586 architecture are placed in
989 <filename>build/tmp/deploy/ipk/i586</filename>, while packages
990 for the qemux86 architecture are placed in
991 <filename>build/tmp/deploy/ipk/qemux86</filename>.
992 </para>
993 </section>
994
995 <section id='bitbake-dev-environment'>
996 <title>BitBake</title>
997
998 <para>
999 The OpenEmbedded build system uses
1000 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
1001 to produce images.
1002 You can see from the
1003 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>,
1004 the BitBake area consists of several functional areas.
1005 This section takes a closer look at each of those areas.
1006 </para>
1007
1008 <para>
1009 Separate documentation exists for the BitBake tool.
1010 See the
1011 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>
1012 for reference material on BitBake.
1013 </para>
1014
1015 <section id='source-fetching-dev-environment'>
1016 <title>Source Fetching</title>
1017
1018 <para>
1019 The first stages of building a recipe are to fetch and
1020 unpack the source code:
1021 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
1022 </para>
1023
1024 <para>
1025 The
1026 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
1027 and
1028 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1029 tasks fetch the source files and unpack them into the work
1030 directory.
1031 <note>
1032 For every local file (e.g. <filename>file://</filename>)
1033 that is part of a recipe's
1034 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1035 statement, the OpenEmbedded build system takes a
1036 checksum of the file for the recipe and inserts the
1037 checksum into the signature for the
1038 <filename>do_fetch</filename>.
1039 If any local file has been modified, the
1040 <filename>do_fetch</filename> task and all tasks that
1041 depend on it are re-executed.
1042 </note>
1043 By default, everything is accomplished in the
1044 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>,
1045 which has a defined structure.
1046 For additional general information on the Build Directory,
1047 see the
1048 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>"
1049 section in the Yocto Project Reference Manual.
1050 </para>
1051
1052 <para>
1053 Unpacked source files are pointed to by the
1054 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1055 variable.
1056 Each recipe has an area in the Build Directory where the
1057 unpacked source code resides.
1058 The name of that directory for any given recipe is defined
1059 from several different variables.
1060 You can see the variables that define these directories
1061 by looking at the figure:
1062 <itemizedlist>
1063 <listitem><para>
1064 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>:
1065 The base directory where the OpenEmbedded build
1066 system performs all its work during the build.
1067 </para></listitem>
1068 <listitem><para>
1069 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1070 The architecture of the built package or packages.
1071 </para></listitem>
1072 <listitem><para>
1073 <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>:
1074 The operating system of the target device.
1075 </para></listitem>
1076 <listitem><para>
1077 <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>:
1078 The name of the built package.
1079 </para></listitem>
1080 <listitem><para>
1081 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1082 The version of the recipe used to build the
1083 package.
1084 </para></listitem>
1085 <listitem><para>
1086 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>:
1087 The revision of the recipe used to build the
1088 package.
1089 </para></listitem>
1090 <listitem><para>
1091 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>:
1092 The location within <filename>TMPDIR</filename>
1093 where a specific package is built.
1094 </para></listitem>
1095 <listitem><para>
1096 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>:
1097 Contains the unpacked source files for a given
1098 recipe.
1099 </para></listitem>
1100 </itemizedlist>
1101 </para>
1102 </section>
1103
1104 <section id='patching-dev-environment'>
1105 <title>Patching</title>
1106
1107 <para>
1108 Once source code is fetched and unpacked, BitBake locates
1109 patch files and applies them to the source files:
1110 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
1111 </para>
1112
1113 <para>
1114 The
1115 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1116 task processes recipes by using the
1117 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1118 variable to locate applicable patch files, which by default
1119 are <filename>*.patch</filename> or
1120 <filename>*.diff</filename> files, or any file if
1121 "apply=yes" is specified for the file in
1122 <filename>SRC_URI</filename>.
1123 </para>
1124
1125 <para>
1126 BitBake finds and applies multiple patches for a single
1127 recipe in the order in which it finds the patches.
1128 Patches are applied to the recipe's source files located
1129 in the
1130 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1131 directory.
1132 </para>
1133
1134 <para>
1135 For more information on how the source directories are
1136 created, see the
1137 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
1138 section.
1139 </para>
1140 </section>
1141
1142 <section id='configuration-and-compilation-dev-environment'>
1143 <title>Configuration and Compilation</title>
1144
1145 <para>
1146 After source code is patched, BitBake executes tasks that
1147 configure and compile the source code:
1148 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
1149 </para>
1150
1151 <para>
1152 This step in the build process consists of three tasks:
1153 <itemizedlist>
1154 <listitem><para>
1155 <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>:
1156 This task sets up the two sysroots in
1157 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>
1158 (i.e. <filename>recipe-sysroot</filename> and
1159 <filename>recipe-sysroot-native</filename>) so that
1160 the sysroots contain the contents of the
1161 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1162 tasks of the recipes on which the recipe
1163 containing the tasks depends.
1164 A sysroot exists for both the target and for the
1165 native binaries, which run on the host system.
1166 </para></listitem>
1167 <listitem><para>
1168 <emphasis><filename>do_configure</filename></emphasis>:
1169 This task configures the source by enabling and
1170 disabling any build-time and configuration options
1171 for the software being built.
1172 Configurations can come from the recipe itself as
1173 well as from an inherited class.
1174 Additionally, the software itself might configure
1175 itself depending on the target for which it is
1176 being built.</para>
1177
1178 <para>The configurations handled by the
1179 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink>
1180 task are specific to source code configuration for
1181 the source code being built by the recipe.</para>
1182
1183 <para>If you are using the
1184 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1185 class, you can add additional configuration options
1186 by using the
1187 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
1188 or
1189 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
1190 variables.
1191 For information on how this variable works within
1192 that class, see the
1193 <filename>meta/classes/autotools.bbclass</filename>
1194 file.
1195 </para></listitem>
1196 <listitem><para>
1197 <emphasis><filename>do_compile</filename></emphasis>:
1198 Once a configuration task has been satisfied, BitBake
1199 compiles the source using the
1200 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink>
1201 task.
1202 Compilation occurs in the directory pointed to by
1203 the
1204 <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink>
1205 variable.
1206 Realize that the <filename>B</filename> directory
1207 is, by default, the same as the
1208 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1209 directory.
1210 </para></listitem>
1211 <listitem><para>
1212 <emphasis><filename>do_install</filename></emphasis>:
1213 Once compilation is done, BitBake executes the
1214 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1215 task.
1216 This task copies files from the
1217 <filename>B</filename> directory and places them
1218 in a holding area pointed to by the
1219 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1220 variable.
1221 </para></listitem>
1222 </itemizedlist>
1223 </para>
1224 </section>
1225
1226 <section id='package-splitting-dev-environment'>
1227 <title>Package Splitting</title>
1228
1229 <para>
1230 After source code is configured and compiled, the
1231 OpenEmbedded build system analyzes
1232 the results and splits the output into packages:
1233 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
1234 </para>
1235
1236 <para>
1237 The
1238 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
1239 and
1240 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
1241 tasks combine to analyze the files found in the
1242 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1243 directory and split them into subsets based on available
1244 packages and files.
1245 The analyzing process involves the following as well as
1246 other items: splitting out debugging symbols, looking at
1247 shared library dependencies between packages, and looking
1248 at package relationships.
1249 The <filename>do_packagedata</filename> task creates
1250 package metadata based on the analysis such that the
1251 OpenEmbedded build system can generate the final packages.
1252 Working, staged, and intermediate results of the analysis
1253 and package splitting process use these areas:
1254 <itemizedlist>
1255 <listitem><para>
1256 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>:
1257 The destination directory for packages before they
1258 are split.
1259 </para></listitem>
1260 <listitem><para>
1261 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>:
1262 A shared, global-state directory that holds data
1263 generated during the packaging process.
1264 </para></listitem>
1265 <listitem><para>
1266 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>:
1267 A temporary work area used by the
1268 <filename>do_package</filename> task.
1269 </para></listitem>
1270 <listitem><para>
1271 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>:
1272 The parent directory for packages after they have
1273 been split.
1274 </para></listitem>
1275 </itemizedlist>
1276 The
1277 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>
1278 variable defines the files that go into each package in
1279 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>.
1280 If you want details on how this is accomplished, you can
1281 look at the
1282 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-package'><filename>package</filename></ulink>
1283 class.
1284 </para>
1285
1286 <para>
1287 Depending on the type of packages being created (RPM, DEB,
1288 or IPK), the
1289 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1290 task creates the actual packages and places them in the
1291 Package Feed area, which is
1292 <filename>${TMPDIR}/deploy</filename>.
1293 You can see the
1294 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
1295 section for more detail on that part of the build process.
1296 <note>
1297 Support for creating feeds directly from the
1298 <filename>deploy/*</filename> directories does not
1299 exist.
1300 Creating such feeds usually requires some kind of feed
1301 maintenance mechanism that would upload the new
1302 packages into an official package feed (e.g. the
1303 Ångström distribution).
1304 This functionality is highly distribution-specific
1305 and thus is not provided out of the box.
1306 </note>
1307 </para>
1308 </section>
1309
1310 <section id='image-generation-dev-environment'>
1311 <title>Image Generation</title>
1312
1313 <para>
1314 Once packages are split and stored in the Package Feeds
1315 area, the OpenEmbedded build system uses BitBake to
1316 generate the root filesystem image:
1317 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
1318 </para>
1319
1320 <para>
1321 The image generation process consists of several stages and
1322 depends on several tasks and variables.
1323 The
1324 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>
1325 task creates the root filesystem (file and directory
1326 structure) for an image.
1327 This task uses several key variables to help create the
1328 list of packages to actually install:
1329 <itemizedlist>
1330 <listitem><para>
1331 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>:
1332 Lists out the base set of packages to install from
1333 the Package Feeds area.
1334 </para></listitem>
1335 <listitem><para>
1336 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>:
1337 Specifies packages that should not be installed.
1338 </para></listitem>
1339 <listitem><para>
1340 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>:
1341 Specifies features to include in the image.
1342 Most of these features map to additional packages
1343 for installation.
1344 </para></listitem>
1345 <listitem><para>
1346 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>:
1347 Specifies the package backend to use and
1348 consequently helps determine where to locate
1349 packages within the Package Feeds area.
1350 </para></listitem>
1351 <listitem><para>
1352 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>:
1353 Determines the language(s) for which additional
1354 language support packages are installed.
1355 </para></listitem>
1356 <listitem><para>
1357 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>:
1358 The final list of packages passed to the package manager
1359 for installation into the image.
1360 </para></listitem>
1361 </itemizedlist>
1362 </para>
1363
1364 <para>
1365 With
1366 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink>
1367 pointing to the location of the filesystem under
1368 construction and the <filename>PACKAGE_INSTALL</filename>
1369 variable providing the final list of packages to install,
1370 the root file system is created.
1371 </para>
1372
1373 <para>
1374 Package installation is under control of the package
1375 manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of
1376 whether or not package management is enabled for the
1377 target.
1378 At the end of the process, if package management is not
1379 enabled for the target, the package manager's data files
1380 are deleted from the root filesystem.
1381 As part of the final stage of package installation,
1382 postinstall scripts that are part of the packages are run.
1383 Any scripts that fail to run
1384 on the build host are run on the target when the target
1385 system is first booted.
1386 If you are using a
1387 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
1388 all the post installation scripts must succeed during the
1389 package installation phase since the root filesystem is
1390 read-only.
1391 </para>
1392
1393 <para>
1394 The final stages of the <filename>do_rootfs</filename> task
1395 handle post processing.
1396 Post processing includes creation of a manifest file and
1397 optimizations.
1398 </para>
1399
1400 <para>
1401 The manifest file (<filename>.manifest</filename>) resides
1402 in the same directory as the root filesystem image.
1403 This file lists out, line-by-line, the installed packages.
1404 The manifest file is useful for the
1405 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink>
1406 class, for example, to determine whether or not to run
1407 specific tests.
1408 See the
1409 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink>
1410 variable for additional information.
1411 </para>
1412
1413 <para>
1414 Optimizing processes run across the image include
1415 <filename>mklibs</filename>, <filename>prelink</filename>,
1416 and any other post-processing commands as defined by the
1417 <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink>
1418 variable.
1419 The <filename>mklibs</filename> process optimizes the size
1420 of the libraries, while the <filename>prelink</filename>
1421 process optimizes the dynamic linking of shared libraries
1422 to reduce start up time of executables.
1423 </para>
1424
1425 <para>
1426 After the root filesystem is built, processing begins on
1427 the image through the
1428 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink>
1429 task.
1430 The build system runs any pre-processing commands as
1431 defined by the
1432 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink>
1433 variable.
1434 This variable specifies a list of functions to call before
1435 the OpenEmbedded build system creates the final image
1436 output files.
1437 </para>
1438
1439 <para>
1440 The OpenEmbedded build system dynamically creates
1441 <filename>do_image_*</filename> tasks as needed, based
1442 on the image types specified in the
1443 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
1444 variable.
1445 The process turns everything into an image file or a set of
1446 image files and can compress the root filesystem image to
1447 reduce the overall size of the image.
1448 The formats used for the root filesystem depend on the
1449 <filename>IMAGE_FSTYPES</filename> variable.
1450 Compression depends on whether the formats support
1451 compression.
1452 </para>
1453
1454 <para>
1455 As an example, a dynamically created task when creating a
1456 particular image <replaceable>type</replaceable> would
1457 take the following form:
1458 <literallayout class='monospaced'>
1459 do_image_<replaceable>type</replaceable>
1460 </literallayout>
1461 So, if the <replaceable>type</replaceable> as specified by
1462 the <filename>IMAGE_FSTYPES</filename> were
1463 <filename>ext4</filename>, the dynamically generated task
1464 would be as follows:
1465 <literallayout class='monospaced'>
1466 do_image_ext4
1467 </literallayout>
1468 </para>
1469
1470 <para>
1471 The final task involved in image creation is the
1472 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink>
1473 task.
1474 This task completes the image by applying any image
1475 post processing as defined through the
1476 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink>
1477 variable.
1478 The variable specifies a list of functions to call once the
1479 OpenEmbedded build system has created the final image
1480 output files.
1481 <note>
1482 The entire image generation process is run under
1483 Pseudo.
1484 Running under Pseudo ensures that the files in the
1485 root filesystem have correct ownership.
1486 </note>
1487 </para>
1488 </section>
1489
1490 <section id='sdk-generation-dev-environment'>
1491 <title>SDK Generation</title>
1492
1493 <para>
1494 The OpenEmbedded build system uses BitBake to generate the
1495 Software Development Kit (SDK) installer script for both
1496 the standard and extensible SDKs:
1497 <imagedata fileref="figures/sdk-generation.png" align="center" />
1498 <note>
1499 For more information on the cross-development toolchain
1500 generation, see the
1501 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1502 section.
1503 For information on advantages gained when building a
1504 cross-development toolchain using the
1505 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1506 task, see the
1507 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
1508 section in the Yocto Project Application Development
1509 and the Extensible Software Development Kit (SDK)
1510 manual.
1511 </note>
1512 </para>
1513
1514 <para>
1515 Like image generation, the SDK script process consists of
1516 several stages and depends on many variables.
1517 The <filename>do_populate_sdk</filename> and
1518 <filename>do_populate_sdk_ext</filename> tasks use these
1519 key variables to help create the list of packages to
1520 actually install.
1521 For information on the variables listed in the figure,
1522 see the
1523 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1524 section.
1525 </para>
1526
1527 <para>
1528 The <filename>do_populate_sdk</filename> task helps create
1529 the standard SDK and handles two parts: a target part and a
1530 host part.
1531 The target part is the part built for the target hardware
1532 and includes libraries and headers.
1533 The host part is the part of the SDK that runs on the
1534 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>.
1535 </para>
1536
1537 <para>
1538 The <filename>do_populate_sdk_ext</filename> task helps
1539 create the extensible SDK and handles host and target parts
1540 differently than its counter part does for the standard SDK.
1541 For the extensible SDK, the task encapsulates the build
1542 system, which includes everything needed (host and target)
1543 for the SDK.
1544 </para>
1545
1546 <para>
1547 Regardless of the type of SDK being constructed, the
1548 tasks perform some cleanup after which a cross-development
1549 environment setup script and any needed configuration files
1550 are created.
1551 The final output is the Cross-development
1552 toolchain installation script (<filename>.sh</filename>
1553 file), which includes the environment setup script.
1554 </para>
1555 </section>
1556
1557 <section id='stamp-files-and-the-rerunning-of-tasks'>
1558 <title>Stamp Files and the Rerunning of Tasks</title>
1559
1560 <para>
1561 For each task that completes successfully, BitBake writes a
1562 stamp file into the
1563 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink>
1564 directory.
1565 The beginning of the stamp file's filename is determined
1566 by the
1567 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
1568 variable, and the end of the name consists of the task's
1569 name and current
1570 <ulink url='&YOCTO_DOCS_GS_URL;#overview-checksums'>input checksum</ulink>.
1571 <note>
1572 This naming scheme assumes that
1573 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
1574 is "OEBasicHash", which is almost always the case in
1575 current OpenEmbedded.
1576 </note>
1577 To determine if a task needs to be rerun, BitBake checks
1578 if a stamp file with a matching input checksum exists
1579 for the task.
1580 If such a stamp file exists, the task's output is
1581 assumed to exist and still be valid.
1582 If the file does not exist, the task is rerun.
1583 <note>
1584 <para>The stamp mechanism is more general than the
1585 shared state (sstate) cache mechanism described in the
1586 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
1587 section.
1588 BitBake avoids rerunning any task that has a valid
1589 stamp file, not just tasks that can be accelerated
1590 through the sstate cache.</para>
1591
1592 <para>However, you should realize that stamp files only
1593 serve as a marker that some work has been done and that
1594 these files do not record task output.
1595 The actual task output would usually be somewhere in
1596 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
1597 (e.g. in some recipe's
1598 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.)
1599 What the sstate cache mechanism adds is a way to cache
1600 task output that can then be shared between build
1601 machines.</para>
1602 </note>
1603 Since <filename>STAMPS_DIR</filename> is usually a
1604 subdirectory of <filename>TMPDIR</filename>, removing
1605 <filename>TMPDIR</filename> will also remove
1606 <filename>STAMPS_DIR</filename>, which means tasks will
1607 properly be rerun to repopulate
1608 <filename>TMPDIR</filename>.
1609 </para>
1610
1611 <para>
1612 If you want some task to always be considered "out of
1613 date", you can mark it with the
1614 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
1615 varflag.
1616 If some other task depends on such a task, then that
1617 task will also always be considered out of date, which
1618 might not be what you want.
1619 </para>
1620
1621 <para>
1622 For details on how to view information about a task's
1623 signature, see the
1624 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
1625 section in the Yocto Project Development Tasks Manual.
1626 </para>
1627 </section>
1628
1629 <section id='setscene-tasks-and-shared-state'>
1630 <title>Setscene Tasks and Shared State</title>
1631
1632 <para>
1633 The description of tasks so far assumes that BitBake needs
1634 to build everything and there are no prebuilt objects
1635 available.
1636 BitBake does support skipping tasks if prebuilt objects are
1637 available.
1638 These objects are usually made available in the form of a
1639 shared state (sstate) cache.
1640 <note>
1641 For information on variables affecting sstate, see the
1642 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
1643 and
1644 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
1645 variables.
1646 </note>
1647 </para>
1648
1649 <para>
1650 The idea of a setscene task (i.e
1651 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
1652 is a version of the task where
1653 instead of building something, BitBake can skip to the end
1654 result and simply place a set of files into specific
1655 locations as needed.
1656 In some cases, it makes sense to have a setscene task
1657 variant (e.g. generating package files in the
1658 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1659 task).
1660 In other cases, it does not make sense, (e.g. a
1661 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1662 task or
1663 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1664 task) since the work involved would be equal to or greater
1665 than the underlying task.
1666 </para>
1667
1668 <para>
1669 In the OpenEmbedded build system, the common tasks that
1670 have setscene variants are
1671 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>,
1672 <filename>do_package_write_*</filename>,
1673 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>,
1674 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>,
1675 and
1676 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>.
1677 Notice that these are most of the tasks whose output is an
1678 end result.
1679 </para>
1680
1681 <para>
1682 The OpenEmbedded build system has knowledge of the
1683 relationship between these tasks and other tasks that
1684 precede them.
1685 For example, if BitBake runs
1686 <filename>do_populate_sysroot_setscene</filename> for
1687 something, there is little point in running any of the
1688 <filename>do_fetch</filename>,
1689 <filename>do_unpack</filename>,
1690 <filename>do_patch</filename>,
1691 <filename>do_configure</filename>,
1692 <filename>do_compile</filename>, and
1693 <filename>do_install</filename> tasks.
1694 However, if <filename>do_package</filename> needs to be
1695 run, BitBake would need to run those other tasks.
1696 </para>
1697
1698 <para>
1699 It becomes more complicated if everything can come
1700 from an sstate cache because some objects are simply
1701 not required at all.
1702 For example, you do not need a compiler or native tools,
1703 such as quilt, if there is nothing to compile or patch.
1704 If the <filename>do_package_write_*</filename> packages
1705 are available from sstate, BitBake does not need the
1706 <filename>do_package</filename> task data.
1707 </para>
1708
1709 <para>
1710 To handle all these complexities, BitBake runs in two
1711 phases.
1712 The first is the "setscene" stage.
1713 During this stage, BitBake first checks the sstate cache
1714 for any targets it is planning to build.
1715 BitBake does a fast check to see if the object exists
1716 rather than a complete download.
1717 If nothing exists, the second phase, which is the setscene
1718 stage, completes and the main build proceeds.
1719 </para>
1720
1721 <para>
1722 If objects are found in the sstate cache, the OpenEmbedded
1723 build system works backwards from the end targets specified
1724 by the user.
1725 For example, if an image is being built, the OpenEmbedded
1726 build system first looks for the packages needed for
1727 that image and the tools needed to construct an image.
1728 If those are available, the compiler is not needed.
1729 Thus, the compiler is not even downloaded.
1730 If something was found to be unavailable, or the
1731 download or setscene task fails, the OpenEmbedded build
1732 system then tries to install dependencies, such as the
1733 compiler, from the cache.
1734 </para>
1735
1736 <para>
1737 The availability of objects in the sstate cache is
1738 handled by the function specified by the
1739 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
1740 variable and returns a list of the objects that are
1741 available.
1742 The function specified by the
1743 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
1744 variable is the function that determines whether a given
1745 dependency needs to be followed, and whether for any given
1746 relationship the function needs to be passed.
1747 The function returns a True or False value.
1748 </para>
1749 </section>
1750 </section>
1751
1752 <section id='images-dev-environment'>
1753 <title>Images</title>
1754
1755 <para>
1756 The images produced by the OpenEmbedded build system
1757 are compressed forms of the
1758 root filesystem that are ready to boot on a target device.
1759 You can see from the
1760 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>
1761 that BitBake output, in part, consists of images.
1762 This section is going to look more closely at this output:
1763 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1764 </para>
1765
1766 <para>
1767 For a list of example images that the Yocto Project provides,
1768 see the
1769 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>"
1770 chapter in the Yocto Project Reference Manual.
1771 </para>
1772
1773 <para>
1774 Images are written out to the
1775 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1776 inside the
1777 <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
1778 folder as shown in the figure.
1779 This folder contains any files expected to be loaded on the
1780 target device.
1781 The
1782 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>
1783 variable points to the <filename>deploy</filename> directory,
1784 while the
1785 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink>
1786 variable points to the appropriate directory containing images
1787 for the current configuration.
1788 <itemizedlist>
1789 <listitem><para>
1790 <filename><replaceable>kernel-image</replaceable></filename>:
1791 A kernel binary file.
1792 The
1793 <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink>
1794 variable setting determines the naming scheme for the
1795 kernel image file.
1796 Depending on that variable, the file could begin with
1797 a variety of naming strings.
1798 The
1799 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1800 directory can contain multiple image files for the
1801 machine.
1802 </para></listitem>
1803 <listitem><para>
1804 <filename><replaceable>root-filesystem-image</replaceable></filename>:
1805 Root filesystems for the target device (e.g.
1806 <filename>*.ext3</filename> or
1807 <filename>*.bz2</filename> files).
1808 The
1809 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
1810 variable setting determines the root filesystem image
1811 type.
1812 The
1813 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1814 directory can contain multiple root filesystems for the
1815 machine.
1816 </para></listitem>
1817 <listitem><para>
1818 <filename><replaceable>kernel-modules</replaceable></filename>:
1819 Tarballs that contain all the modules built for the
1820 kernel.
1821 Kernel module tarballs exist for legacy purposes and
1822 can be suppressed by setting the
1823 <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink>
1824 variable to "0".
1825 The
1826 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1827 directory can contain multiple kernel module tarballs
1828 for the machine.
1829 </para></listitem>
1830 <listitem><para>
1831 <filename><replaceable>bootloaders</replaceable></filename>:
1832 Bootloaders supporting the image, if applicable to the
1833 target machine.
1834 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1835 directory can contain multiple bootloaders for the
1836 machine.
1837 </para></listitem>
1838 <listitem><para>
1839 <filename><replaceable>symlinks</replaceable></filename>:
1840 The
1841 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1842 folder contains a symbolic link that points to the
1843 most recently built file for each machine.
1844 These links might be useful for external scripts that
1845 need to obtain the latest version of each file.
1846 </para></listitem>
1847 </itemizedlist>
1848 </para>
1849 </section>
1850
1851 <section id='sdk-dev-environment'>
1852 <title>Application Development SDK</title>
1853
1854 <para>
1855 In the
1856 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
1857 the output labeled "Application Development SDK" represents an
1858 SDK.
1859 The SDK generation process differs depending on whether you
1860 build a standard SDK (e.g.
1861 <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>)
1862 or an extensible SDK (e.g.
1863 <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>).
1864 This section is going to take a closer look at this output:
1865 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
1866 </para>
1867
1868 <para>
1869 The specific form of this output is a self-extracting
1870 SDK installer (<filename>*.sh</filename>) that, when run,
1871 installs the SDK, which consists of a cross-development
1872 toolchain, a set of libraries and headers, and an SDK
1873 environment setup script.
1874 Running this installer essentially sets up your
1875 cross-development environment.
1876 You can think of the cross-toolchain as the "host"
1877 part because it runs on the SDK machine.
1878 You can think of the libraries and headers as the "target"
1879 part because they are built for the target hardware.
1880 The environment setup script is added so that you can
1881 initialize the environment before using the tools.
1882 </para>
1883
1884 <note><title>Notes</title>
1885 <itemizedlist>
1886 <listitem><para>
1887 The Yocto Project supports several methods by which
1888 you can set up this cross-development environment.
1889 These methods include downloading pre-built SDK
1890 installers or building and installing your own SDK
1891 installer.
1892 </para></listitem>
1893 <listitem><para>
1894 For background information on cross-development
1895 toolchains in the Yocto Project development
1896 environment, see the
1897 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1898 section.
1899 </para></listitem>
1900 <listitem><para>
1901 For information on setting up a cross-development
1902 environment, see the
1903 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
1904 manual.
1905 </para></listitem>
1906 </itemizedlist>
1907 </note>
1908
1909 <para>
1910 Once built, the SDK installers are written out to the
1911 <filename>deploy/sdk</filename> folder inside the
1912 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1913 as shown in the figure at the beginning of this section.
1914 Depending on the type of SDK, several variables exist that help
1915 configure these files.
1916 The following list shows the variables associated with
1917 a standard SDK:
1918 <itemizedlist>
1919 <listitem><para>
1920 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1921 Points to the <filename>deploy</filename>
1922 directory.
1923 </para></listitem>
1924 <listitem><para>
1925 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>:
1926 Specifies the architecture of the machine
1927 on which the cross-development tools are run to
1928 create packages for the target hardware.
1929 </para></listitem>
1930 <listitem><para>
1931 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>:
1932 Lists the features to include in the "target" part
1933 of the SDK.
1934 </para></listitem>
1935 <listitem><para>
1936 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>:
1937 Lists packages that make up the host
1938 part of the SDK (i.e. the part that runs on
1939 the <filename>SDKMACHINE</filename>).
1940 When you use
1941 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
1942 to create the SDK, a set of default packages
1943 apply.
1944 This variable allows you to add more packages.
1945 </para></listitem>
1946 <listitem><para>
1947 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>:
1948 Lists packages that make up the target part
1949 of the SDK (i.e. the part built for the
1950 target hardware).
1951 </para></listitem>
1952 <listitem><para>
1953 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>:
1954 Defines the default SDK installation path offered
1955 by the installation script.
1956 </para></listitem>
1957 </itemizedlist>
1958 This next list, shows the variables associated with an
1959 extensible SDK:
1960 <itemizedlist>
1961 <listitem><para>
1962 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1963 Points to the <filename>deploy</filename> directory.
1964 </para></listitem>
1965 <listitem><para>
1966 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>:
1967 Controls whether or not shared state artifacts are
1968 copied into the extensible SDK.
1969 By default, all required shared state artifacts are
1970 copied into the SDK.
1971 </para></listitem>
1972 <listitem><para>
1973 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>:
1974 Specifies whether or not packagedata will be
1975 included in the extensible SDK for all recipes in
1976 the "world" target.
1977 </para></listitem>
1978 <listitem><para>
1979 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>:
1980 Specifies whether or not the toolchain will be included
1981 when building the extensible SDK.
1982 </para></listitem>
1983 <listitem><para>
1984 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>:
1985 A list of variables allowed through from the build
1986 system configuration into the extensible SDK
1987 configuration.
1988 </para></listitem>
1989 <listitem><para>
1990 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>:
1991 A list of variables not allowed through from the build
1992 system configuration into the extensible SDK
1993 configuration.
1994 </para></listitem>
1995 <listitem><para>
1996 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>:
1997 A list of classes to remove from the
1998 <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink>
1999 value globally within the extensible SDK configuration.
2000 </para></listitem>
2001 </itemizedlist>
2002 </para>
2003 </section>
2004 </section>
2005
2006 <section id="cross-development-toolchain-generation">
2007 <title>Cross-Development Toolchain Generation</title>
2008
2009 <para>
2010 The Yocto Project does most of the work for you when it comes to
2011 creating
2012 <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>.
2013 This section provides some technical background on how
2014 cross-development toolchains are created and used.
2015 For more information on toolchains, you can also see the
2016 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2017 manual.
2018 </para>
2019
2020 <para>
2021 In the Yocto Project development environment, cross-development
2022 toolchains are used to build the image and applications that run
2023 on the target hardware.
2024 With just a few commands, the OpenEmbedded build system creates
2025 these necessary toolchains for you.
2026 </para>
2027
2028 <para>
2029 The following figure shows a high-level build environment regarding
2030 toolchain construction and use.
2031 </para>
2032
2033 <para>
2034 <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" />
2035 </para>
2036
2037 <para>
2038 Most of the work occurs on the Build Host.
2039 This is the machine used to build images and generally work within
2040 the the Yocto Project environment.
2041 When you run
2042 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2043 to create an image, the OpenEmbedded build system
2044 uses the host <filename>gcc</filename> compiler to bootstrap a
2045 cross-compiler named <filename>gcc-cross</filename>.
2046 The <filename>gcc-cross</filename> compiler is what BitBake uses to
2047 compile source files when creating the target image.
2048 You can think of <filename>gcc-cross</filename> simply as an
2049 automatically generated cross-compiler that is used internally
2050 within BitBake only.
2051 <note>
2052 The extensible SDK does not use
2053 <filename>gcc-cross-canadian</filename> since this SDK
2054 ships a copy of the OpenEmbedded build system and the sysroot
2055 within it contains <filename>gcc-cross</filename>.
2056 </note>
2057 </para>
2058
2059 <para>
2060 The chain of events that occurs when <filename>gcc-cross</filename> is
2061 bootstrapped is as follows:
2062 <literallayout class='monospaced'>
2063 gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime
2064 </literallayout>
2065 <itemizedlist>
2066 <listitem><para>
2067 <filename>gcc</filename>:
2068 The build host's GNU Compiler Collection (GCC).
2069 </para></listitem>
2070 <listitem><para>
2071 <filename>binutils-cross</filename>:
2072 The bare minimum binary utilities needed in order to run
2073 the <filename>gcc-cross-initial</filename> phase of the
2074 bootstrap operation.
2075 </para></listitem>
2076 <listitem><para>
2077 <filename>gcc-cross-initial</filename>:
2078 An early stage of the bootstrap process for creating
2079 the cross-compiler.
2080 This stage builds enough of the <filename>gcc-cross</filename>,
2081 the C library, and other pieces needed to finish building the
2082 final cross-compiler in later stages.
2083 This tool is a "native" package (i.e. it is designed to run on
2084 the build host).
2085 </para></listitem>
2086 <listitem><para>
2087 <filename>linux-libc-headers</filename>:
2088 Headers needed for the cross-compiler.
2089 </para></listitem>
2090 <listitem><para>
2091 <filename>glibc-initial</filename>:
2092 An initial version of the Embedded GLIBC needed to bootstrap
2093 <filename>glibc</filename>.
2094 </para></listitem>
2095 <listitem><para>
2096 <filename>gcc-cross</filename>:
2097 The final stage of the bootstrap process for the
2098 cross-compiler.
2099 This stage results in the actual cross-compiler that
2100 BitBake uses when it builds an image for a targeted
2101 device.
2102 <note>
2103 If you are replacing this cross compiler toolchain
2104 with a custom version, you must replace
2105 <filename>gcc-cross</filename>.
2106 </note>
2107 This tool is also a "native" package (i.e. it is
2108 designed to run on the build host).
2109 </para></listitem>
2110 <listitem><para>
2111 <filename>gcc-runtime</filename>:
2112 Runtime libraries resulting from the toolchain bootstrapping
2113 process.
2114 This tool produces a binary that consists of the
2115 runtime libraries need for the targeted device.
2116 </para></listitem>
2117 </itemizedlist>
2118 </para>
2119
2120 <para>
2121 You can use the OpenEmbedded build system to build an installer for
2122 the relocatable SDK used to develop applications.
2123 When you run the installer, it installs the toolchain, which
2124 contains the development tools (e.g.,
2125 <filename>gcc-cross-canadian</filename>,
2126 <filename>binutils-cross-canadian</filename>, and other
2127 <filename>nativesdk-*</filename> tools),
2128 which are tools native to the SDK (i.e. native to
2129 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>),
2130 you need to cross-compile and test your software.
2131 The figure shows the commands you use to easily build out this
2132 toolchain.
2133 This cross-development toolchain is built to execute on the
2134 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2135 which might or might not be the same
2136 machine as the Build Host.
2137 <note>
2138 If your target architecture is supported by the Yocto Project,
2139 you can take advantage of pre-built images that ship with the
2140 Yocto Project and already contain cross-development toolchain
2141 installers.
2142 </note>
2143 </para>
2144
2145 <para>
2146 Here is the bootstrap process for the relocatable toolchain:
2147 <literallayout class='monospaced'>
2148 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers ->
2149 glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
2150 </literallayout>
2151 <itemizedlist>
2152 <listitem><para>
2153 <filename>gcc</filename>:
2154 The build host's GNU Compiler Collection (GCC).
2155 </para></listitem>
2156 <listitem><para>
2157 <filename>binutils-crosssdk</filename>:
2158 The bare minimum binary utilities needed in order to run
2159 the <filename>gcc-crosssdk-initial</filename> phase of the
2160 bootstrap operation.
2161 </para></listitem>
2162 <listitem><para>
2163 <filename>gcc-crosssdk-initial</filename>:
2164 An early stage of the bootstrap process for creating
2165 the cross-compiler.
2166 This stage builds enough of the
2167 <filename>gcc-crosssdk</filename> and supporting pieces so that
2168 the final stage of the bootstrap process can produce the
2169 finished cross-compiler.
2170 This tool is a "native" binary that runs on the build host.
2171 </para></listitem>
2172 <listitem><para>
2173 <filename>linux-libc-headers</filename>:
2174 Headers needed for the cross-compiler.
2175 </para></listitem>
2176 <listitem><para>
2177 <filename>glibc-initial</filename>:
2178 An initial version of the Embedded GLIBC needed to bootstrap
2179 <filename>nativesdk-glibc</filename>.
2180 </para></listitem>
2181 <listitem><para>
2182 <filename>nativesdk-glibc</filename>:
2183 The Embedded GLIBC needed to bootstrap the
2184 <filename>gcc-crosssdk</filename>.
2185 </para></listitem>
2186 <listitem><para>
2187 <filename>gcc-crosssdk</filename>:
2188 The final stage of the bootstrap process for the
2189 relocatable cross-compiler.
2190 The <filename>gcc-crosssdk</filename> is a transitory compiler
2191 and never leaves the build host.
2192 Its purpose is to help in the bootstrap process to create the
2193 eventual relocatable <filename>gcc-cross-canadian</filename>
2194 compiler, which is relocatable.
2195 This tool is also a "native" package (i.e. it is
2196 designed to run on the build host).
2197 </para></listitem>
2198 <listitem><para>
2199 <filename>gcc-cross-canadian</filename>:
2200 The final relocatable cross-compiler.
2201 When run on the
2202 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2203 this tool
2204 produces executable code that runs on the target device.
2205 Only one cross-canadian compiler is produced per architecture
2206 since they can be targeted at different processor optimizations
2207 using configurations passed to the compiler through the
2208 compile commands.
2209 This circumvents the need for multiple compilers and thus
2210 reduces the size of the toolchains.
2211 </para></listitem>
2212 </itemizedlist>
2213 </para>
2214
2215 <note>
2216 For information on advantages gained when building a
2217 cross-development toolchain installer, see the
2218 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
2219 section in the Yocto Project Application Development and the
2220 Extensible Software Development Kit (eSDK) manual.
2221 </note>
2222 </section>
2223
2224 <section id="shared-state-cache">
2225 <title>Shared State Cache</title>
2226
2227 <para>
2228 By design, the OpenEmbedded build system builds everything from
2229 scratch unless
2230 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2231 can determine that parts do not need to be rebuilt.
2232 Fundamentally, building from scratch is attractive as it means all
2233 parts are built fresh and there is no possibility of stale data
2234 causing problems.
2235 When developers hit problems, they typically default back to
2236 building from scratch so they know the state of things from the
2237 start.
2238 </para>
2239
2240 <para>
2241 Building an image from scratch is both an advantage and a
2242 disadvantage to the process.
2243 As mentioned in the previous paragraph, building from scratch
2244 ensures that everything is current and starts from a known state.
2245 However, building from scratch also takes much longer as it
2246 generally means rebuilding things that do not necessarily need
2247 to be rebuilt.
2248 </para>
2249
2250 <para>
2251 The Yocto Project implements shared state code that supports
2252 incremental builds.
2253 The implementation of the shared state code answers the following
2254 questions that were fundamental roadblocks within the OpenEmbedded
2255 incremental build support system:
2256 <itemizedlist>
2257 <listitem><para>
2258 What pieces of the system have changed and what pieces have
2259 not changed?
2260 </para></listitem>
2261 <listitem><para>
2262 How are changed pieces of software removed and replaced?
2263 </para></listitem>
2264 <listitem><para>
2265 How are pre-built components that do not need to be rebuilt
2266 from scratch used when they are available?
2267 </para></listitem>
2268 </itemizedlist>
2269 </para>
2270
2271 <para>
2272 For the first question, the
2273 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
2274 detects changes in the "inputs" to a given task by creating a
2275 checksum (or signature) of the task's inputs.
2276 If the checksum changes, the system assumes the inputs have changed
2277 and the task needs to be rerun.
2278 For the second question, the shared state (sstate) code tracks
2279 which tasks add which output to the build process.
2280 This means the output from a given task can be removed, upgraded
2281 or otherwise manipulated.
2282 The third question is partly addressed by the solution for the
2283 second question assuming the build system can fetch the sstate
2284 objects from remote locations and install them if they are deemed
2285 to be valid.
2286 <note>
2287 The OpenEmbedded build system does not maintain
2288 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2289 information as part of the shared state packages.
2290 Consequently, considerations exist that affect maintaining
2291 shared state feeds.
2292 For information on how the OpenEmbedded build system
2293 works with packages and can track incrementing
2294 <filename>PR</filename> information, see the
2295 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
2296 section in the Yocto Project Development Tasks Manual.
2297 </note>
2298 </para>
2299
2300 <para>
2301 The rest of this section goes into detail about the overall
2302 incremental build architecture, the checksums (signatures), shared
2303 state, and some tips and tricks.
2304 </para>
2305
2306 <section id='concepts-overall-architecture'>
2307 <title>Overall Architecture</title>
2308
2309 <para>
2310 When determining what parts of the system need to be built,
2311 BitBake works on a per-task basis rather than a per-recipe
2312 basis.
2313 You might wonder why using a per-task basis is preferred over
2314 a per-recipe basis.
2315 To help explain, consider having the IPK packaging backend
2316 enabled and then switching to DEB.
2317 In this case, the
2318 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2319 and
2320 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2321 task outputs are still valid.
2322 However, with a per-recipe approach, the build would not
2323 include the <filename>.deb</filename> files.
2324 Consequently, you would have to invalidate the whole build and
2325 rerun it.
2326 Rerunning everything is not the best solution.
2327 Also, in this case, the core must be "taught" much about
2328 specific tasks.
2329 This methodology does not scale well and does not allow users
2330 to easily add new tasks in layers or as external recipes
2331 without touching the packaged-staging core.
2332 </para>
2333 </section>
2334
2335 <section id='overview-checksums'>
2336 <title>Checksums (Signatures)</title>
2337
2338 <para>
2339 The shared state code uses a checksum, which is a unique
2340 signature of a task's inputs, to determine if a task needs to
2341 be run again.
2342 Because it is a change in a task's inputs that triggers a
2343 rerun, the process needs to detect all the inputs to a given
2344 task.
2345 For shell tasks, this turns out to be fairly easy because
2346 the build process generates a "run" shell script for each task
2347 and it is possible to create a checksum that gives you a good
2348 idea of when the task's data changes.
2349 </para>
2350
2351 <para>
2352 To complicate the problem, there are things that should not be
2353 included in the checksum.
2354 First, there is the actual specific build path of a given
2355 task - the
2356 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2357 It does not matter if the work directory changes because it
2358 should not affect the output for target packages.
2359 Also, the build process has the objective of making native
2360 or cross packages relocatable.
2361 <note>
2362 Both native and cross packages run on the
2363 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>.
2364 However, cross packages generate output for the target
2365 architecture.
2366 </note>
2367 The checksum therefore needs to exclude
2368 <filename>WORKDIR</filename>.
2369 The simplistic approach for excluding the work directory is to
2370 set <filename>WORKDIR</filename> to some fixed value and
2371 create the checksum for the "run" script.
2372 </para>
2373
2374 <para>
2375 Another problem results from the "run" scripts containing
2376 functions that might or might not get called.
2377 The incremental build solution contains code that figures out
2378 dependencies between shell functions.
2379 This code is used to prune the "run" scripts down to the
2380 minimum set, thereby alleviating this problem and making the
2381 "run" scripts much more readable as a bonus.
2382 </para>
2383
2384 <para>
2385 So far, solutions for shell scripts exist.
2386 What about Python tasks?
2387 The same approach applies even though these tasks are more
2388 difficult.
2389 The process needs to figure out what variables a Python
2390 function accesses and what functions it calls.
2391 Again, the incremental build solution contains code that first
2392 figures out the variable and function dependencies, and then
2393 creates a checksum for the data used as the input to the task.
2394 </para>
2395
2396 <para>
2397 Like the <filename>WORKDIR</filename> case, situations exist
2398 where dependencies should be ignored.
2399 For these situations, you can instruct the build process to
2400 ignore a dependency by using a line like the following:
2401 <literallayout class='monospaced'>
2402 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
2403 </literallayout>
2404 This example ensures that the
2405 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
2406 variable does not depend on the value of
2407 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
2408 even if it does reference it.
2409 </para>
2410
2411 <para>
2412 Equally, there are cases where you need to add dependencies
2413 BitBake is not able to find.
2414 You can accomplish this by using a line like the following:
2415 <literallayout class='monospaced'>
2416 PACKAGE_ARCHS[vardeps] = "MACHINE"
2417 </literallayout>
2418 This example explicitly adds the <filename>MACHINE</filename>
2419 variable as a dependency for
2420 <filename>PACKAGE_ARCHS</filename>.
2421 </para>
2422
2423 <para>
2424 As an example, consider a case with in-line Python where
2425 BitBake is not able to figure out dependencies.
2426 When running in debug mode (i.e. using
2427 <filename>-DDD</filename>), BitBake produces output when it
2428 discovers something for which it cannot figure out dependencies.
2429 The Yocto Project team has currently not managed to cover
2430 those dependencies in detail and is aware of the need to fix
2431 this situation.
2432 </para>
2433
2434 <para>
2435 Thus far, this section has limited discussion to the direct
2436 inputs into a task.
2437 Information based on direct inputs is referred to as the
2438 "basehash" in the code.
2439 However, there is still the question of a task's indirect
2440 inputs - the things that were already built and present in the
2441 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
2442 The checksum (or signature) for a particular task needs to add
2443 the hashes of all the tasks on which the particular task
2444 depends.
2445 Choosing which dependencies to add is a policy decision.
2446 However, the effect is to generate a master checksum that
2447 combines the basehash and the hashes of the task's
2448 dependencies.
2449 </para>
2450
2451 <para>
2452 At the code level, a variety of ways exist by which both the
2453 basehash and the dependent task hashes can be influenced.
2454 Within the BitBake configuration file, you can give BitBake
2455 some extra information to help it construct the basehash.
2456 The following statement effectively results in a list of
2457 global variable dependency excludes - variables never
2458 included in any checksum:
2459 <literallayout class='monospaced'>
2460 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
2461 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
2462 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
2463 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
2464 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
2465 </literallayout>
2466 The previous example excludes
2467 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
2468 since that variable is actually constructed as a path within
2469 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
2470 which is on the whitelist.
2471 </para>
2472
2473 <para>
2474 The rules for deciding which hashes of dependent tasks to
2475 include through dependency chains are more complex and are
2476 generally accomplished with a Python function.
2477 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
2478 two examples of this and also illustrates how you can insert
2479 your own policy into the system if so desired.
2480 This file defines the two basic signature generators
2481 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
2482 uses: "OEBasic" and "OEBasicHash".
2483 By default, there is a dummy "noop" signature handler enabled
2484 in BitBake.
2485 This means that behavior is unchanged from previous versions.
2486 OE-Core uses the "OEBasicHash" signature handler by default
2487 through this setting in the <filename>bitbake.conf</filename>
2488 file:
2489 <literallayout class='monospaced'>
2490 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
2491 </literallayout>
2492 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
2493 is the same as the "OEBasic" version but adds the task hash to
2494 the stamp files.
2495 This results in any
2496 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
2497 change that changes the task hash, automatically
2498 causing the task to be run again.
2499 This removes the need to bump
2500 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2501 values, and changes to Metadata automatically ripple across
2502 the build.
2503 </para>
2504
2505 <para>
2506 It is also worth noting that the end result of these
2507 signature generators is to make some dependency and hash
2508 information available to the build.
2509 This information includes:
2510 <itemizedlist>
2511 <listitem><para>
2512 <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
2513 The base hashes for each task in the recipe.
2514 </para></listitem>
2515 <listitem><para>
2516 <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2517 The base hashes for each dependent task.
2518 </para></listitem>
2519 <listitem><para>
2520 <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2521 The task dependencies for each task.
2522 </para></listitem>
2523 <listitem><para>
2524 <filename>BB_TASKHASH</filename>:
2525 The hash of the currently running task.
2526 </para></listitem>
2527 </itemizedlist>
2528 </para>
2529 </section>
2530
2531 <section id='shared-state'>
2532 <title>Shared State</title>
2533
2534 <para>
2535 Checksums and dependencies, as discussed in the previous
2536 section, solve half the problem of supporting a shared state.
2537 The other part of the problem is being able to use checksum
2538 information during the build and being able to reuse or rebuild
2539 specific components.
2540 </para>
2541
2542 <para>
2543 The
2544 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2545 class is a relatively generic implementation of how to
2546 "capture" a snapshot of a given task.
2547 The idea is that the build process does not care about the
2548 source of a task's output.
2549 Output could be freshly built or it could be downloaded and
2550 unpacked from somewhere - the build process does not need to
2551 worry about its origin.
2552 </para>
2553
2554 <para>
2555 Two types of output exist.
2556 One type is just about creating a directory in
2557 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2558 A good example is the output of either
2559 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2560 or
2561 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
2562 The other type of output occurs when a set of data is merged
2563 into a shared directory tree such as the sysroot.
2564 </para>
2565
2566 <para>
2567 The Yocto Project team has tried to keep the details of the
2568 implementation hidden in <filename>sstate</filename> class.
2569 From a user's perspective, adding shared state wrapping to a task
2570 is as simple as this
2571 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2572 example taken from the
2573 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
2574 class:
2575 <literallayout class='monospaced'>
2576 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
2577 SSTATETASKS += "do_deploy"
2578 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
2579 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
2580
2581 python do_deploy_setscene () {
2582 sstate_setscene(d)
2583 }
2584 addtask do_deploy_setscene
2585 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
2586 </literallayout>
2587 The following list explains the previous example:
2588 <itemizedlist>
2589 <listitem><para>
2590 Adding "do_deploy" to <filename>SSTATETASKS</filename>
2591 adds some required sstate-related processing, which is
2592 implemented in the
2593 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2594 class, to before and after the
2595 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2596 task.
2597 </para></listitem>
2598 <listitem><para>
2599 The
2600 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
2601 declares that <filename>do_deploy</filename> places its
2602 output in <filename>${DEPLOYDIR}</filename> when run
2603 normally (i.e. when not using the sstate cache).
2604 This output becomes the input to the shared state cache.
2605 </para></listitem>
2606 <listitem><para>
2607 The
2608 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
2609 line causes the contents of the shared state cache to be
2610 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
2611 <note>
2612 If <filename>do_deploy</filename> is not already in
2613 the shared state cache or if its input checksum
2614 (signature) has changed from when the output was
2615 cached, the task will be run to populate the shared
2616 state cache, after which the contents of the shared
2617 state cache is copied to
2618 <filename>${DEPLOY_DIR_IMAGE}</filename>.
2619 If <filename>do_deploy</filename> is in the shared
2620 state cache and its signature indicates that the
2621 cached output is still valid (i.e. if no
2622 relevant task inputs have changed), then the
2623 contents of the shared state cache will be copied
2624 directly to
2625 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
2626 <filename>do_deploy_setscene</filename> task
2627 instead, skipping the
2628 <filename>do_deploy</filename> task.
2629 </note>
2630 </para></listitem>
2631 <listitem><para>
2632 The following task definition is glue logic needed to
2633 make the previous settings effective:
2634 <literallayout class='monospaced'>
2635 python do_deploy_setscene () {
2636 sstate_setscene(d)
2637 }
2638 addtask do_deploy_setscene
2639 </literallayout>
2640 <filename>sstate_setscene()</filename> takes the flags
2641 above as input and accelerates the
2642 <filename>do_deploy</filename> task through the
2643 shared state cache if possible.
2644 If the task was accelerated,
2645 <filename>sstate_setscene()</filename> returns True.
2646 Otherwise, it returns False, and the normal
2647 <filename>do_deploy</filename> task runs.
2648 For more information, see the
2649 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
2650 section in the BitBake User Manual.
2651 </para></listitem>
2652 <listitem><para>
2653 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
2654 line creates <filename>${DEPLOYDIR}</filename> and
2655 <filename>${B}</filename> before the
2656 <filename>do_deploy</filename> task runs, and also sets
2657 the current working directory of
2658 <filename>do_deploy</filename> to
2659 <filename>${B}</filename>.
2660 For more information, see the
2661 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
2662 section in the BitBake User Manual.
2663 <note>
2664 In cases where
2665 <filename>sstate-inputdirs</filename> and
2666 <filename>sstate-outputdirs</filename> would be the
2667 same, you can use
2668 <filename>sstate-plaindirs</filename>.
2669 For example, to preserve the
2670 <filename>${PKGD}</filename> and
2671 <filename>${PKGDEST}</filename> output from the
2672 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2673 task, use the following:
2674 <literallayout class='monospaced'>
2675 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
2676 </literallayout>
2677 </note>
2678 </para></listitem>
2679 <listitem><para>
2680 <filename>sstate-inputdirs</filename> and
2681 <filename>sstate-outputdirs</filename> can also be used
2682 with multiple directories.
2683 For example, the following declares
2684 <filename>PKGDESTWORK</filename> and
2685 <filename>SHLIBWORK</filename> as shared state
2686 input directories, which populates the shared state
2687 cache, and <filename>PKGDATA_DIR</filename> and
2688 <filename>SHLIBSDIR</filename> as the corresponding
2689 shared state output directories:
2690 <literallayout class='monospaced'>
2691 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
2692 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
2693 </literallayout>
2694 </para></listitem>
2695 <listitem><para>
2696 These methods also include the ability to take a
2697 lockfile when manipulating shared state directory
2698 structures, for cases where file additions or removals
2699 are sensitive:
2700 <literallayout class='monospaced'>
2701 do_package[sstate-lockfile] = "${PACKAGELOCK}"
2702 </literallayout>
2703 </para></listitem>
2704 </itemizedlist>
2705 </para>
2706
2707 <para>
2708 Behind the scenes, the shared state code works by looking in
2709 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
2710 and
2711 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
2712 for shared state files.
2713 Here is an example:
2714 <literallayout class='monospaced'>
2715 SSTATE_MIRRORS ?= "\
2716 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
2717 file://.* file:///some/local/dir/sstate/PATH"
2718 </literallayout>
2719 <note>
2720 The shared state directory
2721 (<filename>SSTATE_DIR</filename>) is organized into
2722 two-character subdirectories, where the subdirectory
2723 names are based on the first two characters of the hash.
2724 If the shared state directory structure for a mirror has the
2725 same structure as <filename>SSTATE_DIR</filename>, you must
2726 specify "PATH" as part of the URI to enable the build system
2727 to map to the appropriate subdirectory.
2728 </note>
2729 </para>
2730
2731 <para>
2732 The shared state package validity can be detected just by
2733 looking at the filename since the filename contains the task
2734 checksum (or signature) as described earlier in this section.
2735 If a valid shared state package is found, the build process
2736 downloads it and uses it to accelerate the task.
2737 </para>
2738
2739 <para>
2740 The build processes use the <filename>*_setscene</filename>
2741 tasks for the task acceleration phase.
2742 BitBake goes through this phase before the main execution
2743 code and tries to accelerate any tasks for which it can find
2744 shared state packages.
2745 If a shared state package for a task is available, the
2746 shared state package is used.
2747 This means the task and any tasks on which it is dependent
2748 are not executed.
2749 </para>
2750
2751 <para>
2752 As a real world example, the aim is when building an IPK-based
2753 image, only the
2754 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
2755 tasks would have their shared state packages fetched and
2756 extracted.
2757 Since the sysroot is not used, it would never get extracted.
2758 This is another reason why a task-based approach is preferred
2759 over a recipe-based approach, which would have to install the
2760 output from every task.n
2761 </para>
2762 </section>
2763
2764 <section id='concepts-tips-and-tricks'>
2765 <title>Tips and Tricks</title>
2766
2767 <para>
2768 The code in the build system that supports incremental builds
2769 is not simple code.
2770 This section presents some tips and tricks that help you work
2771 around issues related to shared state code.
2772 </para>
2773
2774 <section id='concepts-overview-debugging'>
2775 <title>Debugging</title>
2776
2777 <para>
2778 Seeing what metadata went into creating the input signature
2779 of a shared state (sstate) task can be a useful debugging
2780 aid.
2781 This information is available in signature information
2782 (<filename>siginfo</filename>) files in
2783 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>.
2784 For information on how to view and interpret information in
2785 <filename>siginfo</filename> files, see the
2786 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
2787 section in the Yocto Project Development Tasks Manual.
2788 </para>
2789 </section>
2790
2791 <section id='concepts-invalidating-shared-state'>
2792 <title>Invalidating Shared State</title>
2793
2794 <para>
2795 The OpenEmbedded build system uses checksums and shared
2796 state cache to avoid unnecessarily rebuilding tasks.
2797 Collectively, this scheme is known as "shared state code."
2798 </para>
2799
2800 <para>
2801 As with all schemes, this one has some drawbacks.
2802 It is possible that you could make implicit changes to your
2803 code that the checksum calculations do not take into
2804 account.
2805 These implicit changes affect a task's output but do not
2806 trigger the shared state code into rebuilding a recipe.
2807 Consider an example during which a tool changes its output.
2808 Assume that the output of <filename>rpmdeps</filename>
2809 changes.
2810 The result of the change should be that all the
2811 <filename>package</filename> and
2812 <filename>package_write_rpm</filename> shared state cache
2813 items become invalid.
2814 However, because the change to the output is
2815 external to the code and therefore implicit,
2816 the associated shared state cache items do not become
2817 invalidated.
2818 In this case, the build process uses the cached items
2819 rather than running the task again.
2820 Obviously, these types of implicit changes can cause
2821 problems.
2822 </para>
2823
2824 <para>
2825 To avoid these problems during the build, you need to
2826 understand the effects of any changes you make.
2827 Realize that changes you make directly to a function
2828 are automatically factored into the checksum calculation.
2829 Thus, these explicit changes invalidate the associated
2830 area of shared state cache.
2831 However, you need to be aware of any implicit changes that
2832 are not obvious changes to the code and could affect
2833 the output of a given task.
2834 </para>
2835
2836 <para>
2837 When you identify an implicit change, you can easily
2838 take steps to invalidate the cache and force the tasks
2839 to run.
2840 The steps you can take are as simple as changing a
2841 function's comments in the source code.
2842 For example, to invalidate package shared state files,
2843 change the comment statements of
2844 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2845 or the comments of one of the functions it calls.
2846 Even though the change is purely cosmetic, it causes the
2847 checksum to be recalculated and forces the OpenEmbedded
2848 build system to run the task again.
2849 <note>
2850 For an example of a commit that makes a cosmetic
2851 change to invalidate shared state, see this
2852 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
2853 </note>
2854 </para>
2855 </section>
2856 </section>
2857 </section>
2858
2859 <section id='automatically-added-runtime-dependencies'>
2860 <title>Automatically Added Runtime Dependencies</title>
2861
2862 <para>
2863 The OpenEmbedded build system automatically adds common types of
2864 runtime dependencies between packages, which means that you do not
2865 need to explicitly declare the packages using
2866 <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>.
2867 Three automatic mechanisms exist (<filename>shlibdeps</filename>,
2868 <filename>pcdeps</filename>, and <filename>depchains</filename>)
2869 that handle shared libraries, package configuration (pkg-config)
2870 modules, and <filename>-dev</filename> and
2871 <filename>-dbg</filename> packages, respectively.
2872 For other types of runtime dependencies, you must manually declare
2873 the dependencies.
2874 <itemizedlist>
2875 <listitem><para>
2876 <filename>shlibdeps</filename>:
2877 During the
2878 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2879 task of each recipe, all shared libraries installed by the
2880 recipe are located.
2881 For each shared library, the package that contains the
2882 shared library is registered as providing the shared
2883 library.
2884 More specifically, the package is registered as providing
2885 the
2886 <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>
2887 of the library.
2888 The resulting shared-library-to-package mapping
2889 is saved globally in
2890 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>
2891 by the
2892 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
2893 task.</para>
2894
2895 <para>Simultaneously, all executables and shared libraries
2896 installed by the recipe are inspected to see what shared
2897 libraries they link against.
2898 For each shared library dependency that is found,
2899 <filename>PKGDATA_DIR</filename> is queried to
2900 see if some package (likely from a different recipe)
2901 contains the shared library.
2902 If such a package is found, a runtime dependency is added
2903 from the package that depends on the shared library to the
2904 package that contains the library.</para>
2905
2906 <para>The automatically added runtime dependency also
2907 includes a version restriction.
2908 This version restriction specifies that at least the
2909 current version of the package that provides the shared
2910 library must be used, as if
2911 "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"
2912 had been added to <filename>RDEPENDS</filename>.
2913 This forces an upgrade of the package containing the shared
2914 library when installing the package that depends on the
2915 library, if needed.</para>
2916
2917 <para>If you want to avoid a package being registered as
2918 providing a particular shared library (e.g. because the library
2919 is for internal use only), then add the library to
2920 <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink>
2921 inside the package's recipe.
2922 </para></listitem>
2923 <listitem><para>
2924 <filename>pcdeps</filename>:
2925 During the <filename>do_package</filename> task of each
2926 recipe, all pkg-config modules
2927 (<filename>*.pc</filename> files) installed by the recipe
2928 are located.
2929 For each module, the package that contains the module is
2930 registered as providing the module.
2931 The resulting module-to-package mapping is saved globally in
2932 <filename>PKGDATA_DIR</filename> by the
2933 <filename>do_packagedata</filename> task.</para>
2934
2935 <para>Simultaneously, all pkg-config modules installed by
2936 the recipe are inspected to see what other pkg-config
2937 modules they depend on.
2938 A module is seen as depending on another module if it
2939 contains a "Requires:" line that specifies the other module.
2940 For each module dependency,
2941 <filename>PKGDATA_DIR</filename> is queried to see if some
2942 package contains the module.
2943 If such a package is found, a runtime dependency is added
2944 from the package that depends on the module to the package
2945 that contains the module.
2946 <note>
2947 The <filename>pcdeps</filename> mechanism most often
2948 infers dependencies between <filename>-dev</filename>
2949 packages.
2950 </note>
2951 </para></listitem>
2952 <listitem><para>
2953 <filename>depchains</filename>:
2954 If a package <filename>foo</filename> depends on a package
2955 <filename>bar</filename>, then <filename>foo-dev</filename>
2956 and <filename>foo-dbg</filename> are also made to depend on
2957 <filename>bar-dev</filename> and
2958 <filename>bar-dbg</filename>, respectively.
2959 Taking the <filename>-dev</filename> packages as an
2960 example, the <filename>bar-dev</filename> package might
2961 provide headers and shared library symlinks needed by
2962 <filename>foo-dev</filename>, which shows the need
2963 for a dependency between the packages.</para>
2964
2965 <para>The dependencies added by
2966 <filename>depchains</filename> are in the form of
2967 <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>.
2968 <note>
2969 By default, <filename>foo-dev</filename> also has an
2970 <filename>RDEPENDS</filename>-style dependency on
2971 <filename>foo</filename>, because the default value of
2972 <filename>RDEPENDS_${PN}-dev</filename> (set in
2973 <filename>bitbake.conf</filename>) includes
2974 "${PN}".
2975 </note></para>
2976
2977 <para>To ensure that the dependency chain is never broken,
2978 <filename>-dev</filename> and <filename>-dbg</filename>
2979 packages are always generated by default, even if the
2980 packages turn out to be empty.
2981 See the
2982 <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink>
2983 variable for more information.
2984 </para></listitem>
2985 </itemizedlist>
2986 </para>
2987
2988 <para>
2989 The <filename>do_package</filename> task depends on the
2990 <filename>do_packagedata</filename> task of each recipe in
2991 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>
2992 through use of a
2993 <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>
2994 declaration, which guarantees that the required
2995 shared-library/module-to-package mapping information will be available
2996 when needed as long as <filename>DEPENDS</filename> has been
2997 correctly set.
2998 </para>
2999 </section>
3000
3001 <section id='fakeroot-and-pseudo'>
3002 <title>Fakeroot and Pseudo</title>
3003
3004 <para>
3005 Some tasks are easier to implement when allowed to perform certain
3006 operations that are normally reserved for the root user (e.g.
3007 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>,
3008 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>,
3009 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>,
3010 and
3011 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>).
3012 For example, the <filename>do_install</filename> task benefits
3013 from being able to set the UID and GID of installed files to
3014 arbitrary values.
3015 </para>
3016
3017 <para>
3018 One approach to allowing tasks to perform root-only operations
3019 would be to require
3020 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
3021 to run as root.
3022 However, this method is cumbersome and has security issues.
3023 The approach that is actually used is to run tasks that benefit
3024 from root privileges in a "fake" root environment.
3025 Within this environment, the task and its child processes believe
3026 that they are running as the root user, and see an internally
3027 consistent view of the filesystem.
3028 As long as generating the final output (e.g. a package or an image)
3029 does not require root privileges, the fact that some earlier
3030 steps ran in a fake root environment does not cause problems.
3031 </para>
3032
3033 <para>
3034 The capability to run tasks in a fake root environment is known as
3035 "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>",
3036 which is derived from the BitBake keyword/variable
3037 flag that requests a fake root environment for a task.
3038 </para>
3039
3040 <para>
3041 In the
3042 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
3043 the program that implements fakeroot is known as Pseudo.
3044 Pseudo overrides system calls by using the environment variable
3045 <filename>LD_PRELOAD</filename>, which results in the illusion
3046 of running as root.
3047 To keep track of "fake" file ownership and permissions resulting
3048 from operations that require root permissions, Pseudo uses
3049 an SQLite 3 database.
3050 This database is stored in
3051 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename>
3052 for individual recipes.
3053 Storing the database in a file as opposed to in memory
3054 gives persistence between tasks and builds, which is not
3055 accomplished using fakeroot.
3056 <note><title>Caution</title>
3057 If you add your own task that manipulates the same files or
3058 directories as a fakeroot task, then that task also needs to
3059 run under fakeroot.
3060 Otherwise, the task cannot run root-only operations, and
3061 cannot see the fake file ownership and permissions set by the
3062 other task.
3063 You need to also add a dependency on
3064 <filename>virtual/fakeroot-native:do_populate_sysroot</filename>,
3065 giving the following:
3066 <literallayout class='monospaced'>
3067 fakeroot do_mytask () {
3068 ...
3069 }
3070 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
3071 </literallayout>
3072 </note>
3073 For more information, see the
3074 <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>
3075 variables in the BitBake User Manual.
3076 You can also reference the
3077 "<ulink url='http://www.ibm.com/developerworks/opensource/library/os-aapseudo1/index.html'>Pseudo</ulink>"
3078 and
3079 "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>"
3080 articles for background information on Pseudo.
3081 </para>
3082 </section>
3083
3084 <section id="wayland">
3085 <title>Wayland</title>
3086
3087 <para>
3088 <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink>
3089 is a computer display server protocol that
3090 provides a method for compositing window managers to communicate
3091 directly with applications and video hardware and expects them to
3092 communicate with input hardware using other libraries.
3093 Using Wayland with supporting targets can result in better control
3094 over graphics frame rendering than an application might otherwise
3095 achieve.
3096 </para>
3097
3098 <para>
3099 The Yocto Project provides the Wayland protocol libraries and the
3100 reference
3101 <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink>
3102 compositor as part of its release.
3103 This section describes what you need to do to implement Wayland and
3104 use the compositor when building an image for a supporting target.
3105 </para>
3106
3107 <section id="wayland-support">
3108 <title>Support</title>
3109
3110 <para>
3111 The Wayland protocol libraries and the reference Weston
3112 compositor ship as integrated packages in the
3113 <filename>meta</filename> layer of the
3114 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
3115 Specifically, you can find the recipes that build both Wayland
3116 and Weston at
3117 <filename>meta/recipes-graphics/wayland</filename>.
3118 </para>
3119
3120 <para>
3121 You can build both the Wayland and Weston packages for use only
3122 with targets that accept the
3123 <ulink url='https://en.wikipedia.org/wiki/Mesa_(computer_graphics)'>Mesa 3D and Direct Rendering Infrastructure</ulink>,
3124 which is also known as Mesa DRI.
3125 This implies that you cannot build and use the packages if your
3126 target uses, for example, the
3127 <trademark class='registered'>Intel</trademark> Embedded Media
3128 and Graphics Driver
3129 (<trademark class='registered'>Intel</trademark> EMGD) that
3130 overrides Mesa DRI.
3131 <note>
3132 Due to lack of EGL support, Weston 1.0.3 will not run
3133 directly on the emulated QEMU hardware.
3134 However, this version of Weston will run under X emulation
3135 without issues.
3136 </note>
3137 </para>
3138 </section>
3139
3140 <section id="enabling-wayland-in-an-image">
3141 <title>Enabling Wayland in an Image</title>
3142
3143 <para>
3144 To enable Wayland, you need to enable it to be built and enable
3145 it to be included in the image.
3146 </para>
3147
3148 <section id="enable-building">
3149 <title>Building</title>
3150
3151 <para>
3152 To cause Mesa to build the <filename>wayland-egl</filename>
3153 platform and Weston to build Wayland with Kernel Mode
3154 Setting
3155 (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>)
3156 support, include the "wayland" flag in the
3157 <ulink url="&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></ulink>
3158 statement in your <filename>local.conf</filename> file:
3159 <literallayout class='monospaced'>
3160 DISTRO_FEATURES_append = " wayland"
3161 </literallayout>
3162 <note>
3163 If X11 has been enabled elsewhere, Weston will build
3164 Wayland with X11 support
3165 </note>
3166 </para>
3167 </section>
3168
3169 <section id="enable-installation-in-an-image">
3170 <title>Installing</title>
3171
3172 <para>
3173 To install the Wayland feature into an image, you must
3174 include the following
3175 <ulink url='&YOCTO_DOCS_REF_URL;#var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></ulink>
3176 statement in your <filename>local.conf</filename> file:
3177 <literallayout class='monospaced'>
3178 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
3179 </literallayout>
3180 </para>
3181 </section>
3182 </section>
3183
3184 <section id="running-weston">
3185 <title>Running Weston</title>
3186
3187 <para>
3188 To run Weston inside X11, enabling it as described earlier and
3189 building a Sato image is sufficient.
3190 If you are running your image under Sato, a Weston Launcher
3191 appears in the "Utility" category.
3192 </para>
3193
3194 <para>
3195 Alternatively, you can run Weston through the command-line
3196 interpretor (CLI), which is better suited for development work.
3197 To run Weston under the CLI, you need to do the following after
3198 your image is built:
3199 <orderedlist>
3200 <listitem><para>
3201 Run these commands to export
3202 <filename>XDG_RUNTIME_DIR</filename>:
3203 <literallayout class='monospaced'>
3204 mkdir -p /tmp/$USER-weston
3205 chmod 0700 /tmp/$USER-weston
3206 export XDG_RUNTIME_DIR=/tmp/$USER-weston
3207 </literallayout>
3208 </para></listitem>
3209 <listitem><para>
3210 Launch Weston in the shell:
3211 <literallayout class='monospaced'>
3212 weston
3213 </literallayout></para></listitem>
3214 </orderedlist>
3215 </para>
3216 </section>
3217 </section>
3218
3219 <section id="overview-licenses">
3220 <title>Licenses</title>
3221
3222 <para>
3223 This section describes the mechanism by which the
3224 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
3225 tracks changes to licensing text.
3226 The section also describes how to enable commercially licensed
3227 recipes, which by default are disabled.
3228 </para>
3229
3230 <para>
3231 For information that can help you maintain compliance with
3232 various open source licensing during the lifecycle of the product,
3233 see the
3234 "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>"
3235 section in the Yocto Project Development Tasks Manual.
3236 </para>
3237
3238 <section id="usingpoky-configuring-LIC_FILES_CHKSUM">
3239 <title>Tracking License Changes</title>
3240
3241 <para>
3242 The license of an upstream project might change in the future.
3243 In order to prevent these changes going unnoticed, the
3244 <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink>
3245 variable tracks changes to the license text. The checksums are
3246 validated at the end of the configure step, and if the
3247 checksums do not match, the build will fail.
3248 </para>
3249
3250 <section id="usingpoky-specifying-LIC_FILES_CHKSUM">
3251 <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>
3252
3253 <para>
3254 The <filename>LIC_FILES_CHKSUM</filename>
3255 variable contains checksums of the license text in the
3256 source code for the recipe.
3257 Following is an example of how to specify
3258 <filename>LIC_FILES_CHKSUM</filename>:
3259 <literallayout class='monospaced'>
3260 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
3261 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
3262 file://licfile2.txt;endline=50;md5=zzzz \
3263 ..."
3264 </literallayout>
3265 <note><title>Notes</title>
3266 <itemizedlist>
3267 <listitem><para>
3268 When using "beginline" and "endline", realize
3269 that line numbering begins with one and not
3270 zero.
3271 Also, the included lines are inclusive (i.e.
3272 lines five through and including 29 in the
3273 previous example for
3274 <filename>licfile1.txt</filename>).
3275 </para></listitem>
3276 <listitem><para>
3277 When a license check fails, the selected license
3278 text is included as part of the QA message.
3279 Using this output, you can determine the exact
3280 start and finish for the needed license text.
3281 </para></listitem>
3282 </itemizedlist>
3283 </note>
3284 </para>
3285
3286 <para>
3287 The build system uses the
3288 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
3289 variable as the default directory when searching files
3290 listed in <filename>LIC_FILES_CHKSUM</filename>.
3291 The previous example employs the default directory.
3292 </para>
3293
3294 <para>
3295 Consider this next example:
3296 <literallayout class='monospaced'>
3297 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
3298 md5=bb14ed3c4cda583abc85401304b5cd4e"
3299 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
3300 </literallayout>
3301 </para>
3302
3303 <para>
3304 The first line locates a file in
3305 <filename>${S}/src/ls.c</filename> and isolates lines five
3306 through 16 as license text.
3307 The second line refers to a file in
3308 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
3309 </para>
3310
3311 <para>
3312 Note that <filename>LIC_FILES_CHKSUM</filename> variable is
3313 mandatory for all recipes, unless the
3314 <filename>LICENSE</filename> variable is set to "CLOSED".
3315 </para>
3316 </section>
3317
3318 <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
3319 <title>Explanation of Syntax</title>
3320
3321 <para>
3322 As mentioned in the previous section, the
3323 <filename>LIC_FILES_CHKSUM</filename> variable lists all
3324 the important files that contain the license text for the
3325 source code.
3326 It is possible to specify a checksum for an entire file,
3327 or a specific section of a file (specified by beginning and
3328 ending line numbers with the "beginline" and "endline"
3329 parameters, respectively).
3330 The latter is useful for source files with a license
3331 notice header, README documents, and so forth.
3332 If you do not use the "beginline" parameter, then it is
3333 assumed that the text begins on the first line of the file.
3334 Similarly, if you do not use the "endline" parameter,
3335 it is assumed that the license text ends with the last
3336 line of the file.
3337 </para>
3338
3339 <para>
3340 The "md5" parameter stores the md5 checksum of the license
3341 text.
3342 If the license text changes in any way as compared to
3343 this parameter then a mismatch occurs.
3344 This mismatch triggers a build failure and notifies
3345 the developer.
3346 Notification allows the developer to review and address
3347 the license text changes.
3348 Also note that if a mismatch occurs during the build,
3349 the correct md5 checksum is placed in the build log and
3350 can be easily copied to the recipe.
3351 </para>
3352
3353 <para>
3354 There is no limit to how many files you can specify using
3355 the <filename>LIC_FILES_CHKSUM</filename> variable.
3356 Generally, however, every project requires a few
3357 specifications for license tracking.
3358 Many projects have a "COPYING" file that stores the
3359 license information for all the source code files.
3360 This practice allows you to just track the "COPYING"
3361 file as long as it is kept up to date.
3362 <note><title>Tips</title>
3363 <itemizedlist>
3364 <listitem><para>
3365 If you specify an empty or invalid "md5"
3366 parameter,
3367 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
3368 returns an md5 mis-match
3369 error and displays the correct "md5" parameter
3370 value during the build.
3371 The correct parameter is also captured in
3372 the build log.
3373 </para></listitem>
3374 <listitem><para>
3375 If the whole file contains only license text,
3376 you do not need to use the "beginline" and
3377 "endline" parameters.
3378 </para></listitem>
3379 </itemizedlist>
3380 </note>
3381 </para>
3382 </section>
3383 </section>
3384
3385 <section id="enabling-commercially-licensed-recipes">
3386 <title>Enabling Commercially Licensed Recipes</title>
3387
3388 <para>
3389 By default, the OpenEmbedded build system disables
3390 components that have commercial or other special licensing
3391 requirements.
3392 Such requirements are defined on a
3393 recipe-by-recipe basis through the
3394 <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></ulink>
3395 variable definition in the affected recipe.
3396 For instance, the
3397 <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
3398 recipe contains the following statement:
3399 <literallayout class='monospaced'>
3400 LICENSE_FLAGS = "commercial"
3401 </literallayout>
3402 Here is a slightly more complicated example that contains both
3403 an explicit recipe name and version (after variable expansion):
3404 <literallayout class='monospaced'>
3405 LICENSE_FLAGS = "license_${PN}_${PV}"
3406 </literallayout>
3407 In order for a component restricted by a
3408 <filename>LICENSE_FLAGS</filename> definition to be enabled and
3409 included in an image, it needs to have a matching entry in the
3410 global
3411 <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></ulink>
3412 variable, which is a variable typically defined in your
3413 <filename>local.conf</filename> file.
3414 For example, to enable the
3415 <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
3416 package, you could add either the string
3417 "commercial_gst-plugins-ugly" or the more general string
3418 "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.
3419 See the
3420 "<link linkend='license-flag-matching'>License Flag Matching</link>"
3421 section for a full
3422 explanation of how <filename>LICENSE_FLAGS</filename> matching
3423 works.
3424 Here is the example:
3425 <literallayout class='monospaced'>
3426 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
3427 </literallayout>
3428 Likewise, to additionally enable the package built from the
3429 recipe containing
3430 <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>,
3431 and assuming that the actual recipe name was
3432 <filename>emgd_1.10.bb</filename>, the following string would
3433 enable that package as well as the original
3434 <filename>gst-plugins-ugly</filename> package:
3435 <literallayout class='monospaced'>
3436 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
3437 </literallayout>
3438 As a convenience, you do not need to specify the complete
3439 license string in the whitelist for every package.
3440 You can use an abbreviated form, which consists
3441 of just the first portion or portions of the license
3442 string before the initial underscore character or characters.
3443 A partial string will match any license that contains the
3444 given string as the first portion of its license.
3445 For example, the following whitelist string will also match
3446 both of the packages previously mentioned as well as any other
3447 packages that have licenses starting with "commercial" or
3448 "license".
3449 <literallayout class='monospaced'>
3450 LICENSE_FLAGS_WHITELIST = "commercial license"
3451 </literallayout>
3452 </para>
3453
3454 <section id="license-flag-matching">
3455 <title>License Flag Matching</title>
3456
3457 <para>
3458 License flag matching allows you to control what recipes
3459 the OpenEmbedded build system includes in the build.
3460 Fundamentally, the build system attempts to match
3461 <filename>LICENSE_FLAGS</filename> strings found in recipes
3462 against <filename>LICENSE_FLAGS_WHITELIST</filename>
3463 strings found in the whitelist.
3464 A match causes the build system to include a recipe in the
3465 build, while failure to find a match causes the build
3466 system to exclude a recipe.
3467 </para>
3468
3469 <para>
3470 In general, license flag matching is simple.
3471 However, understanding some concepts will help you
3472 correctly and effectively use matching.
3473 </para>
3474
3475 <para>
3476 Before a flag
3477 defined by a particular recipe is tested against the
3478 contents of the whitelist, the expanded string
3479 <filename>_${PN}</filename> is appended to the flag.
3480 This expansion makes each
3481 <filename>LICENSE_FLAGS</filename> value recipe-specific.
3482 After expansion, the string is then matched against the
3483 whitelist.
3484 Thus, specifying
3485 <filename>LICENSE_FLAGS = "commercial"</filename>
3486 in recipe "foo", for example, results in the string
3487 <filename>"commercial_foo"</filename>.
3488 And, to create a match, that string must appear in the
3489 whitelist.
3490 </para>
3491
3492 <para>
3493 Judicious use of the <filename>LICENSE_FLAGS</filename>
3494 strings and the contents of the
3495 <filename>LICENSE_FLAGS_WHITELIST</filename> variable
3496 allows you a lot of flexibility for including or excluding
3497 recipes based on licensing.
3498 For example, you can broaden the matching capabilities by
3499 using license flags string subsets in the whitelist.
3500 <note>
3501 When using a string subset, be sure to use the part of
3502 the expanded string that precedes the appended
3503 underscore character (e.g.
3504 <filename>usethispart_1.3</filename>,
3505 <filename>usethispart_1.4</filename>, and so forth).
3506 </note>
3507 For example, simply specifying the string "commercial" in
3508 the whitelist matches any expanded
3509 <filename>LICENSE_FLAGS</filename> definition that starts
3510 with the string "commercial" such as "commercial_foo" and
3511 "commercial_bar", which are the strings the build system
3512 automatically generates for hypothetical recipes named
3513 "foo" and "bar" assuming those recipes simply specify the
3514 following:
3515 <literallayout class='monospaced'>
3516 LICENSE_FLAGS = "commercial"
3517 </literallayout>
3518 Thus, you can choose to exhaustively
3519 enumerate each license flag in the whitelist and
3520 allow only specific recipes into the image, or
3521 you can use a string subset that causes a broader range of
3522 matches to allow a range of recipes into the image.
3523 </para>
3524
3525 <para>
3526 This scheme works even if the
3527 <filename>LICENSE_FLAGS</filename> string already
3528 has <filename>_${PN}</filename> appended.
3529 For example, the build system turns the license flag
3530 "commercial_1.2_foo" into "commercial_1.2_foo_foo" and
3531 would match both the general "commercial" and the specific
3532 "commercial_1.2_foo" strings found in the whitelist, as
3533 expected.
3534 </para>
3535
3536 <para>
3537 Here are some other scenarios:
3538 <itemizedlist>
3539 <listitem><para>
3540 You can specify a versioned string in the recipe
3541 such as "commercial_foo_1.2" in a "foo" recipe.
3542 The build system expands this string to
3543 "commercial_foo_1.2_foo".
3544 Combine this license flag with a whitelist that has
3545 the string "commercial" and you match the flag
3546 along with any other flag that starts with the
3547 string "commercial".
3548 </para></listitem>
3549 <listitem><para>
3550 Under the same circumstances, you can use
3551 "commercial_foo" in the whitelist and the build
3552 system not only matches "commercial_foo_1.2" but
3553 also matches any license flag with the string
3554 "commercial_foo", regardless of the version.
3555 </para></listitem>
3556 <listitem><para>
3557 You can be very specific and use both the
3558 package and version parts in the whitelist (e.g.
3559 "commercial_foo_1.2") to specifically match a
3560 versioned recipe.
3561 </para></listitem>
3562 </itemizedlist>
3563 </para>
3564 </section>
3565
3566 <section id="other-variables-related-to-commercial-licenses">
3567 <title>Other Variables Related to Commercial Licenses</title>
3568
3569 <para>
3570 Other helpful variables related to commercial
3571 license handling exist and are defined in the
3572 <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file:
3573 <literallayout class='monospaced'>
3574 COMMERCIAL_AUDIO_PLUGINS ?= ""
3575 COMMERCIAL_VIDEO_PLUGINS ?= ""
3576 </literallayout>
3577 If you want to enable these components, you can do so by
3578 making sure you have statements similar to the following
3579 in your <filename>local.conf</filename> configuration file:
3580 <literallayout class='monospaced'>
3581 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
3582 gst-plugins-ugly-mpegaudioparse"
3583 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
3584 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
3585 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
3586 </literallayout>
3587 Of course, you could also create a matching whitelist
3588 for those components using the more general "commercial"
3589 in the whitelist, but that would also enable all the
3590 other packages with <filename>LICENSE_FLAGS</filename>
3591 containing "commercial", which you may or may not want:
3592 <literallayout class='monospaced'>
3593 LICENSE_FLAGS_WHITELIST = "commercial"
3594 </literallayout>
3595 </para>
3596
3597 <para>
3598 Specifying audio and video plug-ins as part of the
3599 <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
3600 <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements
3601 (along with the enabling
3602 <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the
3603 plug-ins or components into built images, thus adding
3604 support for media formats or components.
3605 </para>
3606 </section>
3607 </section>
3608 </section>
3609</chapter>
3610<!--
3611vim: expandtab tw=80 ts=4
3612-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-03-20 13:43:28 +0000