summaryrefslogtreecommitdiffstats
path: root/documentation/overview-manual
diff options
context:
space:
mode:
authorNicolas Dechesne <nicolas.dechesne@linaro.org>2020-10-05 16:30:32 +0200
committerRichard Purdie <richard.purdie@linuxfoundation.org>2020-10-06 13:56:17 +0100
commit43d07a285181e64c30d98d10ff93ef50391efe59 (patch)
tree78918fc94d55d44d35e1e3e61c7a6fccc28bca24 /documentation/overview-manual
parent1fd9c4b2c0ae927df29f7a0d34c3e595bcf48e89 (diff)
downloadpoky-43d07a285181e64c30d98d10ff93ef50391efe59.tar.gz
sphinx: remove DocBook files
The Yocto Project documentation was migrated to Sphinx. Let's remove the deprecated DocBook files. (From yocto-docs rev: 28fb0e63b2fbfd6426b00498bf2682bb53fdd862) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/overview-manual')
-rw-r--r--documentation/overview-manual/overview-manual-concepts.xml3235
-rw-r--r--documentation/overview-manual/overview-manual-customization.xsl29
-rw-r--r--documentation/overview-manual/overview-manual-development-environment.xml954
-rw-r--r--documentation/overview-manual/overview-manual-intro.xml113
-rw-r--r--documentation/overview-manual/overview-manual-style.css990
-rw-r--r--documentation/overview-manual/overview-manual-yp-intro.xml1333
-rwxr-xr-xdocumentation/overview-manual/overview-manual.xml130
7 files changed, 0 insertions, 6784 deletions
diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml
deleted file mode 100644
index 58b64bd269..0000000000
--- a/documentation/overview-manual/overview-manual-concepts.xml
+++ /dev/null
@@ -1,3235 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id=' overview-manual-concepts'>
7<title>Yocto Project Concepts</title>
8
9 <para>
10 This chapter provides explanations for Yocto Project concepts that
11 go beyond the surface of "how-to" information and reference (or
12 look-up) material.
13 Concepts such as components, the
14 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
15 workflow, cross-development toolchains, shared state cache, and so
16 forth are explained.
17 </para>
18
19 <section id='yocto-project-components'>
20 <title>Yocto Project Components</title>
21
22 <para>
23 The
24 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
25 task executor together with various types of configuration files
26 form the
27 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>.
28 This section overviews these components by describing their use and
29 how they interact.
30 </para>
31
32 <para>
33 BitBake handles the parsing and execution of the data files.
34 The data itself is of various types:
35 <itemizedlist>
36 <listitem><para>
37 <emphasis>Recipes:</emphasis>
38 Provides details about particular pieces of software.
39 </para></listitem>
40 <listitem><para>
41 <emphasis>Class Data:</emphasis>
42 Abstracts common build information (e.g. how to build a
43 Linux kernel).
44 </para></listitem>
45 <listitem><para>
46 <emphasis>Configuration Data:</emphasis>
47 Defines machine-specific settings, policy decisions, and
48 so forth.
49 Configuration data acts as the glue to bind everything
50 together.
51 </para></listitem>
52 </itemizedlist>
53 </para>
54
55 <para>
56 BitBake knows how to combine multiple data sources together and
57 refers to each data source as a layer.
58 For information on layers, see the
59 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
60 section of the Yocto Project Development Tasks Manual.
61 </para>
62
63 <para>
64 Following are some brief details on these core components.
65 For additional information on how these components interact during
66 a build, see the
67 "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>"
68 section.
69 </para>
70
71 <section id='usingpoky-components-bitbake'>
72 <title>BitBake</title>
73
74 <para>
75 BitBake is the tool at the heart of the
76 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
77 and is responsible for parsing the
78 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>,
79 generating a list of tasks from it, and then executing those
80 tasks.
81 </para>
82
83 <para>
84 This section briefly introduces BitBake.
85 If you want more information on BitBake, see the
86 <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>.
87 </para>
88
89 <para>
90 To see a list of the options BitBake supports, use either of
91 the following commands:
92 <literallayout class='monospaced'>
93 $ bitbake -h
94 $ bitbake --help
95 </literallayout>
96 </para>
97
98 <para>
99 The most common usage for BitBake is
100 <filename>bitbake <replaceable>packagename</replaceable></filename>,
101 where <filename>packagename</filename> is the name of the
102 package you want to build (referred to as the "target").
103 The target often equates to the first part of a recipe's
104 filename (e.g. "foo" for a recipe named
105 <filename>foo_1.3.0-r0.bb</filename>).
106 So, to process the
107 <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you
108 might type the following:
109 <literallayout class='monospaced'>
110 $ bitbake matchbox-desktop
111 </literallayout>
112 Several different versions of
113 <filename>matchbox-desktop</filename> might exist.
114 BitBake chooses the one selected by the distribution
115 configuration.
116 You can get more details about how BitBake chooses between
117 different target versions and providers in the
118 "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>"
119 section of the BitBake User Manual.
120 </para>
121
122 <para>
123 BitBake also tries to execute any dependent tasks first.
124 So for example, before building
125 <filename>matchbox-desktop</filename>, BitBake would build a
126 cross compiler and <filename>glibc</filename> if they had not
127 already been built.
128 </para>
129
130 <para>
131 A useful BitBake option to consider is the
132 <filename>-k</filename> or <filename>--continue</filename>
133 option.
134 This option instructs BitBake to try and continue processing
135 the job as long as possible even after encountering an error.
136 When an error occurs, the target that failed and those that
137 depend on it cannot be remade.
138 However, when you use this option other dependencies can
139 still be processed.
140 </para>
141 </section>
142
143 <section id='overview-components-recipes'>
144 <title>Recipes</title>
145
146 <para>
147 Files that have the <filename>.bb</filename> suffix are
148 "recipes" files.
149 In general, a recipe contains information about a single piece
150 of software.
151 This information includes the location from which to download
152 the unaltered source, any source patches to be applied to that
153 source (if needed), which special configuration options to
154 apply, how to compile the source files, and how to package the
155 compiled output.
156 </para>
157
158 <para>
159 The term "package" is sometimes used to refer to recipes.
160 However, since the word "package" is used for the packaged
161 output from the OpenEmbedded build system (i.e.
162 <filename>.ipk</filename> or <filename>.deb</filename> files),
163 this document avoids using the term "package" when referring
164 to recipes.
165 </para>
166 </section>
167
168 <section id='overview-components-classes'>
169 <title>Classes</title>
170
171 <para>
172 Class files (<filename>.bbclass</filename>) contain information
173 that is useful to share between recipes files.
174 An example is the
175 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
176 class, which contains common settings for any application that
177 Autotools uses.
178 The
179 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
180 chapter in the Yocto Project Reference Manual provides
181 details about classes and how to use them.
182 </para>
183 </section>
184
185 <section id='overview-components-configurations'>
186 <title>Configurations</title>
187
188 <para>
189 The configuration files (<filename>.conf</filename>) define
190 various configuration variables that govern the OpenEmbedded
191 build process.
192 These files fall into several areas that define machine
193 configuration options, distribution configuration options,
194 compiler tuning options, general common configuration options,
195 and user configuration options in
196 <filename>conf/local.conf</filename>, which is found in the
197 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
198 </para>
199 </section>
200 </section>
201
202 <section id='overview-layers'>
203 <title>Layers</title>
204
205 <para>
206 Layers are repositories that contain related metadata (i.e.
207 sets of instructions) that tell the OpenEmbedded build system how
208 to build a target.
209 Yocto Project's
210 <link linkend='the-yocto-project-layer-model'>layer model</link>
211 facilitates collaboration, sharing, customization, and reuse
212 within the Yocto Project development environment.
213 Layers logically separate information for your project.
214 For example, you can use a layer to hold all the configurations
215 for a particular piece of hardware.
216 Isolating hardware-specific configurations allows you to share
217 other metadata by using a different layer where that metadata
218 might be common across several pieces of hardware.
219 </para>
220
221 <para>
222 Many layers exist that work in the Yocto Project development
223 environment.
224 The
225 <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project Curated Layer Index</ulink>
226 and
227 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>
228 both contain layers from which you can use or leverage.
229 </para>
230
231 <para>
232 By convention, layers in the Yocto Project follow a specific form.
233 Conforming to a known structure allows BitBake to make assumptions
234 during builds on where to find types of metadata.
235 You can find procedures and learn about tools (i.e.
236 <filename>bitbake-layers</filename>) for creating layers suitable
237 for the Yocto Project in the
238 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
239 section of the Yocto Project Development Tasks Manual.
240 </para>
241 </section>
242
243 <section id="openembedded-build-system-build-concepts">
244 <title>OpenEmbedded Build System Concepts</title>
245
246 <para>
247 This section takes a more detailed look inside the build
248 process used by the
249 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
250 which is the build system specific to the Yocto Project.
251 At the heart of the build system is BitBake, the task executor.
252 </para>
253
254 <para>
255 The following diagram represents the high-level workflow of a
256 build.
257 The remainder of this section expands on the fundamental input,
258 output, process, and metadata logical blocks that make up the
259 workflow.
260 </para>
261
262 <para id='general-workflow-figure'>
263 <imagedata fileref="figures/YP-flow-diagram.png" format="PNG" align='center' width="8in"/>
264 </para>
265
266 <para>
267 In general, the build's workflow consists of several functional
268 areas:
269 <itemizedlist>
270 <listitem><para>
271 <emphasis>User Configuration:</emphasis>
272 metadata you can use to control the build process.
273 </para></listitem>
274 <listitem><para>
275 <emphasis>Metadata Layers:</emphasis>
276 Various layers that provide software, machine, and
277 distro metadata.
278 </para></listitem>
279 <listitem><para>
280 <emphasis>Source Files:</emphasis>
281 Upstream releases, local projects, and SCMs.
282 </para></listitem>
283 <listitem><para>
284 <emphasis>Build System:</emphasis>
285 Processes under the control of
286 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>.
287 This block expands on how BitBake fetches source, applies
288 patches, completes compilation, analyzes output for package
289 generation, creates and tests packages, generates images,
290 and generates cross-development tools.
291 </para></listitem>
292 <listitem><para>
293 <emphasis>Package Feeds:</emphasis>
294 Directories containing output packages (RPM, DEB or IPK),
295 which are subsequently used in the construction of an
296 image or Software Development Kit (SDK), produced by the
297 build system.
298 These feeds can also be copied and shared using a web
299 server or other means to facilitate extending or updating
300 existing images on devices at runtime if runtime package
301 management is enabled.
302 </para></listitem>
303 <listitem><para>
304 <emphasis>Images:</emphasis>
305 Images produced by the workflow.
306 </para></listitem>
307 <listitem><para>
308 <emphasis>Application Development SDK:</emphasis>
309 Cross-development tools that are produced along with
310 an image or separately with BitBake.
311 </para></listitem>
312 </itemizedlist>
313 </para>
314
315 <section id="user-configuration">
316 <title>User Configuration</title>
317
318 <para>
319 User configuration helps define the build.
320 Through user configuration, you can tell BitBake the
321 target architecture for which you are building the image,
322 where to store downloaded source, and other build properties.
323 </para>
324
325 <para>
326 The following figure shows an expanded representation of the
327 "User Configuration" box of the
328 <link linkend='general-workflow-figure'>general workflow figure</link>:
329 </para>
330
331 <para>
332 <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" />
333 </para>
334
335 <para>
336 BitBake needs some basic configuration files in order to
337 complete a build.
338 These files are <filename>*.conf</filename> files.
339 The minimally necessary ones reside as example files in the
340 <filename>build/conf</filename> directory of the
341 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
342 For simplicity, this section refers to the Source Directory as
343 the "Poky Directory."
344 </para>
345
346 <para>
347 When you clone the
348 <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>
349 Git repository or you download and unpack a Yocto Project
350 release, you can set up the Source Directory to be named
351 anything you want.
352 For this discussion, the cloned repository uses the default
353 name <filename>poky</filename>.
354 <note>
355 The Poky repository is primarily an aggregation of existing
356 repositories.
357 It is not a canonical upstream source.
358 </note>
359 </para>
360
361 <para>
362 The <filename>meta-poky</filename> layer inside Poky contains
363 a <filename>conf</filename> directory that has example
364 configuration files.
365 These example files are used as a basis for creating actual
366 configuration files when you source
367 <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>,
368 which is the build environment script.
369 </para>
370
371 <para>
372 Sourcing the build environment script creates a
373 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
374 if one does not already exist.
375 BitBake uses the Build Directory for all its work during
376 builds.
377 The Build Directory has a <filename>conf</filename> directory
378 that contains default versions of your
379 <filename>local.conf</filename> and
380 <filename>bblayers.conf</filename> configuration files.
381 These default configuration files are created only if versions
382 do not already exist in the Build Directory at the time you
383 source the build environment setup script.
384 </para>
385
386 <para>
387 Because the Poky repository is fundamentally an aggregation of
388 existing repositories, some users might be familiar with
389 running the <filename>&OE_INIT_FILE;</filename> script
390 in the context of separate
391 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>
392 and BitBake repositories rather than a single Poky repository.
393 This discussion assumes the script is executed from
394 within a cloned or unpacked version of Poky.
395 </para>
396
397 <para>
398 Depending on where the script is sourced, different
399 sub-scripts are called to set up the Build Directory
400 (Yocto or OpenEmbedded).
401 Specifically, the script
402 <filename>scripts/oe-setup-builddir</filename> inside the
403 poky directory sets up the Build Directory and seeds the
404 directory (if necessary) with configuration files appropriate
405 for the Yocto Project development environment.
406 <note>
407 The <filename>scripts/oe-setup-builddir</filename> script
408 uses the <filename>$TEMPLATECONF</filename> variable to
409 determine which sample configuration files to locate.
410 </note>
411 </para>
412
413 <para>
414 The <filename>local.conf</filename> file provides many
415 basic variables that define a build environment.
416 Here is a list of a few.
417 To see the default configurations in a
418 <filename>local.conf</filename> file created by the build
419 environment script, see the
420 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample'><filename>local.conf.sample</filename></ulink>
421 in the <filename>meta-poky</filename> layer:
422 <itemizedlist>
423 <listitem><para>
424 <emphasis>Target Machine Selection:</emphasis>
425 Controlled by the
426 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>
427 variable.
428 </para></listitem>
429 <listitem><para>
430 <emphasis>Download Directory:</emphasis>
431 Controlled by the
432 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
433 variable.
434 </para></listitem>
435 <listitem><para>
436 <emphasis>Shared State Directory:</emphasis>
437 Controlled by the
438 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
439 variable.
440 </para></listitem>
441 <listitem><para>
442 <emphasis>Build Output:</emphasis>
443 Controlled by the
444 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
445 variable.
446 </para></listitem>
447 <listitem><para>
448 <emphasis>Distribution Policy:</emphasis>
449 Controlled by the
450 <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink>
451 variable.
452 </para></listitem>
453 <listitem><para>
454 <emphasis>Packaging Format:</emphasis>
455 Controlled by the
456 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
457 variable.
458 </para></listitem>
459 <listitem><para>
460 <emphasis>SDK Target Architecture:</emphasis>
461 Controlled by the
462 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>
463 variable.
464 </para></listitem>
465 <listitem><para>
466 <emphasis>Extra Image Packages:</emphasis>
467 Controlled by the
468 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink>
469 variable.
470 </para></listitem>
471 </itemizedlist>
472 <note>
473 Configurations set in the
474 <filename>conf/local.conf</filename> file can also be set
475 in the <filename>conf/site.conf</filename> and
476 <filename>conf/auto.conf</filename> configuration files.
477 </note>
478 </para>
479
480 <para>
481 The <filename>bblayers.conf</filename> file tells BitBake what
482 layers you want considered during the build.
483 By default, the layers listed in this file include layers
484 minimally needed by the build system.
485 However, you must manually add any custom layers you have
486 created.
487 You can find more information on working with the
488 <filename>bblayers.conf</filename> file in the
489 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
490 section in the Yocto Project Development Tasks Manual.
491 </para>
492
493 <para>
494 The files <filename>site.conf</filename> and
495 <filename>auto.conf</filename> are not created by the
496 environment initialization script.
497 If you want the <filename>site.conf</filename> file, you
498 need to create that yourself.
499 The <filename>auto.conf</filename> file is typically created by
500 an autobuilder:
501 <itemizedlist>
502 <listitem><para>
503 <emphasis><filename>site.conf</filename>:</emphasis>
504 You can use the <filename>conf/site.conf</filename>
505 configuration file to configure multiple
506 build directories.
507 For example, suppose you had several build environments
508 and they shared some common features.
509 You can set these default build properties here.
510 A good example is perhaps the packaging format to use
511 through the
512 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
513 variable.</para>
514
515 <para>One useful scenario for using the
516 <filename>conf/site.conf</filename> file is to extend
517 your
518 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
519 variable to include the path to a
520 <filename>conf/site.conf</filename>.
521 Then, when BitBake looks for Metadata using
522 <filename>BBPATH</filename>, it finds the
523 <filename>conf/site.conf</filename> file and applies
524 your common configurations found in the file.
525 To override configurations in a particular build
526 directory, alter the similar configurations within
527 that build directory's
528 <filename>conf/local.conf</filename> file.
529 </para></listitem>
530 <listitem><para>
531 <emphasis><filename>auto.conf</filename>:</emphasis>
532 The file is usually created and written to by
533 an autobuilder.
534 The settings put into the file are typically the
535 same as you would find in the
536 <filename>conf/local.conf</filename> or the
537 <filename>conf/site.conf</filename> files.
538 </para></listitem>
539 </itemizedlist>
540 </para>
541
542 <para>
543 You can edit all configuration files to further define
544 any particular build environment.
545 This process is represented by the "User Configuration Edits"
546 box in the figure.
547 </para>
548
549 <para>
550 When you launch your build with the
551 <filename>bitbake <replaceable>target</replaceable></filename>
552 command, BitBake sorts out the configurations to ultimately
553 define your build environment.
554 It is important to understand that the
555 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
556 reads the configuration files in a specific order:
557 <filename>site.conf</filename>, <filename>auto.conf</filename>,
558 and <filename>local.conf</filename>.
559 And, the build system applies the normal assignment statement
560 rules as described in the
561 "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>"
562 chapter of the BitBake User Manual.
563 Because the files are parsed in a specific order, variable
564 assignments for the same variable could be affected.
565 For example, if the <filename>auto.conf</filename> file and
566 the <filename>local.conf</filename> set
567 <replaceable>variable1</replaceable> to different values,
568 because the build system parses <filename>local.conf</filename>
569 after <filename>auto.conf</filename>,
570 <replaceable>variable1</replaceable> is assigned the value from
571 the <filename>local.conf</filename> file.
572 </para>
573 </section>
574
575 <section id="metadata-machine-configuration-and-policy-configuration">
576 <title>Metadata, Machine Configuration, and Policy Configuration</title>
577
578 <para>
579 The previous section described the user configurations that
580 define BitBake's global behavior.
581 This section takes a closer look at the layers the build system
582 uses to further control the build.
583 These layers provide Metadata for the software, machine, and
584 policies.
585 </para>
586
587 <para>
588 In general, three types of layer input exists.
589 You can see them below the "User Configuration" box in the
590 <link linkend='general-workflow-figure'>general workflow figure</link>:
591 <itemizedlist>
592 <listitem><para>
593 <emphasis>Metadata (<filename>.bb</filename> + Patches):</emphasis>
594 Software layers containing user-supplied recipe files,
595 patches, and append files.
596 A good example of a software layer might be the
597 <ulink url='https://github.com/meta-qt5/meta-qt5'><filename>meta-qt5</filename></ulink>
598 layer from the
599 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>.
600 This layer is for version 5.0 of the popular
601 <ulink url='https://wiki.qt.io/About_Qt'>Qt</ulink>
602 cross-platform application development framework for
603 desktop, embedded and mobile.
604 </para></listitem>
605 <listitem><para>
606 <emphasis>Machine BSP Configuration:</emphasis>
607 Board Support Package (BSP) layers (i.e. "BSP Layer"
608 in the following figure) providing machine-specific
609 configurations.
610 This type of information is specific to a particular
611 target architecture.
612 A good example of a BSP layer from the
613 <link linkend='gs-reference-distribution-poky'>Poky Reference Distribution</link>
614 is the
615 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink>
616 layer.
617 </para></listitem>
618 <listitem><para>
619 <emphasis>Policy Configuration:</emphasis>
620 Distribution Layers (i.e. "Distro Layer" in the
621 following figure) providing top-level or general
622 policies for the images or SDKs being built for a
623 particular distribution.
624 For example, in the Poky Reference Distribution the
625 distro layer is the
626 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink>
627 layer.
628 Within the distro layer is a
629 <filename>conf/distro</filename> directory that
630 contains distro configuration files (e.g.
631 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink>
632 that contain many policy configurations for the
633 Poky distribution.
634 </para></listitem>
635 </itemizedlist>
636 </para>
637
638 <para>
639 The following figure shows an expanded representation of
640 these three layers from the
641 <link linkend='general-workflow-figure'>general workflow figure</link>:
642 </para>
643
644 <para>
645 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="8in" />
646 </para>
647
648 <para>
649 In general, all layers have a similar structure.
650 They all contain a licensing file
651 (e.g. <filename>COPYING.MIT</filename>) if the layer is to be
652 distributed, a <filename>README</filename> file as good
653 practice and especially if the layer is to be distributed, a
654 configuration directory, and recipe directories.
655 You can learn about the general structure for layers used with
656 the Yocto Project in the
657 "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-your-own-layer'>Creating Your Own Layer</ulink>"
658 section in the Yocto Project Development Tasks Manual.
659 For a general discussion on layers and the many layers from
660 which you can draw, see the
661 "<link linkend='overview-layers'>Layers</link>" and
662 "<link linkend='the-yocto-project-layer-model'>The Yocto Project Layer Model</link>"
663 sections both earlier in this manual.
664 </para>
665
666 <para>
667 If you explored the previous links, you discovered some
668 areas where many layers that work with the Yocto Project
669 exist.
670 The
671 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
672 also shows layers categorized under "Yocto Metadata Layers."
673 <note>
674 Layers exist in the Yocto Project Source Repositories that
675 cannot be found in the OpenEmbedded Layer Index.
676 These layers are either deprecated or experimental
677 in nature.
678 </note>
679 </para>
680
681 <para>
682 BitBake uses the <filename>conf/bblayers.conf</filename> file,
683 which is part of the user configuration, to find what layers it
684 should be using as part of the build.
685 </para>
686
687 <section id="distro-layer">
688 <title>Distro Layer</title>
689
690 <para>
691 The distribution layer provides policy configurations for
692 your distribution.
693 Best practices dictate that you isolate these types of
694 configurations into their own layer.
695 Settings you provide in
696 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
697 similar settings that BitBake finds in your
698 <filename>conf/local.conf</filename> file in the Build
699 Directory.
700 </para>
701
702 <para>
703 The following list provides some explanation and references
704 for what you typically find in the distribution layer:
705 <itemizedlist>
706 <listitem><para>
707 <emphasis>classes:</emphasis>
708 Class files (<filename>.bbclass</filename>) hold
709 common functionality that can be shared among
710 recipes in the distribution.
711 When your recipes inherit a class, they take on the
712 settings and functions for that class.
713 You can read more about class files in the
714 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
715 chapter of the Yocto Reference Manual.
716 </para></listitem>
717 <listitem><para>
718 <emphasis>conf:</emphasis>
719 This area holds configuration files for the
720 layer (<filename>conf/layer.conf</filename>),
721 the distribution
722 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
723 and any distribution-wide include files.
724 </para></listitem>
725 <listitem><para>
726 <emphasis>recipes-*:</emphasis>
727 Recipes and append files that affect common
728 functionality across the distribution.
729 This area could include recipes and append files
730 to add distribution-specific configuration,
731 initialization scripts, custom image recipes,
732 and so forth.
733 Examples of <filename>recipes-*</filename>
734 directories are <filename>recipes-core</filename>
735 and <filename>recipes-extra</filename>.
736 Hierarchy and contents within a
737 <filename>recipes-*</filename> directory can vary.
738 Generally, these directories contain recipe files
739 (<filename>*.bb</filename>), recipe append files
740 (<filename>*.bbappend</filename>), directories
741 that are distro-specific for configuration files,
742 and so forth.
743 </para></listitem>
744 </itemizedlist>
745 </para>
746 </section>
747
748 <section id="bsp-layer">
749 <title>BSP Layer</title>
750
751 <para>
752 The BSP Layer provides machine configurations that
753 target specific hardware.
754 Everything in this layer is specific to the machine for
755 which you are building the image or the SDK.
756 A common structure or form is defined for BSP layers.
757 You can learn more about this structure in the
758 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
759 <note>
760 In order for a BSP layer to be considered compliant
761 with the Yocto Project, it must meet some structural
762 requirements.
763 </note>
764 </para>
765
766 <para>
767 The BSP Layer's configuration directory contains
768 configuration files for the machine
769 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>)
770 and, of course, the layer
771 (<filename>conf/layer.conf</filename>).
772 </para>
773
774 <para>
775 The remainder of the layer is dedicated to specific recipes
776 by function: <filename>recipes-bsp</filename>,
777 <filename>recipes-core</filename>,
778 <filename>recipes-graphics</filename>,
779 <filename>recipes-kernel</filename>, and so forth.
780 Metadata can exist for multiple formfactors, graphics
781 support systems, and so forth.
782 <note>
783 While the figure shows several
784 <filename>recipes-*</filename> directories, not all
785 these directories appear in all BSP layers.
786 </note>
787 </para>
788 </section>
789
790 <section id="software-layer">
791 <title>Software Layer</title>
792
793 <para>
794 The software layer provides the Metadata for additional
795 software packages used during the build.
796 This layer does not include Metadata that is specific to
797 the distribution or the machine, which are found in their
798 respective layers.
799 </para>
800
801 <para>
802 This layer contains any recipes, append files, and
803 patches, that your project needs.
804 </para>
805 </section>
806 </section>
807
808 <section id="sources-dev-environment">
809 <title>Sources</title>
810
811 <para>
812 In order for the OpenEmbedded build system to create an
813 image or any target, it must be able to access source files.
814 The
815 <link linkend='general-workflow-figure'>general workflow figure</link>
816 represents source files using the "Upstream Project Releases",
817 "Local Projects", and "SCMs (optional)" boxes.
818 The figure represents mirrors, which also play a role in
819 locating source files, with the "Source Materials" box.
820 </para>
821
822 <para>
823 The method by which source files are ultimately organized is
824 a function of the project.
825 For example, for released software, projects tend to use
826 tarballs or other archived files that can capture the
827 state of a release guaranteeing that it is statically
828 represented.
829 On the other hand, for a project that is more dynamic or
830 experimental in nature, a project might keep source files in a
831 repository controlled by a Source Control Manager (SCM) such as
832 Git.
833 Pulling source from a repository allows you to control
834 the point in the repository (the revision) from which you
835 want to build software.
836 Finally, a combination of the two might exist, which would
837 give the consumer a choice when deciding where to get
838 source files.
839 </para>
840
841 <para>
842 BitBake uses the
843 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
844 variable to point to source files regardless of their location.
845 Each recipe must have a <filename>SRC_URI</filename> variable
846 that points to the source.
847 </para>
848
849 <para>
850 Another area that plays a significant role in where source
851 files come from is pointed to by the
852 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
853 variable.
854 This area is a cache that can hold previously downloaded
855 source.
856 You can also instruct the OpenEmbedded build system to create
857 tarballs from Git repositories, which is not the default
858 behavior, and store them in the <filename>DL_DIR</filename>
859 by using the
860 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
861 variable.
862 </para>
863
864 <para>
865 Judicious use of a <filename>DL_DIR</filename> directory can
866 save the build system a trip across the Internet when looking
867 for files.
868 A good method for using a download directory is to have
869 <filename>DL_DIR</filename> point to an area outside of your
870 Build Directory.
871 Doing so allows you to safely delete the Build Directory
872 if needed without fear of removing any downloaded source file.
873 </para>
874
875 <para>
876 The remainder of this section provides a deeper look into the
877 source files and the mirrors.
878 Here is a more detailed look at the source file area of the
879 <link linkend='general-workflow-figure'>general workflow figure</link>:
880 </para>
881
882 <para>
883 <imagedata fileref="figures/source-input.png" width="6in" depth="6in" align="center" />
884 </para>
885
886 <section id='upstream-project-releases'>
887 <title>Upstream Project Releases</title>
888
889 <para>
890 Upstream project releases exist anywhere in the form of an
891 archived file (e.g. tarball or zip file).
892 These files correspond to individual recipes.
893 For example, the figure uses specific releases each for
894 BusyBox, Qt, and Dbus.
895 An archive file can be for any released product that can be
896 built using a recipe.
897 </para>
898 </section>
899
900 <section id='local-projects'>
901 <title>Local Projects</title>
902
903 <para>
904 Local projects are custom bits of software the user
905 provides.
906 These bits reside somewhere local to a project - perhaps
907 a directory into which the user checks in items (e.g.
908 a local directory containing a development source tree
909 used by the group).
910 </para>
911
912 <para>
913 The canonical method through which to include a local
914 project is to use the
915 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
916 class to include that local project.
917 You use either the <filename>local.conf</filename> or a
918 recipe's append file to override or set the
919 recipe to point to the local directory on your disk to pull
920 in the whole source tree.
921 </para>
922 </section>
923
924 <section id='scms'>
925 <title>Source Control Managers (Optional)</title>
926
927 <para>
928 Another place from which the build system can get source
929 files is with
930 <ulink url='&YOCTO_DOCS_BB_URL;#bb-fetchers'>fetchers</ulink>
931 employing various Source Control Managers (SCMs) such as
932 Git or Subversion.
933 In such cases, a repository is cloned or checked out.
934 The
935 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
936 task inside BitBake uses
937 the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
938 variable and the argument's prefix to determine the correct
939 fetcher module.
940 <note>
941 For information on how to have the OpenEmbedded build
942 system generate tarballs for Git repositories and place
943 them in the
944 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
945 directory, see the
946 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
947 variable in the Yocto Project Reference Manual.
948 </note>
949 </para>
950
951 <para>
952 When fetching a repository, BitBake uses the
953 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
954 variable to determine the specific revision from which to
955 build.
956 </para>
957 </section>
958
959 <section id='source-mirrors'>
960 <title>Source Mirror(s)</title>
961
962 <para>
963 Two kinds of mirrors exist: pre-mirrors and regular
964 mirrors.
965 The
966 <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink>
967 and
968 <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink>
969 variables point to these, respectively.
970 BitBake checks pre-mirrors before looking upstream for any
971 source files.
972 Pre-mirrors are appropriate when you have a shared
973 directory that is not a directory defined by the
974 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
975 variable.
976 A Pre-mirror typically points to a shared directory that is
977 local to your organization.
978 </para>
979
980 <para>
981 Regular mirrors can be any site across the Internet
982 that is used as an alternative location for source
983 code should the primary site not be functioning for
984 some reason or another.
985 </para>
986 </section>
987 </section>
988
989 <section id="package-feeds-dev-environment">
990 <title>Package Feeds</title>
991
992 <para>
993 When the OpenEmbedded build system generates an image or an
994 SDK, it gets the packages from a package feed area located
995 in the
996 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
997 The
998 <link linkend='general-workflow-figure'>general workflow figure</link>
999 shows this package feeds area in the upper-right corner.
1000 </para>
1001
1002 <para>
1003 This section looks a little closer into the package feeds
1004 area used by the build system.
1005 Here is a more detailed look at the area:
1006 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
1007 </para>
1008
1009 <para>
1010 Package feeds are an intermediary step in the build process.
1011 The OpenEmbedded build system provides classes to generate
1012 different package types, and you specify which classes to
1013 enable through the
1014 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
1015 variable.
1016 Before placing the packages into package feeds,
1017 the build process validates them with generated output quality
1018 assurance checks through the
1019 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink>
1020 class.
1021 </para>
1022
1023 <para>
1024 The package feed area resides in the Build Directory.
1025 The directory the build system uses to temporarily store
1026 packages is determined by a combination of variables and the
1027 particular package manager in use.
1028 See the "Package Feeds" box in the illustration and note the
1029 information to the right of that area.
1030 In particular, the following defines where package files are
1031 kept:
1032 <itemizedlist>
1033 <listitem><para>
1034 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1035 Defined as <filename>tmp/deploy</filename> in the Build
1036 Directory.
1037 </para></listitem>
1038 <listitem><para>
1039 <filename>DEPLOY_DIR_*</filename>:
1040 Depending on the package manager used, the package type
1041 sub-folder.
1042 Given RPM, IPK, or DEB packaging and tarball creation,
1043 the
1044 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>,
1045 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>,
1046 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>,
1047 or
1048 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>,
1049 variables are used, respectively.
1050 </para></listitem>
1051 <listitem><para>
1052 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1053 Defines architecture-specific sub-folders.
1054 For example, packages could exist for the i586 or
1055 qemux86 architectures.
1056 </para></listitem>
1057 </itemizedlist>
1058 </para>
1059
1060 <para>
1061 BitBake uses the
1062 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1063 tasks to generate packages and place them into the package
1064 holding area (e.g. <filename>do_package_write_ipk</filename>
1065 for IPK packages).
1066 See the
1067 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>",
1068 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>",
1069 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>",
1070 and
1071 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>"
1072 sections in the Yocto Project Reference Manual
1073 for additional information.
1074 As an example, consider a scenario where an IPK packaging
1075 manager is being used and package architecture support for
1076 both i586 and qemux86 exist.
1077 Packages for the i586 architecture are placed in
1078 <filename>build/tmp/deploy/ipk/i586</filename>, while packages
1079 for the qemux86 architecture are placed in
1080 <filename>build/tmp/deploy/ipk/qemux86</filename>.
1081 </para>
1082 </section>
1083
1084 <section id='bitbake-dev-environment'>
1085 <title>BitBake</title>
1086
1087 <para>
1088 The OpenEmbedded build system uses
1089 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
1090 to produce images and Software Development Kits (SDKs).
1091 You can see from the
1092 <link linkend='general-workflow-figure'>general workflow figure</link>,
1093 the BitBake area consists of several functional areas.
1094 This section takes a closer look at each of those areas.
1095 <note>
1096 Separate documentation exists for the BitBake tool.
1097 See the
1098 <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>
1099 for reference material on BitBake.
1100 </note>
1101 </para>
1102
1103 <section id='source-fetching-dev-environment'>
1104 <title>Source Fetching</title>
1105
1106 <para>
1107 The first stages of building a recipe are to fetch and
1108 unpack the source code:
1109 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
1110 </para>
1111
1112 <para>
1113 The
1114 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
1115 and
1116 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1117 tasks fetch the source files and unpack them into the
1118 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
1119 <note>
1120 For every local file (e.g. <filename>file://</filename>)
1121 that is part of a recipe's
1122 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1123 statement, the OpenEmbedded build system takes a
1124 checksum of the file for the recipe and inserts the
1125 checksum into the signature for the
1126 <filename>do_fetch</filename> task.
1127 If any local file has been modified, the
1128 <filename>do_fetch</filename> task and all tasks that
1129 depend on it are re-executed.
1130 </note>
1131 By default, everything is accomplished in the Build
1132 Directory, which has a defined structure.
1133 For additional general information on the Build Directory,
1134 see the
1135 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>"
1136 section in the Yocto Project Reference Manual.
1137 </para>
1138
1139 <para>
1140 Each recipe has an area in the Build Directory where the
1141 unpacked source code resides.
1142 The
1143 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1144 variable points to this area for a recipe's unpacked source
1145 code.
1146 The name of that directory for any given recipe is defined
1147 from several different variables.
1148 The preceding figure and the following list describe
1149 the Build Directory's hierarchy:
1150 <itemizedlist>
1151 <listitem><para>
1152 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>:
1153 The base directory where the OpenEmbedded build
1154 system performs all its work during the build.
1155 The default base directory is the
1156 <filename>tmp</filename> directory.
1157 </para></listitem>
1158 <listitem><para>
1159 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1160 The architecture of the built package or packages.
1161 Depending on the eventual destination of the
1162 package or packages (i.e. machine architecture,
1163 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>,
1164 SDK, or specific machine),
1165 <filename>PACKAGE_ARCH</filename> varies.
1166 See the variable's description for details.
1167 </para></listitem>
1168 <listitem><para>
1169 <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>:
1170 The operating system of the target device.
1171 A typical value would be "linux" (e.g.
1172 "qemux86-poky-linux").
1173 </para></listitem>
1174 <listitem><para>
1175 <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>:
1176 The name of the recipe used to build the package.
1177 This variable can have multiple meanings.
1178 However, when used in the context of input files,
1179 <filename>PN</filename> represents the the name
1180 of the recipe.
1181 </para></listitem>
1182 <listitem><para>
1183 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>:
1184 The location where the OpenEmbedded build system
1185 builds a recipe (i.e. does the work to create the
1186 package).
1187 <itemizedlist>
1188 <listitem><para>
1189 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1190 The version of the recipe used to build the
1191 package.
1192 </para></listitem>
1193 <listitem><para>
1194 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>:
1195 The revision of the recipe used to build the
1196 package.
1197 </para></listitem>
1198 </itemizedlist>
1199 </para></listitem>
1200 <listitem><para>
1201 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>:
1202 Contains the unpacked source files for a given
1203 recipe.
1204 <itemizedlist>
1205 <listitem><para>
1206 <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>:
1207 The name of the recipe used to build the
1208 package.
1209 The <filename>BPN</filename> variable is
1210 a version of the <filename>PN</filename>
1211 variable but with common prefixes and
1212 suffixes removed.
1213 </para></listitem>
1214 <listitem><para>
1215 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1216 The version of the recipe used to build the
1217 package.
1218 </para></listitem>
1219 </itemizedlist>
1220 </para></listitem>
1221 </itemizedlist>
1222 <note>
1223 In the previous figure, notice that two sample
1224 hierarchies exist: one based on package architecture (i.e.
1225 <filename>PACKAGE_ARCH</filename>) and one based on a
1226 machine (i.e. <filename>MACHINE</filename>).
1227 The underlying structures are identical.
1228 The differentiator being what the OpenEmbedded build
1229 system is using as a build target (e.g. general
1230 architecture, a build host, an SDK, or a specific
1231 machine).
1232 </note>
1233 </para>
1234 </section>
1235
1236 <section id='patching-dev-environment'>
1237 <title>Patching</title>
1238
1239 <para>
1240 Once source code is fetched and unpacked, BitBake locates
1241 patch files and applies them to the source files:
1242 <imagedata fileref="figures/patching.png" align="center" width="7in" depth="6in" />
1243 </para>
1244
1245 <para>
1246 The
1247 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1248 task uses a recipe's
1249 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1250 statements and the
1251 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink>
1252 variable to locate applicable patch files.
1253 </para>
1254
1255 <para>
1256 Default processing for patch files assumes the files have
1257 either <filename>*.patch</filename> or
1258 <filename>*.diff</filename> file types.
1259 You can use <filename>SRC_URI</filename> parameters to
1260 change the way the build system recognizes patch files.
1261 See the
1262 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1263 task for more information.
1264 </para>
1265
1266 <para>
1267 BitBake finds and applies multiple patches for a single
1268 recipe in the order in which it locates the patches.
1269 The <filename>FILESPATH</filename> variable defines the
1270 default set of directories that the build system uses to
1271 search for patch files.
1272 Once found, patches are applied to the recipe's source
1273 files, which are located in the
1274 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1275 directory.
1276 </para>
1277
1278 <para>
1279 For more information on how the source directories are
1280 created, see the
1281 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
1282 section.
1283 For more information on how to create patches and how the
1284 build system processes patches, see the
1285 "<ulink url='&YOCTO_DOCS_DEV_URL;#new-recipe-patching-code'>Patching Code</ulink>"
1286 section in the Yocto Project Development Tasks Manual.
1287 You can also see the
1288 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-modify-to-modify-the-source-of-an-existing-component'>Use <filename>devtool modify</filename> to Modify the Source of an Existing Component</ulink>"
1289 section in the Yocto Project Application Development and
1290 the Extensible Software Development Kit (SDK) manual and
1291 the
1292 "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</ulink>"
1293 section in the Yocto Project Linux Kernel Development
1294 Manual.
1295 </para>
1296 </section>
1297
1298 <section id='configuration-compilation-and-staging-dev-environment'>
1299 <title>Configuration, Compilation, and Staging</title>
1300
1301 <para>
1302 After source code is patched, BitBake executes tasks that
1303 configure and compile the source code.
1304 Once compilation occurs, the files are copied to a holding
1305 area (staged) in preparation for packaging:
1306 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
1307 </para>
1308
1309 <para>
1310 This step in the build process consists of the following
1311 tasks:
1312 <itemizedlist>
1313 <listitem><para>
1314 <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>:
1315 This task sets up the two sysroots in
1316 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>
1317 (i.e. <filename>recipe-sysroot</filename> and
1318 <filename>recipe-sysroot-native</filename>) so that
1319 during the packaging phase the sysroots can contain
1320 the contents of the
1321 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1322 tasks of the recipes on which the recipe
1323 containing the tasks depends.
1324 A sysroot exists for both the target and for the
1325 native binaries, which run on the host system.
1326 </para></listitem>
1327 <listitem><para>
1328 <emphasis><filename>do_configure</filename></emphasis>:
1329 This task configures the source by enabling and
1330 disabling any build-time and configuration options
1331 for the software being built.
1332 Configurations can come from the recipe itself as
1333 well as from an inherited class.
1334 Additionally, the software itself might configure
1335 itself depending on the target for which it is
1336 being built.</para>
1337
1338 <para>The configurations handled by the
1339 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink>
1340 task are specific to configurations for the source
1341 code being built by the recipe.</para>
1342
1343 <para>If you are using the
1344 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1345 class, you can add additional configuration options
1346 by using the
1347 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
1348 or
1349 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
1350 variables.
1351 For information on how this variable works within
1352 that class, see the
1353 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1354 class
1355 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass'>here</ulink>.
1356 </para></listitem>
1357 <listitem><para>
1358 <emphasis><filename>do_compile</filename></emphasis>:
1359 Once a configuration task has been satisfied,
1360 BitBake compiles the source using the
1361 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink>
1362 task.
1363 Compilation occurs in the directory pointed to by
1364 the
1365 <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink>
1366 variable.
1367 Realize that the <filename>B</filename> directory
1368 is, by default, the same as the
1369 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1370 directory.
1371 </para></listitem>
1372 <listitem><para>
1373 <emphasis><filename>do_install</filename></emphasis>:
1374 After compilation completes, BitBake executes the
1375 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1376 task.
1377 This task copies files from the
1378 <filename>B</filename> directory and places them
1379 in a holding area pointed to by the
1380 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1381 variable.
1382 Packaging occurs later using files from this
1383 holding directory.
1384 </para></listitem>
1385 </itemizedlist>
1386 </para>
1387 </section>
1388
1389 <section id='package-splitting-dev-environment'>
1390 <title>Package Splitting</title>
1391
1392 <para>
1393 After source code is configured, compiled, and staged, the
1394 build system analyzes the results and splits the output
1395 into packages:
1396 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
1397 </para>
1398
1399 <para>
1400 The
1401 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
1402 and
1403 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
1404 tasks combine to analyze the files found in the
1405 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1406 directory and split them into subsets based on available
1407 packages and files.
1408 Analysis involves the following as well as other items:
1409 splitting out debugging symbols, looking at shared library
1410 dependencies between packages, and looking at package
1411 relationships.
1412 </para>
1413
1414 <para>
1415 The <filename>do_packagedata</filename> task creates
1416 package metadata based on the analysis such that the
1417 build system can generate the final packages.
1418 The
1419 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1420 task stages (copies) a subset of the files installed by
1421 the
1422 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1423 task into the appropriate sysroot.
1424 Working, staged, and intermediate results of the analysis
1425 and package splitting process use several areas:
1426 <itemizedlist>
1427 <listitem><para>
1428 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>:
1429 The destination directory
1430 (i.e. <filename>package</filename>) for packages
1431 before they are split into individual packages.
1432 </para></listitem>
1433 <listitem><para>
1434 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>:
1435 A temporary work area (i.e.
1436 <filename>pkgdata</filename>) used by the
1437 <filename>do_package</filename> task to save
1438 package metadata.
1439 </para></listitem>
1440 <listitem><para>
1441 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>:
1442 The parent directory (i.e.
1443 <filename>packages-split</filename>) for packages
1444 after they have been split.
1445 </para></listitem>
1446 <listitem><para>
1447 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>:
1448 A shared, global-state directory that holds
1449 packaging metadata generated during the packaging
1450 process.
1451 The packaging process copies metadata from
1452 <filename>PKGDESTWORK</filename> to the
1453 <filename>PKGDATA_DIR</filename> area where it
1454 becomes globally available.
1455 </para></listitem>
1456 <listitem><para>
1457 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></ulink>:
1458 The path for the sysroot for the system on which
1459 a component is built to run (i.e.
1460 <filename>recipe-sysroot</filename>).
1461 </para></listitem>
1462 <listitem><para>
1463 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_NATIVE'><filename>STAGING_DIR_NATIVE</filename></ulink>:
1464 The path for the sysroot used when building
1465 components for the build host (i.e.
1466 <filename>recipe-sysroot-native</filename>).
1467 </para></listitem>
1468 <listitem><para>
1469 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_TARGET'><filename>STAGING_DIR_TARGET</filename></ulink>:
1470 The path for the sysroot used when a component that
1471 is built to execute on a system and it generates
1472 code for yet another machine (e.g. cross-canadian
1473 recipes).
1474 </para></listitem>
1475 </itemizedlist>
1476 The
1477 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>
1478 variable defines the files that go into each package in
1479 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>.
1480 If you want details on how this is accomplished, you can
1481 look at
1482 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass'><filename>package.bbclass</filename></ulink>.
1483 </para>
1484
1485 <para>
1486 Depending on the type of packages being created (RPM, DEB,
1487 or IPK), the
1488 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1489 task creates the actual packages and places them in the
1490 Package Feed area, which is
1491 <filename>${TMPDIR}/deploy</filename>.
1492 You can see the
1493 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
1494 section for more detail on that part of the build process.
1495 <note>
1496 Support for creating feeds directly from the
1497 <filename>deploy/*</filename> directories does not
1498 exist.
1499 Creating such feeds usually requires some kind of feed
1500 maintenance mechanism that would upload the new
1501 packages into an official package feed (e.g. the
1502 Ångström distribution).
1503 This functionality is highly distribution-specific
1504 and thus is not provided out of the box.
1505 </note>
1506 </para>
1507 </section>
1508
1509 <section id='image-generation-dev-environment'>
1510 <title>Image Generation</title>
1511
1512 <para>
1513 Once packages are split and stored in the Package Feeds
1514 area, the build system uses BitBake to generate the root
1515 filesystem image:
1516 <imagedata fileref="figures/image-generation.png" align="center" width="7.5in" depth="7.5in" />
1517 </para>
1518
1519 <para>
1520 The image generation process consists of several stages and
1521 depends on several tasks and variables.
1522 The
1523 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>
1524 task creates the root filesystem (file and directory
1525 structure) for an image.
1526 This task uses several key variables to help create the
1527 list of packages to actually install:
1528 <itemizedlist>
1529 <listitem><para>
1530 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>:
1531 Lists out the base set of packages from which to
1532 install from the Package Feeds area.
1533 </para></listitem>
1534 <listitem><para>
1535 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>:
1536 Specifies packages that should not be installed
1537 into the image.
1538 </para></listitem>
1539 <listitem><para>
1540 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>:
1541 Specifies features to include in the image.
1542 Most of these features map to additional packages
1543 for installation.
1544 </para></listitem>
1545 <listitem><para>
1546 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>:
1547 Specifies the package backend (e.g. RPM, DEB, or
1548 IPK) to use and consequently helps determine where
1549 to locate packages within the Package Feeds area.
1550 </para></listitem>
1551 <listitem><para>
1552 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>:
1553 Determines the language(s) for which additional
1554 language support packages are installed.
1555 </para></listitem>
1556 <listitem><para>
1557 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>:
1558 The final list of packages passed to the package
1559 manager for installation into the image.
1560 </para></listitem>
1561 </itemizedlist>
1562 </para>
1563
1564 <para>
1565 With
1566 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink>
1567 pointing to the location of the filesystem under
1568 construction and the <filename>PACKAGE_INSTALL</filename>
1569 variable providing the final list of packages to install,
1570 the root file system is created.
1571 </para>
1572
1573 <para>
1574 Package installation is under control of the package
1575 manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of
1576 whether or not package management is enabled for the
1577 target.
1578 At the end of the process, if package management is not
1579 enabled for the target, the package manager's data files
1580 are deleted from the root filesystem.
1581 As part of the final stage of package installation,
1582 post installation scripts that are part of the packages
1583 are run.
1584 Any scripts that fail to run on the build host are run on
1585 the target when the target system is first booted.
1586 If you are using a
1587 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
1588 all the post installation scripts must succeed on the
1589 build host during the package installation phase since the
1590 root filesystem on the target is read-only.
1591 </para>
1592
1593 <para>
1594 The final stages of the <filename>do_rootfs</filename> task
1595 handle post processing.
1596 Post processing includes creation of a manifest file and
1597 optimizations.
1598 </para>
1599
1600 <para>
1601 The manifest file (<filename>.manifest</filename>) resides
1602 in the same directory as the root filesystem image.
1603 This file lists out, line-by-line, the installed packages.
1604 The manifest file is useful for the
1605 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink>
1606 class, for example, to determine whether or not to run
1607 specific tests.
1608 See the
1609 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink>
1610 variable for additional information.
1611 </para>
1612
1613 <para>
1614 Optimizing processes that are run across the image include
1615 <filename>mklibs</filename>, <filename>prelink</filename>,
1616 and any other post-processing commands as defined by the
1617 <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink>
1618 variable.
1619 The <filename>mklibs</filename> process optimizes the size
1620 of the libraries, while the <filename>prelink</filename>
1621 process optimizes the dynamic linking of shared libraries
1622 to reduce start up time of executables.
1623 </para>
1624
1625 <para>
1626 After the root filesystem is built, processing begins on
1627 the image through the
1628 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink>
1629 task.
1630 The build system runs any pre-processing commands as
1631 defined by the
1632 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink>
1633 variable.
1634 This variable specifies a list of functions to call before
1635 the build system creates the final image output files.
1636 </para>
1637
1638 <para>
1639 The build system dynamically creates
1640 <filename>do_image_*</filename> tasks as needed, based
1641 on the image types specified in the
1642 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
1643 variable.
1644 The process turns everything into an image file or a set of
1645 image files and can compress the root filesystem image to
1646 reduce the overall size of the image.
1647 The formats used for the root filesystem depend on the
1648 <filename>IMAGE_FSTYPES</filename> variable.
1649 Compression depends on whether the formats support
1650 compression.
1651 </para>
1652
1653 <para>
1654 As an example, a dynamically created task when creating a
1655 particular image <replaceable>type</replaceable> would
1656 take the following form:
1657 <literallayout class='monospaced'>
1658 do_image_<replaceable>type</replaceable>
1659 </literallayout>
1660 So, if the <replaceable>type</replaceable> as specified by
1661 the <filename>IMAGE_FSTYPES</filename> were
1662 <filename>ext4</filename>, the dynamically generated task
1663 would be as follows:
1664 <literallayout class='monospaced'>
1665 do_image_ext4
1666 </literallayout>
1667 </para>
1668
1669 <para>
1670 The final task involved in image creation is the
1671 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink>
1672 task.
1673 This task completes the image by applying any image
1674 post processing as defined through the
1675 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink>
1676 variable.
1677 The variable specifies a list of functions to call once the
1678 build system has created the final image output files.
1679 <note>
1680 The entire image generation process is run under
1681 <link linkend='fakeroot-and-pseudo'>Pseudo</link>.
1682 Running under Pseudo ensures that the files in the
1683 root filesystem have correct ownership.
1684 </note>
1685 </para>
1686 </section>
1687
1688 <section id='sdk-generation-dev-environment'>
1689 <title>SDK Generation</title>
1690
1691 <para>
1692 The OpenEmbedded build system uses BitBake to generate the
1693 Software Development Kit (SDK) installer scripts for both
1694 the standard SDK and the extensible SDK (eSDK):
1695 </para>
1696
1697 <para>
1698 <imagedata fileref="figures/sdk-generation.png" width="9in" align="center" />
1699 <note>
1700 For more information on the cross-development toolchain
1701 generation, see the
1702 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1703 section.
1704 For information on advantages gained when building a
1705 cross-development toolchain using the
1706 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1707 task, see the
1708 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
1709 section in the Yocto Project Application Development
1710 and the Extensible Software Development Kit (eSDK)
1711 manual.
1712 </note>
1713 </para>
1714
1715 <para>
1716 Like image generation, the SDK script process consists of
1717 several stages and depends on many variables.
1718 The
1719 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1720 and
1721 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk_ext'><filename>do_populate_sdk_ext</filename></ulink>
1722 tasks use these key variables to help create the list of
1723 packages to actually install.
1724 For information on the variables listed in the figure,
1725 see the
1726 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1727 section.
1728 </para>
1729
1730 <para>
1731 The <filename>do_populate_sdk</filename> task helps create
1732 the standard SDK and handles two parts: a target part and a
1733 host part.
1734 The target part is the part built for the target hardware
1735 and includes libraries and headers.
1736 The host part is the part of the SDK that runs on the
1737 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>.
1738 </para>
1739
1740 <para>
1741 The <filename>do_populate_sdk_ext</filename> task helps
1742 create the extensible SDK and handles host and target parts
1743 differently than its counter part does for the standard SDK.
1744 For the extensible SDK, the task encapsulates the build
1745 system, which includes everything needed (host and target)
1746 for the SDK.
1747 </para>
1748
1749 <para>
1750 Regardless of the type of SDK being constructed, the
1751 tasks perform some cleanup after which a cross-development
1752 environment setup script and any needed configuration files
1753 are created.
1754 The final output is the Cross-development
1755 toolchain installation script (<filename>.sh</filename>
1756 file), which includes the environment setup script.
1757 </para>
1758 </section>
1759
1760 <section id='stamp-files-and-the-rerunning-of-tasks'>
1761 <title>Stamp Files and the Rerunning of Tasks</title>
1762
1763 <para>
1764 For each task that completes successfully, BitBake writes a
1765 stamp file into the
1766 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink>
1767 directory.
1768 The beginning of the stamp file's filename is determined
1769 by the
1770 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
1771 variable, and the end of the name consists of the task's
1772 name and current
1773 <link linkend='overview-checksums'>input checksum</link>.
1774 <note>
1775 This naming scheme assumes that
1776 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
1777 is "OEBasicHash", which is almost always the case in
1778 current OpenEmbedded.
1779 </note>
1780 To determine if a task needs to be rerun, BitBake checks
1781 if a stamp file with a matching input checksum exists
1782 for the task.
1783 If such a stamp file exists, the task's output is
1784 assumed to exist and still be valid.
1785 If the file does not exist, the task is rerun.
1786 <note>
1787 <para>The stamp mechanism is more general than the
1788 shared state (sstate) cache mechanism described in the
1789 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
1790 section.
1791 BitBake avoids rerunning any task that has a valid
1792 stamp file, not just tasks that can be accelerated
1793 through the sstate cache.</para>
1794
1795 <para>However, you should realize that stamp files only
1796 serve as a marker that some work has been done and that
1797 these files do not record task output.
1798 The actual task output would usually be somewhere in
1799 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
1800 (e.g. in some recipe's
1801 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.)
1802 What the sstate cache mechanism adds is a way to cache
1803 task output that can then be shared between build
1804 machines.</para>
1805 </note>
1806 Since <filename>STAMPS_DIR</filename> is usually a
1807 subdirectory of <filename>TMPDIR</filename>, removing
1808 <filename>TMPDIR</filename> will also remove
1809 <filename>STAMPS_DIR</filename>, which means tasks will
1810 properly be rerun to repopulate
1811 <filename>TMPDIR</filename>.
1812 </para>
1813
1814 <para>
1815 If you want some task to always be considered "out of
1816 date", you can mark it with the
1817 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
1818 varflag.
1819 If some other task depends on such a task, then that
1820 task will also always be considered out of date, which
1821 might not be what you want.
1822 </para>
1823
1824 <para>
1825 For details on how to view information about a task's
1826 signature, see the
1827 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
1828 section in the Yocto Project Development Tasks Manual.
1829 </para>
1830 </section>
1831
1832 <section id='setscene-tasks-and-shared-state'>
1833 <title>Setscene Tasks and Shared State</title>
1834
1835 <para>
1836 The description of tasks so far assumes that BitBake needs
1837 to build everything and no available prebuilt objects
1838 exist.
1839 BitBake does support skipping tasks if prebuilt objects are
1840 available.
1841 These objects are usually made available in the form of a
1842 shared state (sstate) cache.
1843 <note>
1844 For information on variables affecting sstate, see the
1845 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
1846 and
1847 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
1848 variables.
1849 </note>
1850 </para>
1851
1852 <para>
1853 The idea of a setscene task (i.e
1854 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
1855 is a version of the task where
1856 instead of building something, BitBake can skip to the end
1857 result and simply place a set of files into specific
1858 locations as needed.
1859 In some cases, it makes sense to have a setscene task
1860 variant (e.g. generating package files in the
1861 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1862 task).
1863 In other cases, it does not make sense (e.g. a
1864 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1865 task or a
1866 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1867 task) since the work involved would be equal to or greater
1868 than the underlying task.
1869 </para>
1870
1871 <para>
1872 In the build system, the common tasks that have setscene
1873 variants are
1874 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>,
1875 <filename>do_package_write_*</filename>,
1876 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>,
1877 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>,
1878 and
1879 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>.
1880 Notice that these tasks represent most of the tasks whose
1881 output is an end result.
1882 </para>
1883
1884 <para>
1885 The build system has knowledge of the relationship between
1886 these tasks and other preceding tasks.
1887 For example, if BitBake runs
1888 <filename>do_populate_sysroot_setscene</filename> for
1889 something, it does not make sense to run any of the
1890 <filename>do_fetch</filename>,
1891 <filename>do_unpack</filename>,
1892 <filename>do_patch</filename>,
1893 <filename>do_configure</filename>,
1894 <filename>do_compile</filename>, and
1895 <filename>do_install</filename> tasks.
1896 However, if <filename>do_package</filename> needs to be
1897 run, BitBake needs to run those other tasks.
1898 </para>
1899
1900 <para>
1901 It becomes more complicated if everything can come
1902 from an sstate cache because some objects are simply
1903 not required at all.
1904 For example, you do not need a compiler or native tools,
1905 such as quilt, if nothing exists to compile or patch.
1906 If the <filename>do_package_write_*</filename> packages
1907 are available from sstate, BitBake does not need the
1908 <filename>do_package</filename> task data.
1909 </para>
1910
1911 <para>
1912 To handle all these complexities, BitBake runs in two
1913 phases.
1914 The first is the "setscene" stage.
1915 During this stage, BitBake first checks the sstate cache
1916 for any targets it is planning to build.
1917 BitBake does a fast check to see if the object exists
1918 rather than a complete download.
1919 If nothing exists, the second phase, which is the setscene
1920 stage, completes and the main build proceeds.
1921 </para>
1922
1923 <para>
1924 If objects are found in the sstate cache, the build system
1925 works backwards from the end targets specified by the user.
1926 For example, if an image is being built, the build system
1927 first looks for the packages needed for that image and the
1928 tools needed to construct an image.
1929 If those are available, the compiler is not needed.
1930 Thus, the compiler is not even downloaded.
1931 If something was found to be unavailable, or the
1932 download or setscene task fails, the build system then
1933 tries to install dependencies, such as the compiler, from
1934 the cache.
1935 </para>
1936
1937 <para>
1938 The availability of objects in the sstate cache is
1939 handled by the function specified by the
1940 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
1941 variable and returns a list of available objects.
1942 The function specified by the
1943 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
1944 variable is the function that determines whether a given
1945 dependency needs to be followed, and whether for any given
1946 relationship the function needs to be passed.
1947 The function returns a True or False value.
1948 </para>
1949 </section>
1950 </section>
1951
1952 <section id='images-dev-environment'>
1953 <title>Images</title>
1954
1955 <para>
1956 The images produced by the build system are compressed forms
1957 of the root filesystem and are ready to boot on a target
1958 device.
1959 You can see from the
1960 <link linkend='general-workflow-figure'>general workflow figure</link>
1961 that BitBake output, in part, consists of images.
1962 This section takes a closer look at this output:
1963 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1964 </para>
1965
1966 <note>
1967 For a list of example images that the Yocto Project provides,
1968 see the
1969 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>"
1970 chapter in the Yocto Project Reference Manual.
1971 </note>
1972
1973 <para>
1974 The build process writes images out to the
1975 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1976 inside the
1977 <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
1978 folder as shown in the figure.
1979 This folder contains any files expected to be loaded on the
1980 target device.
1981 The
1982 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>
1983 variable points to the <filename>deploy</filename> directory,
1984 while the
1985 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink>
1986 variable points to the appropriate directory containing images
1987 for the current configuration.
1988 <itemizedlist>
1989 <listitem><para>
1990 <replaceable>kernel-image</replaceable>:
1991 A kernel binary file.
1992 The
1993 <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink>
1994 variable determines the naming scheme for the
1995 kernel image file.
1996 Depending on this variable, the file could begin with
1997 a variety of naming strings.
1998 The
1999 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2000 directory can contain multiple image files for the
2001 machine.
2002 </para></listitem>
2003 <listitem><para>
2004 <replaceable>root-filesystem-image</replaceable>:
2005 Root filesystems for the target device (e.g.
2006 <filename>*.ext3</filename> or
2007 <filename>*.bz2</filename> files).
2008 The
2009 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
2010 variable determines the root filesystem image type.
2011 The
2012 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2013 directory can contain multiple root filesystems for the
2014 machine.
2015 </para></listitem>
2016 <listitem><para>
2017 <replaceable>kernel-modules</replaceable>:
2018 Tarballs that contain all the modules built for the
2019 kernel.
2020 Kernel module tarballs exist for legacy purposes and
2021 can be suppressed by setting the
2022 <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink>
2023 variable to "0".
2024 The
2025 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2026 directory can contain multiple kernel module tarballs
2027 for the machine.
2028 </para></listitem>
2029 <listitem><para>
2030 <replaceable>bootloaders</replaceable>:
2031 If applicable to the target machine, bootloaders
2032 supporting the image.
2033 The <filename>deploy/images/</filename><replaceable>machine</replaceable>
2034 directory can contain multiple bootloaders for the
2035 machine.
2036 </para></listitem>
2037 <listitem><para>
2038 <replaceable>symlinks</replaceable>:
2039 The
2040 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2041 folder contains a symbolic link that points to the
2042 most recently built file for each machine.
2043 These links might be useful for external scripts that
2044 need to obtain the latest version of each file.
2045 </para></listitem>
2046 </itemizedlist>
2047 </para>
2048 </section>
2049
2050 <section id='sdk-dev-environment'>
2051 <title>Application Development SDK</title>
2052
2053 <para>
2054 In the
2055 <link linkend='general-workflow-figure'>general workflow figure</link>,
2056 the output labeled "Application Development SDK" represents an
2057 SDK.
2058 The SDK generation process differs depending on whether you
2059 build an extensible SDK (e.g.
2060 <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>)
2061 or a standard SDK (e.g.
2062 <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>).
2063 This section takes a closer look at this output:
2064 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
2065 </para>
2066
2067 <para>
2068 The specific form of this output is a set of files that
2069 includes a self-extracting SDK installer
2070 (<filename>*.sh</filename>), host and target manifest files,
2071 and files used for SDK testing.
2072 When the SDK installer file is run, it installs the SDK.
2073 The SDK consists of a cross-development toolchain, a set of
2074 libraries and headers, and an SDK environment setup script.
2075 Running this installer essentially sets up your
2076 cross-development environment.
2077 You can think of the cross-toolchain as the "host"
2078 part because it runs on the SDK machine.
2079 You can think of the libraries and headers as the "target"
2080 part because they are built for the target hardware.
2081 The environment setup script is added so that you can
2082 initialize the environment before using the tools.
2083 </para>
2084
2085 <note><title>Notes</title>
2086 <itemizedlist>
2087 <listitem><para>
2088 The Yocto Project supports several methods by which
2089 you can set up this cross-development environment.
2090 These methods include downloading pre-built SDK
2091 installers or building and installing your own SDK
2092 installer.
2093 </para></listitem>
2094 <listitem><para>
2095 For background information on cross-development
2096 toolchains in the Yocto Project development
2097 environment, see the
2098 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
2099 section.
2100 </para></listitem>
2101 <listitem><para>
2102 For information on setting up a cross-development
2103 environment, see the
2104 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2105 manual.
2106 </para></listitem>
2107 </itemizedlist>
2108 </note>
2109
2110 <para>
2111 All the output files for an SDK are written to the
2112 <filename>deploy/sdk</filename> folder inside the
2113 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
2114 as shown in the previous figure.
2115 Depending on the type of SDK, several variables exist that help
2116 configure these files.
2117 The following list shows the variables associated with an
2118 extensible SDK:
2119 <itemizedlist>
2120 <listitem><para>
2121 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2122 Points to the <filename>deploy</filename> directory.
2123 </para></listitem>
2124 <listitem><para>
2125 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>:
2126 Controls whether or not shared state artifacts are
2127 copied into the extensible SDK.
2128 By default, all required shared state artifacts are
2129 copied into the SDK.
2130 </para></listitem>
2131 <listitem><para>
2132 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>:
2133 Specifies whether or not packagedata is included in the
2134 extensible SDK for all recipes in the "world" target.
2135 </para></listitem>
2136 <listitem><para>
2137 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>:
2138 Specifies whether or not the toolchain is included
2139 when building the extensible SDK.
2140 </para></listitem>
2141 <listitem><para>
2142 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>:
2143 A list of variables allowed through from the build
2144 system configuration into the extensible SDK
2145 configuration.
2146 </para></listitem>
2147 <listitem><para>
2148 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>:
2149 A list of variables not allowed through from the build
2150 system configuration into the extensible SDK
2151 configuration.
2152 </para></listitem>
2153 <listitem><para>
2154 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>:
2155 A list of classes to remove from the
2156 <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink>
2157 value globally within the extensible SDK configuration.
2158 </para></listitem>
2159 </itemizedlist>
2160 This next list, shows the variables associated with a standard
2161 SDK:
2162 <itemizedlist>
2163 <listitem><para>
2164 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2165 Points to the <filename>deploy</filename> directory.
2166 </para></listitem>
2167 <listitem><para>
2168 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>:
2169 Specifies the architecture of the machine on which the
2170 cross-development tools are run to create packages for
2171 the target hardware.
2172 </para></listitem>
2173 <listitem><para>
2174 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>:
2175 Lists the features to include in the "target" part
2176 of the SDK.
2177 </para></listitem>
2178 <listitem><para>
2179 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>:
2180 Lists packages that make up the host part of the SDK
2181 (i.e. the part that runs on the
2182 <filename>SDKMACHINE</filename>).
2183 When you use
2184 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
2185 to create the SDK, a set of default packages apply.
2186 This variable allows you to add more packages.
2187 </para></listitem>
2188 <listitem><para>
2189 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>:
2190 Lists packages that make up the target part of the SDK
2191 (i.e. the part built for the target hardware).
2192 </para></listitem>
2193 <listitem><para>
2194 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>:
2195 Defines the default SDK installation path offered by
2196 the installation script.
2197 </para></listitem>
2198 <listitem><para>
2199 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_HOST_MANIFEST'><filename>SDK_HOST_MANIFEST</filename></ulink>:
2200 Lists all the installed packages that make up the host
2201 part of the SDK.
2202 This variable also plays a minor role for extensible
2203 SDK development as well.
2204 However, it is mainly used for the standard SDK.
2205 </para></listitem>
2206 <listitem><para>
2207 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_TARGET_MANIFEST'><filename>SDK_TARGET_MANIFEST</filename></ulink>:
2208 Lists all the installed packages that make up the
2209 target part of the SDK.
2210 This variable also plays a minor role for extensible
2211 SDK development as well.
2212 However, it is mainly used for the standard SDK.
2213 </para></listitem>
2214 </itemizedlist>
2215 </para>
2216 </section>
2217 </section>
2218
2219 <section id="cross-development-toolchain-generation">
2220 <title>Cross-Development Toolchain Generation</title>
2221
2222 <para>
2223 The Yocto Project does most of the work for you when it comes to
2224 creating
2225 <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>.
2226 This section provides some technical background on how
2227 cross-development toolchains are created and used.
2228 For more information on toolchains, you can also see the
2229 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2230 manual.
2231 </para>
2232
2233 <para>
2234 In the Yocto Project development environment, cross-development
2235 toolchains are used to build images and applications that run
2236 on the target hardware.
2237 With just a few commands, the OpenEmbedded build system creates
2238 these necessary toolchains for you.
2239 </para>
2240
2241 <para>
2242 The following figure shows a high-level build environment regarding
2243 toolchain construction and use.
2244 </para>
2245
2246 <para>
2247 <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" />
2248 </para>
2249
2250 <para>
2251 Most of the work occurs on the Build Host.
2252 This is the machine used to build images and generally work within
2253 the the Yocto Project environment.
2254 When you run
2255 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2256 to create an image, the OpenEmbedded build system
2257 uses the host <filename>gcc</filename> compiler to bootstrap a
2258 cross-compiler named <filename>gcc-cross</filename>.
2259 The <filename>gcc-cross</filename> compiler is what BitBake uses to
2260 compile source files when creating the target image.
2261 You can think of <filename>gcc-cross</filename> simply as an
2262 automatically generated cross-compiler that is used internally
2263 within BitBake only.
2264 <note>
2265 The extensible SDK does not use
2266 <filename>gcc-cross-canadian</filename> since this SDK
2267 ships a copy of the OpenEmbedded build system and the sysroot
2268 within it contains <filename>gcc-cross</filename>.
2269 </note>
2270 </para>
2271
2272 <para>
2273 The chain of events that occurs when <filename>gcc-cross</filename> is
2274 bootstrapped is as follows:
2275 <literallayout class='monospaced'>
2276 gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime
2277 </literallayout>
2278 <itemizedlist>
2279 <listitem><para>
2280 <filename>gcc</filename>:
2281 The build host's GNU Compiler Collection (GCC).
2282 </para></listitem>
2283 <listitem><para>
2284 <filename>binutils-cross</filename>:
2285 The bare minimum binary utilities needed in order to run
2286 the <filename>gcc-cross-initial</filename> phase of the
2287 bootstrap operation.
2288 </para></listitem>
2289 <listitem><para>
2290 <filename>gcc-cross-initial</filename>:
2291 An early stage of the bootstrap process for creating
2292 the cross-compiler.
2293 This stage builds enough of the <filename>gcc-cross</filename>,
2294 the C library, and other pieces needed to finish building the
2295 final cross-compiler in later stages.
2296 This tool is a "native" package (i.e. it is designed to run on
2297 the build host).
2298 </para></listitem>
2299 <listitem><para>
2300 <filename>linux-libc-headers</filename>:
2301 Headers needed for the cross-compiler.
2302 </para></listitem>
2303 <listitem><para>
2304 <filename>glibc-initial</filename>:
2305 An initial version of the Embedded GNU C Library
2306 (GLIBC) needed to bootstrap <filename>glibc</filename>.
2307 </para></listitem>
2308 <listitem><para>
2309 <filename>glibc</filename>:
2310 The GNU C Library.
2311 </para></listitem>
2312 <listitem><para>
2313 <filename>gcc-cross</filename>:
2314 The final stage of the bootstrap process for the
2315 cross-compiler.
2316 This stage results in the actual cross-compiler that
2317 BitBake uses when it builds an image for a targeted
2318 device.
2319 <note>
2320 If you are replacing this cross compiler toolchain
2321 with a custom version, you must replace
2322 <filename>gcc-cross</filename>.
2323 </note>
2324 This tool is also a "native" package (i.e. it is
2325 designed to run on the build host).
2326 </para></listitem>
2327 <listitem><para>
2328 <filename>gcc-runtime</filename>:
2329 Runtime libraries resulting from the toolchain bootstrapping
2330 process.
2331 This tool produces a binary that consists of the
2332 runtime libraries need for the targeted device.
2333 </para></listitem>
2334 </itemizedlist>
2335 </para>
2336
2337 <para>
2338 You can use the OpenEmbedded build system to build an installer for
2339 the relocatable SDK used to develop applications.
2340 When you run the installer, it installs the toolchain, which
2341 contains the development tools (e.g.,
2342 <filename>gcc-cross-canadian</filename>,
2343 <filename>binutils-cross-canadian</filename>, and other
2344 <filename>nativesdk-*</filename> tools),
2345 which are tools native to the SDK (i.e. native to
2346 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>),
2347 you need to cross-compile and test your software.
2348 The figure shows the commands you use to easily build out this
2349 toolchain.
2350 This cross-development toolchain is built to execute on the
2351 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2352 which might or might not be the same
2353 machine as the Build Host.
2354 <note>
2355 If your target architecture is supported by the Yocto Project,
2356 you can take advantage of pre-built images that ship with the
2357 Yocto Project and already contain cross-development toolchain
2358 installers.
2359 </note>
2360 </para>
2361
2362 <para>
2363 Here is the bootstrap process for the relocatable toolchain:
2364 <literallayout class='monospaced'>
2365 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers ->
2366 glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
2367 </literallayout>
2368 <itemizedlist>
2369 <listitem><para>
2370 <filename>gcc</filename>:
2371 The build host's GNU Compiler Collection (GCC).
2372 </para></listitem>
2373 <listitem><para>
2374 <filename>binutils-crosssdk</filename>:
2375 The bare minimum binary utilities needed in order to run
2376 the <filename>gcc-crosssdk-initial</filename> phase of the
2377 bootstrap operation.
2378 </para></listitem>
2379 <listitem><para>
2380 <filename>gcc-crosssdk-initial</filename>:
2381 An early stage of the bootstrap process for creating
2382 the cross-compiler.
2383 This stage builds enough of the
2384 <filename>gcc-crosssdk</filename> and supporting pieces so that
2385 the final stage of the bootstrap process can produce the
2386 finished cross-compiler.
2387 This tool is a "native" binary that runs on the build host.
2388 </para></listitem>
2389 <listitem><para>
2390 <filename>linux-libc-headers</filename>:
2391 Headers needed for the cross-compiler.
2392 </para></listitem>
2393 <listitem><para>
2394 <filename>glibc-initial</filename>:
2395 An initial version of the Embedded GLIBC needed to bootstrap
2396 <filename>nativesdk-glibc</filename>.
2397 </para></listitem>
2398 <listitem><para>
2399 <filename>nativesdk-glibc</filename>:
2400 The Embedded GLIBC needed to bootstrap the
2401 <filename>gcc-crosssdk</filename>.
2402 </para></listitem>
2403 <listitem><para>
2404 <filename>gcc-crosssdk</filename>:
2405 The final stage of the bootstrap process for the
2406 relocatable cross-compiler.
2407 The <filename>gcc-crosssdk</filename> is a transitory
2408 compiler and never leaves the build host.
2409 Its purpose is to help in the bootstrap process to create
2410 the eventual <filename>gcc-cross-canadian</filename>
2411 compiler, which is relocatable.
2412 This tool is also a "native" package (i.e. it is
2413 designed to run on the build host).
2414 </para></listitem>
2415 <listitem><para>
2416 <filename>gcc-cross-canadian</filename>:
2417 The final relocatable cross-compiler.
2418 When run on the
2419 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2420 this tool
2421 produces executable code that runs on the target device.
2422 Only one cross-canadian compiler is produced per architecture
2423 since they can be targeted at different processor optimizations
2424 using configurations passed to the compiler through the
2425 compile commands.
2426 This circumvents the need for multiple compilers and thus
2427 reduces the size of the toolchains.
2428 </para></listitem>
2429 </itemizedlist>
2430 </para>
2431
2432 <note>
2433 For information on advantages gained when building a
2434 cross-development toolchain installer, see the
2435 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
2436 appendix in the Yocto Project Application Development and the
2437 Extensible Software Development Kit (eSDK) manual.
2438 </note>
2439 </section>
2440
2441 <section id="shared-state-cache">
2442 <title>Shared State Cache</title>
2443
2444 <para>
2445 By design, the OpenEmbedded build system builds everything from
2446 scratch unless
2447 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2448 can determine that parts do not need to be rebuilt.
2449 Fundamentally, building from scratch is attractive as it means all
2450 parts are built fresh and no possibility of stale data exists that
2451 can cause problems.
2452 When developers hit problems, they typically default back to
2453 building from scratch so they have a know state from the
2454 start.
2455 </para>
2456
2457 <para>
2458 Building an image from scratch is both an advantage and a
2459 disadvantage to the process.
2460 As mentioned in the previous paragraph, building from scratch
2461 ensures that everything is current and starts from a known state.
2462 However, building from scratch also takes much longer as it
2463 generally means rebuilding things that do not necessarily need
2464 to be rebuilt.
2465 </para>
2466
2467 <para>
2468 The Yocto Project implements shared state code that supports
2469 incremental builds.
2470 The implementation of the shared state code answers the following
2471 questions that were fundamental roadblocks within the OpenEmbedded
2472 incremental build support system:
2473 <itemizedlist>
2474 <listitem><para>
2475 What pieces of the system have changed and what pieces have
2476 not changed?
2477 </para></listitem>
2478 <listitem><para>
2479 How are changed pieces of software removed and replaced?
2480 </para></listitem>
2481 <listitem><para>
2482 How are pre-built components that do not need to be rebuilt
2483 from scratch used when they are available?
2484 </para></listitem>
2485 </itemizedlist>
2486 </para>
2487
2488 <para>
2489 For the first question, the build system detects changes in the
2490 "inputs" to a given task by creating a checksum (or signature) of
2491 the task's inputs.
2492 If the checksum changes, the system assumes the inputs have changed
2493 and the task needs to be rerun.
2494 For the second question, the shared state (sstate) code tracks
2495 which tasks add which output to the build process.
2496 This means the output from a given task can be removed, upgraded
2497 or otherwise manipulated.
2498 The third question is partly addressed by the solution for the
2499 second question assuming the build system can fetch the sstate
2500 objects from remote locations and install them if they are deemed
2501 to be valid.
2502 <note><title>Notes</title>
2503 <itemizedlist>
2504 <listitem><para>
2505 The build system does not maintain
2506 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2507 information as part of the shared state packages.
2508 Consequently, considerations exist that affect
2509 maintaining shared state feeds.
2510 For information on how the build system works with
2511 packages and can track incrementing
2512 <filename>PR</filename> information, see the
2513 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
2514 section in the Yocto Project Development Tasks Manual.
2515 </para></listitem>
2516 <listitem><para>
2517 The code in the build system that supports incremental
2518 builds is not simple code.
2519 For techniques that help you work around issues related
2520 to shared state code, see the
2521 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'>Viewing Metadata Used to Create the Input Signature of a Shared State Task</ulink>"
2522 and
2523 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-invalidating-shared-state-to-force-a-task-to-run'>Invalidating Shared State to Force a Task to Run</ulink>"
2524 sections both in the Yocto Project Development Tasks
2525 Manual.
2526 </para></listitem>
2527 </itemizedlist>
2528 </note>
2529 </para>
2530
2531 <para>
2532 The rest of this section goes into detail about the overall
2533 incremental build architecture, the checksums (signatures), and
2534 shared state.
2535 </para>
2536
2537 <section id='concepts-overall-architecture'>
2538 <title>Overall Architecture</title>
2539
2540 <para>
2541 When determining what parts of the system need to be built,
2542 BitBake works on a per-task basis rather than a per-recipe
2543 basis.
2544 You might wonder why using a per-task basis is preferred over
2545 a per-recipe basis.
2546 To help explain, consider having the IPK packaging backend
2547 enabled and then switching to DEB.
2548 In this case, the
2549 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2550 and
2551 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2552 task outputs are still valid.
2553 However, with a per-recipe approach, the build would not
2554 include the <filename>.deb</filename> files.
2555 Consequently, you would have to invalidate the whole build and
2556 rerun it.
2557 Rerunning everything is not the best solution.
2558 Also, in this case, the core must be "taught" much about
2559 specific tasks.
2560 This methodology does not scale well and does not allow users
2561 to easily add new tasks in layers or as external recipes
2562 without touching the packaged-staging core.
2563 </para>
2564 </section>
2565
2566 <section id='overview-checksums'>
2567 <title>Checksums (Signatures)</title>
2568
2569 <para>
2570 The shared state code uses a checksum, which is a unique
2571 signature of a task's inputs, to determine if a task needs to
2572 be run again.
2573 Because it is a change in a task's inputs that triggers a
2574 rerun, the process needs to detect all the inputs to a given
2575 task.
2576 For shell tasks, this turns out to be fairly easy because
2577 the build process generates a "run" shell script for each task
2578 and it is possible to create a checksum that gives you a good
2579 idea of when the task's data changes.
2580 </para>
2581
2582 <para>
2583 To complicate the problem, there are things that should not be
2584 included in the checksum.
2585 First, there is the actual specific build path of a given
2586 task - the
2587 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2588 It does not matter if the work directory changes because it
2589 should not affect the output for target packages.
2590 Also, the build process has the objective of making native
2591 or cross packages relocatable.
2592 <note>
2593 Both native and cross packages run on the
2594 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>.
2595 However, cross packages generate output for the target
2596 architecture.
2597 </note>
2598 The checksum therefore needs to exclude
2599 <filename>WORKDIR</filename>.
2600 The simplistic approach for excluding the work directory is to
2601 set <filename>WORKDIR</filename> to some fixed value and
2602 create the checksum for the "run" script.
2603 </para>
2604
2605 <para>
2606 Another problem results from the "run" scripts containing
2607 functions that might or might not get called.
2608 The incremental build solution contains code that figures out
2609 dependencies between shell functions.
2610 This code is used to prune the "run" scripts down to the
2611 minimum set, thereby alleviating this problem and making the
2612 "run" scripts much more readable as a bonus.
2613 </para>
2614
2615 <para>
2616 So far, solutions for shell scripts exist.
2617 What about Python tasks?
2618 The same approach applies even though these tasks are more
2619 difficult.
2620 The process needs to figure out what variables a Python
2621 function accesses and what functions it calls.
2622 Again, the incremental build solution contains code that first
2623 figures out the variable and function dependencies, and then
2624 creates a checksum for the data used as the input to the task.
2625 </para>
2626
2627 <para>
2628 Like the <filename>WORKDIR</filename> case, situations exist
2629 where dependencies should be ignored.
2630 For these situations, you can instruct the build process to
2631 ignore a dependency by using a line like the following:
2632 <literallayout class='monospaced'>
2633 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
2634 </literallayout>
2635 This example ensures that the
2636 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
2637 variable does not depend on the value of
2638 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
2639 even if it does reference it.
2640 </para>
2641
2642 <para>
2643 Equally, there are cases where you need to add dependencies
2644 BitBake is not able to find.
2645 You can accomplish this by using a line like the following:
2646 <literallayout class='monospaced'>
2647 PACKAGE_ARCHS[vardeps] = "MACHINE"
2648 </literallayout>
2649 This example explicitly adds the <filename>MACHINE</filename>
2650 variable as a dependency for
2651 <filename>PACKAGE_ARCHS</filename>.
2652 </para>
2653
2654 <para>
2655 As an example, consider a case with in-line Python where
2656 BitBake is not able to figure out dependencies.
2657 When running in debug mode (i.e. using
2658 <filename>-DDD</filename>), BitBake produces output when it
2659 discovers something for which it cannot figure out dependencies.
2660 The Yocto Project team has currently not managed to cover
2661 those dependencies in detail and is aware of the need to fix
2662 this situation.
2663 </para>
2664
2665 <para>
2666 Thus far, this section has limited discussion to the direct
2667 inputs into a task.
2668 Information based on direct inputs is referred to as the
2669 "basehash" in the code.
2670 However, the question of a task's indirect inputs still
2671 exits - items already built and present in the
2672 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
2673 The checksum (or signature) for a particular task needs to add
2674 the hashes of all the tasks on which the particular task
2675 depends.
2676 Choosing which dependencies to add is a policy decision.
2677 However, the effect is to generate a master checksum that
2678 combines the basehash and the hashes of the task's
2679 dependencies.
2680 </para>
2681
2682 <para>
2683 At the code level, a variety of ways exist by which both the
2684 basehash and the dependent task hashes can be influenced.
2685 Within the BitBake configuration file, you can give BitBake
2686 some extra information to help it construct the basehash.
2687 The following statement effectively results in a list of
2688 global variable dependency excludes (i.e. variables never
2689 included in any checksum):
2690 <literallayout class='monospaced'>
2691 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
2692 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
2693 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
2694 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
2695 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
2696 </literallayout>
2697 The previous example excludes
2698 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
2699 since that variable is actually constructed as a path within
2700 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
2701 which is on the whitelist.
2702 </para>
2703
2704 <para>
2705 The rules for deciding which hashes of dependent tasks to
2706 include through dependency chains are more complex and are
2707 generally accomplished with a Python function.
2708 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
2709 two examples of this and also illustrates how you can insert
2710 your own policy into the system if so desired.
2711 This file defines the two basic signature generators
2712 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
2713 uses: "OEBasic" and "OEBasicHash".
2714 By default, a dummy "noop" signature handler is enabled
2715 in BitBake.
2716 This means that behavior is unchanged from previous versions.
2717 OE-Core uses the "OEBasicHash" signature handler by default
2718 through this setting in the <filename>bitbake.conf</filename>
2719 file:
2720 <literallayout class='monospaced'>
2721 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
2722 </literallayout>
2723 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
2724 is the same as the "OEBasic" version but adds the task hash to
2725 the
2726 <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp files</link>.
2727 This results in any metadata change that changes the task hash,
2728 automatically causing the task to be run again.
2729 This removes the need to bump
2730 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2731 values, and changes to metadata automatically ripple across
2732 the build.
2733 </para>
2734
2735 <para>
2736 It is also worth noting that the end result of these
2737 signature generators is to make some dependency and hash
2738 information available to the build.
2739 This information includes:
2740 <itemizedlist>
2741 <listitem><para>
2742 <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
2743 The base hashes for each task in the recipe.
2744 </para></listitem>
2745 <listitem><para>
2746 <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2747 The base hashes for each dependent task.
2748 </para></listitem>
2749 <listitem><para>
2750 <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2751 The task dependencies for each task.
2752 </para></listitem>
2753 <listitem><para>
2754 <filename>BB_TASKHASH</filename>:
2755 The hash of the currently running task.
2756 </para></listitem>
2757 </itemizedlist>
2758 </para>
2759 </section>
2760
2761 <section id='shared-state'>
2762 <title>Shared State</title>
2763
2764 <para>
2765 Checksums and dependencies, as discussed in the previous
2766 section, solve half the problem of supporting a shared state.
2767 The other half of the problem is being able to use checksum
2768 information during the build and being able to reuse or rebuild
2769 specific components.
2770 </para>
2771
2772 <para>
2773 The
2774 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2775 class is a relatively generic implementation of how to
2776 "capture" a snapshot of a given task.
2777 The idea is that the build process does not care about the
2778 source of a task's output.
2779 Output could be freshly built or it could be downloaded and
2780 unpacked from somewhere.
2781 In other words, the build process does not need to worry about
2782 its origin.
2783 </para>
2784
2785 <para>
2786 Two types of output exist.
2787 One type is just about creating a directory in
2788 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2789 A good example is the output of either
2790 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2791 or
2792 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
2793 The other type of output occurs when a set of data is merged
2794 into a shared directory tree such as the sysroot.
2795 </para>
2796
2797 <para>
2798 The Yocto Project team has tried to keep the details of the
2799 implementation hidden in <filename>sstate</filename> class.
2800 From a user's perspective, adding shared state wrapping to a
2801 task is as simple as this
2802 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2803 example taken from the
2804 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
2805 class:
2806 <literallayout class='monospaced'>
2807 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
2808 SSTATETASKS += "do_deploy"
2809 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
2810 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
2811
2812 python do_deploy_setscene () {
2813 sstate_setscene(d)
2814 }
2815 addtask do_deploy_setscene
2816 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
2817 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
2818 </literallayout>
2819 The following list explains the previous example:
2820 <itemizedlist>
2821 <listitem><para>
2822 Adding "do_deploy" to <filename>SSTATETASKS</filename>
2823 adds some required sstate-related processing, which is
2824 implemented in the
2825 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2826 class, to before and after the
2827 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2828 task.
2829 </para></listitem>
2830 <listitem><para>
2831 The
2832 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
2833 declares that <filename>do_deploy</filename> places its
2834 output in <filename>${DEPLOYDIR}</filename> when run
2835 normally (i.e. when not using the sstate cache).
2836 This output becomes the input to the shared state cache.
2837 </para></listitem>
2838 <listitem><para>
2839 The
2840 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
2841 line causes the contents of the shared state cache to be
2842 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
2843 <note>
2844 If <filename>do_deploy</filename> is not already in
2845 the shared state cache or if its input checksum
2846 (signature) has changed from when the output was
2847 cached, the task runs to populate the shared
2848 state cache, after which the contents of the shared
2849 state cache is copied to
2850 <filename>${DEPLOY_DIR_IMAGE}</filename>.
2851 If <filename>do_deploy</filename> is in the shared
2852 state cache and its signature indicates that the
2853 cached output is still valid (i.e. if no
2854 relevant task inputs have changed), then the
2855 contents of the shared state cache copies
2856 directly to
2857 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
2858 <filename>do_deploy_setscene</filename> task
2859 instead, skipping the
2860 <filename>do_deploy</filename> task.
2861 </note>
2862 </para></listitem>
2863 <listitem><para>
2864 The following task definition is glue logic needed to
2865 make the previous settings effective:
2866 <literallayout class='monospaced'>
2867 python do_deploy_setscene () {
2868 sstate_setscene(d)
2869 }
2870 addtask do_deploy_setscene
2871 </literallayout>
2872 <filename>sstate_setscene()</filename> takes the flags
2873 above as input and accelerates the
2874 <filename>do_deploy</filename> task through the
2875 shared state cache if possible.
2876 If the task was accelerated,
2877 <filename>sstate_setscene()</filename> returns True.
2878 Otherwise, it returns False, and the normal
2879 <filename>do_deploy</filename> task runs.
2880 For more information, see the
2881 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
2882 section in the BitBake User Manual.
2883 </para></listitem>
2884 <listitem><para>
2885 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
2886 line creates <filename>${DEPLOYDIR}</filename> and
2887 <filename>${B}</filename> before the
2888 <filename>do_deploy</filename> task runs, and also sets
2889 the current working directory of
2890 <filename>do_deploy</filename> to
2891 <filename>${B}</filename>.
2892 For more information, see the
2893 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
2894 section in the BitBake User Manual.
2895 <note>
2896 In cases where
2897 <filename>sstate-inputdirs</filename> and
2898 <filename>sstate-outputdirs</filename> would be the
2899 same, you can use
2900 <filename>sstate-plaindirs</filename>.
2901 For example, to preserve the
2902 <filename>${PKGD}</filename> and
2903 <filename>${PKGDEST}</filename> output from the
2904 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2905 task, use the following:
2906 <literallayout class='monospaced'>
2907 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
2908 </literallayout>
2909 </note>
2910 </para></listitem>
2911 <listitem><para>
2912 The <filename>do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"</filename>
2913 line appends extra metadata to the
2914 <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp file</link>.
2915 In this case, the metadata makes the task specific
2916 to a machine's architecture.
2917 See
2918 "<ulink url='&YOCTO_DOCS_BB_URL;#ref-bitbake-tasklist'>The Task List</ulink>"
2919 section in the BitBake User Manual for more
2920 information on the <filename>stamp-extra-info</filename>
2921 flag.
2922 </para></listitem>
2923 <listitem><para>
2924 <filename>sstate-inputdirs</filename> and
2925 <filename>sstate-outputdirs</filename> can also be used
2926 with multiple directories.
2927 For example, the following declares
2928 <filename>PKGDESTWORK</filename> and
2929 <filename>SHLIBWORK</filename> as shared state
2930 input directories, which populates the shared state
2931 cache, and <filename>PKGDATA_DIR</filename> and
2932 <filename>SHLIBSDIR</filename> as the corresponding
2933 shared state output directories:
2934 <literallayout class='monospaced'>
2935 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
2936 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
2937 </literallayout>
2938 </para></listitem>
2939 <listitem><para>
2940 These methods also include the ability to take a
2941 lockfile when manipulating shared state directory
2942 structures, for cases where file additions or removals
2943 are sensitive:
2944 <literallayout class='monospaced'>
2945 do_package[sstate-lockfile] = "${PACKAGELOCK}"
2946 </literallayout>
2947 </para></listitem>
2948 </itemizedlist>
2949 </para>
2950
2951 <para>
2952 Behind the scenes, the shared state code works by looking in
2953 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
2954 and
2955 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
2956 for shared state files.
2957 Here is an example:
2958 <literallayout class='monospaced'>
2959 SSTATE_MIRRORS ?= "\
2960 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
2961 file://.* file:///some/local/dir/sstate/PATH"
2962 </literallayout>
2963 <note>
2964 The shared state directory
2965 (<filename>SSTATE_DIR</filename>) is organized into
2966 two-character subdirectories, where the subdirectory
2967 names are based on the first two characters of the hash.
2968 If the shared state directory structure for a mirror has the
2969 same structure as <filename>SSTATE_DIR</filename>, you must
2970 specify "PATH" as part of the URI to enable the build system
2971 to map to the appropriate subdirectory.
2972 </note>
2973 </para>
2974
2975 <para>
2976 The shared state package validity can be detected just by
2977 looking at the filename since the filename contains the task
2978 checksum (or signature) as described earlier in this section.
2979 If a valid shared state package is found, the build process
2980 downloads it and uses it to accelerate the task.
2981 </para>
2982
2983 <para>
2984 The build processes use the <filename>*_setscene</filename>
2985 tasks for the task acceleration phase.
2986 BitBake goes through this phase before the main execution
2987 code and tries to accelerate any tasks for which it can find
2988 shared state packages.
2989 If a shared state package for a task is available, the
2990 shared state package is used.
2991 This means the task and any tasks on which it is dependent
2992 are not executed.
2993 </para>
2994
2995 <para>
2996 As a real world example, the aim is when building an IPK-based
2997 image, only the
2998 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
2999 tasks would have their shared state packages fetched and
3000 extracted.
3001 Since the sysroot is not used, it would never get extracted.
3002 This is another reason why a task-based approach is preferred
3003 over a recipe-based approach, which would have to install the
3004 output from every task.
3005 </para>
3006 </section>
3007 </section>
3008
3009 <section id='automatically-added-runtime-dependencies'>
3010 <title>Automatically Added Runtime Dependencies</title>
3011
3012 <para>
3013 The OpenEmbedded build system automatically adds common types of
3014 runtime dependencies between packages, which means that you do not
3015 need to explicitly declare the packages using
3016 <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>.
3017 Three automatic mechanisms exist (<filename>shlibdeps</filename>,
3018 <filename>pcdeps</filename>, and <filename>depchains</filename>)
3019 that handle shared libraries, package configuration (pkg-config)
3020 modules, and <filename>-dev</filename> and
3021 <filename>-dbg</filename> packages, respectively.
3022 For other types of runtime dependencies, you must manually declare
3023 the dependencies.
3024 <itemizedlist>
3025 <listitem><para>
3026 <filename>shlibdeps</filename>:
3027 During the
3028 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
3029 task of each recipe, all shared libraries installed by the
3030 recipe are located.
3031 For each shared library, the package that contains the
3032 shared library is registered as providing the shared
3033 library.
3034 More specifically, the package is registered as providing
3035 the
3036 <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>
3037 of the library.
3038 The resulting shared-library-to-package mapping
3039 is saved globally in
3040 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>
3041 by the
3042 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
3043 task.</para>
3044
3045 <para>Simultaneously, all executables and shared libraries
3046 installed by the recipe are inspected to see what shared
3047 libraries they link against.
3048 For each shared library dependency that is found,
3049 <filename>PKGDATA_DIR</filename> is queried to
3050 see if some package (likely from a different recipe)
3051 contains the shared library.
3052 If such a package is found, a runtime dependency is added
3053 from the package that depends on the shared library to the
3054 package that contains the library.</para>
3055
3056 <para>The automatically added runtime dependency also
3057 includes a version restriction.
3058 This version restriction specifies that at least the
3059 current version of the package that provides the shared
3060 library must be used, as if
3061 "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"
3062 had been added to <filename>RDEPENDS</filename>.
3063 This forces an upgrade of the package containing the shared
3064 library when installing the package that depends on the
3065 library, if needed.</para>
3066
3067 <para>If you want to avoid a package being registered as
3068 providing a particular shared library (e.g. because the library
3069 is for internal use only), then add the library to
3070 <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink>
3071 inside the package's recipe.
3072 </para></listitem>
3073 <listitem><para>
3074 <filename>pcdeps</filename>:
3075 During the <filename>do_package</filename> task of each
3076 recipe, all pkg-config modules
3077 (<filename>*.pc</filename> files) installed by the recipe
3078 are located.
3079 For each module, the package that contains the module is
3080 registered as providing the module.
3081 The resulting module-to-package mapping is saved globally in
3082 <filename>PKGDATA_DIR</filename> by the
3083 <filename>do_packagedata</filename> task.</para>
3084
3085 <para>Simultaneously, all pkg-config modules installed by
3086 the recipe are inspected to see what other pkg-config
3087 modules they depend on.
3088 A module is seen as depending on another module if it
3089 contains a "Requires:" line that specifies the other module.
3090 For each module dependency,
3091 <filename>PKGDATA_DIR</filename> is queried to see if some
3092 package contains the module.
3093 If such a package is found, a runtime dependency is added
3094 from the package that depends on the module to the package
3095 that contains the module.
3096 <note>
3097 The <filename>pcdeps</filename> mechanism most often
3098 infers dependencies between <filename>-dev</filename>
3099 packages.
3100 </note>
3101 </para></listitem>
3102 <listitem><para>
3103 <filename>depchains</filename>:
3104 If a package <filename>foo</filename> depends on a package
3105 <filename>bar</filename>, then <filename>foo-dev</filename>
3106 and <filename>foo-dbg</filename> are also made to depend on
3107 <filename>bar-dev</filename> and
3108 <filename>bar-dbg</filename>, respectively.
3109 Taking the <filename>-dev</filename> packages as an
3110 example, the <filename>bar-dev</filename> package might
3111 provide headers and shared library symlinks needed by
3112 <filename>foo-dev</filename>, which shows the need
3113 for a dependency between the packages.</para>
3114
3115 <para>The dependencies added by
3116 <filename>depchains</filename> are in the form of
3117 <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>.
3118 <note>
3119 By default, <filename>foo-dev</filename> also has an
3120 <filename>RDEPENDS</filename>-style dependency on
3121 <filename>foo</filename>, because the default value of
3122 <filename>RDEPENDS_${PN}-dev</filename> (set in
3123 <filename>bitbake.conf</filename>) includes
3124 "${PN}".
3125 </note></para>
3126
3127 <para>To ensure that the dependency chain is never broken,
3128 <filename>-dev</filename> and <filename>-dbg</filename>
3129 packages are always generated by default, even if the
3130 packages turn out to be empty.
3131 See the
3132 <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink>
3133 variable for more information.
3134 </para></listitem>
3135 </itemizedlist>
3136 </para>
3137
3138 <para>
3139 The <filename>do_package</filename> task depends on the
3140 <filename>do_packagedata</filename> task of each recipe in
3141 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>
3142 through use of a
3143 <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>
3144 declaration, which guarantees that the required
3145 shared-library/module-to-package mapping information will be available
3146 when needed as long as <filename>DEPENDS</filename> has been
3147 correctly set.
3148 </para>
3149 </section>
3150
3151 <section id='fakeroot-and-pseudo'>
3152 <title>Fakeroot and Pseudo</title>
3153
3154 <para>
3155 Some tasks are easier to implement when allowed to perform certain
3156 operations that are normally reserved for the root user (e.g.
3157 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>,
3158 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>,
3159 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>,
3160 and
3161 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>).
3162 For example, the <filename>do_install</filename> task benefits
3163 from being able to set the UID and GID of installed files to
3164 arbitrary values.
3165 </para>
3166
3167 <para>
3168 One approach to allowing tasks to perform root-only operations
3169 would be to require
3170 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
3171 to run as root.
3172 However, this method is cumbersome and has security issues.
3173 The approach that is actually used is to run tasks that benefit
3174 from root privileges in a "fake" root environment.
3175 Within this environment, the task and its child processes believe
3176 that they are running as the root user, and see an internally
3177 consistent view of the filesystem.
3178 As long as generating the final output (e.g. a package or an image)
3179 does not require root privileges, the fact that some earlier
3180 steps ran in a fake root environment does not cause problems.
3181 </para>
3182
3183 <para>
3184 The capability to run tasks in a fake root environment is known as
3185 "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>",
3186 which is derived from the BitBake keyword/variable
3187 flag that requests a fake root environment for a task.
3188 </para>
3189
3190 <para>
3191 In the
3192 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
3193 the program that implements fakeroot is known as
3194 <ulink url='https://www.yoctoproject.org/software-item/pseudo/'>Pseudo</ulink>.
3195 Pseudo overrides system calls by using the environment variable
3196 <filename>LD_PRELOAD</filename>, which results in the illusion
3197 of running as root.
3198 To keep track of "fake" file ownership and permissions resulting
3199 from operations that require root permissions, Pseudo uses
3200 an SQLite 3 database.
3201 This database is stored in
3202 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename>
3203 for individual recipes.
3204 Storing the database in a file as opposed to in memory
3205 gives persistence between tasks and builds, which is not
3206 accomplished using fakeroot.
3207 <note><title>Caution</title>
3208 If you add your own task that manipulates the same files or
3209 directories as a fakeroot task, then that task also needs to
3210 run under fakeroot.
3211 Otherwise, the task cannot run root-only operations, and
3212 cannot see the fake file ownership and permissions set by the
3213 other task.
3214 You need to also add a dependency on
3215 <filename>virtual/fakeroot-native:do_populate_sysroot</filename>,
3216 giving the following:
3217 <literallayout class='monospaced'>
3218 fakeroot do_mytask () {
3219 ...
3220 }
3221 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
3222 </literallayout>
3223 </note>
3224 For more information, see the
3225 <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>
3226 variables in the BitBake User Manual.
3227 You can also reference the
3228 "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>"
3229 article for background information on Fakeroot and Pseudo.
3230 </para>
3231 </section>
3232</chapter>
3233<!--
3234vim: expandtab tw=80 ts=4
3235-->
diff --git a/documentation/overview-manual/overview-manual-customization.xsl b/documentation/overview-manual/overview-manual-customization.xsl
deleted file mode 100644
index 1dd91bde80..0000000000
--- a/documentation/overview-manual/overview-manual-customization.xsl
+++ /dev/null
@@ -1,29 +0,0 @@
1<?xml version='1.0'?>
2<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
3
4<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
5
6 <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
7
8<!--
9
10 <xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
11
12 <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/xhtml/docbook.xsl" />
13
14-->
15
16 <xsl:include href="../template/permalinks.xsl"/>
17 <xsl:include href="../template/section.title.xsl"/>
18 <xsl:include href="../template/component.title.xsl"/>
19 <xsl:include href="../template/division.title.xsl"/>
20 <xsl:include href="../template/formal.object.heading.xsl"/>
21
22 <xsl:param name="html.stylesheet" select="'overview-manual-style.css'" />
23 <xsl:param name="chapter.autolabel" select="1" />
24 <xsl:param name="appendix.autolabel" select="A" />
25 <xsl:param name="section.autolabel" select="1" />
26 <xsl:param name="section.label.includes.component.label" select="1" />
27 <xsl:param name="generate.id.attributes" select="1" />
28
29</xsl:stylesheet>
diff --git a/documentation/overview-manual/overview-manual-development-environment.xml b/documentation/overview-manual/overview-manual-development-environment.xml
deleted file mode 100644
index 08ad071316..0000000000
--- a/documentation/overview-manual/overview-manual-development-environment.xml
+++ /dev/null
@@ -1,954 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='overview-development-environment'>
7<title>The Yocto Project Development Environment</title>
8
9<para>
10 This chapter takes a look at the Yocto Project development
11 environment.
12 The chapter provides Yocto Project Development environment concepts that
13 help you understand how work is accomplished in an open source environment,
14 which is very different as compared to work accomplished in a closed,
15 proprietary environment.
16</para>
17
18<para>
19 Specifically, this chapter addresses open source philosophy, source
20 repositories, workflows, Git, and licensing.
21</para>
22
23<section id='open-source-philosophy'>
24 <title>Open Source Philosophy</title>
25
26 <para>
27 Open source philosophy is characterized by software development
28 directed by peer production and collaboration through an active
29 community of developers.
30 Contrast this to the more standard centralized development models
31 used by commercial software companies where a finite set of developers
32 produces a product for sale using a defined set of procedures that
33 ultimately result in an end product whose architecture and source
34 material are closed to the public.
35 </para>
36
37 <para>
38 Open source projects conceptually have differing concurrent agendas,
39 approaches, and production.
40 These facets of the development process can come from anyone in the
41 public (community) who has a stake in the software project.
42 The open source environment contains new copyright, licensing, domain,
43 and consumer issues that differ from the more traditional development
44 environment.
45 In an open source environment, the end product, source material,
46 and documentation are all available to the public at no cost.
47 </para>
48
49 <para>
50 A benchmark example of an open source project is the Linux kernel,
51 which was initially conceived and created by Finnish computer science
52 student Linus Torvalds in 1991.
53 Conversely, a good example of a non-open source project is the
54 <trademark class='registered'>Windows</trademark> family of operating
55 systems developed by
56 <trademark class='registered'>Microsoft</trademark> Corporation.
57 </para>
58
59 <para>
60 Wikipedia has a good historical description of the Open Source
61 Philosophy
62 <ulink url='http://en.wikipedia.org/wiki/Open_source'>here</ulink>.
63 You can also find helpful information on how to participate in the
64 Linux Community
65 <ulink url='http://ldn.linuxfoundation.org/book/how-participate-linux-community'>here</ulink>.
66 </para>
67</section>
68
69<section id='gs-the-development-host'>
70 <title>The Development Host</title>
71
72 <para>
73 A development host or
74 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>
75 is key to using the Yocto Project.
76 Because the goal of the Yocto Project is to develop images or
77 applications that run on embedded hardware, development of those
78 images and applications generally takes place on a system not
79 intended to run the software - the development host.
80 </para>
81
82 <para>
83 You need to set up a development host in order to use it with the
84 Yocto Project.
85 Most find that it is best to have a native Linux machine function as
86 the development host.
87 However, it is possible to use a system that does not run Linux
88 as its operating system as your development host.
89 When you have a Mac or Windows-based system, you can set it up
90 as the development host by using
91 <ulink url='https://github.com/crops/poky-container'>CROPS</ulink>,
92 which leverages
93 <ulink url='https://www.docker.com/'>Docker Containers</ulink>.
94 Once you take the steps to set up a CROPS machine, you effectively
95 have access to a shell environment that is similar to what you see
96 when using a Linux-based development host.
97 For the steps needed to set up a system using CROPS, see the
98 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops'>Setting Up to Use CROss PlatformS (CROPS)</ulink>"
99 section in the Yocto Project Development Tasks Manual.
100 </para>
101
102 <para>
103 If your development host is going to be a system that runs a Linux
104 distribution, steps still exist that you must take to prepare the
105 system for use with the Yocto Project.
106 You need to be sure that the Linux distribution on the system is
107 one that supports the Yocto Project.
108 You also need to be sure that the correct set of host packages are
109 installed that allow development using the Yocto Project.
110 For the steps needed to set up a development host that runs Linux,
111 see the
112 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host'>Setting Up a Native Linux Host</ulink>"
113 section in the Yocto Project Development Tasks Manual.
114 </para>
115
116 <para>
117 Once your development host is set up to use the Yocto Project,
118 several methods exist for you to do work in the Yocto Project
119 environment:
120 <itemizedlist>
121 <listitem><para>
122 <emphasis>Command Lines, BitBake, and Shells:</emphasis>
123 Traditional development in the Yocto Project involves using the
124 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
125 which uses BitBake, in a command-line environment from a shell
126 on your development host.
127 You can accomplish this from a host that is a native Linux
128 machine or from a host that has been set up with CROPS.
129 Either way, you create, modify, and build images and
130 applications all within a shell-based environment using
131 components and tools available through your Linux distribution
132 and the Yocto Project.</para>
133
134 <para>For a general flow of the build procedures, see the
135 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-building-a-simple-image'>Building a Simple Image</ulink>"
136 section in the Yocto Project Development Tasks Manual.
137 </para></listitem>
138 <listitem><para>
139 <emphasis>Board Support Package (BSP) Development:</emphasis>
140 Development of BSPs involves using the Yocto Project to
141 create and test layers that allow easy development of
142 images and applications targeted for specific hardware.
143 To development BSPs, you need to take some additional steps
144 beyond what was described in setting up a development host.
145 </para>
146
147 <para>The
148 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>
149 provides BSP-related development information.
150 For specifics on development host preparation, see the
151 "<ulink url='&YOCTO_DOCS_BSP_URL;#preparing-your-build-host-to-work-with-bsp-layers'>Preparing Your Build Host to Work With BSP Layers</ulink>"
152 section in the Yocto Project Board Support Package (BSP)
153 Developer's Guide.
154 </para></listitem>
155 <listitem><para>
156 <emphasis>Kernel Development:</emphasis>
157 If you are going to be developing kernels using the Yocto
158 Project you likely will be using <filename>devtool</filename>.
159 A workflow using <filename>devtool</filename> makes kernel
160 development quicker by reducing iteration cycle times.</para>
161
162 <para>The
163 <ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;'>Yocto Project Linux Kernel Development Manual</ulink>
164 provides kernel-related development information.
165 For specifics on development host preparation, see the
166 "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#preparing-the-build-host-to-work-on-the-kernel'>Preparing the Build Host to Work on the Kernel</ulink>"
167 section in the Yocto Project Linux Kernel Development Manual.
168 </para></listitem>
169 <listitem><para>
170 <emphasis>Using Toaster:</emphasis>
171 The other Yocto Project development method that involves an
172 interface that effectively puts the Yocto Project into the
173 background is Toaster.
174 Toaster provides an interface to the OpenEmbedded build system.
175 The interface enables you to configure and run your builds.
176 Information about builds is collected and stored in a database.
177 You can use Toaster to configure and start builds on multiple
178 remote build servers.</para>
179
180 <para>For steps that show you how to set up your development
181 host to use Toaster and on how to use Toaster in general,
182 see the
183 <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>.
184 </para></listitem>
185 </itemizedlist>
186 </para>
187</section>
188
189<section id='yocto-project-repositories'>
190 <title>Yocto Project Source Repositories</title>
191
192 <para>
193 The Yocto Project team maintains complete source repositories for all
194 Yocto Project files at
195 <ulink url='&YOCTO_GIT_URL;'></ulink>.
196 This web-based source code browser is organized into categories by
197 function such as IDE Plugins, Matchbox, Poky, Yocto Linux Kernel, and
198 so forth.
199 From the interface, you can click on any particular item in the "Name"
200 column and see the URL at the bottom of the page that you need to clone
201 a Git repository for that particular item.
202 Having a local Git repository of the
203 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>,
204 which is usually named "poky", allows
205 you to make changes, contribute to the history, and ultimately enhance
206 the Yocto Project's tools, Board Support Packages, and so forth.
207 </para>
208
209 <para>
210 For any supported release of Yocto Project, you can also go to the
211 <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink> and
212 select the "DOWNLOADS" item from the "SOFTWARE" menu and get a
213 released tarball of the <filename>poky</filename> repository, any
214 supported BSP tarball, or Yocto Project tools.
215 Unpacking these tarballs gives you a snapshot of the released
216 files.
217 <note><title>Notes</title>
218 <itemizedlist>
219 <listitem><para>
220 The recommended method for setting up the Yocto Project
221 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
222 and the files for supported BSPs
223 (e.g., <filename>meta-intel</filename>) is to use
224 <link linkend='git'>Git</link> to create a local copy of
225 the upstream repositories.
226 </para></listitem>
227 <listitem><para>
228 Be sure to always work in matching branches for both
229 the selected BSP repository and the Source Directory
230 (i.e. <filename>poky</filename>) repository.
231 For example, if you have checked out the "master" branch
232 of <filename>poky</filename> and you are going to use
233 <filename>meta-intel</filename>, be sure to checkout the
234 "master" branch of <filename>meta-intel</filename>.
235 </para></listitem>
236 </itemizedlist>
237 </note>
238 </para>
239
240 <para>
241 In summary, here is where you can get the project files needed for
242 development:
243 <itemizedlist>
244 <listitem><para id='source-repositories'>
245 <emphasis>
246 <ulink url='&YOCTO_GIT_URL;'>Source Repositories:</ulink>
247 </emphasis>
248 This area contains IDE Plugins, Matchbox, Poky, Poky Support,
249 Tools, Yocto Linux Kernel, and Yocto Metadata Layers.
250 You can create local copies of Git repositories for each of
251 these areas.</para>
252
253 <para>
254 <imagedata fileref="figures/source-repos.png" align="center" width="6in" depth="4in" />
255 For steps on how to view and access these upstream Git
256 repositories, see the
257 "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-source-repositories'>Accessing Source Repositories</ulink>"
258 Section in the Yocto Project Development Tasks Manual.
259 </para></listitem>
260 <listitem><para><anchor id='index-downloads' />
261 <emphasis>
262 <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink>
263 </emphasis>
264 This is an index of releases such as Poky, Pseudo, installers
265 for cross-development toolchains, miscellaneous support
266 and all released versions of Yocto Project in the form of
267 images or tarballs.
268 Downloading and extracting these files does not produce a local
269 copy of the Git repository but rather a snapshot of a
270 particular release or image.</para>
271
272 <para>
273 <imagedata fileref="figures/index-downloads.png" align="center" width="6in" depth="3.5in" />
274 For steps on how to view and access these files, see the
275 "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-index-of-releases'>Accessing Index of Releases</ulink>"
276 section in the Yocto Project Development Tasks Manual.
277 </para></listitem>
278 <listitem><para id='downloads-page'>
279 <emphasis>"DOWNLOADS" page for the
280 <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>:
281 </emphasis></para>
282
283 <para>The Yocto Project website includes a "DOWNLOADS" page
284 accessible through the "SOFTWARE" menu that allows you to
285 download any Yocto Project release, tool, and Board Support
286 Package (BSP) in tarball form.
287 The tarballs are similar to those found in the
288 <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink>
289 area.</para>
290
291 <para>
292 <imagedata fileref="figures/yp-download.png" align="center" width="6in" depth="4in" />
293 For steps on how to use the "DOWNLOADS" page, see the
294 "<ulink url='&YOCTO_DOCS_DEV_URL;#using-the-downloads-page'>Using the Downloads Page</ulink>"
295 section in the Yocto Project Development Tasks Manual.
296 </para></listitem>
297 </itemizedlist>
298 </para>
299</section>
300
301<section id='gs-git-workflows-and-the-yocto-project'>
302 <title>Git Workflows and the Yocto Project</title>
303
304 <para>
305 Developing using the Yocto Project likely requires the use of
306 <link linkend='git'>Git</link>.
307 Git is a free, open source distributed version control system
308 used as part of many collaborative design environments.
309 This section provides workflow concepts using the Yocto Project and
310 Git.
311 In particular, the information covers basic practices that describe
312 roles and actions in a collaborative development environment.
313 <note>
314 If you are familiar with this type of development environment, you
315 might not want to read this section.
316 </note>
317 </para>
318
319 <para>
320 The Yocto Project files are maintained using Git in "branches"
321 whose Git histories track every change and whose structures
322 provide branches for all diverging functionality.
323 Although there is no need to use Git, many open source projects do so.
324 <para>
325
326 </para>
327 For the Yocto Project, a key individual called the "maintainer" is
328 responsible for the integrity of the "master" branch of a given Git
329 repository.
330 The "master" branch is the "upstream" repository from which final or
331 most recent builds of a project occur.
332 The maintainer is responsible for accepting changes from other
333 developers and for organizing the underlying branch structure to
334 reflect release strategies and so forth.
335 <note>
336 For information on finding out who is responsible for (maintains)
337 a particular area of code in the Yocto Project, see the
338 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
339 section of the Yocto Project Development Tasks Manual.
340 </note>
341 </para>
342
343 <para>
344 The Yocto Project <filename>poky</filename> Git repository also has an
345 upstream contribution Git repository named
346 <filename>poky-contrib</filename>.
347 You can see all the branches in this repository using the web interface
348 of the
349 <ulink url='&YOCTO_GIT_URL;'>Source Repositories</ulink> organized
350 within the "Poky Support" area.
351 These branches hold changes (commits) to the project that have been
352 submitted or committed by the Yocto Project development team and by
353 community members who contribute to the project.
354 The maintainer determines if the changes are qualified to be moved
355 from the "contrib" branches into the "master" branch of the Git
356 repository.
357 </para>
358
359 <para>
360 Developers (including contributing community members) create and
361 maintain cloned repositories of upstream branches.
362 The cloned repositories are local to their development platforms and
363 are used to develop changes.
364 When a developer is satisfied with a particular feature or change,
365 they "push" the change to the appropriate "contrib" repository.
366 </para>
367
368 <para>
369 Developers are responsible for keeping their local repository
370 up-to-date with whatever upstream branch they are working against.
371 They are also responsible for straightening out any conflicts that
372 might arise within files that are being worked on simultaneously by
373 more than one person.
374 All this work is done locally on the development host before
375 anything is pushed to a "contrib" area and examined at the maintainer's
376 level.
377 </para>
378
379 <para>
380 A somewhat formal method exists by which developers commit changes
381 and push them into the "contrib" area and subsequently request that
382 the maintainer include them into an upstream branch.
383 This process is called "submitting a patch" or "submitting a change."
384 For information on submitting patches and changes, see the
385 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
386 section in the Yocto Project Development Tasks Manual.
387 </para>
388
389 <para>
390 In summary, a single point of entry
391 exists for changes into a "master" or development branch of the
392 Git repository, which is controlled by the project's maintainer.
393 And, a set of developers exist who independently develop, test, and
394 submit changes to "contrib" areas for the maintainer to examine.
395 The maintainer then chooses which changes are going to become a
396 permanent part of the project.
397 </para>
398
399 <para>
400 <imagedata fileref="figures/git-workflow.png" width="6in" depth="3in" align="left" scalefit="1" />
401 </para>
402
403 <para>
404 While each development environment is unique, there are some best
405 practices or methods that help development run smoothly.
406 The following list describes some of these practices.
407 For more information about Git workflows, see the workflow topics in
408 the
409 <ulink url='http://book.git-scm.com'>Git Community Book</ulink>.
410 <itemizedlist>
411 <listitem><para>
412 <emphasis>Make Small Changes:</emphasis>
413 It is best to keep the changes you commit small as compared to
414 bundling many disparate changes into a single commit.
415 This practice not only keeps things manageable but also allows
416 the maintainer to more easily include or refuse changes.
417 </para></listitem>
418 <listitem><para>
419 <emphasis>Make Complete Changes:</emphasis>
420 It is also good practice to leave the repository in a
421 state that allows you to still successfully build your project.
422 In other words, do not commit half of a feature,
423 then add the other half as a separate, later commit.
424 Each commit should take you from one buildable project state
425 to another buildable state.
426 </para></listitem>
427 <listitem><para>
428 <emphasis>Use Branches Liberally:</emphasis>
429 It is very easy to create, use, and delete local branches in
430 your working Git repository on the development host.
431 You can name these branches anything you like.
432 It is helpful to give them names associated with the particular
433 feature or change on which you are working.
434 Once you are done with a feature or change and have merged it
435 into your local master branch, simply discard the temporary
436 branch.
437 </para></listitem>
438 <listitem><para>
439 <emphasis>Merge Changes:</emphasis>
440 The <filename>git merge</filename> command allows you to take
441 the changes from one branch and fold them into another branch.
442 This process is especially helpful when more than a single
443 developer might be working on different parts of the same
444 feature.
445 Merging changes also automatically identifies any collisions
446 or "conflicts" that might happen as a result of the same lines
447 of code being altered by two different developers.
448 </para></listitem>
449 <listitem><para>
450 <emphasis>Manage Branches:</emphasis>
451 Because branches are easy to use, you should use a system
452 where branches indicate varying levels of code readiness.
453 For example, you can have a "work" branch to develop in, a
454 "test" branch where the code or change is tested, a "stage"
455 branch where changes are ready to be committed, and so forth.
456 As your project develops, you can merge code across the
457 branches to reflect ever-increasing stable states of the
458 development.
459 </para></listitem>
460 <listitem><para>
461 <emphasis>Use Push and Pull:</emphasis>
462 The push-pull workflow is based on the concept of developers
463 "pushing" local commits to a remote repository, which is
464 usually a contribution repository.
465 This workflow is also based on developers "pulling" known
466 states of the project down into their local development
467 repositories.
468 The workflow easily allows you to pull changes submitted by
469 other developers from the upstream repository into your
470 work area ensuring that you have the most recent software
471 on which to develop.
472 The Yocto Project has two scripts named
473 <filename>create-pull-request</filename> and
474 <filename>send-pull-request</filename> that ship with the
475 release to facilitate this workflow.
476 You can find these scripts in the <filename>scripts</filename>
477 folder of the
478 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
479 For information on how to use these scripts, see the
480 "<ulink url='&YOCTO_DOCS_DEV_URL;#pushing-a-change-upstream'>Using Scripts to Push a Change Upstream and Request a Pull</ulink>"
481 section in the Yocto Project Development Tasks Manual.
482 </para></listitem>
483 <listitem><para>
484 <emphasis>Patch Workflow:</emphasis>
485 This workflow allows you to notify the maintainer through an
486 email that you have a change (or patch) you would like
487 considered for the "master" branch of the Git repository.
488 To send this type of change, you format the patch and then
489 send the email using the Git commands
490 <filename>git format-patch</filename> and
491 <filename>git send-email</filename>.
492 For information on how to use these scripts, see the
493 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
494 section in the Yocto Project Development Tasks Manual.
495 </para></listitem>
496 </itemizedlist>
497 </para>
498</section>
499
500<section id='git'>
501 <title>Git</title>
502
503 <para>
504 The Yocto Project makes extensive use of Git, which is a
505 free, open source distributed version control system.
506 Git supports distributed development, non-linear development,
507 and can handle large projects.
508 It is best that you have some fundamental understanding
509 of how Git tracks projects and how to work with Git if
510 you are going to use the Yocto Project for development.
511 This section provides a quick overview of how Git works and
512 provides you with a summary of some essential Git commands.
513 <note><title>Notes</title>
514 <itemizedlist>
515 <listitem><para>
516 For more information on Git, see
517 <ulink url='http://git-scm.com/documentation'></ulink>.
518 </para></listitem>
519 <listitem><para>
520 If you need to download Git, it is recommended that you add
521 Git to your system through your distribution's "software
522 store" (e.g. for Ubuntu, use the Ubuntu Software feature).
523 For the Git download page, see
524 <ulink url='http://git-scm.com/download'></ulink>.
525 </para></listitem>
526 <listitem><para>
527 For information beyond the introductory nature in this
528 section, see the
529 "<ulink url='&YOCTO_DOCS_DEV_URL;#locating-yocto-project-source-files'>Locating Yocto Project Source Files</ulink>"
530 section in the Yocto Project Development Tasks Manual.
531 </para></listitem>
532 </itemizedlist>
533 </note>
534 </para>
535
536 <section id='repositories-tags-and-branches'>
537 <title>Repositories, Tags, and Branches</title>
538
539 <para>
540 As mentioned briefly in the previous section and also in the
541 "<link linkend='gs-git-workflows-and-the-yocto-project'>Git Workflows and the Yocto Project</link>"
542 section, the Yocto Project maintains source repositories at
543 <ulink url='&YOCTO_GIT_URL;'></ulink>.
544 If you look at this web-interface of the repositories, each item
545 is a separate Git repository.
546 </para>
547
548 <para>
549 Git repositories use branching techniques that track content
550 change (not files) within a project (e.g. a new feature or updated
551 documentation).
552 Creating a tree-like structure based on project divergence allows
553 for excellent historical information over the life of a project.
554 This methodology also allows for an environment from which you can
555 do lots of local experimentation on projects as you develop
556 changes or new features.
557 </para>
558
559 <para>
560 A Git repository represents all development efforts for a given
561 project.
562 For example, the Git repository <filename>poky</filename> contains
563 all changes and developments for that repository over the course
564 of its entire life.
565 That means that all changes that make up all releases are captured.
566 The repository maintains a complete history of changes.
567 </para>
568
569 <para>
570 You can create a local copy of any repository by "cloning" it
571 with the <filename>git clone</filename> command.
572 When you clone a Git repository, you end up with an identical
573 copy of the repository on your development system.
574 Once you have a local copy of a repository, you can take steps to
575 develop locally.
576 For examples on how to clone Git repositories, see the
577 "<ulink url='&YOCTO_DOCS_DEV_URL;#locating-yocto-project-source-files'>Locating Yocto Project Source Files</ulink>"
578 section in the Yocto Project Development Tasks Manual.
579 </para>
580
581 <para>
582 It is important to understand that Git tracks content change and
583 not files.
584 Git uses "branches" to organize different development efforts.
585 For example, the <filename>poky</filename> repository has
586 several branches that include the current "&DISTRO_NAME_NO_CAP;"
587 branch, the "master" branch, and many branches for past
588 Yocto Project releases.
589 You can see all the branches by going to
590 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and
591 clicking on the
592 <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/heads'>[...]</ulink></filename>
593 link beneath the "Branch" heading.
594 </para>
595
596 <para>
597 Each of these branches represents a specific area of development.
598 The "master" branch represents the current or most recent
599 development.
600 All other branches represent offshoots of the "master" branch.
601 </para>
602
603 <para>
604 When you create a local copy of a Git repository, the copy has
605 the same set of branches as the original.
606 This means you can use Git to create a local working area
607 (also called a branch) that tracks a specific development branch
608 from the upstream source Git repository.
609 in other words, you can define your local Git environment to
610 work on any development branch in the repository.
611 To help illustrate, consider the following example Git commands:
612 <literallayout class='monospaced'>
613 $ cd ~
614 $ git clone git://git.yoctoproject.org/poky
615 $ cd poky
616 $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP;
617 </literallayout>
618 In the previous example after moving to the home directory, the
619 <filename>git clone</filename> command creates a
620 local copy of the upstream <filename>poky</filename> Git repository.
621 By default, Git checks out the "master" branch for your work.
622 After changing the working directory to the new local repository
623 (i.e. <filename>poky</filename>), the
624 <filename>git checkout</filename> command creates
625 and checks out a local branch named "&DISTRO_NAME_NO_CAP;", which
626 tracks the upstream "origin/&DISTRO_NAME_NO_CAP;" branch.
627 Changes you make while in this branch would ultimately affect
628 the upstream "&DISTRO_NAME_NO_CAP;" branch of the
629 <filename>poky</filename> repository.
630 </para>
631
632 <para>
633 It is important to understand that when you create and checkout a
634 local working branch based on a branch name,
635 your local environment matches the "tip" of that particular
636 development branch at the time you created your local branch,
637 which could be different from the files in the "master" branch
638 of the upstream repository.
639 In other words, creating and checking out a local branch based on
640 the "&DISTRO_NAME_NO_CAP;" branch name is not the same as
641 checking out the "master" branch in the repository.
642 Keep reading to see how you create a local snapshot of a Yocto
643 Project Release.
644 </para>
645
646 <para>
647 Git uses "tags" to mark specific changes in a repository branch
648 structure.
649 Typically, a tag is used to mark a special point such as the final
650 change (or commit) before a project is released.
651 You can see the tags used with the <filename>poky</filename> Git
652 repository by going to
653 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and
654 clicking on the
655 <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/tags'>[...]</ulink></filename>
656 link beneath the "Tag" heading.
657 </para>
658
659 <para>
660 Some key tags for the <filename>poky</filename> repository are
661 <filename>jethro-14.0.3</filename>,
662 <filename>morty-16.0.1</filename>,
663 <filename>pyro-17.0.0</filename>, and
664 <filename>&DISTRO_NAME_NO_CAP;-&POKYVERSION;</filename>.
665 These tags represent Yocto Project releases.
666 </para>
667
668 <para>
669 When you create a local copy of the Git repository, you also
670 have access to all the tags in the upstream repository.
671 Similar to branches, you can create and checkout a local working
672 Git branch based on a tag name.
673 When you do this, you get a snapshot of the Git repository that
674 reflects the state of the files when the change was made associated
675 with that tag.
676 The most common use is to checkout a working branch that matches
677 a specific Yocto Project release.
678 Here is an example:
679 <literallayout class='monospaced'>
680 $ cd ~
681 $ git clone git://git.yoctoproject.org/poky
682 $ cd poky
683 $ git fetch --tags
684 $ git checkout tags/rocko-18.0.0 -b my_rocko-18.0.0
685 </literallayout>
686 In this example, the name of the top-level directory of your
687 local Yocto Project repository is <filename>poky</filename>.
688 After moving to the <filename>poky</filename> directory, the
689 <filename>git fetch</filename> command makes all the upstream
690 tags available locally in your repository.
691 Finally, the <filename>git checkout</filename> command
692 creates and checks out a branch named "my-rocko-18.0.0" that is
693 based on the upstream branch whose "HEAD" matches the
694 commit in the repository associated with the "rocko-18.0.0" tag.
695 The files in your repository now exactly match that particular
696 Yocto Project release as it is tagged in the upstream Git
697 repository.
698 It is important to understand that when you create and
699 checkout a local working branch based on a tag, your environment
700 matches a specific point in time and not the entire development
701 branch (i.e. from the "tip" of the branch backwards).
702 </para>
703 </section>
704
705 <section id='basic-commands'>
706 <title>Basic Commands</title>
707
708 <para>
709 Git has an extensive set of commands that lets you manage changes
710 and perform collaboration over the life of a project.
711 Conveniently though, you can manage with a small set of basic
712 operations and workflows once you understand the basic
713 philosophy behind Git.
714 You do not have to be an expert in Git to be functional.
715 A good place to look for instruction on a minimal set of Git
716 commands is
717 <ulink url='http://git-scm.com/documentation'>here</ulink>.
718 </para>
719
720 <para>
721 The following list of Git commands briefly describes some basic
722 Git operations as a way to get started.
723 As with any set of commands, this list (in most cases) simply shows
724 the base command and omits the many arguments it supports.
725 See the Git documentation for complete descriptions and strategies
726 on how to use these commands:
727 <itemizedlist>
728 <listitem><para>
729 <emphasis><filename>git init</filename>:</emphasis>
730 Initializes an empty Git repository.
731 You cannot use Git commands unless you have a
732 <filename>.git</filename> repository.
733 </para></listitem>
734 <listitem><para id='git-commands-clone'>
735 <emphasis><filename>git clone</filename>:</emphasis>
736 Creates a local clone of a Git repository that is on
737 equal footing with a fellow developer's Git repository
738 or an upstream repository.
739 </para></listitem>
740 <listitem><para>
741 <emphasis><filename>git add</filename>:</emphasis>
742 Locally stages updated file contents to the index that
743 Git uses to track changes.
744 You must stage all files that have changed before you
745 can commit them.
746 </para></listitem>
747 <listitem><para>
748 <emphasis><filename>git commit</filename>:</emphasis>
749 Creates a local "commit" that documents the changes you
750 made.
751 Only changes that have been staged can be committed.
752 Commits are used for historical purposes, for determining
753 if a maintainer of a project will allow the change,
754 and for ultimately pushing the change from your local
755 Git repository into the project's upstream repository.
756 </para></listitem>
757 <listitem><para>
758 <emphasis><filename>git status</filename>:</emphasis>
759 Reports any modified files that possibly need to be
760 staged and gives you a status of where you stand regarding
761 local commits as compared to the upstream repository.
762 </para></listitem>
763 <listitem><para>
764 <emphasis><filename>git checkout</filename> <replaceable>branch-name</replaceable>:</emphasis>
765 Changes your local working branch and in this form
766 assumes the local branch already exists.
767 This command is analogous to "cd".
768 </para></listitem>
769 <listitem><para>
770 <emphasis><filename>git checkout –b</filename> <replaceable>working-branch</replaceable> <replaceable>upstream-branch</replaceable>:</emphasis>
771 Creates and checks out a working branch on your local
772 machine.
773 The local branch tracks the upstream branch.
774 You can use your local branch to isolate your work.
775 It is a good idea to use local branches when adding
776 specific features or changes.
777 Using isolated branches facilitates easy removal of
778 changes if they do not work out.
779 </para></listitem>
780 <listitem><para><emphasis><filename>git branch</filename>:</emphasis>
781 Displays the existing local branches associated with your
782 local repository.
783 The branch that you have currently checked out is noted
784 with an asterisk character.
785 </para></listitem>
786 <listitem><para>
787 <emphasis><filename>git branch -D</filename> <replaceable>branch-name</replaceable>:</emphasis>
788 Deletes an existing local branch.
789 You need to be in a local branch other than the one you
790 are deleting in order to delete
791 <replaceable>branch-name</replaceable>.
792 </para></listitem>
793 <listitem><para>
794 <emphasis><filename>git pull --rebase</filename>:</emphasis>
795 Retrieves information from an upstream Git repository
796 and places it in your local Git repository.
797 You use this command to make sure you are synchronized with
798 the repository from which you are basing changes
799 (.e.g. the "master" branch).
800 The "--rebase" option ensures that any local commits you
801 have in your branch are preserved at the top of your
802 local branch.
803 </para></listitem>
804 <listitem><para>
805 <emphasis><filename>git push</filename> <replaceable>repo-name</replaceable> <replaceable>local-branch</replaceable><filename>:</filename><replaceable>upstream-branch</replaceable>:</emphasis>
806 Sends all your committed local changes to the upstream Git
807 repository that your local repository is tracking
808 (e.g. a contribution repository).
809 The maintainer of the project draws from these repositories
810 to merge changes (commits) into the appropriate branch
811 of project's upstream repository.
812 </para></listitem>
813 <listitem><para>
814 <emphasis><filename>git merge</filename>:</emphasis>
815 Combines or adds changes from one
816 local branch of your repository with another branch.
817 When you create a local Git repository, the default branch
818 is named "master".
819 A typical workflow is to create a temporary branch that is
820 based off "master" that you would use for isolated work.
821 You would make your changes in that isolated branch,
822 stage and commit them locally, switch to the "master"
823 branch, and then use the <filename>git merge</filename>
824 command to apply the changes from your isolated branch
825 into the currently checked out branch (e.g. "master").
826 After the merge is complete and if you are done with
827 working in that isolated branch, you can safely delete
828 the isolated branch.
829 </para></listitem>
830 <listitem><para>
831 <emphasis><filename>git cherry-pick</filename> <replaceable>commits</replaceable>:</emphasis>
832 Choose and apply specific commits from one branch
833 into another branch.
834 There are times when you might not be able to merge
835 all the changes in one branch with
836 another but need to pick out certain ones.
837 </para></listitem>
838 <listitem><para>
839 <emphasis><filename>gitk</filename>:</emphasis>
840 Provides a GUI view of the branches and changes in your
841 local Git repository.
842 This command is a good way to graphically see where things
843 have diverged in your local repository.
844 <note>
845 You need to install the <filename>gitk</filename>
846 package on your development system to use this
847 command.
848 </note>
849 </para></listitem>
850 <listitem><para>
851 <emphasis><filename>git log</filename>:</emphasis>
852 Reports a history of your commits to the repository.
853 This report lists all commits regardless of whether you
854 have pushed them upstream or not.
855 </para></listitem>
856 <listitem><para>
857 <emphasis><filename>git diff</filename>:</emphasis>
858 Displays line-by-line differences between a local
859 working file and the same file as understood by Git.
860 This command is useful to see what you have changed
861 in any given file.
862 </para></listitem>
863 </itemizedlist>
864 </para>
865 </section>
866</section>
867
868<section id='licensing'>
869 <title>Licensing</title>
870
871 <para>
872 Because open source projects are open to the public, they have
873 different licensing structures in place.
874 License evolution for both Open Source and Free Software has an
875 interesting history.
876 If you are interested in this history, you can find basic information
877 here:
878 <itemizedlist>
879 <listitem><para>
880 <ulink url='http://en.wikipedia.org/wiki/Open-source_license'>Open source license history</ulink>
881 </para></listitem>
882 <listitem><para>
883 <ulink url='http://en.wikipedia.org/wiki/Free_software_license'>Free software license history</ulink>
884 </para></listitem>
885 </itemizedlist>
886 </para>
887
888 <para>
889 In general, the Yocto Project is broadly licensed under the
890 Massachusetts Institute of Technology (MIT) License.
891 MIT licensing permits the reuse of software within proprietary
892 software as long as the license is distributed with that software.
893 MIT is also compatible with the GNU General Public License (GPL).
894 Patches to the Yocto Project follow the upstream licensing scheme.
895 You can find information on the MIT license
896 <ulink url='http://www.opensource.org/licenses/mit-license.php'>here</ulink>.
897 You can find information on the GNU GPL
898 <ulink url='http://www.opensource.org/licenses/LGPL-3.0'>here</ulink>.
899 </para>
900
901 <para>
902 When you build an image using the Yocto Project, the build process
903 uses a known list of licenses to ensure compliance.
904 You can find this list in the
905 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
906 at <filename>meta/files/common-licenses</filename>.
907 Once the build completes, the list of all licenses found and used
908 during that build are kept in the
909 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
910 at <filename>tmp/deploy/licenses</filename>.
911 </para>
912
913 <para>
914 If a module requires a license that is not in the base list, the
915 build process generates a warning during the build.
916 These tools make it easier for a developer to be certain of the
917 licenses with which their shipped products must comply.
918 However, even with these tools it is still up to the developer to
919 resolve potential licensing issues.
920 </para>
921
922 <para>
923 The base list of licenses used by the build process is a combination
924 of the Software Package Data Exchange (SPDX) list and the Open
925 Source Initiative (OSI) projects.
926 <ulink url='http://spdx.org'>SPDX Group</ulink> is a working group of
927 the Linux Foundation that maintains a specification for a standard
928 format for communicating the components, licenses, and copyrights
929 associated with a software package.
930 <ulink url='http://opensource.org'>OSI</ulink> is a corporation
931 dedicated to the Open Source Definition and the effort for reviewing
932 and approving licenses that conform to the Open Source Definition
933 (OSD).
934 </para>
935
936 <para>
937 You can find a list of the combined SPDX and OSI licenses that the
938 Yocto Project uses in the
939 <filename>meta/files/common-licenses</filename> directory in your
940 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
941 </para>
942
943 <para>
944 For information that can help you maintain compliance with various
945 open source licensing during the lifecycle of a product created using
946 the Yocto Project, see the
947 "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Product's Lifecycle</ulink>"
948 section in the Yocto Project Development Tasks Manual.
949 </para>
950</section>
951</chapter>
952<!--
953vim: expandtab tw=80 ts=4
954-->
diff --git a/documentation/overview-manual/overview-manual-intro.xml b/documentation/overview-manual/overview-manual-intro.xml
deleted file mode 100644
index 0e0bfed6e5..0000000000
--- a/documentation/overview-manual/overview-manual-intro.xml
+++ /dev/null
@@ -1,113 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='overview-manual-intro'>
7
8<title>The Yocto Project Overview and Concepts Manual</title>
9 <section id='overview-manual-welcome'>
10 <title>Welcome</title>
11
12 <para>
13 Welcome to the Yocto Project Overview and Concepts Manual!
14 This manual introduces the Yocto Project by providing concepts,
15 software overviews, best-known-methods (BKMs), and any other
16 high-level introductory information suitable for a new Yocto
17 Project user.
18 </para>
19
20 <para>
21 The following list describes what you can get from this manual:
22 <itemizedlist>
23 <listitem><para>
24 <emphasis><link linkend='overview-yp'>Introducing the Yocto Project</link>:</emphasis>
25 This chapter provides an introduction to the Yocto
26 Project.
27 You will learn about features and challenges of the
28 Yocto Project, the layer model, components and tools,
29 development methods, the
30 <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>
31 reference distribution, the OpenEmbedded build system
32 workflow, and some basic Yocto terms.
33 </para></listitem>
34 <listitem><para>
35 <emphasis><link linkend='overview-development-environment'>The Yocto Project Development Environment</link>:</emphasis>
36 This chapter helps you get started understanding the
37 Yocto Project development environment.
38 You will learn about open source, development hosts,
39 Yocto Project source repositories, workflows using Git
40 and the Yocto Project, a Git primer, and information
41 about licensing.
42 </para></listitem>
43 <listitem><para>
44 <emphasis><link linkend='overview-manual-concepts'>Yocto Project Concepts</link>:</emphasis>
45 This chapter presents various concepts regarding the
46 Yocto Project.
47 You can find conceptual information about components,
48 development, cross-toolchains, and so forth.
49 </para></listitem>
50 </itemizedlist>
51 </para>
52
53 <para>
54 This manual does not give you the following:
55 <itemizedlist>
56 <listitem><para>
57 <emphasis>Step-by-step Instructions for Development Tasks:</emphasis>
58 Instructional procedures reside in other manuals within
59 the Yocto Project documentation set.
60 For example, the
61 <ulink url='&YOCTO_DOCS_DEV_URL;'>Yocto Project Development Tasks Manual</ulink>
62 provides examples on how to perform various development
63 tasks.
64 As another example, the
65 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
66 manual contains detailed instructions on how to install an
67 SDK, which is used to develop applications for target
68 hardware.
69 </para></listitem>
70 <listitem><para>
71 <emphasis>Reference Material:</emphasis>
72 This type of material resides in an appropriate reference
73 manual.
74 For example, system variables are documented in the
75 <ulink url='&YOCTO_DOCS_REF_URL;'>Yocto Project Reference Manual</ulink>.
76 As another example, the
77 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>
78 contains reference information on BSPs.
79 </para></listitem>
80 <listitem><para>
81 <emphasis>Detailed Public Information Not Specific to the
82 Yocto Project:</emphasis>
83 For example, exhaustive information on how to use the
84 Source Control Manager Git is better covered with Internet
85 searches and official Git Documentation than through the
86 Yocto Project documentation.
87 </para></listitem>
88 </itemizedlist>
89 </para>
90 </section>
91
92 <section id='overview-manual-other-information'>
93 <title>Other Information</title>
94
95 <para>
96 Because this manual presents information for many different
97 topics, supplemental information is recommended for full
98 comprehension.
99 For additional introductory information on the Yocto Project, see
100 the <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>.
101 If you want to build an image with no knowledge of Yocto Project
102 as a way of quickly testing it out, see the
103 <ulink url='&YOCTO_DOCS_BRIEF_URL;'>Yocto Project Quick Build</ulink>
104 document.
105 For a comprehensive list of links and other documentation, see the
106 "<ulink url='&YOCTO_DOCS_REF_URL;#resources-links-and-related-documentation'>Links and Related Documentation</ulink>"
107 section in the Yocto Project Reference Manual.
108 </para>
109 </section>
110</chapter>
111<!--
112vim: expandtab tw=80 ts=4
113-->
diff --git a/documentation/overview-manual/overview-manual-style.css b/documentation/overview-manual/overview-manual-style.css
deleted file mode 100644
index eec934161a..0000000000
--- a/documentation/overview-manual/overview-manual-style.css
+++ /dev/null
@@ -1,990 +0,0 @@
1/*
2 SPDX-License-Identifier: CC-BY-2.0-UK
3
4 Generic XHTML / DocBook XHTML CSS Stylesheet.
5
6 Browser wrangling and typographic design by
7 Oyvind Kolas / pippin@gimp.org
8
9 Customised for Poky by
10 Matthew Allum / mallum@o-hand.com
11
12 Thanks to:
13 Liam R. E. Quin
14 William Skaggs
15 Jakub Steiner
16
17 Structure
18 ---------
19
20 The stylesheet is divided into the following sections:
21
22 Positioning
23 Margins, paddings, width, font-size, clearing.
24 Decorations
25 Borders, style
26 Colors
27 Colors
28 Graphics
29 Graphical backgrounds
30 Nasty IE tweaks
31 Workarounds needed to make it work in internet explorer,
32 currently makes the stylesheet non validating, but up until
33 this point it is validating.
34 Mozilla extensions
35 Transparency for footer
36 Rounded corners on boxes
37
38*/
39
40
41 /*************** /
42 / Positioning /
43/ ***************/
44
45body {
46 font-family: Verdana, Sans, sans-serif;
47
48 min-width: 640px;
49 width: 80%;
50 margin: 0em auto;
51 padding: 2em 5em 5em 5em;
52 color: #333;
53}
54
55h1,h2,h3,h4,h5,h6,h7 {
56 font-family: Arial, Sans;
57 color: #00557D;
58 clear: both;
59}
60
61h1 {
62 font-size: 2em;
63 text-align: left;
64 padding: 0em 0em 0em 0em;
65 margin: 2em 0em 0em 0em;
66}
67
68h2.subtitle {
69 margin: 0.10em 0em 3.0em 0em;
70 padding: 0em 0em 0em 0em;
71 font-size: 1.8em;
72 padding-left: 20%;
73 font-weight: normal;
74 font-style: italic;
75}
76
77h2 {
78 margin: 2em 0em 0.66em 0em;
79 padding: 0.5em 0em 0em 0em;
80 font-size: 1.5em;
81 font-weight: bold;
82}
83
84h3.subtitle {
85 margin: 0em 0em 1em 0em;
86 padding: 0em 0em 0em 0em;
87 font-size: 142.14%;
88 text-align: right;
89}
90
91h3 {
92 margin: 1em 0em 0.5em 0em;
93 padding: 1em 0em 0em 0em;
94 font-size: 140%;
95 font-weight: bold;
96}
97
98h4 {
99 margin: 1em 0em 0.5em 0em;
100 padding: 1em 0em 0em 0em;
101 font-size: 120%;
102 font-weight: bold;
103}
104
105h5 {
106 margin: 1em 0em 0.5em 0em;
107 padding: 1em 0em 0em 0em;
108 font-size: 110%;
109 font-weight: bold;
110}
111
112h6 {
113 margin: 1em 0em 0em 0em;
114 padding: 1em 0em 0em 0em;
115 font-size: 110%;
116 font-weight: bold;
117}
118
119.authorgroup {
120 background-color: transparent;
121 background-repeat: no-repeat;
122 padding-top: 256px;
123 background-image: url("figures/overview-manual-title.png");
124 background-position: left top;
125 margin-top: -256px;
126 padding-right: 50px;
127 margin-left: 0px;
128 text-align: right;
129 width: 740px;
130}
131
132h3.author {
133 margin: 0em 0me 0em 0em;
134 padding: 0em 0em 0em 0em;
135 font-weight: normal;
136 font-size: 100%;
137 color: #333;
138 clear: both;
139}
140
141.author tt.email {
142 font-size: 66%;
143}
144
145.titlepage hr {
146 width: 0em;
147 clear: both;
148}
149
150.revhistory {
151 padding-top: 2em;
152 clear: both;
153}
154
155.toc,
156.list-of-tables,
157.list-of-examples,
158.list-of-figures {
159 padding: 1.33em 0em 2.5em 0em;
160 color: #00557D;
161}
162
163.toc p,
164.list-of-tables p,
165.list-of-figures p,
166.list-of-examples p {
167 padding: 0em 0em 0em 0em;
168 padding: 0em 0em 0.3em;
169 margin: 1.5em 0em 0em 0em;
170}
171
172.toc p b,
173.list-of-tables p b,
174.list-of-figures p b,
175.list-of-examples p b{
176 font-size: 100.0%;
177 font-weight: bold;
178}
179
180.toc dl,
181.list-of-tables dl,
182.list-of-figures dl,
183.list-of-examples dl {
184 margin: 0em 0em 0.5em 0em;
185 padding: 0em 0em 0em 0em;
186}
187
188.toc dt {
189 margin: 0em 0em 0em 0em;
190 padding: 0em 0em 0em 0em;
191}
192
193.toc dd {
194 margin: 0em 0em 0em 2.6em;
195 padding: 0em 0em 0em 0em;
196}
197
198div.glossary dl,
199div.variablelist dl {
200}
201
202.glossary dl dt,
203.variablelist dl dt,
204.variablelist dl dt span.term {
205 font-weight: normal;
206 width: 20em;
207 text-align: right;
208}
209
210.variablelist dl dt {
211 margin-top: 0.5em;
212}
213
214.glossary dl dd,
215.variablelist dl dd {
216 margin-top: -1em;
217 margin-left: 25.5em;
218}
219
220.glossary dd p,
221.variablelist dd p {
222 margin-top: 0em;
223 margin-bottom: 1em;
224}
225
226
227div.calloutlist table td {
228 padding: 0em 0em 0em 0em;
229 margin: 0em 0em 0em 0em;
230}
231
232div.calloutlist table td p {
233 margin-top: 0em;
234 margin-bottom: 1em;
235}
236
237div p.copyright {
238 text-align: left;
239}
240
241div.legalnotice p.legalnotice-title {
242 margin-bottom: 0em;
243}
244
245p {
246 line-height: 1.5em;
247 margin-top: 0em;
248
249}
250
251dl {
252 padding-top: 0em;
253}
254
255hr {
256 border: solid 1px;
257}
258
259
260.mediaobject,
261.mediaobjectco {
262 text-align: center;
263}
264
265img {
266 border: none;
267}
268
269ul {
270 padding: 0em 0em 0em 1.5em;
271}
272
273ul li {
274 padding: 0em 0em 0em 0em;
275}
276
277ul li p {
278 text-align: left;
279}
280
281table {
282 width :100%;
283}
284
285th {
286 padding: 0.25em;
287 text-align: left;
288 font-weight: normal;
289 vertical-align: top;
290}
291
292td {
293 padding: 0.25em;
294 vertical-align: top;
295}
296
297p a[id] {
298 margin: 0px;
299 padding: 0px;
300 display: inline;
301 background-image: none;
302}
303
304a {
305 text-decoration: underline;
306 color: #444;
307}
308
309pre {
310 overflow: auto;
311}
312
313a:hover {
314 text-decoration: underline;
315 /*font-weight: bold;*/
316}
317
318/* This style defines how the permalink character
319 appears by itself and when hovered over with
320 the mouse. */
321
322[alt='Permalink'] { color: #eee; }
323[alt='Permalink']:hover { color: black; }
324
325
326div.informalfigure,
327div.informalexample,
328div.informaltable,
329div.figure,
330div.table,
331div.example {
332 margin: 1em 0em;
333 padding: 1em;
334 page-break-inside: avoid;
335}
336
337
338div.informalfigure p.title b,
339div.informalexample p.title b,
340div.informaltable p.title b,
341div.figure p.title b,
342div.example p.title b,
343div.table p.title b{
344 padding-top: 0em;
345 margin-top: 0em;
346 font-size: 100%;
347 font-weight: normal;
348}
349
350.mediaobject .caption,
351.mediaobject .caption p {
352 text-align: center;
353 font-size: 80%;
354 padding-top: 0.5em;
355 padding-bottom: 0.5em;
356}
357
358.epigraph {
359 padding-left: 55%;
360 margin-bottom: 1em;
361}
362
363.epigraph p {
364 text-align: left;
365}
366
367.epigraph .quote {
368 font-style: italic;
369}
370.epigraph .attribution {
371 font-style: normal;
372 text-align: right;
373}
374
375span.application {
376 font-style: italic;
377}
378
379.programlisting {
380 font-family: monospace;
381 font-size: 80%;
382 white-space: pre;
383 margin: 1.33em 0em;
384 padding: 1.33em;
385}
386
387.tip,
388.warning,
389.caution,
390.note {
391 margin-top: 1em;
392 margin-bottom: 1em;
393
394}
395
396/* force full width of table within div */
397.tip table,
398.warning table,
399.caution table,
400.note table {
401 border: none;
402 width: 100%;
403}
404
405
406.tip table th,
407.warning table th,
408.caution table th,
409.note table th {
410 padding: 0.8em 0.0em 0.0em 0.0em;
411 margin : 0em 0em 0em 0em;
412}
413
414.tip p,
415.warning p,
416.caution p,
417.note p {
418 margin-top: 0.5em;
419 margin-bottom: 0.5em;
420 padding-right: 1em;
421 text-align: left;
422}
423
424.acronym {
425 text-transform: uppercase;
426}
427
428b.keycap,
429.keycap {
430 padding: 0.09em 0.3em;
431 margin: 0em;
432}
433
434.itemizedlist li {
435 clear: none;
436}
437
438.filename {
439 font-size: medium;
440 font-family: Courier, monospace;
441}
442
443
444div.navheader, div.heading{
445 position: absolute;
446 left: 0em;
447 top: 0em;
448 width: 100%;
449 background-color: #cdf;
450 width: 100%;
451}
452
453div.navfooter, div.footing{
454 position: fixed;
455 left: 0em;
456 bottom: 0em;
457 background-color: #eee;
458 width: 100%;
459}
460
461
462div.navheader td,
463div.navfooter td {
464 font-size: 66%;
465}
466
467div.navheader table th {
468 /*font-family: Georgia, Times, serif;*/
469 /*font-size: x-large;*/
470 font-size: 80%;
471}
472
473div.navheader table {
474 border-left: 0em;
475 border-right: 0em;
476 border-top: 0em;
477 width: 100%;
478}
479
480div.navfooter table {
481 border-left: 0em;
482 border-right: 0em;
483 border-bottom: 0em;
484 width: 100%;
485}
486
487div.navheader table td a,
488div.navfooter table td a {
489 color: #777;
490 text-decoration: none;
491}
492
493/* normal text in the footer */
494div.navfooter table td {
495 color: black;
496}
497
498div.navheader table td a:visited,
499div.navfooter table td a:visited {
500 color: #444;
501}
502
503
504/* links in header and footer */
505div.navheader table td a:hover,
506div.navfooter table td a:hover {
507 text-decoration: underline;
508 background-color: transparent;
509 color: #33a;
510}
511
512div.navheader hr,
513div.navfooter hr {
514 display: none;
515}
516
517
518.qandaset tr.question td p {
519 margin: 0em 0em 1em 0em;
520 padding: 0em 0em 0em 0em;
521}
522
523.qandaset tr.answer td p {
524 margin: 0em 0em 1em 0em;
525 padding: 0em 0em 0em 0em;
526}
527.answer td {
528 padding-bottom: 1.5em;
529}
530
531.emphasis {
532 font-weight: bold;
533}
534
535
536 /************* /
537 / decorations /
538/ *************/
539
540.titlepage {
541}
542
543.part .title {
544}
545
546.subtitle {
547 border: none;
548}
549
550/*
551h1 {
552 border: none;
553}
554
555h2 {
556 border-top: solid 0.2em;
557 border-bottom: solid 0.06em;
558}
559
560h3 {
561 border-top: 0em;
562 border-bottom: solid 0.06em;
563}
564
565h4 {
566 border: 0em;
567 border-bottom: solid 0.06em;
568}
569
570h5 {
571 border: 0em;
572}
573*/
574
575.programlisting {
576 border: solid 1px;
577}
578
579div.figure,
580div.table,
581div.informalfigure,
582div.informaltable,
583div.informalexample,
584div.example {
585 border: 1px solid;
586}
587
588
589
590.tip,
591.warning,
592.caution,
593.note {
594 border: 1px solid;
595}
596
597.tip table th,
598.warning table th,
599.caution table th,
600.note table th {
601 border-bottom: 1px solid;
602}
603
604.question td {
605 border-top: 1px solid black;
606}
607
608.answer {
609}
610
611
612b.keycap,
613.keycap {
614 border: 1px solid;
615}
616
617
618div.navheader, div.heading{
619 border-bottom: 1px solid;
620}
621
622
623div.navfooter, div.footing{
624 border-top: 1px solid;
625}
626
627 /********* /
628 / colors /
629/ *********/
630
631body {
632 color: #333;
633 background: white;
634}
635
636a {
637 background: transparent;
638}
639
640a:hover {
641 background-color: #dedede;
642}
643
644
645h1,
646h2,
647h3,
648h4,
649h5,
650h6,
651h7,
652h8 {
653 background-color: transparent;
654}
655
656hr {
657 border-color: #aaa;
658}
659
660
661.tip, .warning, .caution, .note {
662 border-color: #fff;
663}
664
665
666.tip table th,
667.warning table th,
668.caution table th,
669.note table th {
670 border-bottom-color: #fff;
671}
672
673
674.warning {
675 background-color: #f0f0f2;
676}
677
678.caution {
679 background-color: #f0f0f2;
680}
681
682.tip {
683 background-color: #f0f0f2;
684}
685
686.note {
687 background-color: #f0f0f2;
688}
689
690.glossary dl dt,
691.variablelist dl dt,
692.variablelist dl dt span.term {
693 color: #044;
694}
695
696div.figure,
697div.table,
698div.example,
699div.informalfigure,
700div.informaltable,
701div.informalexample {
702 border-color: #aaa;
703}
704
705pre.programlisting {
706 color: black;
707 background-color: #fff;
708 border-color: #aaa;
709 border-width: 2px;
710}
711
712.guimenu,
713.guilabel,
714.guimenuitem {
715 background-color: #eee;
716}
717
718
719b.keycap,
720.keycap {
721 background-color: #eee;
722 border-color: #999;
723}
724
725
726div.navheader {
727 border-color: black;
728}
729
730
731div.navfooter {
732 border-color: black;
733}
734
735.writernotes {
736 color: red;
737}
738
739
740 /*********** /
741 / graphics /
742/ ***********/
743
744/*
745body {
746 background-image: url("images/body_bg.jpg");
747 background-attachment: fixed;
748}
749
750.navheader,
751.note,
752.tip {
753 background-image: url("images/note_bg.jpg");
754 background-attachment: fixed;
755}
756
757.warning,
758.caution {
759 background-image: url("images/warning_bg.jpg");
760 background-attachment: fixed;
761}
762
763.figure,
764.informalfigure,
765.example,
766.informalexample,
767.table,
768.informaltable {
769 background-image: url("images/figure_bg.jpg");
770 background-attachment: fixed;
771}
772
773*/
774h1,
775h2,
776h3,
777h4,
778h5,
779h6,
780h7{
781}
782
783/*
784Example of how to stick an image as part of the title.
785
786div.article .titlepage .title
787{
788 background-image: url("figures/white-on-black.png");
789 background-position: center;
790 background-repeat: repeat-x;
791}
792*/
793
794div.preface .titlepage .title,
795div.colophon .title,
796div.chapter .titlepage .title,
797div.article .titlepage .title
798{
799}
800
801div.section div.section .titlepage .title,
802div.sect2 .titlepage .title {
803 background: none;
804}
805
806
807h1.title {
808 background-color: transparent;
809 background-repeat: no-repeat;
810 height: 256px;
811 text-indent: -9000px;
812 overflow:hidden;
813}
814
815h2.subtitle {
816 background-color: transparent;
817 text-indent: -9000px;
818 overflow:hidden;
819 width: 0px;
820 display: none;
821}
822
823 /*************************************** /
824 / pippin.gimp.org specific alterations /
825/ ***************************************/
826
827/*
828div.heading, div.navheader {
829 color: #777;
830 font-size: 80%;
831 padding: 0;
832 margin: 0;
833 text-align: left;
834 position: absolute;
835 top: 0px;
836 left: 0px;
837 width: 100%;
838 height: 50px;
839 background: url('/gfx/heading_bg.png') transparent;
840 background-repeat: repeat-x;
841 background-attachment: fixed;
842 border: none;
843}
844
845div.heading a {
846 color: #444;
847}
848
849div.footing, div.navfooter {
850 border: none;
851 color: #ddd;
852 font-size: 80%;
853 text-align:right;
854
855 width: 100%;
856 padding-top: 10px;
857 position: absolute;
858 bottom: 0px;
859 left: 0px;
860
861 background: url('/gfx/footing_bg.png') transparent;
862}
863*/
864
865
866
867 /****************** /
868 / nasty ie tweaks /
869/ ******************/
870
871/*
872div.heading, div.navheader {
873 width:expression(document.body.clientWidth + "px");
874}
875
876div.footing, div.navfooter {
877 width:expression(document.body.clientWidth + "px");
878 margin-left:expression("-5em");
879}
880body {
881 padding:expression("4em 5em 0em 5em");
882}
883*/
884
885 /**************************************** /
886 / mozilla vendor specific css extensions /
887/ ****************************************/
888/*
889div.navfooter, div.footing{
890 -moz-opacity: 0.8em;
891}
892
893div.figure,
894div.table,
895div.informalfigure,
896div.informaltable,
897div.informalexample,
898div.example,
899.tip,
900.warning,
901.caution,
902.note {
903 -moz-border-radius: 0.5em;
904}
905
906b.keycap,
907.keycap {
908 -moz-border-radius: 0.3em;
909}
910*/
911
912table tr td table tr td {
913 display: none;
914}
915
916
917hr {
918 display: none;
919}
920
921table {
922 border: 0em;
923}
924
925 .photo {
926 float: right;
927 margin-left: 1.5em;
928 margin-bottom: 1.5em;
929 margin-top: 0em;
930 max-width: 17em;
931 border: 1px solid gray;
932 padding: 3px;
933 background: white;
934}
935 .seperator {
936 padding-top: 2em;
937 clear: both;
938 }
939
940 #validators {
941 margin-top: 5em;
942 text-align: right;
943 color: #777;
944 }
945 @media print {
946 body {
947 font-size: 8pt;
948 }
949 .noprint {
950 display: none;
951 }
952 }
953
954
955.tip,
956.note {
957 background: #f0f0f2;
958 color: #333;
959 padding: 20px;
960 margin: 20px;
961}
962
963.tip h3,
964.note h3 {
965 padding: 0em;
966 margin: 0em;
967 font-size: 2em;
968 font-weight: bold;
969 color: #333;
970}
971
972.tip a,
973.note a {
974 color: #333;
975 text-decoration: underline;
976}
977
978.footnote {
979 font-size: small;
980 color: #333;
981}
982
983/* Changes the announcement text */
984.tip h3,
985.warning h3,
986.caution h3,
987.note h3 {
988 font-size:large;
989 color: #00557D;
990}
diff --git a/documentation/overview-manual/overview-manual-yp-intro.xml b/documentation/overview-manual/overview-manual-yp-intro.xml
deleted file mode 100644
index a2a1f494bb..0000000000
--- a/documentation/overview-manual/overview-manual-yp-intro.xml
+++ /dev/null
@@ -1,1333 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='overview-yp'>
7 <title>Introducing the Yocto Project</title>
8
9 <section id='what-is-the-yocto-project'>
10 <title>What is the Yocto Project?</title>
11
12 <para>
13 The Yocto Project is an open source collaboration project
14 that helps developers create custom Linux-based systems that are
15 designed for embedded products regardless of the product's hardware
16 architecture.
17 Yocto Project provides a flexible toolset and a development
18 environment that allows embedded device developers across the
19 world to collaborate through shared technologies, software stacks,
20 configurations, and best practices used to create these tailored
21 Linux images.
22 </para>
23
24 <para>
25 Thousands of developers worldwide have discovered that Yocto
26 Project provides advantages in both systems and applications
27 development, archival and management benefits, and customizations
28 used for speed, footprint, and memory utilization.
29 The project is a standard when it comes to delivering embedded
30 software stacks.
31 The project allows software customizations and build interchange
32 for multiple hardware platforms as well as software stacks that
33 can be maintained and scaled.
34 </para>
35
36 <para id='yp-key-dev-elements'>
37 <imagedata fileref="figures/key-dev-elements.png" format="PNG" align='center' width="8in"/>
38 </para>
39
40 <para>
41 For further introductory information on the Yocto Project, you
42 might be interested in this
43 <ulink url='https://www.embedded.com/electronics-blogs/say-what-/4458600/Why-the-Yocto-Project-for-my-IoT-Project-'>article</ulink>
44 by Drew Moseley and in this short introductory
45 <ulink url='https://www.youtube.com/watch?v=utZpKM7i5Z4'>video</ulink>.
46 </para>
47
48 <para>
49 The remainder of this section overviews advantages and challenges
50 tied to the Yocto Project.
51 </para>
52
53 <section id='gs-features'>
54 <title>Features</title>
55
56 <para>
57 The following list describes features and advantages of the
58 Yocto Project:
59 <itemizedlist>
60 <listitem><para>
61 <emphasis>Widely Adopted Across the Industry:</emphasis>
62 Semiconductor, operating system, software, and
63 service vendors exist whose products and services
64 adopt and support the Yocto Project.
65 For a look at the Yocto Project community and
66 the companies involved with the Yocto
67 Project, see the "COMMUNITY" and "ECOSYSTEM" tabs
68 on the
69 <ulink url='&YOCTO_HOME_URL;'>Yocto Project</ulink>
70 home page.
71 </para></listitem>
72 <listitem><para>
73 <emphasis>Architecture Agnostic:</emphasis>
74 Yocto Project supports Intel, ARM, MIPS, AMD, PPC
75 and other architectures.
76 Most ODMs, OSVs, and chip vendors create and supply
77 BSPs that support their hardware.
78 If you have custom silicon, you can create a BSP
79 that supports that architecture.</para>
80
81 <para>Aside from lots of architecture support, the
82 Yocto Project fully supports a wide range of device
83 emulation through the Quick EMUlator (QEMU).
84 </para></listitem>
85 <listitem><para>
86 <emphasis>Images and Code Transfer Easily:</emphasis>
87 Yocto Project output can easily move between
88 architectures without moving to new development
89 environments.
90 Additionally, if you have used the Yocto Project to
91 create an image or application and you find yourself
92 not able to support it, commercial Linux vendors such
93 as Wind River, Mentor Graphics, Timesys, and ENEA could
94 take it and provide ongoing support.
95 These vendors have offerings that are built using
96 the Yocto Project.
97 </para></listitem>
98 <listitem><para>
99 <emphasis>Flexibility:</emphasis>
100 Corporations use the Yocto Project many different ways.
101 One example is to create an internal Linux distribution
102 as a code base the corporation can use across multiple
103 product groups.
104 Through customization and layering, a project group
105 can leverage the base Linux distribution to create
106 a distribution that works for their product needs.
107 </para></listitem>
108 <listitem><para>
109 <emphasis>Ideal for Constrained Embedded and IoT devices:</emphasis>
110 Unlike a full Linux distribution, you can use the
111 Yocto Project to create exactly what you need for
112 embedded devices.
113 You only add the feature support or packages that you
114 absolutely need for the device.
115 For devices that have display hardware, you can use
116 available system components such as X11, GTK+, Qt,
117 Clutter, and SDL (among others) to create a rich user
118 experience.
119 For devices that do not have a display or where you
120 want to use alternative UI frameworks, you can choose
121 to not install these components.
122 </para></listitem>
123 <listitem><para>
124 <emphasis>Comprehensive Toolchain Capabilities:</emphasis>
125 Toolchains for supported architectures satisfy most
126 use cases.
127 However, if your hardware supports features that are
128 not part of a standard toolchain, you can easily
129 customize that toolchain through specification of
130 platform-specific tuning parameters.
131 And, should you need to use a third-party toolchain,
132 mechanisms built into the Yocto Project allow for that.
133 </para></listitem>
134 <listitem><para>
135 <emphasis>Mechanism Rules Over Policy:</emphasis>
136 Focusing on mechanism rather than policy ensures that
137 you are free to set policies based on the needs of your
138 design instead of adopting decisions enforced by some
139 system software provider.
140 </para></listitem>
141 <listitem><para>
142 <emphasis>Uses a Layer Model:</emphasis>
143 The Yocto Project
144 <link linkend='the-yocto-project-layer-model'>layer infrastructure</link>
145 groups related functionality into separate bundles.
146 You can incrementally add these grouped functionalities
147 to your project as needed.
148 Using layers to isolate and group functionality
149 reduces project complexity and redundancy, allows you
150 to easily extend the system, make customizations,
151 and keep functionality organized.
152 </para></listitem>
153 <listitem><para>
154 <emphasis>Supports Partial Builds:</emphasis>
155 You can build and rebuild individual packages as
156 needed.
157 Yocto Project accomplishes this through its
158 <link linkend='shared-state-cache'>shared-state cache</link>
159 (sstate) scheme.
160 Being able to build and debug components individually
161 eases project development.
162 </para></listitem>
163 <listitem><para>
164 <emphasis>Releases According to a Strict Schedule:</emphasis>
165 Major releases occur on a
166 <ulink url='&YOCTO_DOCS_REF_URL;#ref-release-process'>six-month cycle</ulink>
167 predictably in October and April.
168 The most recent two releases support point releases
169 to address common vulnerabilities and exposures.
170 This predictability is crucial for projects based on
171 the Yocto Project and allows development teams to
172 plan activities.
173 </para></listitem>
174 <listitem><para>
175 <emphasis>Rich Ecosystem of Individuals and Organizations:</emphasis>
176 For open source projects, the value of community is
177 very important.
178 Support forums, expertise, and active developers who
179 continue to push the Yocto Project forward are readily
180 available.
181 </para></listitem>
182 <listitem><para>
183 <emphasis>Binary Reproducibility:</emphasis>
184 The Yocto Project allows you to be very specific about
185 dependencies and achieves very high percentages of
186 binary reproducibility (e.g. 99.8% for
187 <filename>core-image-minimal</filename>).
188 When distributions are not specific about which
189 packages are pulled in and in what order to support
190 dependencies, other build systems can arbitrarily
191 include packages.
192 </para></listitem>
193 <listitem><para>
194 <emphasis>License Manifest:</emphasis>
195 The Yocto Project provides a
196 <ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>license manifest</ulink>
197 for review by people who need to track the use of open
198 source licenses (e.g.legal teams).
199 </para></listitem>
200 </itemizedlist>
201 </para>
202 </section>
203
204 <section id='gs-challenges'>
205 <title>Challenges</title>
206
207 <para>
208 The following list presents challenges you might encounter
209 when developing using the Yocto Project:
210 <itemizedlist>
211 <listitem><para>
212 <emphasis>Steep Learning Curve:</emphasis>
213 The Yocto Project has a steep learning curve and has
214 many different ways to accomplish similar tasks.
215 It can be difficult to choose how to proceed when
216 varying methods exist by which to accomplish a given
217 task.
218 </para></listitem>
219 <listitem><para>
220 <emphasis>Understanding What Changes You Need to Make
221 For Your Design Requires Some Research:</emphasis>
222 Beyond the simple tutorial stage, understanding what
223 changes need to be made for your particular design
224 can require a significant amount of research and
225 investigation.
226 For information that helps you transition from
227 trying out the Yocto Project to using it for your
228 project, see the
229 "<ulink url='&YOCTO_DOCS_URL;/what-i-wish-id-known/'>What I wish I'd Known</ulink>"
230 and
231 "<ulink url='&YOCTO_DOCS_URL;/transitioning-to-a-custom-environment/'>Transitioning to a Custom Environment for Systems Development</ulink>"
232 documents on the Yocto Project website.
233 </para></listitem>
234 <listitem><para>
235 <emphasis>Project Workflow Could Be Confusing:</emphasis>
236 The
237 <link linkend='overview-development-environment'>Yocto Project workflow</link>
238 could be confusing if you are used to traditional
239 desktop and server software development.
240 In a desktop development environment, mechanisms exist
241 to easily pull and install new packages, which are
242 typically pre-compiled binaries from servers accessible
243 over the Internet.
244 Using the Yocto Project, you must modify your
245 configuration and rebuild to add additional packages.
246 </para></listitem>
247 <listitem><para>
248 <emphasis>Working in a Cross-Build Environment Can
249 Feel Unfamiliar:</emphasis>
250 When developing code to run on a target, compilation,
251 execution, and testing done on the actual target
252 can be faster than running a BitBake build on a
253 development host and then deploying binaries to the
254 target for test.
255 While the Yocto Project does support development tools
256 on the target, the additional step of integrating your
257 changes back into the Yocto Project build environment
258 would be required.
259 Yocto Project supports an intermediate approach that
260 involves making changes on the development system
261 within the BitBake environment and then deploying only
262 the updated packages to the target.</para>
263
264 <para>The Yocto Project
265 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
266 produces packages in standard formats (i.e. RPM,
267 DEB, IPK, and TAR).
268 You can deploy these packages into the running system
269 on the target by using utilities on the target such
270 as <filename>rpm</filename> or
271 <filename>ipk</filename>.
272 </para></listitem>
273 <listitem><para>
274 <emphasis>Initial Build Times Can be Significant:</emphasis>
275 Long initial build times are unfortunately unavoidable
276 due to the large number of packages initially built
277 from scratch for a fully functioning Linux system.
278 Once that initial build is completed, however, the
279 shared-state (sstate) cache mechanism Yocto Project
280 uses keeps the system from rebuilding packages that
281 have not been "touched" since the last build.
282 The sstate mechanism significantly reduces times
283 for successive builds.
284 </para></listitem>
285 </itemizedlist>
286 </para>
287 </section>
288 </section>
289
290 <section id='the-yocto-project-layer-model'>
291 <title>The Yocto Project Layer Model</title>
292
293 <para>
294 The Yocto Project's "Layer Model" is a development model for
295 embedded and IoT Linux creation that distinguishes the
296 Yocto Project from other simple build systems.
297 The Layer Model simultaneously supports collaboration and
298 customization.
299 Layers are repositories that contain related sets of instructions
300 that tell the
301 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
302 what to do.
303 You can collaborate, share, and reuse layers.
304 </para>
305
306 <para>
307 Layers can contain changes to previous instructions or settings
308 at any time.
309 This powerful override capability is what allows you to customize
310 previously supplied collaborative or community layers to suit your
311 product requirements.
312 </para>
313
314 <para>
315 You use different layers to logically separate information in your
316 build.
317 As an example, you could have BSP, GUI, distro configuration,
318 middleware, or application layers.
319 Putting your entire build into one layer limits and complicates
320 future customization and reuse.
321 Isolating information into layers, on the other hand, helps
322 simplify future customizations and reuse.
323 You might find it tempting to keep everything in one layer when
324 working on a single project.
325 However, the more modular your Metadata, the easier
326 it is to cope with future changes.
327 <note><title>Notes</title>
328 <itemizedlist>
329 <listitem><para>
330 Use Board Support Package (BSP) layers from silicon
331 vendors when possible.
332 </para></listitem>
333 <listitem><para>
334 Familiarize yourself with the
335 <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project curated layer index</ulink>
336 or the
337 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded layer index</ulink>.
338 The latter contains more layers but they are less
339 universally validated.
340 </para></listitem>
341 <listitem><para>
342 Layers support the inclusion of technologies, hardware
343 components, and software components.
344 The
345 <ulink url='&YOCTO_DOCS_DEV_URL;#making-sure-your-layer-is-compatible-with-yocto-project'>Yocto Project Compatible</ulink>
346 designation provides a minimum level of standardization
347 that contributes to a strong ecosystem.
348 "YP Compatible" is applied to appropriate products and
349 software components such as BSPs, other OE-compatible
350 layers, and related open-source projects, allowing the
351 producer to use Yocto Project badges and branding
352 assets.
353 </para></listitem>
354 </itemizedlist>
355 </note>
356 </para>
357
358 <para>
359 To illustrate how layers are used to keep things modular, consider
360 machine customizations.
361 These types of customizations typically reside in a special layer,
362 rather than a general layer, called a BSP Layer.
363 Furthermore, the machine customizations should be isolated from
364 recipes and Metadata that support a new GUI environment,
365 for example.
366 This situation gives you a couple of layers: one for the machine
367 configurations, and one for the GUI environment.
368 It is important to understand, however, that the BSP layer can
369 still make machine-specific additions to recipes within the GUI
370 environment layer without polluting the GUI layer itself
371 with those machine-specific changes.
372 You can accomplish this through a recipe that is a BitBake append
373 (<filename>.bbappend</filename>) file, which is described later
374 in this section.
375 <note>
376 For general information on BSP layer structure, see the
377 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Packages (BSP) Developer's Guide</ulink>.
378 </note>
379 </para>
380
381 <para>
382 The
383 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
384 contains both general layers and BSP layers right out of the box.
385 You can easily identify layers that ship with a Yocto Project
386 release in the Source Directory by their names.
387 Layers typically have names that begin with the string
388 <filename>meta-</filename>.
389 <note>
390 It is not a requirement that a layer name begin with the
391 prefix <filename>meta-</filename>, but it is a commonly
392 accepted standard in the Yocto Project community.
393 </note>
394 For example, if you were to examine the
395 <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/'>tree view</ulink>
396 of the <filename>poky</filename> repository, you will see several
397 layers: <filename>meta</filename>,
398 <filename>meta-skeleton</filename>,
399 <filename>meta-selftest</filename>,
400 <filename>meta-poky</filename>, and
401 <filename>meta-yocto-bsp</filename>.
402 Each of these repositories represents a distinct layer.
403 </para>
404
405 <para>
406 For procedures on how to create layers, see the
407 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
408 section in the Yocto Project Development Tasks Manual.
409 </para>
410 </section>
411
412 <section id='components-and-tools'>
413 <title>Components and Tools</title>
414
415 <para>
416 The Yocto Project employs a collection of components and
417 tools used by the project itself, by project developers,
418 and by those using the Yocto Project.
419 These components and tools are open source projects and
420 metadata that are separate from the reference distribution
421 (<ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>)
422 and the
423 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>.
424 Most of the components and tools are downloaded separately.
425 </para>
426
427 <para>
428 This section provides brief overviews of the components and
429 tools associated with the Yocto Project.
430 </para>
431
432 <section id='gs-development-tools'>
433 <title>Development Tools</title>
434
435 <para>
436 The following list consists of tools that help you develop
437 images and applications using the Yocto Project:
438 <itemizedlist>
439 <listitem><para id='gs-crops-overview'>
440 <emphasis>CROPS:</emphasis>
441 <ulink url='https://github.com/crops/poky-container/'>CROPS</ulink>
442 is an open source, cross-platform development framework
443 that leverages
444 <ulink url='https://www.docker.com/'>Docker Containers</ulink>.
445 CROPS provides an easily managed, extensible environment
446 that allows you to build binaries for a variety of
447 architectures on Windows, Linux and Mac OS X hosts.
448 </para></listitem>
449 <listitem><para>
450 <emphasis><filename>devtool</filename>:</emphasis>
451 This command-line tool is available as part of the
452 extensible SDK (eSDK) and is its cornerstone.
453 You can use <filename>devtool</filename> to help build,
454 test, and package software within the eSDK.
455 You can use the tool to optionally integrate what you
456 build into an image built by the OpenEmbedded build
457 system.</para>
458
459 <para>The <filename>devtool</filename> command employs
460 a number of sub-commands that allow you to add, modify,
461 and upgrade recipes.
462 As with the OpenEmbedded build system, "recipes"
463 represent software packages within
464 <filename>devtool</filename>.
465 When you use <filename>devtool add</filename>, a recipe
466 is automatically created.
467 When you use <filename>devtool modify</filename>, the
468 specified existing recipe is used in order to determine
469 where to get the source code and how to patch it.
470 In both cases, an environment is set up so that when
471 you build the recipe a source tree that is under your
472 control is used in order to allow you to make changes
473 to the source as desired.
474 By default, both new recipes and the source go into
475 a "workspace" directory under the eSDK.
476 The <filename>devtool upgrade</filename> command
477 updates an existing recipe so that you can build it
478 for an updated set of source files.</para>
479
480 <para>You can read about the
481 <filename>devtool</filename> workflow in the Yocto
482 Project Application Development and Extensible
483 Software Development Kit (eSDK) Manual in the
484 "<ulink url='&YOCTO_DOCS_SDK_URL;#using-devtool-in-your-sdk-workflow'>Using <filename>devtool</filename> in Your SDK Workflow'</ulink>"
485 section.
486 </para></listitem>
487 <listitem><para>
488 <emphasis>Extensible Software Development Kit (eSDK):</emphasis>
489 The eSDK provides a cross-development toolchain and
490 libraries tailored to the contents of a specific image.
491 The eSDK makes it easy to add new applications and
492 libraries to an image, modify the source for an
493 existing component, test changes on the target
494 hardware, and integrate into the rest of the
495 OpenEmbedded build system.
496 The eSDK gives you a toolchain experience supplemented
497 with the powerful set of <filename>devtool</filename>
498 commands tailored for the Yocto Project environment.
499 </para>
500
501 <para>For information on the eSDK, see the
502 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
503 Manual.
504 </para></listitem>
505 <listitem><para>
506 <emphasis>Toaster:</emphasis>
507 Toaster is a web interface to the Yocto Project
508 OpenEmbedded build system.
509 Toaster allows you to configure, run, and view
510 information about builds.
511 For information on Toaster, see the
512 <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>.
513 </para></listitem>
514 </itemizedlist>
515 </para>
516 </section>
517
518 <section id='gs-production-tools'>
519 <title>Production Tools</title>
520
521 <para>
522 The following list consists of tools that help production
523 related activities using the Yocto Project:
524 <itemizedlist>
525 <listitem><para>
526 <emphasis>Auto Upgrade Helper:</emphasis>
527 This utility when used in conjunction with the
528 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
529 (BitBake and OE-Core) automatically generates upgrades
530 for recipes that are based on new versions of the
531 recipes published upstream.
532 </para></listitem>
533 <listitem><para>
534 <emphasis>Recipe Reporting System:</emphasis>
535 The Recipe Reporting System tracks recipe versions
536 available for Yocto Project.
537 The main purpose of the system is to help you
538 manage the recipes you maintain and to offer a dynamic
539 overview of the project.
540 The Recipe Reporting System is built on top of the
541 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Layer Index</ulink>,
542 which is a website that indexes OpenEmbedded-Core
543 layers.
544 </para></listitem>
545 <listitem><para>
546 <emphasis>Patchwork:</emphasis>
547 <ulink url='http://jk.ozlabs.org/projects/patchwork/'>Patchwork</ulink>
548 is a fork of a project originally started by
549 <ulink url='http://ozlabs.org/'>OzLabs</ulink>.
550 The project is a web-based tracking system designed
551 to streamline the process of bringing contributions
552 into a project.
553 The Yocto Project uses Patchwork as an organizational
554 tool to handle patches, which number in the thousands
555 for every release.
556 </para></listitem>
557 <listitem><para>
558 <emphasis>AutoBuilder:</emphasis>
559 AutoBuilder is a project that automates build tests
560 and quality assurance (QA).
561 By using the public AutoBuilder, anyone can determine
562 the status of the current "master" branch of Poky.
563 <note>
564 AutoBuilder is based on
565 <ulink url='https://buildbot.net/'>buildbot</ulink>.
566 </note></para>
567
568 <para>A goal of the Yocto Project is to lead the
569 open source industry with a project that automates
570 testing and QA procedures.
571 In doing so, the project encourages a development
572 community that publishes QA and test plans, publicly
573 demonstrates QA and test plans, and encourages
574 development of tools that automate and test and QA
575 procedures for the benefit of the development
576 community.</para>
577
578 <para>You can learn more about the AutoBuilder used
579 by the Yocto Project
580 <ulink url='&YOCTO_AB_URL;'>here</ulink>.
581 </para></listitem>
582 <listitem><para>
583 <emphasis>Cross-Prelink:</emphasis>
584 Prelinking is the process of pre-computing the load
585 addresses and link tables generated by the dynamic
586 linker as compared to doing this at runtime.
587 Doing this ahead of time results in performance
588 improvements when the application is launched and
589 reduced memory usage for libraries shared by many
590 applications.</para>
591
592 <para>Historically, cross-prelink is a variant of
593 prelink, which was conceived by
594 <ulink url='http://people.redhat.com/jakub/prelink.pdf'>Jakub Jel&iacute;nek</ulink>
595 a number of years ago.
596 Both prelink and cross-prelink are maintained in the
597 same repository albeit on separate branches.
598 By providing an emulated runtime dynamic linker
599 (i.e. <filename>glibc</filename>-derived
600 <filename>ld.so</filename> emulation), the
601 cross-prelink project extends the prelink software's
602 ability to prelink a sysroot environment.
603 Additionally, the cross-prelink software enables the
604 ability to work in sysroot style environments.</para>
605
606 <para>The dynamic linker determines standard load
607 address calculations based on a variety of factors
608 such as mapping addresses, library usage, and library
609 function conflicts.
610 The prelink tool uses this information, from the
611 dynamic linker, to determine unique load addresses
612 for executable and linkable format (ELF) binaries
613 that are shared libraries and dynamically linked.
614 The prelink tool modifies these ELF binaries with the
615 pre-computed information.
616 The result is faster loading and often lower memory
617 consumption because more of the library code can
618 be re-used from shared Copy-On-Write (COW) pages.
619 </para>
620
621 <para>The original upstream prelink project only
622 supports running prelink on the end target device
623 due to the reliance on the target device's dynamic
624 linker.
625 This restriction causes issues when developing a
626 cross-compiled system.
627 The cross-prelink adds a synthesized dynamic loader
628 that runs on the host, thus permitting cross-prelinking
629 without ever having to run on a read-write target
630 filesystem.
631 </para></listitem>
632 <listitem><para>
633 <emphasis>Pseudo:</emphasis>
634 Pseudo is the Yocto Project implementation of
635 <ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>,
636 which is used to run commands in an environment
637 that seemingly has root privileges.</para>
638
639 <para>During a build, it can be necessary to perform
640 operations that require system administrator
641 privileges.
642 For example, file ownership or permissions might need
643 definition.
644 Pseudo is a tool that you can either use directly or
645 through the environment variable
646 <filename>LD_PRELOAD</filename>.
647 Either method allows these operations to succeed as
648 if system administrator privileges exist even
649 when they do not.</para>
650
651 <para>You can read more about Pseudo in the
652 "<link linkend='fakeroot-and-pseudo'>Fakeroot and Pseudo</link>"
653 section.
654 </para></listitem>
655 </itemizedlist>
656 </para>
657 </section>
658
659 <section id='gs-openembedded-build-system'>
660 <title>Open-Embedded Build System Components</title>
661
662 <para>
663 The following list consists of components associated with the
664 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>:
665 <itemizedlist>
666 <listitem><para>
667 <emphasis>BitBake:</emphasis>
668 BitBake is a core component of the Yocto Project and is
669 used by the OpenEmbedded build system to build images.
670 While BitBake is key to the build system, BitBake
671 is maintained separately from the Yocto Project.</para>
672
673 <para>BitBake is a generic task execution engine that
674 allows shell and Python tasks to be run efficiently
675 and in parallel while working within complex inter-task
676 dependency constraints.
677 In short, BitBake is a build engine that works
678 through recipes written in a specific format in order
679 to perform sets of tasks.</para>
680
681 <para>You can learn more about BitBake in the
682 <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>.
683 </para></listitem>
684 <listitem><para>
685 <emphasis>OpenEmbedded-Core:</emphasis>
686 OpenEmbedded-Core (OE-Core) is a common layer of
687 metadata (i.e. recipes, classes, and associated files)
688 used by OpenEmbedded-derived systems, which includes
689 the Yocto Project.
690 The Yocto Project and the OpenEmbedded Project both
691 maintain the OpenEmbedded-Core.
692 You can find the OE-Core metadata in the Yocto Project
693 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta'>Source Repositories</ulink>.
694 </para>
695
696 <para>Historically, the Yocto Project integrated the
697 OE-Core metadata throughout the Yocto Project
698 source repository reference system (Poky).
699 After Yocto Project Version 1.0, the Yocto Project
700 and OpenEmbedded agreed to work together and share a
701 common core set of metadata (OE-Core), which contained
702 much of the functionality previously found in Poky.
703 This collaboration achieved a long-standing
704 OpenEmbedded objective for having a more tightly
705 controlled and quality-assured core.
706 The results also fit well with the Yocto Project
707 objective of achieving a smaller number of fully
708 featured tools as compared to many different ones.
709 </para>
710
711 <para>Sharing a core set of metadata results in Poky
712 as an integration layer on top of OE-Core.
713 You can see that in this
714 <link linkend='yp-key-dev-elements'>figure</link>.
715 The Yocto Project combines various components such as
716 BitBake, OE-Core, script "glue", and documentation
717 for its build system.
718 </para></listitem>
719 </itemizedlist>
720 </para>
721 </section>
722
723 <section id='gs-reference-distribution-poky'>
724 <title>Reference Distribution (Poky)</title>
725
726 <para>
727 Poky is the Yocto Project reference distribution.
728 It contains the
729 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>Open-Embedded build system</ulink>
730 (BitBake and OE-Core) as well as a set of metadata to get you
731 started building your own distribution.
732 See the
733 <link linkend='what-is-the-yocto-project'>figure</link> in
734 "What is the Yocto Project?" section for an illustration
735 that shows Poky and its relationship with other parts of the
736 Yocto Project.</para>
737
738 <para>To use the Yocto Project tools and components, you
739 can download (<filename>clone</filename>) Poky and use it
740 to bootstrap your own distribution.
741 <note>
742 Poky does not contain binary files.
743 It is a working example of how to build your own custom
744 Linux distribution from source.
745 </note>
746 You can read more about Poky in the
747 "<link linkend='reference-embedded-distribution'>Reference Embedded Distribution (Poky)</link>"
748 section.
749 </para>
750 </section>
751
752 <section id='gs-packages-for-finished-targets'>
753 <title>Packages for Finished Targets</title>
754
755 <para>
756 The following lists components associated with packages
757 for finished targets:
758 <itemizedlist>
759 <listitem><para>
760 <emphasis>Matchbox:</emphasis>
761 Matchbox is an Open Source, base environment for the
762 X Window System running on non-desktop, embedded
763 platforms such as handhelds, set-top boxes, kiosks,
764 and anything else for which screen space, input
765 mechanisms, or system resources are limited.</para>
766
767 <para>Matchbox consists of a number of interchangeable
768 and optional applications that you can tailor to a
769 specific, non-desktop platform to enhance usability
770 in constrained environments.</para>
771
772 <para>You can find the Matchbox source in the Yocto
773 Project
774 <ulink url='&YOCTO_GIT_URL;'>Source Repositories</ulink>.
775 </para></listitem>
776 <listitem><para>
777 <emphasis>Opkg</emphasis>
778 Open PacKaGe management (opkg) is a lightweight
779 package management system based on the itsy package
780 (ipkg) management system.
781 Opkg is written in C and resembles Advanced Package
782 Tool (APT) and Debian Package (dpkg) in operation.
783 </para>
784
785 <para>Opkg is intended for use on embedded Linux
786 devices and is used in this capacity in the
787 <ulink url='http://www.openembedded.org/wiki/Main_Page'>OpenEmbedded</ulink>
788 and
789 <ulink url='https://openwrt.org/'>OpenWrt</ulink>
790 projects, as well as the Yocto Project.
791 <note>
792 As best it can, opkg maintains backwards
793 compatibility with ipkg and conforms to a subset
794 of Debian's policy manual regarding control files.
795 </note>
796 </para></listitem>
797 </itemizedlist>
798 </para>
799 </section>
800
801 <section id='gs-archived-components'>
802 <title>Archived Components</title>
803
804 <para>
805 The Build Appliance is a virtual machine image that enables
806 you to build and boot a custom embedded Linux image with
807 the Yocto Project using a non-Linux development system.
808 </para>
809
810 <para>
811 Historically, the Build Appliance was the second of three
812 methods by which you could use the Yocto Project on a system
813 that was not native to Linux.
814 <orderedlist>
815 <listitem><para>
816 <emphasis>Hob:</emphasis>
817 Hob, which is now deprecated and is no longer available
818 since the 2.1 release of the Yocto Project provided
819 a rudimentary, GUI-based interface to the Yocto
820 Project.
821 Toaster has fully replaced Hob.
822 </para></listitem>
823 <listitem><para>
824 <emphasis>Build Appliance:</emphasis>
825 Post Hob, the Build Appliance became available.
826 It was never recommended that you use the Build
827 Appliance as a day-to-day production development
828 environment with the Yocto Project.
829 Build Appliance was useful as a way to try out
830 development in the Yocto Project environment.
831 </para></listitem>
832 <listitem><para>
833 <emphasis>CROPS:</emphasis>
834 The final and best solution available now for
835 developing using the Yocto Project on a system
836 not native to Linux is with
837 <link linkend='gs-crops-overview'>CROPS</link>.
838 </para></listitem>
839 </orderedlist>
840 </para>
841 </section>
842 </section>
843
844 <section id='gs-development-methods'>
845 <title>Development Methods</title>
846
847 <para>
848 The Yocto Project development environment usually involves a
849 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>Build Host</ulink>
850 and target hardware.
851 You use the Build Host to build images and develop applications,
852 while you use the target hardware to test deployed software.
853 </para>
854
855 <para>
856 This section provides an introduction to the choices or
857 development methods you have when setting up your Build Host.
858 Depending on the your particular workflow preference and the
859 type of operating system your Build Host runs, several choices
860 exist that allow you to use the Yocto Project.
861 <note>
862 For additional detail about the Yocto Project development
863 environment, see the
864 "<link linkend='overview-development-environment'>The Yocto Project Development Environment</link>"
865 chapter.
866 </note>
867 <itemizedlist>
868 <listitem><para>
869 <emphasis>Native Linux Host:</emphasis>
870 By far the best option for a Build Host.
871 A system running Linux as its native operating system
872 allows you to develop software by directly using the
873 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
874 tool.
875 You can accomplish all aspects of development from a
876 familiar shell of a supported Linux distribution.</para>
877
878 <para>For information on how to set up a Build Host on
879 a system running Linux as its native operating system,
880 see the
881 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host'>Setting Up a Native Linux Host</ulink>"
882 section in the Yocto Project Development Tasks Manual.
883 </para></listitem>
884 <listitem><para>
885 <emphasis>CROss PlatformS (CROPS):</emphasis>
886 Typically, you use
887 <ulink url='https://github.com/crops/poky-container/'>CROPS</ulink>,
888 which leverages
889 <ulink url='https://www.docker.com/'>Docker Containers</ulink>,
890 to set up a Build Host that is not running Linux (e.g.
891 <trademark class='registered'>Microsoft</trademark>
892 <trademark class='trademark'>Windows</trademark>
893 or
894 <trademark class='registered'>macOS</trademark>).
895 <note>
896 You can, however, use CROPS on a Linux-based system.
897 </note>
898 CROPS is an open source, cross-platform development
899 framework that provides an easily managed, extensible
900 environment for building binaries targeted for a variety
901 of architectures on Windows, macOS, or Linux hosts.
902 Once the Build Host is set up using CROPS, you can prepare
903 a shell environment to mimic that of a shell being used
904 on a system natively running Linux.</para>
905
906 <para>For information on how to set up a Build Host with
907 CROPS, see the
908 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops'>Setting Up to Use CROss PlatformS (CROPS)</ulink>"
909 section in the Yocto Project Development Tasks Manual.
910 </para></listitem>
911 <listitem><para>
912 <emphasis>Windows Subsystem For Linux (WSLv2):</emphasis>
913 You may use Windows Subsystem For Linux v2 to set up a build
914 host using Windows 10.
915 <note>
916 The Yocto Project is not compatible with WSLv1, it is
917 compatible but not officially supported nor validated
918 with WSLv2, if you still decide to use WSL please upgrade
919 to WSLv2.
920 </note>
921 The Windows Subsystem For Linux allows Windows 10 to run a real
922 Linux kernel inside of a lightweight utility virtual
923 machine (VM) using virtualization technology.</para>
924 <para>For information on how to set up a Build Host with
925 WSLv2, see the
926 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-wsl'>Setting Up to Use Windows Subsystem For Linux</ulink>"
927 section in the Yocto Project Development Tasks Manual.
928 </para></listitem>
929 <listitem><para>
930 <emphasis>Toaster:</emphasis>
931 Regardless of what your Build Host is running, you can
932 use Toaster to develop software using the Yocto Project.
933 Toaster is a web interface to the Yocto Project's
934 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>Open-Embedded build system</ulink>.
935 The interface enables you to configure and run your
936 builds.
937 Information about builds is collected and stored in a
938 database.
939 You can use Toaster to configure and start builds on
940 multiple remote build servers.</para>
941
942 <para>For information about and how to use Toaster,
943 see the
944 <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>.
945 </para></listitem>
946 </itemizedlist>
947 </para>
948 </section>
949
950 <section id='reference-embedded-distribution'>
951 <title>Reference Embedded Distribution (Poky)</title>
952
953 <para>
954 "Poky", which is pronounced <emphasis>Pock</emphasis>-ee, is the
955 name of the Yocto Project's reference distribution or Reference OS
956 Kit.
957 Poky contains the
958 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded Build System</ulink>
959 (<ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> and
960 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>)
961 as well as a set of
962 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>metadata</ulink> to get
963 you started building your own distro.
964 In other words, Poky is a base specification of the functionality
965 needed for a typical embedded system as well as the components
966 from the Yocto Project that allow you to build a distribution into
967 a usable binary image.
968 </para>
969
970 <para>
971 Poky is a combined repository of BitBake, OpenEmbedded-Core
972 (which is found in <filename>meta</filename>),
973 <filename>meta-poky</filename>,
974 <filename>meta-yocto-bsp</filename>, and documentation provided
975 all together and known to work well together.
976 You can view these items that make up the Poky repository in the
977 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/'>Source Repositories</ulink>.
978 <note>
979 If you are interested in all the contents of the
980 <filename>poky</filename> Git repository, see the
981 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core'>Top-Level Core Components</ulink>"
982 section in the Yocto Project Reference Manual.
983 </note>
984 </para>
985
986 <para id='gs-poky-reference-distribution'>
987 The following figure illustrates what generally comprises Poky:
988 <imagedata fileref="figures/poky-reference-distribution.png" format="PNG" align='center' width="8in"/>
989 <itemizedlist>
990 <listitem><para>
991 BitBake is a task executor and scheduler that is the heart of
992 the OpenEmbedded build system.
993 </para></listitem>
994 <listitem><para>
995 <filename>meta-poky</filename>, which is Poky-specific
996 metadata.
997 </para></listitem>
998 <listitem><para>
999 <filename>meta-yocto-bsp</filename>, which are Yocto
1000 Project-specific Board Support Packages (BSPs).
1001 </para></listitem>
1002 <listitem><para>
1003 OpenEmbedded-Core (OE-Core) metadata, which includes
1004 shared configurations, global variable definitions,
1005 shared classes, packaging, and recipes.
1006 Classes define the encapsulation and inheritance of build
1007 logic.
1008 Recipes are the logical units of software and images
1009 to be built.
1010 </para></listitem>
1011 <listitem><para>
1012 Documentation, which contains the Yocto Project source
1013 files used to make the set of user manuals.
1014 </para></listitem>
1015 </itemizedlist>
1016 <note>
1017 While Poky is a "complete" distribution specification and is
1018 tested and put through QA, you cannot use it as a product
1019 "out of the box" in its current form.
1020 </note>
1021 </para>
1022
1023 <para>
1024 To use the Yocto Project tools, you can use Git to clone (download)
1025 the Poky repository then use your local copy of the reference
1026 distribution to bootstrap your own distribution.
1027 <note>
1028 Poky does not contain binary files.
1029 It is a working example of how to build your own custom Linux distribution
1030 from source.
1031 </note>
1032 </para>
1033
1034 <para>
1035 Poky has a regular, well established, six-month release cycle
1036 under its own version.
1037 Major releases occur at the same time major releases (point
1038 releases) occur for the Yocto Project, which are typically in the
1039 Spring and Fall.
1040 For more information on the Yocto Project release schedule and
1041 cadence, see the
1042 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-release-process'>Yocto Project Releases and the Stable Release Process</ulink>"
1043 chapter in the Yocto Project Reference Manual.
1044 </para>
1045
1046 <para>
1047 Much has been said about Poky being a "default configuration."
1048 A default configuration provides a starting image footprint.
1049 You can use Poky out of the box to create an image ranging from a
1050 shell-accessible minimal image all the way up to a Linux
1051 Standard Base-compliant image that uses a GNOME Mobile and
1052 Embedded (GMAE) based reference user interface called Sato.
1053 </para>
1054
1055 <para>
1056 One of the most powerful properties of Poky is that every aspect
1057 of a build is controlled by the metadata.
1058 You can use metadata to augment these base image types by
1059 adding metadata
1060 <link linkend='the-yocto-project-layer-model'>layers</link>
1061 that extend functionality.
1062 These layers can provide, for example, an additional software
1063 stack for an image type, add a board support package (BSP) for
1064 additional hardware, or even create a new image type.
1065 </para>
1066
1067 <para>
1068 Metadata is loosely grouped into configuration files or package
1069 recipes.
1070 A recipe is a collection of non-executable metadata used by
1071 BitBake to set variables or define additional build-time tasks.
1072 A recipe contains fields such as the recipe description, the recipe
1073 version, the license of the package and the upstream source
1074 repository.
1075 A recipe might also indicate that the build process uses autotools,
1076 make, distutils or any other build process, in which case the basic
1077 functionality can be defined by the classes it inherits from
1078 the OE-Core layer's class definitions in
1079 <filename>./meta/classes</filename>.
1080 Within a recipe you can also define additional tasks as well as
1081 task prerequisites.
1082 Recipe syntax through BitBake also supports both
1083 <filename>_prepend</filename> and <filename>_append</filename>
1084 operators as a method of extending task functionality.
1085 These operators inject code into the beginning or end of a task.
1086 For information on these BitBake operators, see the
1087 "<ulink url='&YOCTO_DOCS_BB_URL;#appending-and-prepending-override-style-syntax'>Appending and Prepending (Override Style Syntax)</ulink>"
1088 section in the BitBake User's Manual.
1089 </para>
1090 </section>
1091
1092 <section id='openembedded-build-system-workflow'>
1093 <title>The OpenEmbedded Build System Workflow</title>
1094
1095 <para>
1096 The
1097 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
1098 uses a "workflow" to accomplish image and SDK generation.
1099 The following figure overviews that workflow:
1100 <imagedata fileref="figures/YP-flow-diagram.png"
1101 format="PNG" align='center' width="8in"/>
1102 Following is a brief summary of the "workflow":
1103 <orderedlist>
1104 <listitem><para>
1105 Developers specify architecture, policies, patches and
1106 configuration details.
1107 </para></listitem>
1108 <listitem><para>
1109 The build system fetches and downloads the source code
1110 from the specified location.
1111 The build system supports standard methods such as tarballs
1112 or source code repositories systems such as Git.
1113 </para></listitem>
1114 <listitem><para>
1115 Once source code is downloaded, the build system extracts
1116 the sources into a local work area where patches are
1117 applied and common steps for configuring and compiling
1118 the software are run.
1119 </para></listitem>
1120 <listitem><para>
1121 The build system then installs the software into a
1122 temporary staging area where the binary package format you
1123 select (DEB, RPM, or IPK) is used to roll up the software.
1124 </para></listitem>
1125 <listitem><para>
1126 Different QA and sanity checks run throughout entire
1127 build process.
1128 </para></listitem>
1129 <listitem><para>
1130 After the binaries are created, the build system
1131 generates a binary package feed that is used to create
1132 the final root file image.
1133 </para></listitem>
1134 <listitem><para>
1135 The build system generates the file system image and a
1136 customized Extensible SDK (eSDK) for application
1137 development in parallel.
1138 </para></listitem>
1139 </orderedlist>
1140 </para>
1141
1142 <para>
1143 For a very detailed look at this workflow, see the
1144 "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>"
1145 section.
1146 </para>
1147 </section>
1148
1149
1150 <section id='some-basic-terms'>
1151 <title>Some Basic Terms</title>
1152
1153 <para>
1154 It helps to understand some basic fundamental terms when
1155 learning the Yocto Project.
1156 Although a list of terms exists in the
1157 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-terms'>Yocto Project Terms</ulink>"
1158 section of the Yocto Project Reference Manual, this section
1159 provides the definitions of some terms helpful for getting started:
1160 <itemizedlist>
1161 <listitem><para>
1162 <emphasis>Configuration Files:</emphasis>
1163 Files that hold global definitions of variables,
1164 user-defined variables, and hardware configuration
1165 information.
1166 These files tell the
1167 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>Open-Embedded build system</ulink>
1168 what to build and what to put into the image to support a
1169 particular platform.
1170 </para></listitem>
1171 <listitem><para>
1172 <emphasis>Extensible Software Development Kit (eSDK):</emphasis>
1173 A custom SDK for application developers.
1174 This eSDK allows developers to incorporate their library
1175 and programming changes back into the image to make
1176 their code available to other application developers.
1177 For information on the eSDK, see the
1178 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
1179 manual.
1180 </para></listitem>
1181 <listitem><para>
1182 <emphasis>Layer:</emphasis>
1183 A collection of related recipes.
1184 Layers allow you to consolidate related metadata to
1185 customize your build.
1186 Layers also isolate information used when building
1187 for multiple architectures.
1188 Layers are hierarchical in their ability to override
1189 previous specifications.
1190 You can include any number of available layers from the
1191 Yocto Project and customize the build by adding your
1192 layers after them.
1193 You can search the Layer Index for layers used within
1194 Yocto Project.</para>
1195
1196 <para>For more detailed information on layers, see the
1197 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
1198 section in the Yocto Project Development Tasks Manual.
1199 For a discussion specifically on BSP Layers, see the
1200 "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>"
1201 section in the Yocto Project Board Support Packages (BSP)
1202 Developer's Guide.
1203 </para></listitem>
1204 <listitem><para>
1205 <emphasis>Metadata:</emphasis>
1206 A key element of the Yocto Project is the Metadata that
1207 is used to construct a Linux distribution and is contained
1208 in the files that the OpenEmbedded build system parses
1209 when building an image.
1210 In general, Metadata includes recipes, configuration
1211 files, and other information that refers to the build
1212 instructions themselves, as well as the data used to
1213 control what things get built and the effects of the
1214 build.
1215 Metadata also includes commands and data used to
1216 indicate what versions of software are used, from
1217 where they are obtained, and changes or additions to the
1218 software itself (patches or auxiliary files) that
1219 are used to fix bugs or customize the software for use
1220 in a particular situation.
1221 OpenEmbedded-Core is an important set of validated
1222 metadata.
1223 </para></listitem>
1224 <listitem><para id='gs-term-openembedded-build-system'>
1225 <emphasis>OpenEmbedded Build System:</emphasis>
1226 The terms "BitBake" and "build system" are sometimes
1227 used for the OpenEmbedded Build System.</para>
1228
1229 <para>BitBake is a task scheduler and execution engine
1230 that parses instructions (i.e. recipes) and configuration
1231 data.
1232 After a parsing phase, BitBake creates a dependency tree
1233 to order the compilation, schedules the compilation of
1234 the included code, and finally executes the building
1235 of the specified custom Linux image (distribution).
1236 BitBake is similar to the <filename>make</filename>
1237 tool.</para>
1238
1239 <para>During a build process, the build system tracks
1240 dependencies and performs a native or cross-compilation
1241 of the package.
1242 As a first step in a cross-build setup, the framework
1243 attempts to create a cross-compiler toolchain
1244 (i.e. Extensible SDK) suited for the target platform.
1245 </para></listitem>
1246 <listitem><para>
1247 <emphasis>OpenEmbedded-Core (OE-Core):</emphasis>
1248 OE-Core is metadata comprised of foundation recipes,
1249 classes, and associated files that are meant to be
1250 common among many different OpenEmbedded-derived systems,
1251 including the Yocto Project.
1252 OE-Core is a curated subset of an original repository
1253 developed by the OpenEmbedded community that has been
1254 pared down into a smaller, core set of continuously
1255 validated recipes.
1256 The result is a tightly controlled and quality-assured
1257 core set of recipes.</para>
1258
1259 <para>You can see the Metadata in the
1260 <filename>meta</filename> directory of the Yocto Project
1261 <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi'>Source Repositories</ulink>.
1262 </para></listitem>
1263 <listitem><para>
1264 <emphasis>Packages:</emphasis>
1265 In the context of the Yocto Project, this term refers to a
1266 recipe's packaged output produced by BitBake (i.e. a
1267 "baked recipe").
1268 A package is generally the compiled binaries produced from the
1269 recipe's sources.
1270 You "bake" something by running it through BitBake.</para>
1271
1272 <para>It is worth noting that the term "package" can,
1273 in general, have subtle meanings.
1274 For example, the packages referred to in the
1275 "<ulink url='&YOCTO_DOCS_REF_URL;#required-packages-for-the-build-host'>Required Packages for the Build Host</ulink>"
1276 section in the Yocto Project Reference Manual are compiled
1277 binaries that, when installed, add functionality to your
1278 Linux distribution.</para>
1279
1280 <para>Another point worth noting is that historically within
1281 the Yocto Project, recipes were referred to as packages - thus,
1282 the existence of several BitBake variables that are seemingly
1283 mis-named,
1284 (e.g. <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>,
1285 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>,
1286 and
1287 <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink>).
1288 </para></listitem>
1289 <listitem><para>
1290 <emphasis>Poky:</emphasis>
1291 Poky is a reference embedded distribution and a reference
1292 test configuration.
1293 Poky provides the following:
1294 <itemizedlist>
1295 <listitem><para>
1296 A base-level functional distro used to illustrate
1297 how to customize a distribution.
1298 </para></listitem>
1299 <listitem><para>
1300 A means by which to test the Yocto Project
1301 components (i.e. Poky is used to validate
1302 the Yocto Project).
1303 </para></listitem>
1304 <listitem><para>
1305 A vehicle through which you can download
1306 the Yocto Project.
1307 </para></listitem>
1308 </itemizedlist>
1309 Poky is not a product level distro.
1310 Rather, it is a good starting point for customization.
1311 <note>
1312 Poky is an integration layer on top of OE-Core.
1313 </note>
1314 </para></listitem>
1315 <listitem><para>
1316 <emphasis>Recipe:</emphasis>
1317 The most common form of metadata.
1318 A recipe contains a list of settings and tasks
1319 (i.e. instructions) for building packages that are then
1320 used to build the binary image.
1321 A recipe describes where you get source code and which
1322 patches to apply.
1323 Recipes describe dependencies for libraries or for other
1324 recipes as well as configuration and compilation options.
1325 Related recipes are consolidated into a layer.
1326 </para></listitem>
1327 </itemizedlist>
1328 </para>
1329 </section>
1330</chapter>
1331<!--
1332vim: expandtab tw=80 ts=4
1333-->
diff --git a/documentation/overview-manual/overview-manual.xml b/documentation/overview-manual/overview-manual.xml
deleted file mode 100755
index 8021a2e95e..0000000000
--- a/documentation/overview-manual/overview-manual.xml
+++ /dev/null
@@ -1,130 +0,0 @@
1<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<book id='overview-manual' lang='en'
7 xmlns:xi="http://www.w3.org/2003/XInclude"
8 xmlns="http://docbook.org/ns/docbook"
9 >
10 <bookinfo>
11
12 <mediaobject>
13 <imageobject>
14 <imagedata fileref='figures/overview-manual-title.png'
15 format='SVG'
16 align='left' scalefit='1' width='100%'/>
17 </imageobject>
18 </mediaobject>
19
20 <title>
21 Yocto Project Overview and Concepts Manual
22 </title>
23
24 <authorgroup>
25 <author>
26 <affiliation>
27 <orgname>&ORGNAME;</orgname>
28 </affiliation>
29 <email>&ORGEMAIL;</email>
30 </author>
31 </authorgroup>
32
33 <revhistory>
34 <revision>
35 <revnumber>2.5</revnumber>
36 <date>May 2018</date>
37 <revremark>The initial document released with the Yocto Project 2.5 Release.</revremark>
38 </revision>
39 <revision>
40 <revnumber>2.6</revnumber>
41 <date>November 2018</date>
42 <revremark>Released with the Yocto Project 2.6 Release.</revremark>
43 </revision>
44 <revision>
45 <revnumber>2.7</revnumber>
46 <date>May 2019</date>
47 <revremark>Released with the Yocto Project 2.7 Release.</revremark>
48 </revision>
49 <revision>
50 <revnumber>3.0</revnumber>
51 <date>October 2019</date>
52 <revremark>Released with the Yocto Project 3.0 Release.</revremark>
53 </revision>
54 <revision>
55 <revnumber>3.1</revnumber>
56 <date>&REL_MONTH_YEAR;</date>
57 <revremark>Released with the Yocto Project 3.1 Release.</revremark>
58 </revision>
59 </revhistory>
60
61 <copyright>
62 <year>&COPYRIGHT_YEAR;</year>
63 <holder>Linux Foundation</holder>
64 </copyright>
65
66 <legalnotice>
67 <para>
68 Permission is granted to copy, distribute and/or modify this document under
69 the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">
70 Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by
71 Creative Commons.
72 </para>
73 <note><title>Manual Notes</title>
74 <itemizedlist>
75 <listitem><para>
76 This version of the
77 <emphasis>Yocto Project Overview and Concepts Manual</emphasis>
78 is for the &YOCTO_DOC_VERSION; release of the
79 Yocto Project.
80 To be sure you have the latest version of the manual
81 for this release, go to the
82 <ulink url='&YOCTO_DOCS_URL;'>Yocto Project documentation page</ulink>
83 and select the manual from that site.
84 Manuals from the site are more up-to-date than manuals
85 derived from the Yocto Project released TAR files.
86 </para></listitem>
87 <listitem><para>
88 If you located this manual through a web search, the
89 version of the manual might not be the one you want
90 (e.g. the search might have returned a manual much
91 older than the Yocto Project version with which you
92 are working).
93 You can see all Yocto Project major releases by
94 visiting the
95 <ulink url='&YOCTO_WIKI_URL;/wiki/Releases'>Releases</ulink>
96 page.
97 If you need a version of this manual for a different
98 Yocto Project release, visit the
99 <ulink url='&YOCTO_DOCS_URL;'>Yocto Project documentation page</ulink>
100 and select the manual set by using the
101 "ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARCHIVE"
102 pull-down menus.
103 </para></listitem>
104 <listitem>
105 <para>
106 To report any inaccuracies or problems with this
107 (or any other Yocto Project) manual, send an email to
108 the Yocto Project documentation mailing list at
109 <filename>docs@lists.yoctoproject.org</filename> or
110 log into the freenode <filename>#yocto</filename> channel.
111 </para>
112 </listitem>
113 </itemizedlist>
114 </note>
115 </legalnotice>
116
117 </bookinfo>
118
119 <xi:include href="overview-manual-intro.xml"/>
120
121 <xi:include href="overview-manual-yp-intro.xml"/>
122
123 <xi:include href="overview-manual-development-environment.xml"/>
124
125 <xi:include href="overview-manual-concepts.xml" />
126
127</book>
128<!--
129vim: expandtab tw=80 ts=4
130-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-19 03:22:44 +0000