summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual/closer-look.xml
diff options
context:
space:
mode:
authorScott Rifenbark <srifenbark@gmail.com>2017-06-14 09:50:55 -0700
committerRichard Purdie <richard.purdie@linuxfoundation.org>2017-06-22 09:16:43 +0100
commitde6d45fefc3000ee8918d7c18448758d4216bae5 (patch)
treed0e7f2913f00d7df940dad84075dd9937e0de635 /documentation/ref-manual/closer-look.xml
parent3f6a9af9272cf30d78aa3de76af3f6ee0dc2bd22 (diff)
downloadpoky-de6d45fefc3000ee8918d7c18448758d4216bae5.tar.gz
documentation: Re-org for "closer-look" chapter
Fixes [YOCTO #11630] The ref-manual needs expansion for the old "closer-look" chapter. This chapter previously held a detailed look at what happens when a user uses the YP to develop something. Now, the chapter needs to also contain YP development environment concepts (e.g. open- source philosophy, etc.), which are coming from the dev-manual. Because of this, I renamed the "closer-look.xml" chapter to be "ref-development-environment.xml". I also renamed the larger section that was formerly the entire chapter into its own section named "Development Concepts". Both these changes caused a few links to break. I fixed all the links from within the various manuals so they would find appropriate targets. I did some re-writing for introductory material to introduce the new chapter and the section on "Development Concepts". A new file ("ref-development-environment.xml") was added by basically renaming the "closer-look.xml" chapter. And, the tracking for "closer-look.xml" was deleted. (From yocto-docs rev: e37806474578b4f0ed137f64d68a39a17ab60644) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/ref-manual/closer-look.xml')
-rw-r--r--documentation/ref-manual/closer-look.xml1627
1 files changed, 0 insertions, 1627 deletions
diff --git a/documentation/ref-manual/closer-look.xml b/documentation/ref-manual/closer-look.xml
deleted file mode 100644
index 5046465fba..0000000000
--- a/documentation/ref-manual/closer-look.xml
+++ /dev/null
@@ -1,1627 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='closer-look'>
6<title>A Closer Look at the Yocto Project Development Environment</title>
7
8 <para>
9 This chapter takes a more detailed look at the Yocto Project
10 development environment.
11 The following diagram represents the development environment at a
12 high level.
13 The remainder of this chapter expands on the fundamental input, output,
14 process, and
15 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>) blocks
16 in the Yocto Project development environment.
17 </para>
18
19 <para id='general-yocto-environment-figure'>
20 <imagedata fileref="figures/yocto-environment-ref.png" align="center" width="8in" depth="4.25in" />
21 </para>
22
23 <para>
24 The generalized Yocto Project Development Environment consists of
25 several functional areas:
26 <itemizedlist>
27 <listitem><para><emphasis>User Configuration:</emphasis>
28 Metadata you can use to control the build process.
29 </para></listitem>
30 <listitem><para><emphasis>Metadata Layers:</emphasis>
31 Various layers that provide software, machine, and
32 distro Metadata.</para></listitem>
33 <listitem><para><emphasis>Source Files:</emphasis>
34 Upstream releases, local projects, and SCMs.</para></listitem>
35 <listitem><para><emphasis>Build System:</emphasis>
36 Processes under the control of
37 <link linkend='bitbake-term'>BitBake</link>.
38 This block expands on how BitBake fetches source, applies
39 patches, completes compilation, analyzes output for package
40 generation, creates and tests packages, generates images, and
41 generates cross-development tools.</para></listitem>
42 <listitem><para><emphasis>Package Feeds:</emphasis>
43 Directories containing output packages (RPM, DEB or IPK),
44 which are subsequently used in the construction of an image or
45 SDK, produced by the build system.
46 These feeds can also be copied and shared using a web server or
47 other means to facilitate extending or updating existing
48 images on devices at runtime if runtime package management is
49 enabled.</para></listitem>
50 <listitem><para><emphasis>Images:</emphasis>
51 Images produced by the development process.
52 </para></listitem>
53 <listitem><para><emphasis>Application Development SDK:</emphasis>
54 Cross-development tools that are produced along with an image
55 or separately with BitBake.</para></listitem>
56 </itemizedlist>
57 </para>
58
59 <section id="user-configuration">
60 <title>User Configuration</title>
61
62 <para>
63 User configuration helps define the build.
64 Through user configuration, you can tell BitBake the
65 target architecture for which you are building the image,
66 where to store downloaded source, and other build properties.
67 </para>
68
69 <para>
70 The following figure shows an expanded representation of the
71 "User Configuration" box of the
72 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
73 </para>
74
75 <para>
76 <imagedata fileref="figures/user-configuration.png" align="center" />
77 </para>
78
79 <para>
80 BitBake needs some basic configuration files in order to complete
81 a build.
82 These files are <filename>*.conf</filename> files.
83 The minimally necessary ones reside as example files in the
84 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
85 For simplicity, this section refers to the Source Directory as
86 the "Poky Directory."
87 </para>
88
89 <para>
90 When you clone the <filename>poky</filename> Git repository or you
91 download and unpack a Yocto Project release, you can set up the
92 Source Directory to be named anything you want.
93 For this discussion, the cloned repository uses the default
94 name <filename>poky</filename>.
95 <note>
96 The Poky repository is primarily an aggregation of existing
97 repositories.
98 It is not a canonical upstream source.
99 </note>
100 </para>
101
102 <para>
103 The <filename>meta-poky</filename> layer inside Poky contains
104 a <filename>conf</filename> directory that has example
105 configuration files.
106 These example files are used as a basis for creating actual
107 configuration files when you source the build environment
108 script
109 (i.e.
110 <link linkend='structure-core-script'><filename>&OE_INIT_FILE;</filename></link>
111 or
112 <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>).
113 </para>
114
115 <para>
116 Sourcing the build environment script creates a
117 <link linkend='build-directory'>Build Directory</link>
118 if one does not already exist.
119 BitBake uses the Build Directory for all its work during builds.
120 The Build Directory has a <filename>conf</filename> directory that
121 contains default versions of your <filename>local.conf</filename>
122 and <filename>bblayers.conf</filename> configuration files.
123 These default configuration files are created only if versions
124 do not already exist in the Build Directory at the time you
125 source the build environment setup script.
126 </para>
127
128 <para>
129 Because the Poky repository is fundamentally an aggregation of
130 existing repositories, some users might be familiar with running
131 the <filename>&OE_INIT_FILE;</filename> or
132 <filename>oe-init-build-env-memres</filename> script in the context
133 of separate OpenEmbedded-Core and BitBake repositories rather than a
134 single Poky repository.
135 This discussion assumes the script is executed from within a cloned
136 or unpacked version of Poky.
137 </para>
138
139 <para>
140 Depending on where the script is sourced, different sub-scripts
141 are called to set up the Build Directory (Yocto or OpenEmbedded).
142 Specifically, the script
143 <filename>scripts/oe-setup-builddir</filename> inside the
144 poky directory sets up the Build Directory and seeds the directory
145 (if necessary) with configuration files appropriate for the
146 Yocto Project development environment.
147 <note>
148 The <filename>scripts/oe-setup-builddir</filename> script
149 uses the <filename>$TEMPLATECONF</filename> variable to
150 determine which sample configuration files to locate.
151 </note>
152 </para>
153
154 <para>
155 The <filename>local.conf</filename> file provides many
156 basic variables that define a build environment.
157 Here is a list of a few.
158 To see the default configurations in a <filename>local.conf</filename>
159 file created by the build environment script, see the
160 <filename>local.conf.sample</filename> in the
161 <filename>meta-poky</filename> layer:
162 <itemizedlist>
163 <listitem><para><emphasis>Parallelism Options:</emphasis>
164 Controlled by the
165 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>,
166 <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></link>,
167 and
168 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></ulink>
169 variables.</para></listitem>
170 <listitem><para><emphasis>Target Machine Selection:</emphasis>
171 Controlled by the
172 <link linkend='var-MACHINE'><filename>MACHINE</filename></link>
173 variable.</para></listitem>
174 <listitem><para><emphasis>Download Directory:</emphasis>
175 Controlled by the
176 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
177 variable.</para></listitem>
178 <listitem><para><emphasis>Shared State Directory:</emphasis>
179 Controlled by the
180 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
181 variable.</para></listitem>
182 <listitem><para><emphasis>Build Output:</emphasis>
183 Controlled by the
184 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
185 variable.</para></listitem>
186 </itemizedlist>
187 <note>
188 Configurations set in the <filename>conf/local.conf</filename>
189 file can also be set in the
190 <filename>conf/site.conf</filename> and
191 <filename>conf/auto.conf</filename> configuration files.
192 </note>
193 </para>
194
195 <para>
196 The <filename>bblayers.conf</filename> file tells BitBake what
197 layers you want considered during the build.
198 By default, the layers listed in this file include layers
199 minimally needed by the build system.
200 However, you must manually add any custom layers you have created.
201 You can find more information on working with the
202 <filename>bblayers.conf</filename> file in the
203 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
204 section in the Yocto Project Development Manual.
205 </para>
206
207 <para>
208 The files <filename>site.conf</filename> and
209 <filename>auto.conf</filename> are not created by the environment
210 initialization script.
211 If you want the <filename>site.conf</filename> file, you need to
212 create that yourself.
213 The <filename>auto.conf</filename> file is typically created by
214 an autobuilder:
215 <itemizedlist>
216 <listitem><para><emphasis><filename>site.conf</filename>:</emphasis>
217 You can use the <filename>conf/site.conf</filename>
218 configuration file to configure multiple build directories.
219 For example, suppose you had several build environments and
220 they shared some common features.
221 You can set these default build properties here.
222 A good example is perhaps the packaging format to use
223 through the
224 <link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>
225 variable.</para>
226 <para>One useful scenario for using the
227 <filename>conf/site.conf</filename> file is to extend your
228 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
229 variable to include the path to a
230 <filename>conf/site.conf</filename>.
231 Then, when BitBake looks for Metadata using
232 <filename>BBPATH</filename>, it finds the
233 <filename>conf/site.conf</filename> file and applies your
234 common configurations found in the file.
235 To override configurations in a particular build directory,
236 alter the similar configurations within that build
237 directory's <filename>conf/local.conf</filename> file.
238 </para></listitem>
239 <listitem><para><emphasis><filename>auto.conf</filename>:</emphasis>
240 The file is usually created and written to by
241 an autobuilder.
242 The settings put into the file are typically the same as
243 you would find in the <filename>conf/local.conf</filename>
244 or the <filename>conf/site.conf</filename> files.
245 </para></listitem>
246 </itemizedlist>
247 </para>
248
249 <para>
250 You can edit all configuration files to further define
251 any particular build environment.
252 This process is represented by the "User Configuration Edits"
253 box in the figure.
254 </para>
255
256 <para>
257 When you launch your build with the
258 <filename>bitbake <replaceable>target</replaceable></filename>
259 command, BitBake sorts out the configurations to ultimately
260 define your build environment.
261 It is important to understand that the OpenEmbedded build system
262 reads the configuration files in a specific order:
263 <filename>site.conf</filename>, <filename>auto.conf</filename>,
264 and <filename>local.conf</filename>.
265 And, the build system applies the normal assignment statement
266 rules.
267 Because the files are parsed in a specific order, variable
268 assignments for the same variable could be affected.
269 For example, if the <filename>auto.conf</filename> file and
270 the <filename>local.conf</filename> set
271 <replaceable>variable1</replaceable> to different values, because
272 the build system parses <filename>local.conf</filename> after
273 <filename>auto.conf</filename>,
274 <replaceable>variable1</replaceable> is assigned the value from
275 the <filename>local.conf</filename> file.
276 </para>
277 </section>
278
279 <section id="metadata-machine-configuration-and-policy-configuration">
280 <title>Metadata, Machine Configuration, and Policy Configuration</title>
281
282 <para>
283 The previous section described the user configurations that
284 define BitBake's global behavior.
285 This section takes a closer look at the layers the build system
286 uses to further control the build.
287 These layers provide Metadata for the software, machine, and
288 policy.
289 </para>
290
291 <para>
292 In general, three types of layer input exist:
293 <itemizedlist>
294 <listitem><para><emphasis>Policy Configuration:</emphasis>
295 Distribution Layers provide top-level or general
296 policies for the image or SDK being built.
297 For example, this layer would dictate whether BitBake
298 produces RPM or IPK packages.</para></listitem>
299 <listitem><para><emphasis>Machine Configuration:</emphasis>
300 Board Support Package (BSP) layers provide machine
301 configurations.
302 This type of information is specific to a particular
303 target architecture.</para></listitem>
304 <listitem><para><emphasis>Metadata:</emphasis>
305 Software layers contain user-supplied recipe files,
306 patches, and append files.
307 </para></listitem>
308 </itemizedlist>
309 </para>
310
311 <para>
312 The following figure shows an expanded representation of the
313 Metadata, Machine Configuration, and Policy Configuration input
314 (layers) boxes of the
315 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
316 </para>
317
318 <para>
319 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="7.5in" />
320 </para>
321
322 <para>
323 In general, all layers have a similar structure.
324 They all contain a licensing file
325 (e.g. <filename>COPYING</filename>) if the layer is to be
326 distributed, a <filename>README</filename> file as good practice
327 and especially if the layer is to be distributed, a
328 configuration directory, and recipe directories.
329 </para>
330
331 <para>
332 The Yocto Project has many layers that can be used.
333 You can see a web-interface listing of them on the
334 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
335 page.
336 The layers are shown at the bottom categorized under
337 "Yocto Metadata Layers."
338 These layers are fundamentally a subset of the
339 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Metadata Index</ulink>,
340 which lists all layers provided by the OpenEmbedded community.
341 <note>
342 Layers exist in the Yocto Project Source Repositories that
343 cannot be found in the OpenEmbedded Metadata Index.
344 These layers are either deprecated or experimental in nature.
345 </note>
346 </para>
347
348 <para>
349 BitBake uses the <filename>conf/bblayers.conf</filename> file,
350 which is part of the user configuration, to find what layers it
351 should be using as part of the build.
352 </para>
353
354 <para>
355 For more information on layers, see the
356 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
357 section in the Yocto Project Development Manual.
358 </para>
359
360 <section id="distro-layer">
361 <title>Distro Layer</title>
362
363 <para>
364 The distribution layer provides policy configurations for your
365 distribution.
366 Best practices dictate that you isolate these types of
367 configurations into their own layer.
368 Settings you provide in
369 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
370 similar
371 settings that BitBake finds in your
372 <filename>conf/local.conf</filename> file in the Build
373 Directory.
374 </para>
375
376 <para>
377 The following list provides some explanation and references
378 for what you typically find in the distribution layer:
379 <itemizedlist>
380 <listitem><para><emphasis>classes:</emphasis>
381 Class files (<filename>.bbclass</filename>) hold
382 common functionality that can be shared among
383 recipes in the distribution.
384 When your recipes inherit a class, they take on the
385 settings and functions for that class.
386 You can read more about class files in the
387 "<link linkend='ref-classes'>Classes</link>" section.
388 </para></listitem>
389 <listitem><para><emphasis>conf:</emphasis>
390 This area holds configuration files for the
391 layer (<filename>conf/layer.conf</filename>),
392 the distribution
393 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
394 and any distribution-wide include files.
395 </para></listitem>
396 <listitem><para><emphasis>recipes-*:</emphasis>
397 Recipes and append files that affect common
398 functionality across the distribution.
399 This area could include recipes and append files
400 to add distribution-specific configuration,
401 initialization scripts, custom image recipes,
402 and so forth.</para></listitem>
403 </itemizedlist>
404 </para>
405 </section>
406
407 <section id="bsp-layer">
408 <title>BSP Layer</title>
409
410 <para>
411 The BSP Layer provides machine configurations.
412 Everything in this layer is specific to the machine for which
413 you are building the image or the SDK.
414 A common structure or form is defined for BSP layers.
415 You can learn more about this structure in the
416 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
417 <note>
418 In order for a BSP layer to be considered compliant with the
419 Yocto Project, it must meet some structural requirements.
420 </note>
421 </para>
422
423 <para>
424 The BSP Layer's configuration directory contains
425 configuration files for the machine
426 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>) and,
427 of course, the layer (<filename>conf/layer.conf</filename>).
428 </para>
429
430 <para>
431 The remainder of the layer is dedicated to specific recipes
432 by function: <filename>recipes-bsp</filename>,
433 <filename>recipes-core</filename>,
434 <filename>recipes-graphics</filename>, and
435 <filename>recipes-kernel</filename>.
436 Metadata can exist for multiple formfactors, graphics
437 support systems, and so forth.
438 <note>
439 While the figure shows several <filename>recipes-*</filename>
440 directories, not all these directories appear in all
441 BSP layers.
442 </note>
443 </para>
444 </section>
445
446 <section id="software-layer">
447 <title>Software Layer</title>
448
449 <para>
450 The software layer provides the Metadata for additional
451 software packages used during the build.
452 This layer does not include Metadata that is specific to the
453 distribution or the machine, which are found in their
454 respective layers.
455 </para>
456
457 <para>
458 This layer contains any new recipes that your project needs
459 in the form of recipe files.
460 </para>
461 </section>
462 </section>
463
464 <section id="sources-dev-environment">
465 <title>Sources</title>
466
467 <para>
468 In order for the OpenEmbedded build system to create an image or
469 any target, it must be able to access source files.
470 The
471 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
472 represents source files using the "Upstream Project Releases",
473 "Local Projects", and "SCMs (optional)" boxes.
474 The figure represents mirrors, which also play a role in locating
475 source files, with the "Source Mirror(s)" box.
476 </para>
477
478 <para>
479 The method by which source files are ultimately organized is
480 a function of the project.
481 For example, for released software, projects tend to use tarballs
482 or other archived files that can capture the state of a release
483 guaranteeing that it is statically represented.
484 On the other hand, for a project that is more dynamic or
485 experimental in nature, a project might keep source files in a
486 repository controlled by a Source Control Manager (SCM) such as
487 Git.
488 Pulling source from a repository allows you to control
489 the point in the repository (the revision) from which you want to
490 build software.
491 Finally, a combination of the two might exist, which would give the
492 consumer a choice when deciding where to get source files.
493 </para>
494
495 <para>
496 BitBake uses the
497 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
498 variable to point to source files regardless of their location.
499 Each recipe must have a <filename>SRC_URI</filename> variable
500 that points to the source.
501 </para>
502
503 <para>
504 Another area that plays a significant role in where source files
505 come from is pointed to by the
506 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
507 variable.
508 This area is a cache that can hold previously downloaded source.
509 You can also instruct the OpenEmbedded build system to create
510 tarballs from Git repositories, which is not the default behavior,
511 and store them in the <filename>DL_DIR</filename> by using the
512 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
513 variable.
514 </para>
515
516 <para>
517 Judicious use of a <filename>DL_DIR</filename> directory can
518 save the build system a trip across the Internet when looking
519 for files.
520 A good method for using a download directory is to have
521 <filename>DL_DIR</filename> point to an area outside of your
522 Build Directory.
523 Doing so allows you to safely delete the Build Directory
524 if needed without fear of removing any downloaded source file.
525 </para>
526
527 <para>
528 The remainder of this section provides a deeper look into the
529 source files and the mirrors.
530 Here is a more detailed look at the source file area of the
531 base figure:
532 <imagedata fileref="figures/source-input.png" align="center" width="7in" depth="7.5in" />
533 </para>
534
535 <section id='upstream-project-releases'>
536 <title>Upstream Project Releases</title>
537
538 <para>
539 Upstream project releases exist anywhere in the form of an
540 archived file (e.g. tarball or zip file).
541 These files correspond to individual recipes.
542 For example, the figure uses specific releases each for
543 BusyBox, Qt, and Dbus.
544 An archive file can be for any released product that can be
545 built using a recipe.
546 </para>
547 </section>
548
549 <section id='local-projects'>
550 <title>Local Projects</title>
551
552 <para>
553 Local projects are custom bits of software the user provides.
554 These bits reside somewhere local to a project - perhaps
555 a directory into which the user checks in items (e.g.
556 a local directory containing a development source tree
557 used by the group).
558 </para>
559
560 <para>
561 The canonical method through which to include a local project
562 is to use the
563 <link linkend='ref-classes-externalsrc'><filename>externalsrc</filename></link>
564 class to include that local project.
565 You use either the <filename>local.conf</filename> or a
566 recipe's append file to override or set the
567 recipe to point to the local directory on your disk to pull
568 in the whole source tree.
569 </para>
570
571 <para>
572 For information on how to use the
573 <filename>externalsrc</filename> class, see the
574 "<link linkend='ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></link>"
575 section.
576 </para>
577 </section>
578
579 <section id='scms'>
580 <title>Source Control Managers (Optional)</title>
581
582 <para>
583 Another place the build system can get source files from is
584 through an SCM such as Git or Subversion.
585 In this case, a repository is cloned or checked out.
586 The
587 <link linkend='ref-tasks-fetch'><filename>do_fetch</filename></link>
588 task inside BitBake uses
589 the <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
590 variable and the argument's prefix to determine the correct
591 fetcher module.
592 </para>
593
594 <note>
595 For information on how to have the OpenEmbedded build system
596 generate tarballs for Git repositories and place them in the
597 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
598 directory, see the
599 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
600 variable.
601 </note>
602
603 <para>
604 When fetching a repository, BitBake uses the
605 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
606 variable to determine the specific revision from which to
607 build.
608 </para>
609 </section>
610
611 <section id='source-mirrors'>
612 <title>Source Mirror(s)</title>
613
614 <para>
615 Two kinds of mirrors exist: pre-mirrors and regular mirrors.
616 The <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
617 and
618 <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
619 variables point to these, respectively.
620 BitBake checks pre-mirrors before looking upstream for any
621 source files.
622 Pre-mirrors are appropriate when you have a shared directory
623 that is not a directory defined by the
624 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
625 variable.
626 A Pre-mirror typically points to a shared directory that is
627 local to your organization.
628 </para>
629
630 <para>
631 Regular mirrors can be any site across the Internet that is
632 used as an alternative location for source code should the
633 primary site not be functioning for some reason or another.
634 </para>
635 </section>
636 </section>
637
638 <section id="package-feeds-dev-environment">
639 <title>Package Feeds</title>
640
641 <para>
642 When the OpenEmbedded build system generates an image or an SDK,
643 it gets the packages from a package feed area located in the
644 <link linkend='build-directory'>Build Directory</link>.
645 The
646 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
647 shows this package feeds area in the upper-right corner.
648 </para>
649
650 <para>
651 This section looks a little closer into the package feeds area used
652 by the build system.
653 Here is a more detailed look at the area:
654 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
655 </para>
656
657 <para>
658 Package feeds are an intermediary step in the build process.
659 The OpenEmbedded build system provides classes to generate
660 different package types, and you specify which classes to enable
661 through the
662 <link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>
663 variable.
664 Before placing the packages into package feeds,
665 the build process validates them with generated output quality
666 assurance checks through the
667 <link linkend='ref-classes-insane'><filename>insane</filename></link>
668 class.
669 </para>
670
671 <para>
672 The package feed area resides in the Build Directory.
673 The directory the build system uses to temporarily store packages
674 is determined by a combination of variables and the particular
675 package manager in use.
676 See the "Package Feeds" box in the illustration and note the
677 information to the right of that area.
678 In particular, the following defines where package files are
679 kept:
680 <itemizedlist>
681 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
682 Defined as <filename>tmp/deploy</filename> in the Build
683 Directory.
684 </para></listitem>
685 <listitem><para><filename>DEPLOY_DIR_*</filename>:
686 Depending on the package manager used, the package type
687 sub-folder.
688 Given RPM, IPK, or DEB packaging and tarball creation, the
689 <link linkend='var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></link>,
690 <link linkend='var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></link>,
691 <link linkend='var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></link>,
692 or
693 <link linkend='var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></link>,
694 variables are used, respectively.
695 </para></listitem>
696 <listitem><para><link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link>:
697 Defines architecture-specific sub-folders.
698 For example, packages could exist for the i586 or qemux86
699 architectures.
700 </para></listitem>
701 </itemizedlist>
702 </para>
703
704 <para>
705 BitBake uses the <filename>do_package_write_*</filename> tasks to
706 generate packages and place them into the package holding area (e.g.
707 <filename>do_package_write_ipk</filename> for IPK packages).
708 See the
709 "<link linkend='ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></link>",
710 "<link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link>",
711 "<link linkend='ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></link>",
712 and
713 "<link linkend='ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></link>"
714 sections for additional information.
715 As an example, consider a scenario where an IPK packaging manager
716 is being used and package architecture support for both i586
717 and qemux86 exist.
718 Packages for the i586 architecture are placed in
719 <filename>build/tmp/deploy/ipk/i586</filename>, while packages for
720 the qemux86 architecture are placed in
721 <filename>build/tmp/deploy/ipk/qemux86</filename>.
722 </para>
723 </section>
724
725 <section id='bitbake-dev-environment'>
726 <title>BitBake</title>
727
728 <para>
729 The OpenEmbedded build system uses
730 <link linkend='bitbake-term'>BitBake</link>
731 to produce images.
732 You can see from the
733 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
734 the BitBake area consists of several functional areas.
735 This section takes a closer look at each of those areas.
736 </para>
737
738 <para>
739 Separate documentation exists for the BitBake tool.
740 See the
741 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>
742 for reference material on BitBake.
743 </para>
744
745 <section id='source-fetching-dev-environment'>
746 <title>Source Fetching</title>
747
748 <para>
749 The first stages of building a recipe are to fetch and unpack
750 the source code:
751 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
752 </para>
753
754 <para>
755 The
756 <link linkend='ref-tasks-fetch'><filename>do_fetch</filename></link>
757 and
758 <link linkend='ref-tasks-unpack'><filename>do_unpack</filename></link>
759 tasks fetch the source files and unpack them into the work
760 directory.
761 <note>
762 For every local file (e.g. <filename>file://</filename>)
763 that is part of a recipe's
764 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
765 statement, the OpenEmbedded build system takes a checksum
766 of the file for the recipe and inserts the checksum into
767 the signature for the <filename>do_fetch</filename>.
768 If any local file has been modified, the
769 <filename>do_fetch</filename> task and all tasks that
770 depend on it are re-executed.
771 </note>
772 By default, everything is accomplished in the
773 <link linkend='build-directory'>Build Directory</link>,
774 which has a defined structure.
775 For additional general information on the Build Directory,
776 see the
777 "<link linkend='structure-core-build'><filename>build/</filename></link>"
778 section.
779 </para>
780
781 <para>
782 Unpacked source files are pointed to by the
783 <link linkend='var-S'><filename>S</filename></link> variable.
784 Each recipe has an area in the Build Directory where the
785 unpacked source code resides.
786 The name of that directory for any given recipe is defined from
787 several different variables.
788 You can see the variables that define these directories
789 by looking at the figure:
790 <itemizedlist>
791 <listitem><para><link linkend='var-TMPDIR'><filename>TMPDIR</filename></link> -
792 The base directory where the OpenEmbedded build system
793 performs all its work during the build.
794 </para></listitem>
795 <listitem><para><link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link> -
796 The architecture of the built package or packages.
797 </para></listitem>
798 <listitem><para><link linkend='var-TARGET_OS'><filename>TARGET_OS</filename></link> -
799 The operating system of the target device.
800 </para></listitem>
801 <listitem><para><link linkend='var-PN'><filename>PN</filename></link> -
802 The name of the built package.
803 </para></listitem>
804 <listitem><para><link linkend='var-PV'><filename>PV</filename></link> -
805 The version of the recipe used to build the package.
806 </para></listitem>
807 <listitem><para><link linkend='var-PR'><filename>PR</filename></link> -
808 The revision of the recipe used to build the package.
809 </para></listitem>
810 <listitem><para><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link> -
811 The location within <filename>TMPDIR</filename> where
812 a specific package is built.
813 </para></listitem>
814 <listitem><para><link linkend='var-S'><filename>S</filename></link> -
815 Contains the unpacked source files for a given recipe.
816 </para></listitem>
817 </itemizedlist>
818 </para>
819 </section>
820
821 <section id='patching-dev-environment'>
822 <title>Patching</title>
823
824 <para>
825 Once source code is fetched and unpacked, BitBake locates
826 patch files and applies them to the source files:
827 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
828 </para>
829
830 <para>
831 The
832 <link linkend='ref-tasks-patch'><filename>do_patch</filename></link>
833 task processes recipes by
834 using the
835 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
836 variable to locate applicable patch files, which by default
837 are <filename>*.patch</filename> or
838 <filename>*.diff</filename> files, or any file if
839 "apply=yes" is specified for the file in
840 <filename>SRC_URI</filename>.
841 </para>
842
843 <para>
844 BitBake finds and applies multiple patches for a single recipe
845 in the order in which it finds the patches.
846 Patches are applied to the recipe's source files located in the
847 <link linkend='var-S'><filename>S</filename></link> directory.
848 </para>
849
850 <para>
851 For more information on how the source directories are
852 created, see the
853 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
854 section.
855 </para>
856 </section>
857
858 <section id='configuration-and-compilation-dev-environment'>
859 <title>Configuration and Compilation</title>
860
861 <para>
862 After source code is patched, BitBake executes tasks that
863 configure and compile the source code:
864 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
865 </para>
866
867 <para>
868 This step in the build process consists of three tasks:
869 <itemizedlist>
870 <listitem><para>
871 <emphasis><link linkend='ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></link>:</emphasis>
872 This task sets up the two sysroots in
873 <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}</filename>
874 (i.e. <filename>recipe-sysroot</filename> and
875 <filename>recipe-sysroot-native</filename>) so that
876 the sysroots contain the contents of the
877 <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
878 tasks of the recipes on which the recipe
879 containing the tasks depends.
880 A sysroot exists for both the target and for the native
881 binaries, which run on the host system.
882 </para></listitem>
883 <listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
884 This task configures the source by enabling and
885 disabling any build-time and configuration options for
886 the software being built.
887 Configurations can come from the recipe itself as well
888 as from an inherited class.
889 Additionally, the software itself might configure itself
890 depending on the target for which it is being built.
891 </para>
892
893 <para>The configurations handled by the
894 <link linkend='ref-tasks-configure'><filename>do_configure</filename></link>
895 task are specific
896 to source code configuration for the source code
897 being built by the recipe.</para>
898
899 <para>If you are using the
900 <link linkend='ref-classes-autotools'><filename>autotools</filename></link>
901 class,
902 you can add additional configuration options by using
903 the <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
904 or
905 <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>
906 variables.
907 For information on how this variable works within
908 that class, see the
909 <filename>meta/classes/autotools.bbclass</filename> file.
910 </para></listitem>
911 <listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
912 Once a configuration task has been satisfied, BitBake
913 compiles the source using the
914 <link linkend='ref-tasks-compile'><filename>do_compile</filename></link>
915 task.
916 Compilation occurs in the directory pointed to by the
917 <link linkend='var-B'><filename>B</filename></link>
918 variable.
919 Realize that the <filename>B</filename> directory is, by
920 default, the same as the
921 <link linkend='var-S'><filename>S</filename></link>
922 directory.</para></listitem>
923 <listitem><para><emphasis><filename>do_install</filename>:</emphasis>
924 Once compilation is done, BitBake executes the
925 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
926 task.
927 This task copies files from the <filename>B</filename>
928 directory and places them in a holding area pointed to
929 by the
930 <link linkend='var-D'><filename>D</filename></link>
931 variable.</para></listitem>
932 </itemizedlist>
933 </para>
934 </section>
935
936 <section id='package-splitting-dev-environment'>
937 <title>Package Splitting</title>
938
939 <para>
940 After source code is configured and compiled, the
941 OpenEmbedded build system analyzes
942 the results and splits the output into packages:
943 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
944 </para>
945
946 <para>
947 The
948 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
949 and
950 <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
951 tasks combine to analyze
952 the files found in the
953 <link linkend='var-D'><filename>D</filename></link> directory
954 and split them into subsets based on available packages and
955 files.
956 The analyzing process involves the following as well as other
957 items: splitting out debugging symbols,
958 looking at shared library dependencies between packages,
959 and looking at package relationships.
960 The <filename>do_packagedata</filename> task creates package
961 metadata based on the analysis such that the
962 OpenEmbedded build system can generate the final packages.
963 Working, staged, and intermediate results of the analysis
964 and package splitting process use these areas:
965 <itemizedlist>
966 <listitem><para><link linkend='var-PKGD'><filename>PKGD</filename></link> -
967 The destination directory for packages before they are
968 split.
969 </para></listitem>
970 <listitem><para><link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link> -
971 A shared, global-state directory that holds data
972 generated during the packaging process.
973 </para></listitem>
974 <listitem><para><link linkend='var-PKGDESTWORK'><filename>PKGDESTWORK</filename></link> -
975 A temporary work area used by the
976 <filename>do_package</filename> task.
977 </para></listitem>
978 <listitem><para><link linkend='var-PKGDEST'><filename>PKGDEST</filename></link> -
979 The parent directory for packages after they have
980 been split.
981 </para></listitem>
982 </itemizedlist>
983 The <link linkend='var-FILES'><filename>FILES</filename></link>
984 variable defines the files that go into each package in
985 <link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>.
986 If you want details on how this is accomplished, you can
987 look at the
988 <link linkend='ref-classes-package'><filename>package</filename></link>
989 class.
990 </para>
991
992 <para>
993 Depending on the type of packages being created (RPM, DEB, or
994 IPK), the <filename>do_package_write_*</filename> task
995 creates the actual packages and places them in the
996 Package Feed area, which is
997 <filename>${TMPDIR}/deploy</filename>.
998 You can see the
999 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
1000 section for more detail on that part of the build process.
1001 <note>
1002 Support for creating feeds directly from the
1003 <filename>deploy/*</filename> directories does not exist.
1004 Creating such feeds usually requires some kind of feed
1005 maintenance mechanism that would upload the new packages
1006 into an official package feed (e.g. the
1007 Ångström distribution).
1008 This functionality is highly distribution-specific
1009 and thus is not provided out of the box.
1010 </note>
1011 </para>
1012 </section>
1013
1014 <section id='image-generation-dev-environment'>
1015 <title>Image Generation</title>
1016
1017 <para>
1018 Once packages are split and stored in the Package Feeds area,
1019 the OpenEmbedded build system uses BitBake to generate the
1020 root filesystem image:
1021 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
1022 </para>
1023
1024 <para>
1025 The image generation process consists of several stages and
1026 depends on several tasks and variables.
1027 The
1028 <link linkend='ref-tasks-rootfs'><filename>do_rootfs</filename></link>
1029 task creates the root filesystem (file and directory structure)
1030 for an image.
1031 This task uses several key variables to help create the list
1032 of packages to actually install:
1033 <itemizedlist>
1034 <listitem><para><link linkend='var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></link>:
1035 Lists out the base set of packages to install from
1036 the Package Feeds area.</para></listitem>
1037 <listitem><para><link linkend='var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></link>:
1038 Specifies packages that should not be installed.
1039 </para></listitem>
1040 <listitem><para><link linkend='var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></link>:
1041 Specifies features to include in the image.
1042 Most of these features map to additional packages for
1043 installation.</para></listitem>
1044 <listitem><para><link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>:
1045 Specifies the package backend to use and consequently
1046 helps determine where to locate packages within the
1047 Package Feeds area.</para></listitem>
1048 <listitem><para><link linkend='var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></link>:
1049 Determines the language(s) for which additional
1050 language support packages are installed.
1051 </para></listitem>
1052 <listitem><para><link linkend='var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></link>:
1053 The final list of packages passed to the package manager
1054 for installation into the image.
1055 </para></listitem>
1056 </itemizedlist>
1057 </para>
1058
1059 <para>
1060 With
1061 <link linkend='var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></link>
1062 pointing to the location of the filesystem under construction and
1063 the <filename>PACKAGE_INSTALL</filename> variable providing the
1064 final list of packages to install, the root file system is
1065 created.
1066 </para>
1067
1068 <para>
1069 Package installation is under control of the package manager
1070 (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of whether or
1071 not package management is enabled for the target.
1072 At the end of the process, if package management is not
1073 enabled for the target, the package manager's data files
1074 are deleted from the root filesystem.
1075 As part of the final stage of package installation, postinstall
1076 scripts that are part of the packages are run.
1077 Any scripts that fail to run
1078 on the build host are run on the target when the target system
1079 is first booted.
1080 If you are using a
1081 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
1082 all the post installation scripts must succeed during the
1083 package installation phase since the root filesystem is
1084 read-only.
1085 </para>
1086
1087 <para>
1088 The final stages of the <filename>do_rootfs</filename> task
1089 handle post processing.
1090 Post processing includes creation of a manifest file and
1091 optimizations.
1092 </para>
1093
1094 <para>
1095 The manifest file (<filename>.manifest</filename>) resides
1096 in the same directory as the root filesystem image.
1097 This file lists out, line-by-line, the installed packages.
1098 The manifest file is useful for the
1099 <link linkend='ref-classes-testimage*'><filename>testimage</filename></link>
1100 class, for example, to determine whether or not to run
1101 specific tests.
1102 See the
1103 <link linkend='var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></link>
1104 variable for additional information.
1105 </para>
1106
1107 <para>
1108 Optimizing processes run across the image include
1109 <filename>mklibs</filename>, <filename>prelink</filename>,
1110 and any other post-processing commands as defined by the
1111 <link linkend='var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></link>
1112 variable.
1113 The <filename>mklibs</filename> process optimizes the size
1114 of the libraries, while the
1115 <filename>prelink</filename> process optimizes the dynamic
1116 linking of shared libraries to reduce start up time of
1117 executables.
1118 </para>
1119
1120 <para>
1121 After the root filesystem is built, processing begins on
1122 the image through the <filename>do_image</filename> task.
1123 The build system runs any pre-processing commands as defined
1124 by the
1125 <link linkend='var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></link>
1126 variable.
1127 This variable specifies a list of functions to call before
1128 the OpenEmbedded build system creates the final image output
1129 files.
1130 </para>
1131
1132 <para>
1133 The <filename>do_image</filename> task dynamically creates
1134 other <filename>do_image_*</filename> tasks as needed, which
1135 include compressing the root filesystem image to reduce the
1136 overall size of the image.
1137 The process turns everything into an image file or a set of
1138 image files.
1139 The formats used for the root filesystem depend on the
1140 <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
1141 variable.
1142 </para>
1143
1144 <para>
1145 The final task involved in image creation is the
1146 <filename>do_image_complete</filename> task.
1147 This task completes the image by applying any image
1148 post processing as defined through the
1149 <link linkend='var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></link>
1150 variable.
1151 The variable specifies a list of functions to call once the
1152 OpenEmbedded build system has created the final image output
1153 files.
1154 </para>
1155
1156 <note>
1157 The entire image generation process is run under Pseudo.
1158 Running under Pseudo ensures that the files in the root
1159 filesystem have correct ownership.
1160 </note>
1161 </section>
1162
1163 <section id='sdk-generation-dev-environment'>
1164 <title>SDK Generation</title>
1165
1166 <para>
1167 The OpenEmbedded build system uses BitBake to generate the
1168 Software Development Kit (SDK) installer script for both the
1169 standard and extensible SDKs:
1170 <imagedata fileref="figures/sdk-generation.png" align="center" />
1171 </para>
1172
1173 <note>
1174 For more information on the cross-development toolchain
1175 generation, see the
1176 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1177 section.
1178 For information on advantages gained when building a
1179 cross-development toolchain using the
1180 <link linkend='ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></link>
1181 task, see the
1182 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
1183 section in the Yocto Project Software Development Kit (SDK)
1184 Developer's Guide.
1185 </note>
1186
1187 <para>
1188 Like image generation, the SDK script process consists of
1189 several stages and depends on many variables.
1190 The <filename>do_populate_sdk</filename> and
1191 <filename>do_populate_sdk_ext</filename> tasks use these
1192 key variables to help create the list of packages to actually
1193 install.
1194 For information on the variables listed in the figure, see the
1195 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1196 section.
1197 </para>
1198
1199 <para>
1200 The <filename>do_populate_sdk</filename> task helps create
1201 the standard SDK and handles two parts: a target part and a
1202 host part.
1203 The target part is the part built for the target hardware and
1204 includes libraries and headers.
1205 The host part is the part of the SDK that runs on the
1206 <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>.
1207 </para>
1208
1209 <para>
1210 The <filename>do_populate_sdk_ext</filename> task helps create
1211 the extensible SDK and handles host and target parts
1212 differently than its counter part does for the standard SDK.
1213 For the extensible SDK, the task encapsulates the build system,
1214 which includes everything needed (host and target) for the SDK.
1215 </para>
1216
1217 <para>
1218 Regardless of the type of SDK being constructed, the
1219 tasks perform some cleanup after which a cross-development
1220 environment setup script and any needed configuration files
1221 are created.
1222 The final output is the Cross-development
1223 toolchain installation script (<filename>.sh</filename> file),
1224 which includes the environment setup script.
1225 </para>
1226 </section>
1227
1228 <section id='stamp-files-and-the-rerunning-of-tasks'>
1229 <title>Stamp Files and the Rerunning of Tasks</title>
1230
1231 <para>
1232 For each task that completes successfully, BitBake writes a
1233 stamp file into the
1234 <link linkend='var-STAMPS_DIR'><filename>STAMPS_DIR</filename></link>
1235 directory.
1236 The beginning of the stamp file's filename is determined by the
1237 <link linkend='var-STAMP'><filename>STAMP</filename></link>
1238 variable, and the end of the name consists of the task's name
1239 and current
1240 <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
1241 <note>
1242 This naming scheme assumes that
1243 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
1244 is "OEBasicHash", which is almost always the case in
1245 current OpenEmbedded.
1246 </note>
1247 To determine if a task needs to be rerun, BitBake checks if a
1248 stamp file with a matching input checksum exists for the task.
1249 If such a stamp file exists, the task's output is assumed to
1250 exist and still be valid.
1251 If the file does not exist, the task is rerun.
1252 <note>
1253 <para>The stamp mechanism is more general than the shared
1254 state (sstate) cache mechanism described in the
1255 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
1256 section.
1257 BitBake avoids rerunning any task that has a valid
1258 stamp file, not just tasks that can be accelerated through
1259 the sstate cache.</para>
1260 <para>However, you should realize that stamp files only
1261 serve as a marker that some work has been done and that
1262 these files do not record task output.
1263 The actual task output would usually be somewhere in
1264 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
1265 (e.g. in some recipe's
1266 <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.)
1267 What the sstate cache mechanism adds is a way to cache task
1268 output that can then be shared between build machines.
1269 </para>
1270 </note>
1271 Since <filename>STAMPS_DIR</filename> is usually a subdirectory
1272 of <filename>TMPDIR</filename>, removing
1273 <filename>TMPDIR</filename> will also remove
1274 <filename>STAMPS_DIR</filename>, which means tasks will
1275 properly be rerun to repopulate <filename>TMPDIR</filename>.
1276 </para>
1277
1278 <para>
1279 If you want some task to always be considered "out of date",
1280 you can mark it with the
1281 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
1282 varflag.
1283 If some other task depends on such a task, then that task will
1284 also always be considered out of date, which might not be what
1285 you want.
1286 </para>
1287
1288 <para>
1289 For details on how to view information about a task's
1290 signature, see the
1291 "<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>"
1292 section.
1293 </para>
1294 </section>
1295
1296 <section id='setscene-tasks-and-shared-state'>
1297 <title>Setscene Tasks and Shared State</title>
1298
1299 <para>
1300 The description of tasks so far assumes that BitBake needs to
1301 build everything and there are no prebuilt objects available.
1302 BitBake does support skipping tasks if prebuilt objects are
1303 available.
1304 These objects are usually made available in the form of a
1305 shared state (sstate) cache.
1306 <note>
1307 For information on variables affecting sstate, see the
1308 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
1309 and
1310 <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
1311 variables.
1312 </note>
1313 </para>
1314
1315 <para>
1316 The idea of a setscene task (i.e
1317 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
1318 is a version of the task where
1319 instead of building something, BitBake can skip to the end
1320 result and simply place a set of files into specific locations
1321 as needed.
1322 In some cases, it makes sense to have a setscene task variant
1323 (e.g. generating package files in the
1324 <filename>do_package_write_*</filename> task).
1325 In other cases, it does not make sense, (e.g. a
1326 <link linkend='ref-tasks-patch'><filename>do_patch</filename></link>
1327 task or
1328 <link linkend='ref-tasks-unpack'><filename>do_unpack</filename></link>
1329 task) since the work involved would be equal to or greater than
1330 the underlying task.
1331 </para>
1332
1333 <para>
1334 In the OpenEmbedded build system, the common tasks that have
1335 setscene variants are <link linkend='ref-tasks-package'><filename>do_package</filename></link>,
1336 <filename>do_package_write_*</filename>,
1337 <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>,
1338 <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>,
1339 and
1340 <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>.
1341 Notice that these are most of the tasks whose output is an
1342 end result.
1343 </para>
1344
1345 <para>
1346 The OpenEmbedded build system has knowledge of the relationship
1347 between these tasks and other tasks that precede them.
1348 For example, if BitBake runs
1349 <filename>do_populate_sysroot_setscene</filename> for
1350 something, there is little point in running any of the
1351 <filename>do_fetch</filename>, <filename>do_unpack</filename>,
1352 <filename>do_patch</filename>,
1353 <filename>do_configure</filename>,
1354 <filename>do_compile</filename>, and
1355 <filename>do_install</filename> tasks.
1356 However, if <filename>do_package</filename> needs to be run,
1357 BitBake would need to run those other tasks.
1358 </para>
1359
1360 <para>
1361 It becomes more complicated if everything can come from an
1362 sstate cache because some objects are simply not required at
1363 all.
1364 For example, you do not need a compiler or native tools, such
1365 as quilt, if there is nothing to compile or patch.
1366 If the <filename>do_package_write_*</filename> packages are
1367 available from sstate, BitBake does not need the
1368 <filename>do_package</filename> task data.
1369 </para>
1370
1371 <para>
1372 To handle all these complexities, BitBake runs in two phases.
1373 The first is the "setscene" stage.
1374 During this stage, BitBake first checks the sstate cache for
1375 any targets it is planning to build.
1376 BitBake does a fast check to see if the object exists rather
1377 than a complete download.
1378 If nothing exists, the second phase, which is the setscene
1379 stage, completes and the main build proceeds.
1380 </para>
1381
1382 <para>
1383 If objects are found in the sstate cache, the OpenEmbedded
1384 build system works backwards from the end targets specified
1385 by the user.
1386 For example, if an image is being built, the OpenEmbedded build
1387 system first looks for the packages needed for that image and
1388 the tools needed to construct an image.
1389 If those are available, the compiler is not needed.
1390 Thus, the compiler is not even downloaded.
1391 If something was found to be unavailable, or the download or
1392 setscene task fails, the OpenEmbedded build system then tries
1393 to install dependencies, such as the compiler, from the cache.
1394 </para>
1395
1396 <para>
1397 The availability of objects in the sstate cache is handled by
1398 the function specified by the
1399 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
1400 variable and returns a list of the objects that are available.
1401 The function specified by the
1402 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
1403 variable is the function that determines whether a given
1404 dependency needs to be followed, and whether for any given
1405 relationship the function needs to be passed.
1406 The function returns a True or False value.
1407 </para>
1408 </section>
1409 </section>
1410
1411 <section id='images-dev-environment'>
1412 <title>Images</title>
1413
1414 <para>
1415 The images produced by the OpenEmbedded build system
1416 are compressed forms of the
1417 root filesystem that are ready to boot on a target device.
1418 You can see from the
1419 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
1420 that BitBake output, in part, consists of images.
1421 This section is going to look more closely at this output:
1422 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1423 </para>
1424
1425 <para>
1426 For a list of example images that the Yocto Project provides,
1427 see the
1428 "<link linkend='ref-images'>Images</link>" chapter.
1429 </para>
1430
1431 <para>
1432 Images are written out to the
1433 <link linkend='build-directory'>Build Directory</link>
1434 inside the <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
1435 folder as shown in the figure.
1436 This folder contains any files expected to be loaded on the
1437 target device.
1438 The
1439 <link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>
1440 variable points to the <filename>deploy</filename> directory,
1441 while the
1442 <link linkend='var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></link>
1443 variable points to the appropriate directory containing images for
1444 the current configuration.
1445 <itemizedlist>
1446 <listitem><para><filename><replaceable>kernel-image</replaceable></filename>:
1447 A kernel binary file.
1448 The <link linkend='var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></link>
1449 variable setting determines the naming scheme for the
1450 kernel image file.
1451 Depending on that variable, the file could begin with
1452 a variety of naming strings.
1453 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1454 directory can contain multiple image files for the
1455 machine.</para></listitem>
1456 <listitem><para><filename><replaceable>root-filesystem-image</replaceable></filename>:
1457 Root filesystems for the target device (e.g.
1458 <filename>*.ext3</filename> or <filename>*.bz2</filename>
1459 files).
1460 The <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
1461 variable setting determines the root filesystem image
1462 type.
1463 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1464 directory can contain multiple root filesystems for the
1465 machine.</para></listitem>
1466 <listitem><para><filename><replaceable>kernel-modules</replaceable></filename>:
1467 Tarballs that contain all the modules built for the kernel.
1468 Kernel module tarballs exist for legacy purposes and
1469 can be suppressed by setting the
1470 <link linkend='var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></link>
1471 variable to "0".
1472 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1473 directory can contain multiple kernel module tarballs
1474 for the machine.</para></listitem>
1475 <listitem><para><filename><replaceable>bootloaders</replaceable></filename>:
1476 Bootloaders supporting the image, if applicable to the
1477 target machine.
1478 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1479 directory can contain multiple bootloaders for the
1480 machine.</para></listitem>
1481 <listitem><para><filename><replaceable>symlinks</replaceable></filename>:
1482 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1483 folder contains
1484 a symbolic link that points to the most recently built file
1485 for each machine.
1486 These links might be useful for external scripts that
1487 need to obtain the latest version of each file.
1488 </para></listitem>
1489 </itemizedlist>
1490 </para>
1491 </section>
1492
1493 <section id='sdk-dev-environment'>
1494 <title>Application Development SDK</title>
1495
1496 <para>
1497 In the
1498 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
1499 the output labeled "Application Development SDK" represents an
1500 SDK.
1501 The SDK generation process differs depending on whether you build
1502 a standard SDK
1503 (e.g. <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>)
1504 or an extensible SDK
1505 (e.g. <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>).
1506 This section is going to take a closer look at this output:
1507 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
1508 </para>
1509
1510 <para>
1511 The specific form of this output is a self-extracting
1512 SDK installer (<filename>*.sh</filename>) that, when run,
1513 installs the SDK, which consists of a cross-development
1514 toolchain, a set of libraries and headers, and an SDK
1515 environment setup script.
1516 Running this installer essentially sets up your
1517 cross-development environment.
1518 You can think of the cross-toolchain as the "host"
1519 part because it runs on the SDK machine.
1520 You can think of the libraries and headers as the "target"
1521 part because they are built for the target hardware.
1522 The environment setup script is added so that you can initialize
1523 the environment before using the tools.
1524 </para>
1525
1526 <note>
1527 <para>
1528 The Yocto Project supports several methods by which you can
1529 set up this cross-development environment.
1530 These methods include downloading pre-built SDK installers
1531 or building and installing your own SDK installer.
1532 </para>
1533
1534 <para>
1535 For background information on cross-development toolchains
1536 in the Yocto Project development environment, see the
1537 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1538 section.
1539 For information on setting up a cross-development
1540 environment, see the
1541 <ulink url='&YOCTO_DOCS_SDK_URL;#sdk-manual'>Yocto Project Software Development Kit (SDK) Developer's Guide</ulink>.
1542 </para>
1543 </note>
1544
1545 <para>
1546 Once built, the SDK installers are written out to the
1547 <filename>deploy/sdk</filename> folder inside the
1548 <link linkend='build-directory'>Build Directory</link>
1549 as shown in the figure at the beginning of this section.
1550 Depending on the type of SDK, several variables exist that help
1551 configure these files.
1552 The following list shows the variables associated with a standard
1553 SDK:
1554 <itemizedlist>
1555 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
1556 Points to the <filename>deploy</filename>
1557 directory.</para></listitem>
1558 <listitem><para><link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>:
1559 Specifies the architecture of the machine
1560 on which the cross-development tools are run to
1561 create packages for the target hardware.
1562 </para></listitem>
1563 <listitem><para><link linkend='var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></link>:
1564 Lists the features to include in the "target" part
1565 of the SDK.
1566 </para></listitem>
1567 <listitem><para><link linkend='var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></link>:
1568 Lists packages that make up the host
1569 part of the SDK (i.e. the part that runs on
1570 the <filename>SDKMACHINE</filename>).
1571 When you use
1572 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
1573 to create the SDK, a set of default packages
1574 apply.
1575 This variable allows you to add more packages.
1576 </para></listitem>
1577 <listitem><para><link linkend='var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></link>:
1578 Lists packages that make up the target part
1579 of the SDK (i.e. the part built for the
1580 target hardware).
1581 </para></listitem>
1582 <listitem><para><link linkend='var-SDKPATH'><filename>SDKPATH</filename></link>:
1583 Defines the default SDK installation path offered by the
1584 installation script.
1585 </para></listitem>
1586 </itemizedlist>
1587 This next list, shows the variables associated with an extensible
1588 SDK:
1589 <itemizedlist>
1590 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
1591 Points to the <filename>deploy</filename> directory.
1592 </para></listitem>
1593 <listitem><para><link linkend='var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></link>:
1594 Controls whether or not shared state artifacts are copied
1595 into the extensible SDK.
1596 By default, all required shared state artifacts are copied
1597 into the SDK.
1598 </para></listitem>
1599 <listitem><para><link linkend='var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></link>:
1600 Specifies whether or not packagedata will be included in
1601 the extensible SDK for all recipes in the "world" target.
1602 </para></listitem>
1603 <listitem><para><link linkend='var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></link>:
1604 Specifies whether or not the toolchain will be included
1605 when building the extensible SDK.
1606 </para></listitem>
1607 <listitem><para><link linkend='var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></link>:
1608 A list of variables allowed through from the build system
1609 configuration into the extensible SDK configuration.
1610 </para></listitem>
1611 <listitem><para><link linkend='var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></link>:
1612 A list of variables not allowed through from the build
1613 system configuration into the extensible SDK configuration.
1614 </para></listitem>
1615 <listitem><para><link linkend='var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></link>:
1616 A list of classes to remove from the
1617 <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
1618 value globally within the extensible SDK configuration.
1619 </para></listitem>
1620 </itemizedlist>
1621 </para>
1622 </section>
1623
1624</chapter>
1625<!--
1626vim: expandtab tw=80 ts=4
1627-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-12 10:22:21 +0000