summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual/ref-development-environment.xml
diff options
context:
space:
mode:
authorScott Rifenbark <srifenbark@gmail.com>2017-06-14 09:50:55 -0700
committerRichard Purdie <richard.purdie@linuxfoundation.org>2017-06-22 09:16:43 +0100
commitde6d45fefc3000ee8918d7c18448758d4216bae5 (patch)
treed0e7f2913f00d7df940dad84075dd9937e0de635 /documentation/ref-manual/ref-development-environment.xml
parent3f6a9af9272cf30d78aa3de76af3f6ee0dc2bd22 (diff)
downloadpoky-de6d45fefc3000ee8918d7c18448758d4216bae5.tar.gz
documentation: Re-org for "closer-look" chapter
Fixes [YOCTO #11630] The ref-manual needs expansion for the old "closer-look" chapter. This chapter previously held a detailed look at what happens when a user uses the YP to develop something. Now, the chapter needs to also contain YP development environment concepts (e.g. open- source philosophy, etc.), which are coming from the dev-manual. Because of this, I renamed the "closer-look.xml" chapter to be "ref-development-environment.xml". I also renamed the larger section that was formerly the entire chapter into its own section named "Development Concepts". Both these changes caused a few links to break. I fixed all the links from within the various manuals so they would find appropriate targets. I did some re-writing for introductory material to introduce the new chapter and the section on "Development Concepts". A new file ("ref-development-environment.xml") was added by basically renaming the "closer-look.xml" chapter. And, the tracking for "closer-look.xml" was deleted. (From yocto-docs rev: e37806474578b4f0ed137f64d68a39a17ab60644) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/ref-manual/ref-development-environment.xml')
-rw-r--r--documentation/ref-manual/ref-development-environment.xml1635
1 files changed, 1635 insertions, 0 deletions
diff --git a/documentation/ref-manual/ref-development-environment.xml b/documentation/ref-manual/ref-development-environment.xml
new file mode 100644
index 0000000000..a30cefc391
--- /dev/null
+++ b/documentation/ref-manual/ref-development-environment.xml
@@ -0,0 +1,1635 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='ref-development-environment'>
6<title>The Yocto Project Development Environment</title>
7
8 <para>
9 This chapter takes a look at the Yocto Project development
10 environment and also provides a detailed look at what goes on during
11 development in that environment.
12 </para>
13
14<section id="development-concepts">
15 <title>Development Concepts</title>
16
17 <para>
18 This section takes a more detailed look inside the development
19 process.
20 The following diagram represents development at a high level.
21 The remainder of this chapter expands on the fundamental input, output,
22 process, and
23 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>) blocks
24 that make up development in the Yocto Project environment.
25 </para>
26
27 <para id='general-yocto-environment-figure'>
28 <imagedata fileref="figures/yocto-environment-ref.png" align="center" width="8in" depth="4.25in" />
29 </para>
30
31 <para>
32 In general, development consists of several functional areas:
33 <itemizedlist>
34 <listitem><para><emphasis>User Configuration:</emphasis>
35 Metadata you can use to control the build process.
36 </para></listitem>
37 <listitem><para><emphasis>Metadata Layers:</emphasis>
38 Various layers that provide software, machine, and
39 distro Metadata.</para></listitem>
40 <listitem><para><emphasis>Source Files:</emphasis>
41 Upstream releases, local projects, and SCMs.</para></listitem>
42 <listitem><para><emphasis>Build System:</emphasis>
43 Processes under the control of
44 <link linkend='bitbake-term'>BitBake</link>.
45 This block expands on how BitBake fetches source, applies
46 patches, completes compilation, analyzes output for package
47 generation, creates and tests packages, generates images, and
48 generates cross-development tools.</para></listitem>
49 <listitem><para><emphasis>Package Feeds:</emphasis>
50 Directories containing output packages (RPM, DEB or IPK),
51 which are subsequently used in the construction of an image or
52 SDK, produced by the build system.
53 These feeds can also be copied and shared using a web server or
54 other means to facilitate extending or updating existing
55 images on devices at runtime if runtime package management is
56 enabled.</para></listitem>
57 <listitem><para><emphasis>Images:</emphasis>
58 Images produced by the development process.
59 </para></listitem>
60 <listitem><para><emphasis>Application Development SDK:</emphasis>
61 Cross-development tools that are produced along with an image
62 or separately with BitBake.</para></listitem>
63 </itemizedlist>
64 </para>
65
66 <section id="user-configuration">
67 <title>User Configuration</title>
68
69 <para>
70 User configuration helps define the build.
71 Through user configuration, you can tell BitBake the
72 target architecture for which you are building the image,
73 where to store downloaded source, and other build properties.
74 </para>
75
76 <para>
77 The following figure shows an expanded representation of the
78 "User Configuration" box of the
79 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
80 </para>
81
82 <para>
83 <imagedata fileref="figures/user-configuration.png" align="center" />
84 </para>
85
86 <para>
87 BitBake needs some basic configuration files in order to complete
88 a build.
89 These files are <filename>*.conf</filename> files.
90 The minimally necessary ones reside as example files in the
91 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
92 For simplicity, this section refers to the Source Directory as
93 the "Poky Directory."
94 </para>
95
96 <para>
97 When you clone the <filename>poky</filename> Git repository or you
98 download and unpack a Yocto Project release, you can set up the
99 Source Directory to be named anything you want.
100 For this discussion, the cloned repository uses the default
101 name <filename>poky</filename>.
102 <note>
103 The Poky repository is primarily an aggregation of existing
104 repositories.
105 It is not a canonical upstream source.
106 </note>
107 </para>
108
109 <para>
110 The <filename>meta-poky</filename> layer inside Poky contains
111 a <filename>conf</filename> directory that has example
112 configuration files.
113 These example files are used as a basis for creating actual
114 configuration files when you source the build environment
115 script
116 (i.e.
117 <link linkend='structure-core-script'><filename>&OE_INIT_FILE;</filename></link>
118 or
119 <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>).
120 </para>
121
122 <para>
123 Sourcing the build environment script creates a
124 <link linkend='build-directory'>Build Directory</link>
125 if one does not already exist.
126 BitBake uses the Build Directory for all its work during builds.
127 The Build Directory has a <filename>conf</filename> directory that
128 contains default versions of your <filename>local.conf</filename>
129 and <filename>bblayers.conf</filename> configuration files.
130 These default configuration files are created only if versions
131 do not already exist in the Build Directory at the time you
132 source the build environment setup script.
133 </para>
134
135 <para>
136 Because the Poky repository is fundamentally an aggregation of
137 existing repositories, some users might be familiar with running
138 the <filename>&OE_INIT_FILE;</filename> or
139 <filename>oe-init-build-env-memres</filename> script in the context
140 of separate OpenEmbedded-Core and BitBake repositories rather than a
141 single Poky repository.
142 This discussion assumes the script is executed from within a cloned
143 or unpacked version of Poky.
144 </para>
145
146 <para>
147 Depending on where the script is sourced, different sub-scripts
148 are called to set up the Build Directory (Yocto or OpenEmbedded).
149 Specifically, the script
150 <filename>scripts/oe-setup-builddir</filename> inside the
151 poky directory sets up the Build Directory and seeds the directory
152 (if necessary) with configuration files appropriate for the
153 Yocto Project development environment.
154 <note>
155 The <filename>scripts/oe-setup-builddir</filename> script
156 uses the <filename>$TEMPLATECONF</filename> variable to
157 determine which sample configuration files to locate.
158 </note>
159 </para>
160
161 <para>
162 The <filename>local.conf</filename> file provides many
163 basic variables that define a build environment.
164 Here is a list of a few.
165 To see the default configurations in a <filename>local.conf</filename>
166 file created by the build environment script, see the
167 <filename>local.conf.sample</filename> in the
168 <filename>meta-poky</filename> layer:
169 <itemizedlist>
170 <listitem><para><emphasis>Parallelism Options:</emphasis>
171 Controlled by the
172 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>,
173 <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></link>,
174 and
175 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></ulink>
176 variables.</para></listitem>
177 <listitem><para><emphasis>Target Machine Selection:</emphasis>
178 Controlled by the
179 <link linkend='var-MACHINE'><filename>MACHINE</filename></link>
180 variable.</para></listitem>
181 <listitem><para><emphasis>Download Directory:</emphasis>
182 Controlled by the
183 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
184 variable.</para></listitem>
185 <listitem><para><emphasis>Shared State Directory:</emphasis>
186 Controlled by the
187 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
188 variable.</para></listitem>
189 <listitem><para><emphasis>Build Output:</emphasis>
190 Controlled by the
191 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
192 variable.</para></listitem>
193 </itemizedlist>
194 <note>
195 Configurations set in the <filename>conf/local.conf</filename>
196 file can also be set in the
197 <filename>conf/site.conf</filename> and
198 <filename>conf/auto.conf</filename> configuration files.
199 </note>
200 </para>
201
202 <para>
203 The <filename>bblayers.conf</filename> file tells BitBake what
204 layers you want considered during the build.
205 By default, the layers listed in this file include layers
206 minimally needed by the build system.
207 However, you must manually add any custom layers you have created.
208 You can find more information on working with the
209 <filename>bblayers.conf</filename> file in the
210 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
211 section in the Yocto Project Development Manual.
212 </para>
213
214 <para>
215 The files <filename>site.conf</filename> and
216 <filename>auto.conf</filename> are not created by the environment
217 initialization script.
218 If you want the <filename>site.conf</filename> file, you need to
219 create that yourself.
220 The <filename>auto.conf</filename> file is typically created by
221 an autobuilder:
222 <itemizedlist>
223 <listitem><para><emphasis><filename>site.conf</filename>:</emphasis>
224 You can use the <filename>conf/site.conf</filename>
225 configuration file to configure multiple build directories.
226 For example, suppose you had several build environments and
227 they shared some common features.
228 You can set these default build properties here.
229 A good example is perhaps the packaging format to use
230 through the
231 <link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>
232 variable.</para>
233 <para>One useful scenario for using the
234 <filename>conf/site.conf</filename> file is to extend your
235 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
236 variable to include the path to a
237 <filename>conf/site.conf</filename>.
238 Then, when BitBake looks for Metadata using
239 <filename>BBPATH</filename>, it finds the
240 <filename>conf/site.conf</filename> file and applies your
241 common configurations found in the file.
242 To override configurations in a particular build directory,
243 alter the similar configurations within that build
244 directory's <filename>conf/local.conf</filename> file.
245 </para></listitem>
246 <listitem><para><emphasis><filename>auto.conf</filename>:</emphasis>
247 The file is usually created and written to by
248 an autobuilder.
249 The settings put into the file are typically the same as
250 you would find in the <filename>conf/local.conf</filename>
251 or the <filename>conf/site.conf</filename> files.
252 </para></listitem>
253 </itemizedlist>
254 </para>
255
256 <para>
257 You can edit all configuration files to further define
258 any particular build environment.
259 This process is represented by the "User Configuration Edits"
260 box in the figure.
261 </para>
262
263 <para>
264 When you launch your build with the
265 <filename>bitbake <replaceable>target</replaceable></filename>
266 command, BitBake sorts out the configurations to ultimately
267 define your build environment.
268 It is important to understand that the OpenEmbedded build system
269 reads the configuration files in a specific order:
270 <filename>site.conf</filename>, <filename>auto.conf</filename>,
271 and <filename>local.conf</filename>.
272 And, the build system applies the normal assignment statement
273 rules.
274 Because the files are parsed in a specific order, variable
275 assignments for the same variable could be affected.
276 For example, if the <filename>auto.conf</filename> file and
277 the <filename>local.conf</filename> set
278 <replaceable>variable1</replaceable> to different values, because
279 the build system parses <filename>local.conf</filename> after
280 <filename>auto.conf</filename>,
281 <replaceable>variable1</replaceable> is assigned the value from
282 the <filename>local.conf</filename> file.
283 </para>
284 </section>
285
286 <section id="metadata-machine-configuration-and-policy-configuration">
287 <title>Metadata, Machine Configuration, and Policy Configuration</title>
288
289 <para>
290 The previous section described the user configurations that
291 define BitBake's global behavior.
292 This section takes a closer look at the layers the build system
293 uses to further control the build.
294 These layers provide Metadata for the software, machine, and
295 policy.
296 </para>
297
298 <para>
299 In general, three types of layer input exist:
300 <itemizedlist>
301 <listitem><para><emphasis>Policy Configuration:</emphasis>
302 Distribution Layers provide top-level or general
303 policies for the image or SDK being built.
304 For example, this layer would dictate whether BitBake
305 produces RPM or IPK packages.</para></listitem>
306 <listitem><para><emphasis>Machine Configuration:</emphasis>
307 Board Support Package (BSP) layers provide machine
308 configurations.
309 This type of information is specific to a particular
310 target architecture.</para></listitem>
311 <listitem><para><emphasis>Metadata:</emphasis>
312 Software layers contain user-supplied recipe files,
313 patches, and append files.
314 </para></listitem>
315 </itemizedlist>
316 </para>
317
318 <para>
319 The following figure shows an expanded representation of the
320 Metadata, Machine Configuration, and Policy Configuration input
321 (layers) boxes of the
322 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
323 </para>
324
325 <para>
326 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="7.5in" />
327 </para>
328
329 <para>
330 In general, all layers have a similar structure.
331 They all contain a licensing file
332 (e.g. <filename>COPYING</filename>) if the layer is to be
333 distributed, a <filename>README</filename> file as good practice
334 and especially if the layer is to be distributed, a
335 configuration directory, and recipe directories.
336 </para>
337
338 <para>
339 The Yocto Project has many layers that can be used.
340 You can see a web-interface listing of them on the
341 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
342 page.
343 The layers are shown at the bottom categorized under
344 "Yocto Metadata Layers."
345 These layers are fundamentally a subset of the
346 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Metadata Index</ulink>,
347 which lists all layers provided by the OpenEmbedded community.
348 <note>
349 Layers exist in the Yocto Project Source Repositories that
350 cannot be found in the OpenEmbedded Metadata Index.
351 These layers are either deprecated or experimental in nature.
352 </note>
353 </para>
354
355 <para>
356 BitBake uses the <filename>conf/bblayers.conf</filename> file,
357 which is part of the user configuration, to find what layers it
358 should be using as part of the build.
359 </para>
360
361 <para>
362 For more information on layers, see the
363 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
364 section in the Yocto Project Development Manual.
365 </para>
366
367 <section id="distro-layer">
368 <title>Distro Layer</title>
369
370 <para>
371 The distribution layer provides policy configurations for your
372 distribution.
373 Best practices dictate that you isolate these types of
374 configurations into their own layer.
375 Settings you provide in
376 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
377 similar
378 settings that BitBake finds in your
379 <filename>conf/local.conf</filename> file in the Build
380 Directory.
381 </para>
382
383 <para>
384 The following list provides some explanation and references
385 for what you typically find in the distribution layer:
386 <itemizedlist>
387 <listitem><para><emphasis>classes:</emphasis>
388 Class files (<filename>.bbclass</filename>) hold
389 common functionality that can be shared among
390 recipes in the distribution.
391 When your recipes inherit a class, they take on the
392 settings and functions for that class.
393 You can read more about class files in the
394 "<link linkend='ref-classes'>Classes</link>" section.
395 </para></listitem>
396 <listitem><para><emphasis>conf:</emphasis>
397 This area holds configuration files for the
398 layer (<filename>conf/layer.conf</filename>),
399 the distribution
400 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
401 and any distribution-wide include files.
402 </para></listitem>
403 <listitem><para><emphasis>recipes-*:</emphasis>
404 Recipes and append files that affect common
405 functionality across the distribution.
406 This area could include recipes and append files
407 to add distribution-specific configuration,
408 initialization scripts, custom image recipes,
409 and so forth.</para></listitem>
410 </itemizedlist>
411 </para>
412 </section>
413
414 <section id="bsp-layer">
415 <title>BSP Layer</title>
416
417 <para>
418 The BSP Layer provides machine configurations.
419 Everything in this layer is specific to the machine for which
420 you are building the image or the SDK.
421 A common structure or form is defined for BSP layers.
422 You can learn more about this structure in the
423 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
424 <note>
425 In order for a BSP layer to be considered compliant with the
426 Yocto Project, it must meet some structural requirements.
427 </note>
428 </para>
429
430 <para>
431 The BSP Layer's configuration directory contains
432 configuration files for the machine
433 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>) and,
434 of course, the layer (<filename>conf/layer.conf</filename>).
435 </para>
436
437 <para>
438 The remainder of the layer is dedicated to specific recipes
439 by function: <filename>recipes-bsp</filename>,
440 <filename>recipes-core</filename>,
441 <filename>recipes-graphics</filename>, and
442 <filename>recipes-kernel</filename>.
443 Metadata can exist for multiple formfactors, graphics
444 support systems, and so forth.
445 <note>
446 While the figure shows several <filename>recipes-*</filename>
447 directories, not all these directories appear in all
448 BSP layers.
449 </note>
450 </para>
451 </section>
452
453 <section id="software-layer">
454 <title>Software Layer</title>
455
456 <para>
457 The software layer provides the Metadata for additional
458 software packages used during the build.
459 This layer does not include Metadata that is specific to the
460 distribution or the machine, which are found in their
461 respective layers.
462 </para>
463
464 <para>
465 This layer contains any new recipes that your project needs
466 in the form of recipe files.
467 </para>
468 </section>
469 </section>
470
471 <section id="sources-dev-environment">
472 <title>Sources</title>
473
474 <para>
475 In order for the OpenEmbedded build system to create an image or
476 any target, it must be able to access source files.
477 The
478 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
479 represents source files using the "Upstream Project Releases",
480 "Local Projects", and "SCMs (optional)" boxes.
481 The figure represents mirrors, which also play a role in locating
482 source files, with the "Source Mirror(s)" box.
483 </para>
484
485 <para>
486 The method by which source files are ultimately organized is
487 a function of the project.
488 For example, for released software, projects tend to use tarballs
489 or other archived files that can capture the state of a release
490 guaranteeing that it is statically represented.
491 On the other hand, for a project that is more dynamic or
492 experimental in nature, a project might keep source files in a
493 repository controlled by a Source Control Manager (SCM) such as
494 Git.
495 Pulling source from a repository allows you to control
496 the point in the repository (the revision) from which you want to
497 build software.
498 Finally, a combination of the two might exist, which would give the
499 consumer a choice when deciding where to get source files.
500 </para>
501
502 <para>
503 BitBake uses the
504 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
505 variable to point to source files regardless of their location.
506 Each recipe must have a <filename>SRC_URI</filename> variable
507 that points to the source.
508 </para>
509
510 <para>
511 Another area that plays a significant role in where source files
512 come from is pointed to by the
513 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
514 variable.
515 This area is a cache that can hold previously downloaded source.
516 You can also instruct the OpenEmbedded build system to create
517 tarballs from Git repositories, which is not the default behavior,
518 and store them in the <filename>DL_DIR</filename> by using the
519 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
520 variable.
521 </para>
522
523 <para>
524 Judicious use of a <filename>DL_DIR</filename> directory can
525 save the build system a trip across the Internet when looking
526 for files.
527 A good method for using a download directory is to have
528 <filename>DL_DIR</filename> point to an area outside of your
529 Build Directory.
530 Doing so allows you to safely delete the Build Directory
531 if needed without fear of removing any downloaded source file.
532 </para>
533
534 <para>
535 The remainder of this section provides a deeper look into the
536 source files and the mirrors.
537 Here is a more detailed look at the source file area of the
538 base figure:
539 <imagedata fileref="figures/source-input.png" align="center" width="7in" depth="7.5in" />
540 </para>
541
542 <section id='upstream-project-releases'>
543 <title>Upstream Project Releases</title>
544
545 <para>
546 Upstream project releases exist anywhere in the form of an
547 archived file (e.g. tarball or zip file).
548 These files correspond to individual recipes.
549 For example, the figure uses specific releases each for
550 BusyBox, Qt, and Dbus.
551 An archive file can be for any released product that can be
552 built using a recipe.
553 </para>
554 </section>
555
556 <section id='local-projects'>
557 <title>Local Projects</title>
558
559 <para>
560 Local projects are custom bits of software the user provides.
561 These bits reside somewhere local to a project - perhaps
562 a directory into which the user checks in items (e.g.
563 a local directory containing a development source tree
564 used by the group).
565 </para>
566
567 <para>
568 The canonical method through which to include a local project
569 is to use the
570 <link linkend='ref-classes-externalsrc'><filename>externalsrc</filename></link>
571 class to include that local project.
572 You use either the <filename>local.conf</filename> or a
573 recipe's append file to override or set the
574 recipe to point to the local directory on your disk to pull
575 in the whole source tree.
576 </para>
577
578 <para>
579 For information on how to use the
580 <filename>externalsrc</filename> class, see the
581 "<link linkend='ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></link>"
582 section.
583 </para>
584 </section>
585
586 <section id='scms'>
587 <title>Source Control Managers (Optional)</title>
588
589 <para>
590 Another place the build system can get source files from is
591 through an SCM such as Git or Subversion.
592 In this case, a repository is cloned or checked out.
593 The
594 <link linkend='ref-tasks-fetch'><filename>do_fetch</filename></link>
595 task inside BitBake uses
596 the <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
597 variable and the argument's prefix to determine the correct
598 fetcher module.
599 </para>
600
601 <note>
602 For information on how to have the OpenEmbedded build system
603 generate tarballs for Git repositories and place them in the
604 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
605 directory, see the
606 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
607 variable.
608 </note>
609
610 <para>
611 When fetching a repository, BitBake uses the
612 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
613 variable to determine the specific revision from which to
614 build.
615 </para>
616 </section>
617
618 <section id='source-mirrors'>
619 <title>Source Mirror(s)</title>
620
621 <para>
622 Two kinds of mirrors exist: pre-mirrors and regular mirrors.
623 The <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
624 and
625 <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
626 variables point to these, respectively.
627 BitBake checks pre-mirrors before looking upstream for any
628 source files.
629 Pre-mirrors are appropriate when you have a shared directory
630 that is not a directory defined by the
631 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
632 variable.
633 A Pre-mirror typically points to a shared directory that is
634 local to your organization.
635 </para>
636
637 <para>
638 Regular mirrors can be any site across the Internet that is
639 used as an alternative location for source code should the
640 primary site not be functioning for some reason or another.
641 </para>
642 </section>
643 </section>
644
645 <section id="package-feeds-dev-environment">
646 <title>Package Feeds</title>
647
648 <para>
649 When the OpenEmbedded build system generates an image or an SDK,
650 it gets the packages from a package feed area located in the
651 <link linkend='build-directory'>Build Directory</link>.
652 The
653 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
654 shows this package feeds area in the upper-right corner.
655 </para>
656
657 <para>
658 This section looks a little closer into the package feeds area used
659 by the build system.
660 Here is a more detailed look at the area:
661 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
662 </para>
663
664 <para>
665 Package feeds are an intermediary step in the build process.
666 The OpenEmbedded build system provides classes to generate
667 different package types, and you specify which classes to enable
668 through the
669 <link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>
670 variable.
671 Before placing the packages into package feeds,
672 the build process validates them with generated output quality
673 assurance checks through the
674 <link linkend='ref-classes-insane'><filename>insane</filename></link>
675 class.
676 </para>
677
678 <para>
679 The package feed area resides in the Build Directory.
680 The directory the build system uses to temporarily store packages
681 is determined by a combination of variables and the particular
682 package manager in use.
683 See the "Package Feeds" box in the illustration and note the
684 information to the right of that area.
685 In particular, the following defines where package files are
686 kept:
687 <itemizedlist>
688 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
689 Defined as <filename>tmp/deploy</filename> in the Build
690 Directory.
691 </para></listitem>
692 <listitem><para><filename>DEPLOY_DIR_*</filename>:
693 Depending on the package manager used, the package type
694 sub-folder.
695 Given RPM, IPK, or DEB packaging and tarball creation, the
696 <link linkend='var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></link>,
697 <link linkend='var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></link>,
698 <link linkend='var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></link>,
699 or
700 <link linkend='var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></link>,
701 variables are used, respectively.
702 </para></listitem>
703 <listitem><para><link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link>:
704 Defines architecture-specific sub-folders.
705 For example, packages could exist for the i586 or qemux86
706 architectures.
707 </para></listitem>
708 </itemizedlist>
709 </para>
710
711 <para>
712 BitBake uses the <filename>do_package_write_*</filename> tasks to
713 generate packages and place them into the package holding area (e.g.
714 <filename>do_package_write_ipk</filename> for IPK packages).
715 See the
716 "<link linkend='ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></link>",
717 "<link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link>",
718 "<link linkend='ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></link>",
719 and
720 "<link linkend='ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></link>"
721 sections for additional information.
722 As an example, consider a scenario where an IPK packaging manager
723 is being used and package architecture support for both i586
724 and qemux86 exist.
725 Packages for the i586 architecture are placed in
726 <filename>build/tmp/deploy/ipk/i586</filename>, while packages for
727 the qemux86 architecture are placed in
728 <filename>build/tmp/deploy/ipk/qemux86</filename>.
729 </para>
730 </section>
731
732 <section id='bitbake-dev-environment'>
733 <title>BitBake</title>
734
735 <para>
736 The OpenEmbedded build system uses
737 <link linkend='bitbake-term'>BitBake</link>
738 to produce images.
739 You can see from the
740 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
741 the BitBake area consists of several functional areas.
742 This section takes a closer look at each of those areas.
743 </para>
744
745 <para>
746 Separate documentation exists for the BitBake tool.
747 See the
748 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>
749 for reference material on BitBake.
750 </para>
751
752 <section id='source-fetching-dev-environment'>
753 <title>Source Fetching</title>
754
755 <para>
756 The first stages of building a recipe are to fetch and unpack
757 the source code:
758 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
759 </para>
760
761 <para>
762 The
763 <link linkend='ref-tasks-fetch'><filename>do_fetch</filename></link>
764 and
765 <link linkend='ref-tasks-unpack'><filename>do_unpack</filename></link>
766 tasks fetch the source files and unpack them into the work
767 directory.
768 <note>
769 For every local file (e.g. <filename>file://</filename>)
770 that is part of a recipe's
771 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
772 statement, the OpenEmbedded build system takes a checksum
773 of the file for the recipe and inserts the checksum into
774 the signature for the <filename>do_fetch</filename>.
775 If any local file has been modified, the
776 <filename>do_fetch</filename> task and all tasks that
777 depend on it are re-executed.
778 </note>
779 By default, everything is accomplished in the
780 <link linkend='build-directory'>Build Directory</link>,
781 which has a defined structure.
782 For additional general information on the Build Directory,
783 see the
784 "<link linkend='structure-core-build'><filename>build/</filename></link>"
785 section.
786 </para>
787
788 <para>
789 Unpacked source files are pointed to by the
790 <link linkend='var-S'><filename>S</filename></link> variable.
791 Each recipe has an area in the Build Directory where the
792 unpacked source code resides.
793 The name of that directory for any given recipe is defined from
794 several different variables.
795 You can see the variables that define these directories
796 by looking at the figure:
797 <itemizedlist>
798 <listitem><para><link linkend='var-TMPDIR'><filename>TMPDIR</filename></link> -
799 The base directory where the OpenEmbedded build system
800 performs all its work during the build.
801 </para></listitem>
802 <listitem><para><link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link> -
803 The architecture of the built package or packages.
804 </para></listitem>
805 <listitem><para><link linkend='var-TARGET_OS'><filename>TARGET_OS</filename></link> -
806 The operating system of the target device.
807 </para></listitem>
808 <listitem><para><link linkend='var-PN'><filename>PN</filename></link> -
809 The name of the built package.
810 </para></listitem>
811 <listitem><para><link linkend='var-PV'><filename>PV</filename></link> -
812 The version of the recipe used to build the package.
813 </para></listitem>
814 <listitem><para><link linkend='var-PR'><filename>PR</filename></link> -
815 The revision of the recipe used to build the package.
816 </para></listitem>
817 <listitem><para><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link> -
818 The location within <filename>TMPDIR</filename> where
819 a specific package is built.
820 </para></listitem>
821 <listitem><para><link linkend='var-S'><filename>S</filename></link> -
822 Contains the unpacked source files for a given recipe.
823 </para></listitem>
824 </itemizedlist>
825 </para>
826 </section>
827
828 <section id='patching-dev-environment'>
829 <title>Patching</title>
830
831 <para>
832 Once source code is fetched and unpacked, BitBake locates
833 patch files and applies them to the source files:
834 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
835 </para>
836
837 <para>
838 The
839 <link linkend='ref-tasks-patch'><filename>do_patch</filename></link>
840 task processes recipes by
841 using the
842 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
843 variable to locate applicable patch files, which by default
844 are <filename>*.patch</filename> or
845 <filename>*.diff</filename> files, or any file if
846 "apply=yes" is specified for the file in
847 <filename>SRC_URI</filename>.
848 </para>
849
850 <para>
851 BitBake finds and applies multiple patches for a single recipe
852 in the order in which it finds the patches.
853 Patches are applied to the recipe's source files located in the
854 <link linkend='var-S'><filename>S</filename></link> directory.
855 </para>
856
857 <para>
858 For more information on how the source directories are
859 created, see the
860 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
861 section.
862 </para>
863 </section>
864
865 <section id='configuration-and-compilation-dev-environment'>
866 <title>Configuration and Compilation</title>
867
868 <para>
869 After source code is patched, BitBake executes tasks that
870 configure and compile the source code:
871 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
872 </para>
873
874 <para>
875 This step in the build process consists of three tasks:
876 <itemizedlist>
877 <listitem><para>
878 <emphasis><link linkend='ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></link>:</emphasis>
879 This task sets up the two sysroots in
880 <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}</filename>
881 (i.e. <filename>recipe-sysroot</filename> and
882 <filename>recipe-sysroot-native</filename>) so that
883 the sysroots contain the contents of the
884 <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
885 tasks of the recipes on which the recipe
886 containing the tasks depends.
887 A sysroot exists for both the target and for the native
888 binaries, which run on the host system.
889 </para></listitem>
890 <listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
891 This task configures the source by enabling and
892 disabling any build-time and configuration options for
893 the software being built.
894 Configurations can come from the recipe itself as well
895 as from an inherited class.
896 Additionally, the software itself might configure itself
897 depending on the target for which it is being built.
898 </para>
899
900 <para>The configurations handled by the
901 <link linkend='ref-tasks-configure'><filename>do_configure</filename></link>
902 task are specific
903 to source code configuration for the source code
904 being built by the recipe.</para>
905
906 <para>If you are using the
907 <link linkend='ref-classes-autotools'><filename>autotools</filename></link>
908 class,
909 you can add additional configuration options by using
910 the <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
911 or
912 <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>
913 variables.
914 For information on how this variable works within
915 that class, see the
916 <filename>meta/classes/autotools.bbclass</filename> file.
917 </para></listitem>
918 <listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
919 Once a configuration task has been satisfied, BitBake
920 compiles the source using the
921 <link linkend='ref-tasks-compile'><filename>do_compile</filename></link>
922 task.
923 Compilation occurs in the directory pointed to by the
924 <link linkend='var-B'><filename>B</filename></link>
925 variable.
926 Realize that the <filename>B</filename> directory is, by
927 default, the same as the
928 <link linkend='var-S'><filename>S</filename></link>
929 directory.</para></listitem>
930 <listitem><para><emphasis><filename>do_install</filename>:</emphasis>
931 Once compilation is done, BitBake executes the
932 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
933 task.
934 This task copies files from the <filename>B</filename>
935 directory and places them in a holding area pointed to
936 by the
937 <link linkend='var-D'><filename>D</filename></link>
938 variable.</para></listitem>
939 </itemizedlist>
940 </para>
941 </section>
942
943 <section id='package-splitting-dev-environment'>
944 <title>Package Splitting</title>
945
946 <para>
947 After source code is configured and compiled, the
948 OpenEmbedded build system analyzes
949 the results and splits the output into packages:
950 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
951 </para>
952
953 <para>
954 The
955 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
956 and
957 <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
958 tasks combine to analyze
959 the files found in the
960 <link linkend='var-D'><filename>D</filename></link> directory
961 and split them into subsets based on available packages and
962 files.
963 The analyzing process involves the following as well as other
964 items: splitting out debugging symbols,
965 looking at shared library dependencies between packages,
966 and looking at package relationships.
967 The <filename>do_packagedata</filename> task creates package
968 metadata based on the analysis such that the
969 OpenEmbedded build system can generate the final packages.
970 Working, staged, and intermediate results of the analysis
971 and package splitting process use these areas:
972 <itemizedlist>
973 <listitem><para><link linkend='var-PKGD'><filename>PKGD</filename></link> -
974 The destination directory for packages before they are
975 split.
976 </para></listitem>
977 <listitem><para><link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link> -
978 A shared, global-state directory that holds data
979 generated during the packaging process.
980 </para></listitem>
981 <listitem><para><link linkend='var-PKGDESTWORK'><filename>PKGDESTWORK</filename></link> -
982 A temporary work area used by the
983 <filename>do_package</filename> task.
984 </para></listitem>
985 <listitem><para><link linkend='var-PKGDEST'><filename>PKGDEST</filename></link> -
986 The parent directory for packages after they have
987 been split.
988 </para></listitem>
989 </itemizedlist>
990 The <link linkend='var-FILES'><filename>FILES</filename></link>
991 variable defines the files that go into each package in
992 <link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>.
993 If you want details on how this is accomplished, you can
994 look at the
995 <link linkend='ref-classes-package'><filename>package</filename></link>
996 class.
997 </para>
998
999 <para>
1000 Depending on the type of packages being created (RPM, DEB, or
1001 IPK), the <filename>do_package_write_*</filename> task
1002 creates the actual packages and places them in the
1003 Package Feed area, which is
1004 <filename>${TMPDIR}/deploy</filename>.
1005 You can see the
1006 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
1007 section for more detail on that part of the build process.
1008 <note>
1009 Support for creating feeds directly from the
1010 <filename>deploy/*</filename> directories does not exist.
1011 Creating such feeds usually requires some kind of feed
1012 maintenance mechanism that would upload the new packages
1013 into an official package feed (e.g. the
1014 Ångström distribution).
1015 This functionality is highly distribution-specific
1016 and thus is not provided out of the box.
1017 </note>
1018 </para>
1019 </section>
1020
1021 <section id='image-generation-dev-environment'>
1022 <title>Image Generation</title>
1023
1024 <para>
1025 Once packages are split and stored in the Package Feeds area,
1026 the OpenEmbedded build system uses BitBake to generate the
1027 root filesystem image:
1028 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
1029 </para>
1030
1031 <para>
1032 The image generation process consists of several stages and
1033 depends on several tasks and variables.
1034 The
1035 <link linkend='ref-tasks-rootfs'><filename>do_rootfs</filename></link>
1036 task creates the root filesystem (file and directory structure)
1037 for an image.
1038 This task uses several key variables to help create the list
1039 of packages to actually install:
1040 <itemizedlist>
1041 <listitem><para><link linkend='var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></link>:
1042 Lists out the base set of packages to install from
1043 the Package Feeds area.</para></listitem>
1044 <listitem><para><link linkend='var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></link>:
1045 Specifies packages that should not be installed.
1046 </para></listitem>
1047 <listitem><para><link linkend='var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></link>:
1048 Specifies features to include in the image.
1049 Most of these features map to additional packages for
1050 installation.</para></listitem>
1051 <listitem><para><link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>:
1052 Specifies the package backend to use and consequently
1053 helps determine where to locate packages within the
1054 Package Feeds area.</para></listitem>
1055 <listitem><para><link linkend='var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></link>:
1056 Determines the language(s) for which additional
1057 language support packages are installed.
1058 </para></listitem>
1059 <listitem><para><link linkend='var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></link>:
1060 The final list of packages passed to the package manager
1061 for installation into the image.
1062 </para></listitem>
1063 </itemizedlist>
1064 </para>
1065
1066 <para>
1067 With
1068 <link linkend='var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></link>
1069 pointing to the location of the filesystem under construction and
1070 the <filename>PACKAGE_INSTALL</filename> variable providing the
1071 final list of packages to install, the root file system is
1072 created.
1073 </para>
1074
1075 <para>
1076 Package installation is under control of the package manager
1077 (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of whether or
1078 not package management is enabled for the target.
1079 At the end of the process, if package management is not
1080 enabled for the target, the package manager's data files
1081 are deleted from the root filesystem.
1082 As part of the final stage of package installation, postinstall
1083 scripts that are part of the packages are run.
1084 Any scripts that fail to run
1085 on the build host are run on the target when the target system
1086 is first booted.
1087 If you are using a
1088 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
1089 all the post installation scripts must succeed during the
1090 package installation phase since the root filesystem is
1091 read-only.
1092 </para>
1093
1094 <para>
1095 The final stages of the <filename>do_rootfs</filename> task
1096 handle post processing.
1097 Post processing includes creation of a manifest file and
1098 optimizations.
1099 </para>
1100
1101 <para>
1102 The manifest file (<filename>.manifest</filename>) resides
1103 in the same directory as the root filesystem image.
1104 This file lists out, line-by-line, the installed packages.
1105 The manifest file is useful for the
1106 <link linkend='ref-classes-testimage*'><filename>testimage</filename></link>
1107 class, for example, to determine whether or not to run
1108 specific tests.
1109 See the
1110 <link linkend='var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></link>
1111 variable for additional information.
1112 </para>
1113
1114 <para>
1115 Optimizing processes run across the image include
1116 <filename>mklibs</filename>, <filename>prelink</filename>,
1117 and any other post-processing commands as defined by the
1118 <link linkend='var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></link>
1119 variable.
1120 The <filename>mklibs</filename> process optimizes the size
1121 of the libraries, while the
1122 <filename>prelink</filename> process optimizes the dynamic
1123 linking of shared libraries to reduce start up time of
1124 executables.
1125 </para>
1126
1127 <para>
1128 After the root filesystem is built, processing begins on
1129 the image through the <filename>do_image</filename> task.
1130 The build system runs any pre-processing commands as defined
1131 by the
1132 <link linkend='var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></link>
1133 variable.
1134 This variable specifies a list of functions to call before
1135 the OpenEmbedded build system creates the final image output
1136 files.
1137 </para>
1138
1139 <para>
1140 The <filename>do_image</filename> task dynamically creates
1141 other <filename>do_image_*</filename> tasks as needed, which
1142 include compressing the root filesystem image to reduce the
1143 overall size of the image.
1144 The process turns everything into an image file or a set of
1145 image files.
1146 The formats used for the root filesystem depend on the
1147 <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
1148 variable.
1149 </para>
1150
1151 <para>
1152 The final task involved in image creation is the
1153 <filename>do_image_complete</filename> task.
1154 This task completes the image by applying any image
1155 post processing as defined through the
1156 <link linkend='var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></link>
1157 variable.
1158 The variable specifies a list of functions to call once the
1159 OpenEmbedded build system has created the final image output
1160 files.
1161 </para>
1162
1163 <note>
1164 The entire image generation process is run under Pseudo.
1165 Running under Pseudo ensures that the files in the root
1166 filesystem have correct ownership.
1167 </note>
1168 </section>
1169
1170 <section id='sdk-generation-dev-environment'>
1171 <title>SDK Generation</title>
1172
1173 <para>
1174 The OpenEmbedded build system uses BitBake to generate the
1175 Software Development Kit (SDK) installer script for both the
1176 standard and extensible SDKs:
1177 <imagedata fileref="figures/sdk-generation.png" align="center" />
1178 </para>
1179
1180 <note>
1181 For more information on the cross-development toolchain
1182 generation, see the
1183 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1184 section.
1185 For information on advantages gained when building a
1186 cross-development toolchain using the
1187 <link linkend='ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></link>
1188 task, see the
1189 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
1190 section in the Yocto Project Software Development Kit (SDK)
1191 Developer's Guide.
1192 </note>
1193
1194 <para>
1195 Like image generation, the SDK script process consists of
1196 several stages and depends on many variables.
1197 The <filename>do_populate_sdk</filename> and
1198 <filename>do_populate_sdk_ext</filename> tasks use these
1199 key variables to help create the list of packages to actually
1200 install.
1201 For information on the variables listed in the figure, see the
1202 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1203 section.
1204 </para>
1205
1206 <para>
1207 The <filename>do_populate_sdk</filename> task helps create
1208 the standard SDK and handles two parts: a target part and a
1209 host part.
1210 The target part is the part built for the target hardware and
1211 includes libraries and headers.
1212 The host part is the part of the SDK that runs on the
1213 <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>.
1214 </para>
1215
1216 <para>
1217 The <filename>do_populate_sdk_ext</filename> task helps create
1218 the extensible SDK and handles host and target parts
1219 differently than its counter part does for the standard SDK.
1220 For the extensible SDK, the task encapsulates the build system,
1221 which includes everything needed (host and target) for the SDK.
1222 </para>
1223
1224 <para>
1225 Regardless of the type of SDK being constructed, the
1226 tasks perform some cleanup after which a cross-development
1227 environment setup script and any needed configuration files
1228 are created.
1229 The final output is the Cross-development
1230 toolchain installation script (<filename>.sh</filename> file),
1231 which includes the environment setup script.
1232 </para>
1233 </section>
1234
1235 <section id='stamp-files-and-the-rerunning-of-tasks'>
1236 <title>Stamp Files and the Rerunning of Tasks</title>
1237
1238 <para>
1239 For each task that completes successfully, BitBake writes a
1240 stamp file into the
1241 <link linkend='var-STAMPS_DIR'><filename>STAMPS_DIR</filename></link>
1242 directory.
1243 The beginning of the stamp file's filename is determined by the
1244 <link linkend='var-STAMP'><filename>STAMP</filename></link>
1245 variable, and the end of the name consists of the task's name
1246 and current
1247 <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
1248 <note>
1249 This naming scheme assumes that
1250 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
1251 is "OEBasicHash", which is almost always the case in
1252 current OpenEmbedded.
1253 </note>
1254 To determine if a task needs to be rerun, BitBake checks if a
1255 stamp file with a matching input checksum exists for the task.
1256 If such a stamp file exists, the task's output is assumed to
1257 exist and still be valid.
1258 If the file does not exist, the task is rerun.
1259 <note>
1260 <para>The stamp mechanism is more general than the shared
1261 state (sstate) cache mechanism described in the
1262 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
1263 section.
1264 BitBake avoids rerunning any task that has a valid
1265 stamp file, not just tasks that can be accelerated through
1266 the sstate cache.</para>
1267 <para>However, you should realize that stamp files only
1268 serve as a marker that some work has been done and that
1269 these files do not record task output.
1270 The actual task output would usually be somewhere in
1271 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
1272 (e.g. in some recipe's
1273 <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.)
1274 What the sstate cache mechanism adds is a way to cache task
1275 output that can then be shared between build machines.
1276 </para>
1277 </note>
1278 Since <filename>STAMPS_DIR</filename> is usually a subdirectory
1279 of <filename>TMPDIR</filename>, removing
1280 <filename>TMPDIR</filename> will also remove
1281 <filename>STAMPS_DIR</filename>, which means tasks will
1282 properly be rerun to repopulate <filename>TMPDIR</filename>.
1283 </para>
1284
1285 <para>
1286 If you want some task to always be considered "out of date",
1287 you can mark it with the
1288 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
1289 varflag.
1290 If some other task depends on such a task, then that task will
1291 also always be considered out of date, which might not be what
1292 you want.
1293 </para>
1294
1295 <para>
1296 For details on how to view information about a task's
1297 signature, see the
1298 "<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>"
1299 section.
1300 </para>
1301 </section>
1302
1303 <section id='setscene-tasks-and-shared-state'>
1304 <title>Setscene Tasks and Shared State</title>
1305
1306 <para>
1307 The description of tasks so far assumes that BitBake needs to
1308 build everything and there are no prebuilt objects available.
1309 BitBake does support skipping tasks if prebuilt objects are
1310 available.
1311 These objects are usually made available in the form of a
1312 shared state (sstate) cache.
1313 <note>
1314 For information on variables affecting sstate, see the
1315 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
1316 and
1317 <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
1318 variables.
1319 </note>
1320 </para>
1321
1322 <para>
1323 The idea of a setscene task (i.e
1324 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
1325 is a version of the task where
1326 instead of building something, BitBake can skip to the end
1327 result and simply place a set of files into specific locations
1328 as needed.
1329 In some cases, it makes sense to have a setscene task variant
1330 (e.g. generating package files in the
1331 <filename>do_package_write_*</filename> task).
1332 In other cases, it does not make sense, (e.g. a
1333 <link linkend='ref-tasks-patch'><filename>do_patch</filename></link>
1334 task or
1335 <link linkend='ref-tasks-unpack'><filename>do_unpack</filename></link>
1336 task) since the work involved would be equal to or greater than
1337 the underlying task.
1338 </para>
1339
1340 <para>
1341 In the OpenEmbedded build system, the common tasks that have
1342 setscene variants are <link linkend='ref-tasks-package'><filename>do_package</filename></link>,
1343 <filename>do_package_write_*</filename>,
1344 <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>,
1345 <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>,
1346 and
1347 <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>.
1348 Notice that these are most of the tasks whose output is an
1349 end result.
1350 </para>
1351
1352 <para>
1353 The OpenEmbedded build system has knowledge of the relationship
1354 between these tasks and other tasks that precede them.
1355 For example, if BitBake runs
1356 <filename>do_populate_sysroot_setscene</filename> for
1357 something, there is little point in running any of the
1358 <filename>do_fetch</filename>, <filename>do_unpack</filename>,
1359 <filename>do_patch</filename>,
1360 <filename>do_configure</filename>,
1361 <filename>do_compile</filename>, and
1362 <filename>do_install</filename> tasks.
1363 However, if <filename>do_package</filename> needs to be run,
1364 BitBake would need to run those other tasks.
1365 </para>
1366
1367 <para>
1368 It becomes more complicated if everything can come from an
1369 sstate cache because some objects are simply not required at
1370 all.
1371 For example, you do not need a compiler or native tools, such
1372 as quilt, if there is nothing to compile or patch.
1373 If the <filename>do_package_write_*</filename> packages are
1374 available from sstate, BitBake does not need the
1375 <filename>do_package</filename> task data.
1376 </para>
1377
1378 <para>
1379 To handle all these complexities, BitBake runs in two phases.
1380 The first is the "setscene" stage.
1381 During this stage, BitBake first checks the sstate cache for
1382 any targets it is planning to build.
1383 BitBake does a fast check to see if the object exists rather
1384 than a complete download.
1385 If nothing exists, the second phase, which is the setscene
1386 stage, completes and the main build proceeds.
1387 </para>
1388
1389 <para>
1390 If objects are found in the sstate cache, the OpenEmbedded
1391 build system works backwards from the end targets specified
1392 by the user.
1393 For example, if an image is being built, the OpenEmbedded build
1394 system first looks for the packages needed for that image and
1395 the tools needed to construct an image.
1396 If those are available, the compiler is not needed.
1397 Thus, the compiler is not even downloaded.
1398 If something was found to be unavailable, or the download or
1399 setscene task fails, the OpenEmbedded build system then tries
1400 to install dependencies, such as the compiler, from the cache.
1401 </para>
1402
1403 <para>
1404 The availability of objects in the sstate cache is handled by
1405 the function specified by the
1406 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
1407 variable and returns a list of the objects that are available.
1408 The function specified by the
1409 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
1410 variable is the function that determines whether a given
1411 dependency needs to be followed, and whether for any given
1412 relationship the function needs to be passed.
1413 The function returns a True or False value.
1414 </para>
1415 </section>
1416 </section>
1417
1418 <section id='images-dev-environment'>
1419 <title>Images</title>
1420
1421 <para>
1422 The images produced by the OpenEmbedded build system
1423 are compressed forms of the
1424 root filesystem that are ready to boot on a target device.
1425 You can see from the
1426 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
1427 that BitBake output, in part, consists of images.
1428 This section is going to look more closely at this output:
1429 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1430 </para>
1431
1432 <para>
1433 For a list of example images that the Yocto Project provides,
1434 see the
1435 "<link linkend='ref-images'>Images</link>" chapter.
1436 </para>
1437
1438 <para>
1439 Images are written out to the
1440 <link linkend='build-directory'>Build Directory</link>
1441 inside the <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
1442 folder as shown in the figure.
1443 This folder contains any files expected to be loaded on the
1444 target device.
1445 The
1446 <link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>
1447 variable points to the <filename>deploy</filename> directory,
1448 while the
1449 <link linkend='var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></link>
1450 variable points to the appropriate directory containing images for
1451 the current configuration.
1452 <itemizedlist>
1453 <listitem><para><filename><replaceable>kernel-image</replaceable></filename>:
1454 A kernel binary file.
1455 The <link linkend='var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></link>
1456 variable setting determines the naming scheme for the
1457 kernel image file.
1458 Depending on that variable, the file could begin with
1459 a variety of naming strings.
1460 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1461 directory can contain multiple image files for the
1462 machine.</para></listitem>
1463 <listitem><para><filename><replaceable>root-filesystem-image</replaceable></filename>:
1464 Root filesystems for the target device (e.g.
1465 <filename>*.ext3</filename> or <filename>*.bz2</filename>
1466 files).
1467 The <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
1468 variable setting determines the root filesystem image
1469 type.
1470 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1471 directory can contain multiple root filesystems for the
1472 machine.</para></listitem>
1473 <listitem><para><filename><replaceable>kernel-modules</replaceable></filename>:
1474 Tarballs that contain all the modules built for the kernel.
1475 Kernel module tarballs exist for legacy purposes and
1476 can be suppressed by setting the
1477 <link linkend='var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></link>
1478 variable to "0".
1479 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1480 directory can contain multiple kernel module tarballs
1481 for the machine.</para></listitem>
1482 <listitem><para><filename><replaceable>bootloaders</replaceable></filename>:
1483 Bootloaders supporting the image, if applicable to the
1484 target machine.
1485 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1486 directory can contain multiple bootloaders for the
1487 machine.</para></listitem>
1488 <listitem><para><filename><replaceable>symlinks</replaceable></filename>:
1489 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1490 folder contains
1491 a symbolic link that points to the most recently built file
1492 for each machine.
1493 These links might be useful for external scripts that
1494 need to obtain the latest version of each file.
1495 </para></listitem>
1496 </itemizedlist>
1497 </para>
1498 </section>
1499
1500 <section id='sdk-dev-environment'>
1501 <title>Application Development SDK</title>
1502
1503 <para>
1504 In the
1505 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
1506 the output labeled "Application Development SDK" represents an
1507 SDK.
1508 The SDK generation process differs depending on whether you build
1509 a standard SDK
1510 (e.g. <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>)
1511 or an extensible SDK
1512 (e.g. <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>).
1513 This section is going to take a closer look at this output:
1514 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
1515 </para>
1516
1517 <para>
1518 The specific form of this output is a self-extracting
1519 SDK installer (<filename>*.sh</filename>) that, when run,
1520 installs the SDK, which consists of a cross-development
1521 toolchain, a set of libraries and headers, and an SDK
1522 environment setup script.
1523 Running this installer essentially sets up your
1524 cross-development environment.
1525 You can think of the cross-toolchain as the "host"
1526 part because it runs on the SDK machine.
1527 You can think of the libraries and headers as the "target"
1528 part because they are built for the target hardware.
1529 The environment setup script is added so that you can initialize
1530 the environment before using the tools.
1531 </para>
1532
1533 <note>
1534 <para>
1535 The Yocto Project supports several methods by which you can
1536 set up this cross-development environment.
1537 These methods include downloading pre-built SDK installers
1538 or building and installing your own SDK installer.
1539 </para>
1540
1541 <para>
1542 For background information on cross-development toolchains
1543 in the Yocto Project development environment, see the
1544 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1545 section.
1546 For information on setting up a cross-development
1547 environment, see the
1548 <ulink url='&YOCTO_DOCS_SDK_URL;#sdk-manual'>Yocto Project Software Development Kit (SDK) Developer's Guide</ulink>.
1549 </para>
1550 </note>
1551
1552 <para>
1553 Once built, the SDK installers are written out to the
1554 <filename>deploy/sdk</filename> folder inside the
1555 <link linkend='build-directory'>Build Directory</link>
1556 as shown in the figure at the beginning of this section.
1557 Depending on the type of SDK, several variables exist that help
1558 configure these files.
1559 The following list shows the variables associated with a standard
1560 SDK:
1561 <itemizedlist>
1562 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
1563 Points to the <filename>deploy</filename>
1564 directory.</para></listitem>
1565 <listitem><para><link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>:
1566 Specifies the architecture of the machine
1567 on which the cross-development tools are run to
1568 create packages for the target hardware.
1569 </para></listitem>
1570 <listitem><para><link linkend='var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></link>:
1571 Lists the features to include in the "target" part
1572 of the SDK.
1573 </para></listitem>
1574 <listitem><para><link linkend='var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></link>:
1575 Lists packages that make up the host
1576 part of the SDK (i.e. the part that runs on
1577 the <filename>SDKMACHINE</filename>).
1578 When you use
1579 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
1580 to create the SDK, a set of default packages
1581 apply.
1582 This variable allows you to add more packages.
1583 </para></listitem>
1584 <listitem><para><link linkend='var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></link>:
1585 Lists packages that make up the target part
1586 of the SDK (i.e. the part built for the
1587 target hardware).
1588 </para></listitem>
1589 <listitem><para><link linkend='var-SDKPATH'><filename>SDKPATH</filename></link>:
1590 Defines the default SDK installation path offered by the
1591 installation script.
1592 </para></listitem>
1593 </itemizedlist>
1594 This next list, shows the variables associated with an extensible
1595 SDK:
1596 <itemizedlist>
1597 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
1598 Points to the <filename>deploy</filename> directory.
1599 </para></listitem>
1600 <listitem><para><link linkend='var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></link>:
1601 Controls whether or not shared state artifacts are copied
1602 into the extensible SDK.
1603 By default, all required shared state artifacts are copied
1604 into the SDK.
1605 </para></listitem>
1606 <listitem><para><link linkend='var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></link>:
1607 Specifies whether or not packagedata will be included in
1608 the extensible SDK for all recipes in the "world" target.
1609 </para></listitem>
1610 <listitem><para><link linkend='var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></link>:
1611 Specifies whether or not the toolchain will be included
1612 when building the extensible SDK.
1613 </para></listitem>
1614 <listitem><para><link linkend='var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></link>:
1615 A list of variables allowed through from the build system
1616 configuration into the extensible SDK configuration.
1617 </para></listitem>
1618 <listitem><para><link linkend='var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></link>:
1619 A list of variables not allowed through from the build
1620 system configuration into the extensible SDK configuration.
1621 </para></listitem>
1622 <listitem><para><link linkend='var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></link>:
1623 A list of classes to remove from the
1624 <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
1625 value globally within the extensible SDK configuration.
1626 </para></listitem>
1627 </itemizedlist>
1628 </para>
1629 </section>
1630</section>
1631
1632</chapter>
1633<!--
1634vim: expandtab tw=80 ts=4
1635-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-02-13 05:58:05 +0000