summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual/closer-look.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/ref-manual/closer-look.xml')
-rw-r--r--documentation/ref-manual/closer-look.xml1224
1 files changed, 1224 insertions, 0 deletions
diff --git a/documentation/ref-manual/closer-look.xml b/documentation/ref-manual/closer-look.xml
new file mode 100644
index 0000000000..25c03e0a52
--- /dev/null
+++ b/documentation/ref-manual/closer-look.xml
@@ -0,0 +1,1224 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='closer-look'>
6<title>A Closer Look at the Yocto Project Development Environment</title>
7
8 <para>
9 This chapter takes a more detailed look at the Yocto Project
10 development environment.
11 The following diagram represents the development environment at a
12 high level.
13 The remainder of this chapter expands on the fundamental input, output,
14 process, and
15 <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>) blocks
16 in the Yocto Project development environment.
17 </para>
18
19 <para id='general-yocto-environment-figure'>
20 <imagedata fileref="figures/yocto-environment-ref.png" align="center" width="8in" depth="4.25in" />
21 </para>
22
23 <para>
24 The generalized Yocto Project Development Environment consists of
25 several functional areas:
26 <itemizedlist>
27 <listitem><para><emphasis>User Configuration:</emphasis>
28 Metadata you can use to control the build process.
29 </para></listitem>
30 <listitem><para><emphasis>Metadata Layers:</emphasis>
31 Various layers that provide software, machine, and
32 distro Metadata.</para></listitem>
33 <listitem><para><emphasis>Source Files:</emphasis>
34 Upstream releases, local projects, and SCMs.</para></listitem>
35 <listitem><para><emphasis>Build System:</emphasis>
36 Processes under the control of BitBake.
37 This block expands on how BitBake fetches source, applies
38 patches, completes compilation, analyzes output for package
39 generation, creates and tests packages, generates images, and
40 generates cross-development tools.</para></listitem>
41 <listitem><para><emphasis>Package Feeds:</emphasis>
42 Directories containing output packages (rpm, deb or ipk),
43 which are subsequently used in the construction of an image or
44 SDK, produced by the build system.
45 These feeds can also be copied and shared using a web server or
46 other means to facilitate extending or updating existing
47 images on devices at runtime if runtime package management is
48 enabled.</para></listitem>
49 <listitem><para><emphasis>Images:</emphasis>
50 Images produced by the development process.
51 Where do they go?
52 Can you mess with them (i.e. freely delete them or move them?).
53 </para></listitem>
54 <listitem><para><emphasis>Application Development SDK:</emphasis>
55 Cross-development tools that are produced along with an image
56 or separately with BitBake.</para></listitem>
57 </itemizedlist>
58 </para>
59
60 <section id="user-configuration">
61 <title>User Configuration</title>
62
63 <para>
64 User configuration helps define the build.
65 Through user configuration, you can tell BitBake the
66 target architecture for which you are building the image,
67 where to store downloaded source, and other build properties.
68 </para>
69
70 <para>
71 The following figure shows an expanded representation of the
72 "User Configuration" box of the
73 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
74 </para>
75
76 <para>
77 <imagedata fileref="figures/user-configuration.png" align="center" width="5.5in" depth="3.5in" />
78 </para>
79
80 <para>
81 BitBake needs some basic configuration files in order to complete
82 a build.
83 These files are <filename>*.conf</filename> files.
84 The minimally necessary ones reside as example files in the
85 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
86 For simplicity, this section refers to the Source Directory as
87 the "Poky Directory."
88 </para>
89
90 <para>
91 When you clone the <filename>poky</filename> Git repository or you
92 download and unpack a Yocto Project release, you can set up the
93 Source Directory to be named anything you want.
94 For this discussion, the cloned repository uses the default
95 name <filename>poky</filename>.
96 <note>
97 The Poky repository is primarily an aggregation of existing
98 repositories.
99 It is not a canonical upstream source.
100 </note>
101 </para>
102
103 <para>
104 The <filename>meta-yocto</filename> layer inside Poky contains
105 a <filename>conf</filename> directory that has example
106 configuration files.
107 These example files are used as a basis for creating actual
108 configuration files when you source the build environment
109 script
110 (i.e.
111 <link linkend='structure-core-script'><filename>&OE_INIT_FILE;</filename></link>
112 or
113 <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>).
114 </para>
115
116 <para>
117 Sourcing the build environment script creates a
118 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
119 if one does not already exist.
120 BitBake uses the Build Directory for all its work during builds.
121 The Build Directory has a <filename>conf</filename> directory that
122 contains default versions of your <filename>local.conf</filename>
123 and <filename>bblayers.conf</filename> configuration files.
124 These default configuration files are created only if versions
125 do not already exist in the Build Directory at the time you
126 source the build environment setup script.
127 </para>
128
129 <para>
130 Because the Poky repository is fundamentally an aggregation of
131 existing repositories, some users might be familiar with running
132 the <filename>&OE_INIT_FILE;</filename> or
133 <filename>oe-init-build-env-memres</filename> script in the context
134 of separate OpenEmbedded-Core and BitBake repositories rather than a
135 single Poky repository.
136 This discussion assumes the script is executed from within a cloned
137 or unpacked version of Poky.
138 </para>
139
140 <para>
141 Depending on where the script is sourced, different sub-scripts
142 are called to set up the Build Directory (Yocto or OpenEmbedded).
143 Specifically, the script
144 <filename>scripts/oe-setup-builddir</filename> inside the
145 poky directory sets up the Build Directory and seeds the directory
146 (if necessary) with configuration files appropriate for the
147 Yocto Project development environment.
148 <note>
149 The <filename>scripts/oe-setup-builddir</filename> script
150 uses the <filename>$TEMPLATECONF</filename> variable to
151 determine which sample configuration files to locate.
152 </note>
153 </para>
154
155 <para>
156 The <filename>local.conf</filename> file provides many
157 basic variables that define a build environment.
158 Here is a list of a few.
159 To see the default configurations in a <filename>local.conf</filename>
160 file created by the build environment script, see the
161 <filename>local.conf.sample</filename> in the
162 <filename>meta-yocto</filename> layer:
163 <itemizedlist>
164 <listitem><para><emphasis>Parallelism Options:</emphasis>
165 Controlled by the
166 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
167 and
168 <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></link>
169 variables.</para></listitem>
170 <listitem><para><emphasis>Target Machine Selection:</emphasis>
171 Controlled by the
172 <link linkend='var-MACHINE'><filename>MACHINE</filename></link>
173 variable.</para></listitem>
174 <listitem><para><emphasis>Download Directory:</emphasis>
175 Controlled by the
176 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
177 variable.</para></listitem>
178 <listitem><para><emphasis>Shared State Directory:</emphasis>
179 Controlled by the
180 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
181 variable.</para></listitem>
182 <listitem><para><emphasis>Build Output:</emphasis>
183 Controlled by the
184 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
185 variable.</para></listitem>
186 </itemizedlist>
187 <note>
188 Configurations set in the <filename>conf/local.conf</filename>
189 file can also be set in the
190 <filename>conf/site.conf</filename> and
191 <filename>conf/auto.conf</filename> configuration files.
192 </note>
193 </para>
194
195 <para>
196 The <filename>bblayers.conf</filename> file tells BitBake what
197 layers you want considered during the build.
198 By default, the layers listed in this file include layers
199 minimally needed by the build system.
200 However, you must manually add any custom layers you have created.
201 You can find more information on working with the
202 <filename>bblayers.conf</filename> file in the
203 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
204 section in the Yocto Project Development Manual.
205 </para>
206
207 <para>
208 The files <filename>site.conf</filename> and
209 <filename>auto.conf</filename> are not created by the environment
210 initialization script.
211 If you want these configuration files, you must create them
212 yourself:
213 <itemizedlist>
214 <listitem><para><emphasis><filename>site.conf</filename>:</emphasis>
215 You can use the <filename>conf/site.conf</filename>
216 configuration file to configure multiple build directories.
217 For example, suppose you had several build environments and
218 they shared some common features.
219 You can set these default build properties here.
220 A good example is perhaps the level of parallelism you want
221 to use through the
222 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
223 and
224 <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></link>
225 variables.</para>
226 <para>One useful scenario for using the
227 <filename>conf/site.conf</filename> file is to extend your
228 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
229 variable to include the path to a
230 <filename>conf/site.conf</filename>.
231 Then, when BitBake looks for Metadata using
232 <filename>BBPATH</filename>, it finds the
233 <filename>conf/site.conf</filename> file and applies your
234 common configurations found in the file.
235 To override configurations in a particular build directory,
236 alter the similar configurations within that build
237 directory's <filename>conf/local.conf</filename> file.
238 </para></listitem>
239 <listitem><para><emphasis><filename>auto.conf</filename>:</emphasis>
240 This file is not hand-created.
241 Rather, the file is usually created and written to by
242 an autobuilder.
243 The settings put into the file are typically the same as
244 you would find in the <filename>conf/local.conf</filename>
245 or the <filename>conf/site.conf</filename> files.
246 </para></listitem>
247 </itemizedlist>
248 </para>
249
250 <para>
251 You can edit all configuration files to further define
252 any particular build environment.
253 This process is represented by the "User Configuration Edits"
254 box in the figure.
255 </para>
256
257 <para>
258 When you launch your build with the
259 <filename>bitbake &lt;target&gt;</filename> command, BitBake
260 sorts out the configurations to ultimately define your build
261 environment.
262 </para>
263 </section>
264
265 <section id="metadata-machine-configuration-and-policy-configuration">
266 <title>Metadata, Machine Configuration, and Policy Configuration</title>
267
268 <para>
269 The previous section described the user configurations that
270 define the BitBake's global behavior.
271 This section takes a closer look at the layers the build system
272 uses to further control the build.
273 These layers provide Metadata for the software, machine, and
274 policy.
275 </para>
276
277 <para>
278 In general, three types of layer input exist:
279 <itemizedlist>
280 <listitem><para><emphasis>Policy Configuration:</emphasis>
281 Distribution Layers provide top-level or general
282 policies for the image or SDK being built.
283 For example, this layer would dictate whether BitBake
284 produces RPM or IPK packages.</para></listitem>
285 <listitem><para><emphasis>Machine Configuration:</emphasis>
286 Board Support Package (BSP) layers provide machine
287 configurations.
288 This type of information is specific to a particular
289 target architecture.</para></listitem>
290 <listitem><para><emphasis>Metadata:</emphasis>
291 Software layers contain user-supplied recipe files,
292 patches, and append files.
293 </para></listitem>
294 </itemizedlist>
295 </para>
296
297 <para>
298 The following figure shows an expanded representation of the
299 Metadata, Machine Configuration, and Policy Configuration input
300 (layers) boxes of the
301 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
302 </para>
303
304 <para>
305 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="7.5in" />
306 </para>
307
308 <para>
309 In general, all layers have a similar structure.
310 They all contain a licensing file
311 (e.g. <filename>COPYING</filename>) if the layer is to be
312 distributed, a <filename>README</filename> file as good practice
313 and especially if the layer is to be distributed, a
314 configuration directory, and recipe directories.
315 </para>
316
317 <para>
318 The Yocto Project has many layers that can be used.
319 You can see a web-interface listing of them on the
320 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
321 page.
322 The layers are shown at the bottom categorized under
323 "Yocto Metadata Layers."
324 These layers are fundamentally a subset of the
325 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Metadata Index</ulink>,
326 which lists all layers provided by the OpenEmbedded community.
327 <note>
328 Layers exist in the Yocto Project Source Repositories that
329 cannot be found in the OpenEmbedded Metadata Index.
330 These layers are either deprecated or experimental in nature.
331 </note>
332 </para>
333
334 <para>
335 BitBake uses the <filename>conf/bblayers.conf</filename> file,
336 which is part of the user configuration, to find what layers it
337 should be using as part of the build.
338 </para>
339
340 <para>
341 For more information on layers, see the
342 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
343 section in the Yocto Project Development Manual.
344 </para>
345
346 <section id="distro-layer">
347 <title>Distro Layer</title>
348
349 <para>
350 The distribution layer provides policy configurations for your
351 distribution.
352 Best practices dictate that you isolate these types of
353 configurations into their own layer.
354 Settings you provide in
355 <filename>conf/&lt;distro&gt;.conf</filename> override similar
356 settings that BitBake finds in your
357 <filename>conf/local.conf</filename> file in the Build
358 Directory.
359 </para>
360
361 <para>
362 The following list provides some explanation and references
363 for what you typically find in the distribution layer:
364 <itemizedlist>
365 <listitem><para><emphasis>classes:</emphasis>
366 Class files (<filename>.bbclass</filename>) holds
367 common functionality that can be shared among
368 recipes in the distribution.
369 When your recipes inherit a class, they take on the
370 settings and functions for that class.
371 You can read more about class files in the
372 "<link linkend='ref-classes'>Classes</link>" section.
373 </para></listitem>
374 <listitem><para><emphasis>conf:</emphasis>
375 This area holds configuration files for the
376 layer (<filename>conf/layer.conf</filename>),
377 the distribution
378 (<filename>conf/distro/&lt;distro&gt;.conf</filename>),
379 and any distribution-wide include files.
380 </para></listitem>
381 <listitem><para><emphasis>recipes-*:</emphasis>
382 Recipes and append files that affect common
383 functionality across the distribution.
384 This area could include recipes and append files to
385 to add distribution-specific configuration,
386 initialization scripts, custom image recipes,
387 and so forth.</para></listitem>
388 </itemizedlist>
389 </para>
390 </section>
391
392 <section id="bsp-layer">
393 <title>BSP Layer</title>
394
395 <para>
396 The BSP Layer provides machine configurations.
397 Everything in this layer is specific to the machine for which
398 you are building the image or the SDK.
399 A common structure or form is defined for BSP layers.
400 You can learn more about this structure in the
401 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
402 <note>
403 In order for a BSP layer to be considered compliant with the
404 Yocto Project, it must meet some structural requirements.
405 </note>
406 </para>
407
408 <para>
409 The BSP Layer's configuration directory contains
410 configuration files for the machine
411 (<filename>conf/machine/&lt;machine&gt;.conf</filename>) and,
412 of course, the layer (<filename>conf/layer.conf</filename>).
413 </para>
414
415 <para>
416 The remainder of the layer is dedicated to specific recipes
417 by function: <filename>recipes-bsp</filename>,
418 <filename>recipes-core</filename>,
419 <filename>recipes-graphics</filename>, and
420 <filename>recipes-kernel</filename>.
421 Metadata can exist for multiple formfactors, graphics
422 support systems, and so forth.
423 <note>
424 While the figure shows several <filename>recipe-*</filename>
425 directories, not all these directories appear in all
426 BSP layers.
427 </note>
428 </para>
429 </section>
430
431 <section id="software-layer">
432 <title>Software Layer</title>
433
434 <para>
435 The software layer provides the Metadata for additional
436 software packages used during the build.
437 This layer does not include Metadata that is specific to the
438 distribution or the machine, which are found in their
439 respective layers.
440 </para>
441
442 <para>
443 This layer contains any new recipes that your project needs
444 in the form of recipe files.
445 </para>
446 </section>
447 </section>
448
449 <section id="sources-dev-environment">
450 <title>Sources</title>
451
452 <para>
453 In order for the OpenEmbedded build system to create an image or
454 any target, it must be able to access source files.
455 The
456 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
457 represents source files using the "Upstream Project Releases",
458 "Local Projects", and "SCMs (optional)" boxes.
459 The figure represents mirrors, which also play a role in locating
460 source files, with the "Source Mirror(s)" box.
461 </para>
462
463 <para>
464 The method by which source files are ultimately organized is
465 a function of the project.
466 For example, for released software, projects tend to use tarballs
467 or other archived files that can capture the state of a release
468 guaranteeing that it is statically represented.
469 On the other hand, for a project that is more dynamic or
470 experimental in nature, a project might keep source files in a
471 repository controlled by a Source Control Manager (SCM) such as
472 Git.
473 Pulling source from a repository allows you to control
474 the point in the repository (the revision) from which you want to
475 build software.
476 Finally, a combination of the two might exist, which would give the
477 consumer a choice when deciding where to get source files.
478 </para>
479
480 <para>
481 BitBake uses the
482 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
483 variable to point to source files regardless of their location.
484 Each recipe must have a <filename>SRC_URI</filename> variable
485 that points to the source.
486 </para>
487
488 <para>
489 Another area that plays a significant role in where source files
490 comes from is pointed to by the
491 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
492 variable.
493 This area is a cache that can hold previously downloaded source.
494 You can also instruct the OpenEmbedded build system to create
495 tarballs from Git repositories, which is not the default behavior,
496 and store them in the <filename>DL_DIR</filename> by using the
497 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
498 variable.
499 </para>
500
501 <para>
502 Judicious use of a <filename>DL_DIR</filename> directory can
503 save the build system a trip across the Internet when looking
504 for files.
505 A good method for using a download directory is to have
506 <filename>DL_DIR</filename> point to an area outside of your
507 Build Directory.
508 Doing so allows you to safely delete the Build Directory
509 if needed without fear of removing any downloaded source file.
510 </para>
511
512 <para>
513 The remainder of this section provides a deeper look into the
514 source files and the mirrors.
515 Here is a more detailed look at the source file area of the
516 base figure:
517 <imagedata fileref="figures/source-input.png" align="center" width="7in" depth="7.5in" />
518 </para>
519
520 <section id='upstream-project-releases'>
521 <title>Upstream Project Releases</title>
522
523 <para>
524 Upstream project releases exist anywhere in the form of an
525 archived file (e.g. tarball or zip file).
526 These files correspond to individual recipes.
527 For example, the figure uses specific releases each for
528 BusyBox, Qt, and Dbus.
529 An archive file can be for any released product that can be
530 built using a recipe.
531 </para>
532 </section>
533
534 <section id='local-projects'>
535 <title>Local Projects</title>
536
537 <para>
538 Local projects are custom bits of software the user provides.
539 These bits reside somewhere local to a project - perhaps
540 a directory into which the user checks in items (e.g.
541 a local directory containing a development source tree
542 used by the group).
543 </para>
544
545 <para>
546 The canonical method through which to include a local project
547 is to use the
548 <link linkend='ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></link>
549 class to include local project.
550 You use either the <filename>local.conf</filename> or a
551 recipe's append file to override or set the
552 recipe to point to the local directory on your disk to pull
553 in the whole source tree.
554 </para>
555
556 <para>
557 For information on how to use the
558 <filename>externalsrc.bbclass</filename>, see the
559 "<link linkend='ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></link>"
560 section.
561 </para>
562 </section>
563
564 <section id='scms'>
565 <title>Source Control Managers (Optional)</title>
566
567 <para>
568 Another place the build system can get source files from is
569 through an SCM such as Git or Subversion.
570 In this case, a repository is cloned or checked out.
571 The <filename>do_fetch</filename> task inside BitBake uses
572 the <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
573 variable and the argument's prefix to determine the correct
574 fetcher module.
575 </para>
576
577 <note>
578 For information on how to have the OpenEmbedded build system
579 generate tarballs for Git repositories and place them in the
580 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
581 directory, see the
582 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
583 variable.
584 </note>
585
586 <para>
587 When fetching a repository, BitBake uses the
588 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
589 variable to determine the specific revision from which to
590 build.
591 </para>
592 </section>
593
594 <section id='source-mirrors'>
595 <title>Source Mirror(s)</title>
596
597 <para>
598 Two kinds of mirrors exist: pre-mirrors and regular mirrors.
599 The <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
600 and
601 <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
602 variables point to these, respectively.
603 BitBake checks pre-mirrors before looking upstream for any
604 source files.
605 Pre-mirrors are appropriate when you have a shared directory
606 that is not a directory defined by the
607 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
608 variable.
609 A Pre-mirror typically points to a shared directory that is
610 local to your organization.
611 </para>
612
613 <para>
614 Regular mirrors can be any site across the Internet that is
615 used as an alternative location for source code should the
616 primary site not be functioning for some reason or another.
617 </para>
618 </section>
619 </section>
620
621 <section id="package-feeds-dev-environment">
622 <title>Package Feeds</title>
623
624 <para>
625 When the OpenEmbedded build system generates an image or an SDK,
626 it gets the packages from a package feed area located in the
627 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
628 The
629 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
630 shows this package feeds area in the upper-right corner.
631 </para>
632
633 <para>
634 This section looks a little closer into the package feeds area used
635 by the build system.
636 Here is a more detailed look at the area:
637 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
638 </para>
639
640 <para>
641 Package feeds are an intermediary step in the build process.
642 BitBake generates packages whose type is defined by the
643 <link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>
644 variable.
645 Before placing the packages into package feeds,
646 the build process validates them with generated output quality
647 assurance checks through the
648 <link linkend='ref-classes-insane'><filename>insane.bbclass</filename></link>
649 class.
650 </para>
651
652 <para>
653 The package feed area resides in
654 <filename>tmp/deploy</filename> of the Build Directory.
655 Folders are created that correspond to the package type
656 (IPK, DEB, or RPM) created.
657 Further organization is derived through the value of the
658 <link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link>
659 variable for each package.
660 For example, packages can exist for the i586 or qemux86
661 architectures.
662 The package files themselves reside within the appropriate
663 architecture folder.
664 </para>
665
666 <para>
667 BitBake uses the <filename>do_package_write_*</filename> task to
668 place generated packages into the package holding area (e.g.
669 <filename>do_package_write_ipk</filename> for IPK packages).
670 </para>
671 </section>
672
673 <section id='bitbake-dev-environment'>
674 <title>BitBake</title>
675
676 <para>
677 The OpenEmbedded build system uses BitBake to produce images.
678 You can see from the
679 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
680 the BitBake area consists of several functional areas.
681 This section takes a closer look at each of those areas.
682 </para>
683
684 <section id='source-fetching-dev-environment'>
685 <title>Source Fetching</title>
686
687 <para>
688 The first stages of building a recipe are to fetch and unpack
689 the source code:
690 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
691 </para>
692
693 <para>
694 The <filename>do_fetch</filename> and
695 <filename>do_unpack</filename> tasks fetch the source files
696 and unpack them into a working directory.
697 By default, everything is accomplished in the
698 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>,
699 which has a defined structure.
700 For additional general information on the Build Directory,
701 see the
702 "<link linkend='structure-core-build'><filename>build/</filename></link>"
703 section.
704 </para>
705
706 <para>
707 Unpacked source source files are pointed to by the
708 <link linkend='var-S'><filename>S</filename></link> variable.
709 Each recipe has an area in the Build Directory where the
710 unpacked source code resides.
711 The name of directory for any given recipe is defined from
712 several different variables.
713 You can see the variables that define these directories
714 by looking at the figure:
715 <itemizedlist>
716 <listitem><para><link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
717 </para></listitem>
718 <listitem><para><link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link>
719 </para></listitem>
720 <listitem><para><link linkend='var-TARGET_OS'><filename>TARGET_OS</filename></link>
721 </para></listitem>
722 <listitem><para><link linkend='var-PN'><filename>PN</filename></link>
723 </para></listitem>
724 <listitem><para><link linkend='var-PV'><filename>PV</filename></link>
725 </para></listitem>
726 <listitem><para><link linkend='var-PR'><filename>PR</filename></link>
727 </para></listitem>
728 <listitem><para><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
729 </para></listitem>
730 <listitem><para><link linkend='var-S'><filename>S</filename></link>
731 </para></listitem>
732 </itemizedlist>
733 </para>
734
735 <para>
736 Briefly, the <filename>S</filename> directory contains the
737 unpacked source files for a recipe.
738 The <filename>WORKDIR</filename> directory is where all the
739 building goes on for a given recipe.
740 </para>
741 </section>
742
743 <section id='patching-dev-environment'>
744 <title>Patching</title>
745
746 <para>
747 Once source code is fetched and unpacked, BitBake locates
748 patch files and applies them to the source files:
749 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
750 </para>
751
752 <para>
753 The <filename>do_patch</filename> task processes recipes by
754 using the
755 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
756 variable to locate applicable patch files, which by default
757 are <filename>*.patch</filename> or
758 <filename>*.diff</filename> files, or any file if
759 "apply=yes" is specified for the file in
760 <filename>SRC_URI</filename>.
761 </para>
762
763 <para>
764 BitBake finds and applies multiple patches for a single recipe
765 in the order in which it finds the patches.
766 Patches are applied to the recipe's source files located in the
767 <link linkend='var-S'><filename>S</filename></link> directory.
768 </para>
769
770 <para>
771 For more information on how the source directories are
772 created, see the
773 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
774 section.
775 </para>
776 </section>
777
778 <section id='configuration-and-compilation-dev-environment'>
779 <title>Configuration and Compilation</title>
780
781 <para>
782 After source code is patched, BitBake executes tasks that
783 configure and compile the source code:
784 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
785 </para>
786
787 <para>
788 This step in the build process consists of three tasks:
789 <itemizedlist>
790 <listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
791 This task configures the source by enabling and
792 disabling any build-time and configuration options for
793 the software being built.
794 Configurations can come from the recipe itself as well
795 as from an inherited class.
796 Additionally, the software itself might configure itself
797 depending on the target for which it is being built.
798 </para>
799
800 <para>The configurations handled by the
801 <filename>do_configure</filename> task are specific
802 to source code configuration for the source code
803 being built by the recipe.</para>
804
805 <para>If you are using
806 <link linkend='ref-classes-autotools'><filename>autotools.bbclass</filename></link>,
807 you can add additional configuration options by using
808 the <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
809 variable.
810 For information on how this variable works within
811 that class, see the
812 <filename>meta/classes/autotools.bbclass</filename>.
813 </para></listitem>
814 <listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
815 Once a configuration task has been satisfied, BitBake
816 compiles the source using the
817 <filename>do_compile</filename> task.
818 Compilation occurs in the directory pointed to by the
819 <link linkend='var-B'><filename>B</filename></link>
820 variable.
821 Realize that the <filename>B</filename> directory, by
822 default, is the same as the
823 <link linkend='var-S'><filename>S</filename></link>
824 directory.</para></listitem>
825 <listitem><para><emphasis><filename>do_install</filename>:</emphasis>
826 Once compilation is done, BitBake executes the
827 <filename>do_install</filename> task.
828 This task copies files from the <filename>B</filename>
829 directory and places them in a holding area pointed to
830 by the
831 <link linkend='var-D'><filename>D</filename></link>
832 variable.</para></listitem>
833 </itemizedlist>
834 </para>
835 </section>
836
837 <section id='package-splitting-dev-environment'>
838 <title>Package Splitting</title>
839
840 <para>
841 After source code is configured and compiled, the
842 OpenEmbedded build system analyzes
843 the results and splits the output into packages:
844 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
845 </para>
846
847 <para>
848 The <filename>do_package</filename> and
849 <filename>do_packagedata</filename> tasks combine to analyze
850 the files found in the
851 <link linkend='var-D'><filename>D</filename></link> directory
852 and split them into subsets based on available packages and
853 files.
854 The analyzing process involves the following as well as other
855 items: splitting out debugging symbols,
856 looking at shared library dependencies between packages,
857 and looking at package relationships.
858 The <filename>do_packagedata</filename> task creates package
859 metadata based on the analysis such that the
860 OpenEmbedded build system can generate the final packages.
861 Working, staged, and intermediate results of the analysis
862 and package splitting process use these areas:
863 <itemizedlist>
864 <listitem><para><link linkend='var-PKGD'><filename>PKGD</filename></link>
865 </para></listitem>
866 <listitem><para><link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
867 </para></listitem>
868 <listitem><para><link linkend='var-PKGDESTWORK'><filename>PKGDESTWORK</filename></link>
869 </para></listitem>
870 <listitem><para><link linkend='var-PKGDEST'><filename>PKGDEST</filename></link>
871 </para></listitem>
872 </itemizedlist>
873 The <link linkend='var-FILES'><filename>FILES</filename></link>
874 variable defines the files that go into each package in
875 <link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>.
876 If you want details on how this is accomplished, you can
877 look at
878 <link linkend='ref-classes-package'><filename>package.bbclass</filename></link>.
879 </para>
880
881 <para>
882 Depending on the type of packages being created (RPM, DEB, or
883 IPK), the <filename>do_package_write_*</filename> task
884 creates the actual packages and places them in the
885 Package Feed area, which is
886 <filename>${TMPDIR}/deploy</filename>.
887 You can see the
888 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
889 section for more detail on that part of the build process.
890 <note>
891 Support for creating feeds directly from the
892 <filename>deploy/*</filename> directories does not exist.
893 Creating such feeds usually requires some kind of feed
894 maintenance mechanism that would upload the new packages
895 into an official package feed (e.g. the
896 Ångström distribution).
897 This functionality is highly distribution-specific
898 and thus is not provided out of the box.
899 </note>
900 </para>
901 </section>
902
903 <section id='image-generation-dev-environment'>
904 <title>Image Generation</title>
905
906 <para>
907 Once packages are split and stored in the Package Feeds area,
908 the OpenEmbedded build system uses BitBake to generate the
909 root filesystem image:
910 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
911 </para>
912
913 <para>
914 The image generation process consists of several stages and
915 depends on many variables.
916 The <filename>do_rootfs</filename> task uses these key variables
917 to help create the list of packages to actually install:
918 <itemizedlist>
919 <listitem><para><link linkend='var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></link>:
920 Lists out the base set of packages to install from
921 the Package Feeds area.</para></listitem>
922 <listitem><para><link linkend='var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></link>:
923 Specifies packages that should not be installed.
924 </para></listitem>
925 <listitem><para><link linkend='var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></link>:
926 Specifies features to include in the image.
927 Most of these features map to additional packages for
928 installation.</para></listitem>
929 <listitem><para><link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>:
930 Specifies the package backend to use and consequently
931 helps determine where to locate packages within the
932 Package Feeds area.</para></listitem>
933 <listitem><para><link linkend='var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></link>:
934 Determines the language(s) for which additional
935 language support packages are installed.
936 </para></listitem>
937 </itemizedlist>
938 </para>
939
940 <para>
941 Package installation is under control of the package manager
942 (e.g. smart/rpm, opkg, or apt/dpkg) regardless of whether or
943 not package management is enabled for the target.
944 At the end of the process, if package management is not
945 enabled for the target, the package manager's data files
946 are deleted from the root filesystem.
947 </para>
948
949 <para>
950 During image generation, the build system attempts to run
951 all post installation scripts.
952 Any that fail to run on the build host are run on the
953 target when the target system is first booted.
954 If you are using a
955 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
956 all the post installation scripts must succeed during the
957 package installation phase since the root filesystem cannot be
958 written into.
959 </para>
960
961 <para>
962 During Optimization, optimizing processes are run across
963 the image.
964 These processes include <filename>mklibs</filename> and
965 <filename>prelink</filename>.
966 The <filename>mklibs</filename> process optimizes the size
967 of the libraries.
968 A <filename>prelink</filename> process optimizes the dynamic
969 linking of shared libraries to reduce start up time of
970 executables.
971 </para>
972
973 <para>
974 Part of the image generation process includes compressing the
975 root filesystem image.
976 Compression is accomplished through several optimization
977 routines designed to reduce the overall size of the image.
978 </para>
979
980 <para>
981 After the root filesystem has been constructed, the image
982 generation process turns everything into an image file or
983 a set of image files.
984 The formats used for the root filesystem depend on the
985 <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
986 variable.
987 </para>
988
989 <note>
990 The entire image generation process is run under Pseudo.
991 Running under Pseudo ensures that the files in the root
992 filesystem have correct ownership.
993 </note>
994 </section>
995
996 <section id='sdk-generation-dev-environment'>
997 <title>SDK Generation</title>
998
999 <para>
1000 The OpenEmbedded build system uses BitBake to generate the
1001 Software Development Kit (SDK) installer script:
1002 <imagedata fileref="figures/sdk-generation.png" align="center" width="6in" depth="7in" />
1003 </para>
1004
1005 <note>
1006 For more information on the cross-development toolchain
1007 generation, see the
1008 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1009 section.
1010 </note>
1011
1012 <para>
1013 Like image generation, the SDK script process consists of
1014 several stages and depends on many variables.
1015 The <filename>do_populate_sdk</filename> task uses these
1016 key variables to help create the list of packages to actually
1017 install.
1018 For information on the variables listed in the figure, see the
1019 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1020 section.
1021 </para>
1022
1023 <para>
1024 The <filename>do_populate_sdk</filename> task handles two
1025 parts: a target part and a host part.
1026 The target part is the part built for the target hardware and
1027 includes libraries and headers.
1028 The host part is the part of the SDK that runs on the
1029 <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>.
1030 </para>
1031
1032 <para>
1033 Once both parts are constructed, the
1034 <filename>do_populate_sdk</filename> task performs some cleanup
1035 on both parts.
1036 After the cleanup, the task creates a cross-development
1037 environment setup script and any configuration files that
1038 might be needed.
1039 </para>
1040
1041 <para>
1042 The final output of the task is the Cross-development
1043 toolchain installation script (<filename>.sh</filename> file),
1044 which includes the environment setup script.
1045 </para>
1046 </section>
1047 </section>
1048
1049 <section id='images-dev-environment'>
1050 <title>Images</title>
1051
1052 <para>
1053 The images produced by the OpenEmbedded build system
1054 are compressed forms of the
1055 root filesystems that are ready to boot on a target device.
1056 You can see from the
1057 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
1058 that BitBake output in part consists of images.
1059 This section is going to look more closely at this output:
1060 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1061 </para>
1062
1063 <para>
1064 For a list of example images that the Yocto Project provides,
1065 the
1066 "<link linkend='ref-images'>Images</link>" chapter.
1067 </para>
1068
1069 <para>
1070 Images are written out to the
1071 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
1072 inside the <filename>deploy/images/&lt;machine&gt;/</filename>
1073 folder as shown in the figure.
1074 This folder contains any files expected to be loaded on the
1075 target device.
1076 The
1077 <link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>
1078 variable points to the <filename>deploy</filename> directory,
1079 while the
1080 <link linkend='var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></link>
1081 variable points to the appropriate directory containing images for
1082 the current configuration.
1083 <itemizedlist>
1084 <listitem><para><filename>&lt;kernel-image&gt;</filename>:
1085 A kernel binary file.
1086 The <link linkend='var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></link>
1087 variable setting determines the naming scheme for the
1088 kernel image file.
1089 Depending on that variable, the file could begin with
1090 a variety of naming strings.
1091 The <filename>deploy/images/&lt;machine&gt;</filename>
1092 directory can contain multiple image files for the
1093 machine.</para></listitem>
1094 <listitem><para><filename>&lt;root-filesystem-image&gt;</filename>:
1095 Root filesystems for the target device (e.g.
1096 <filename>*.ext3</filename> or <filename>*.bz2</filename>
1097 files).
1098 The <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
1099 variable setting determines the root filesystem image
1100 type.
1101 The <filename>deploy/images/&lt;machine&gt;</filename>
1102 directory can contain multiple root filesystems for the
1103 machine.</para></listitem>
1104 <listitem><para><filename>&lt;kernel-modules&gt;</filename>:
1105 Tarballs that contain all the modules built for the kernel.
1106 Kernel module tarballs exist for legacy purposes and
1107 can be suppressed by setting the
1108 <link linkend='var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></link>
1109 variable to "0".
1110 The <filename>deploy/images/&lt;machine&gt;</filename>
1111 directory can contain multiple kernel module tarballs
1112 for the machine.</para></listitem>
1113 <listitem><para><filename>&lt;bootloaders&gt;</filename>:
1114 Bootloaders supporting the image, if applicable to the
1115 target machine.
1116 The <filename>deploy/images/&lt;machine&gt;</filename>
1117 directory can contain multiple bootloaders for the
1118 machine.</para></listitem>
1119 <listitem><para><filename>&lt;symlinks&gt;</filename>:
1120 The <filename>deploy/images/&lt;machine&gt;</filename>
1121 folder contains
1122 a symbolic link that points to the most recently built file
1123 for each machine.
1124 These links might be useful for external scripts that
1125 need to obtain the latest version of each file.
1126 </para></listitem>
1127 </itemizedlist>
1128 </para>
1129 </section>
1130
1131 <section id='sdk-dev-environment'>
1132 <title>Application Development SDK</title>
1133
1134 <para>
1135 In the
1136 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
1137 the output labeled "Application Development SDK" represents an
1138 SDK.
1139 This section is going to take a closer look at this output:
1140 <imagedata fileref="figures/sdk.png" align="center" width="5in" depth="4in" />
1141 </para>
1142
1143 <para>
1144 The specific form of this output is a self-extracting
1145 SDK installer (<filename>*.sh</filename>) that, when run,
1146 installs the SDK, which consists of a cross-development
1147 toolchain, a set of libraries and headers, and an SDK
1148 environment setup script.
1149 Running this installer essentially sets up your
1150 cross-development environment.
1151 You can think of the cross-toolchain as the "host"
1152 part because it runs on the SDK machine.
1153 You can think of the libraries and headers as the "target"
1154 part because they are built for the target hardware.
1155 The setup script is added so that you can initialize the
1156 environment before using the tools.
1157 </para>
1158
1159 <note>
1160 <para>
1161 The Yocto Project supports several methods by which you can
1162 set up this cross-development environment.
1163 These methods include downloading pre-built SDK installers,
1164 building and installing your own SDK installer, or running
1165 an Application Development Toolkit (ADT) installer to
1166 install not just cross-development toolchains
1167 but also additional tools to help in this type of
1168 development.
1169 </para>
1170
1171 <para>
1172 For background information on cross-development toolchains
1173 in the Yocto Project development environment, see the
1174 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1175 section.
1176 For information on setting up a cross-development
1177 environment, see the
1178 "<ulink url='&YOCTO_DOCS_ADT_URL;#installing-the-adt'>Installing the ADT and Toolchains</ulink>"
1179 section in the Yocto Project Application Developer's Guide.
1180 </para>
1181 </note>
1182
1183 <para>
1184 Once built, the SDK installers are written out to the
1185 <filename>deploy/sdk</filename> folder inside the
1186 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
1187 as shown in the figure at the beginning of this section.
1188 Several variables exist that help configure these files:
1189 <itemizedlist>
1190 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
1191 Points to the <filename>deploy</filename>
1192 directory.</para></listitem>
1193 <listitem><para><link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>:
1194 Specifies the architecture of the machine
1195 on which the cross-development tools are run to
1196 create packages for the target hardware.
1197 </para></listitem>
1198 <listitem><para><link linkend='var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></link>:
1199 Lists the features to include in the "target" part
1200 of the SDK.
1201 </para></listitem>
1202 <listitem><para><link linkend='var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></link>:
1203 Lists packages that make up the host
1204 part of the SDK (i.e. the part that runs on
1205 the <filename>SDKMACHINE</filename>).
1206 When you use
1207 <filename>bitbake -c populate_sdk &lt;imagename&gt;</filename>
1208 to create the SDK, a set of default packages
1209 apply.
1210 This variable allows you to add more packages.
1211 </para></listitem>
1212 <listitem><para><link linkend='var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></link>:
1213 Lists packages that make up the target part
1214 of the SDK (i.e. the part built for the
1215 target hardware).
1216 </para></listitem>
1217 </itemizedlist>
1218 </para>
1219 </section>
1220
1221</chapter>
1222<!--
1223vim: expandtab tw=80 ts=4
1224-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-04-25 08:50:20 +0000