summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual/closer-look.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/ref-manual/closer-look.xml')
-rw-r--r--documentation/ref-manual/closer-look.xml1270
1 files changed, 1270 insertions, 0 deletions
diff --git a/documentation/ref-manual/closer-look.xml b/documentation/ref-manual/closer-look.xml
new file mode 100644
index 0000000000..c223bbb0ce
--- /dev/null
+++ b/documentation/ref-manual/closer-look.xml
@@ -0,0 +1,1270 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='closer-look'>
6<title>A Closer Look at the Yocto Project Development Environment</title>
7
8 <para>
9 This chapter takes a more detailed look at the Yocto Project
10 development environment.
11 The following diagram represents the development environment at a
12 high level.
13 The remainder of this chapter expands on the fundamental input, output,
14 process, and
15 <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>) blocks
16 in the Yocto Project development environment.
17 </para>
18
19 <para id='general-yocto-environment-figure'>
20 <imagedata fileref="figures/yocto-environment-ref.png" align="center" width="8in" depth="4.25in" />
21 </para>
22
23 <para>
24 The generalized Yocto Project Development Environment consists of
25 several functional areas:
26 <itemizedlist>
27 <listitem><para><emphasis>User Configuration:</emphasis>
28 Metadata you can use to control the build process.
29 </para></listitem>
30 <listitem><para><emphasis>Metadata Layers:</emphasis>
31 Various layers that provide software, machine, and
32 distro Metadata.</para></listitem>
33 <listitem><para><emphasis>Source Files:</emphasis>
34 Upstream releases, local projects, and SCMs.</para></listitem>
35 <listitem><para><emphasis>Build System:</emphasis>
36 Processes under the control of
37 <ulink url='&YOCTO_DOCS_DEV_URL;#bitbake-term'>BitBake</ulink>.
38 This block expands on how BitBake fetches source, applies
39 patches, completes compilation, analyzes output for package
40 generation, creates and tests packages, generates images, and
41 generates cross-development tools.</para></listitem>
42 <listitem><para><emphasis>Package Feeds:</emphasis>
43 Directories containing output packages (RPM, DEB or IPK),
44 which are subsequently used in the construction of an image or
45 SDK, produced by the build system.
46 These feeds can also be copied and shared using a web server or
47 other means to facilitate extending or updating existing
48 images on devices at runtime if runtime package management is
49 enabled.</para></listitem>
50 <listitem><para><emphasis>Images:</emphasis>
51 Images produced by the development process.
52 </para></listitem>
53 <listitem><para><emphasis>Application Development SDK:</emphasis>
54 Cross-development tools that are produced along with an image
55 or separately with BitBake.</para></listitem>
56 </itemizedlist>
57 </para>
58
59 <section id="user-configuration">
60 <title>User Configuration</title>
61
62 <para>
63 User configuration helps define the build.
64 Through user configuration, you can tell BitBake the
65 target architecture for which you are building the image,
66 where to store downloaded source, and other build properties.
67 </para>
68
69 <para>
70 The following figure shows an expanded representation of the
71 "User Configuration" box of the
72 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
73 </para>
74
75 <para>
76 <imagedata fileref="figures/user-configuration.png" align="center" width="5.5in" depth="3.5in" />
77 </para>
78
79 <para>
80 BitBake needs some basic configuration files in order to complete
81 a build.
82 These files are <filename>*.conf</filename> files.
83 The minimally necessary ones reside as example files in the
84 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
85 For simplicity, this section refers to the Source Directory as
86 the "Poky Directory."
87 </para>
88
89 <para>
90 When you clone the <filename>poky</filename> Git repository or you
91 download and unpack a Yocto Project release, you can set up the
92 Source Directory to be named anything you want.
93 For this discussion, the cloned repository uses the default
94 name <filename>poky</filename>.
95 <note>
96 The Poky repository is primarily an aggregation of existing
97 repositories.
98 It is not a canonical upstream source.
99 </note>
100 </para>
101
102 <para>
103 The <filename>meta-yocto</filename> layer inside Poky contains
104 a <filename>conf</filename> directory that has example
105 configuration files.
106 These example files are used as a basis for creating actual
107 configuration files when you source the build environment
108 script
109 (i.e.
110 <link linkend='structure-core-script'><filename>&OE_INIT_FILE;</filename></link>
111 or
112 <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>).
113 </para>
114
115 <para>
116 Sourcing the build environment script creates a
117 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
118 if one does not already exist.
119 BitBake uses the Build Directory for all its work during builds.
120 The Build Directory has a <filename>conf</filename> directory that
121 contains default versions of your <filename>local.conf</filename>
122 and <filename>bblayers.conf</filename> configuration files.
123 These default configuration files are created only if versions
124 do not already exist in the Build Directory at the time you
125 source the build environment setup script.
126 </para>
127
128 <para>
129 Because the Poky repository is fundamentally an aggregation of
130 existing repositories, some users might be familiar with running
131 the <filename>&OE_INIT_FILE;</filename> or
132 <filename>oe-init-build-env-memres</filename> script in the context
133 of separate OpenEmbedded-Core and BitBake repositories rather than a
134 single Poky repository.
135 This discussion assumes the script is executed from within a cloned
136 or unpacked version of Poky.
137 </para>
138
139 <para>
140 Depending on where the script is sourced, different sub-scripts
141 are called to set up the Build Directory (Yocto or OpenEmbedded).
142 Specifically, the script
143 <filename>scripts/oe-setup-builddir</filename> inside the
144 poky directory sets up the Build Directory and seeds the directory
145 (if necessary) with configuration files appropriate for the
146 Yocto Project development environment.
147 <note>
148 The <filename>scripts/oe-setup-builddir</filename> script
149 uses the <filename>$TEMPLATECONF</filename> variable to
150 determine which sample configuration files to locate.
151 </note>
152 </para>
153
154 <para>
155 The <filename>local.conf</filename> file provides many
156 basic variables that define a build environment.
157 Here is a list of a few.
158 To see the default configurations in a <filename>local.conf</filename>
159 file created by the build environment script, see the
160 <filename>local.conf.sample</filename> in the
161 <filename>meta-yocto</filename> layer:
162 <itemizedlist>
163 <listitem><para><emphasis>Parallelism Options:</emphasis>
164 Controlled by the
165 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
166 and
167 <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></link>
168 variables.</para></listitem>
169 <listitem><para><emphasis>Target Machine Selection:</emphasis>
170 Controlled by the
171 <link linkend='var-MACHINE'><filename>MACHINE</filename></link>
172 variable.</para></listitem>
173 <listitem><para><emphasis>Download Directory:</emphasis>
174 Controlled by the
175 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
176 variable.</para></listitem>
177 <listitem><para><emphasis>Shared State Directory:</emphasis>
178 Controlled by the
179 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
180 variable.</para></listitem>
181 <listitem><para><emphasis>Build Output:</emphasis>
182 Controlled by the
183 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
184 variable.</para></listitem>
185 </itemizedlist>
186 <note>
187 Configurations set in the <filename>conf/local.conf</filename>
188 file can also be set in the
189 <filename>conf/site.conf</filename> and
190 <filename>conf/auto.conf</filename> configuration files.
191 </note>
192 </para>
193
194 <para>
195 The <filename>bblayers.conf</filename> file tells BitBake what
196 layers you want considered during the build.
197 By default, the layers listed in this file include layers
198 minimally needed by the build system.
199 However, you must manually add any custom layers you have created.
200 You can find more information on working with the
201 <filename>bblayers.conf</filename> file in the
202 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
203 section in the Yocto Project Development Manual.
204 </para>
205
206 <para>
207 The files <filename>site.conf</filename> and
208 <filename>auto.conf</filename> are not created by the environment
209 initialization script.
210 If you want these configuration files, you must create them
211 yourself:
212 <itemizedlist>
213 <listitem><para><emphasis><filename>site.conf</filename>:</emphasis>
214 You can use the <filename>conf/site.conf</filename>
215 configuration file to configure multiple build directories.
216 For example, suppose you had several build environments and
217 they shared some common features.
218 You can set these default build properties here.
219 A good example is perhaps the level of parallelism you want
220 to use through the
221 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
222 and
223 <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></link>
224 variables.</para>
225 <para>One useful scenario for using the
226 <filename>conf/site.conf</filename> file is to extend your
227 <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
228 variable to include the path to a
229 <filename>conf/site.conf</filename>.
230 Then, when BitBake looks for Metadata using
231 <filename>BBPATH</filename>, it finds the
232 <filename>conf/site.conf</filename> file and applies your
233 common configurations found in the file.
234 To override configurations in a particular build directory,
235 alter the similar configurations within that build
236 directory's <filename>conf/local.conf</filename> file.
237 </para></listitem>
238 <listitem><para><emphasis><filename>auto.conf</filename>:</emphasis>
239 This file is not hand-created.
240 Rather, the file is usually created and written to by
241 an autobuilder.
242 The settings put into the file are typically the same as
243 you would find in the <filename>conf/local.conf</filename>
244 or the <filename>conf/site.conf</filename> files.
245 </para></listitem>
246 </itemizedlist>
247 </para>
248
249 <para>
250 You can edit all configuration files to further define
251 any particular build environment.
252 This process is represented by the "User Configuration Edits"
253 box in the figure.
254 </para>
255
256 <para>
257 When you launch your build with the
258 <filename>bitbake &lt;target&gt;</filename> command, BitBake
259 sorts out the configurations to ultimately define your build
260 environment.
261 </para>
262 </section>
263
264 <section id="metadata-machine-configuration-and-policy-configuration">
265 <title>Metadata, Machine Configuration, and Policy Configuration</title>
266
267 <para>
268 The previous section described the user configurations that
269 define BitBake's global behavior.
270 This section takes a closer look at the layers the build system
271 uses to further control the build.
272 These layers provide Metadata for the software, machine, and
273 policy.
274 </para>
275
276 <para>
277 In general, three types of layer input exist:
278 <itemizedlist>
279 <listitem><para><emphasis>Policy Configuration:</emphasis>
280 Distribution Layers provide top-level or general
281 policies for the image or SDK being built.
282 For example, this layer would dictate whether BitBake
283 produces RPM or IPK packages.</para></listitem>
284 <listitem><para><emphasis>Machine Configuration:</emphasis>
285 Board Support Package (BSP) layers provide machine
286 configurations.
287 This type of information is specific to a particular
288 target architecture.</para></listitem>
289 <listitem><para><emphasis>Metadata:</emphasis>
290 Software layers contain user-supplied recipe files,
291 patches, and append files.
292 </para></listitem>
293 </itemizedlist>
294 </para>
295
296 <para>
297 The following figure shows an expanded representation of the
298 Metadata, Machine Configuration, and Policy Configuration input
299 (layers) boxes of the
300 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
301 </para>
302
303 <para>
304 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="7.5in" />
305 </para>
306
307 <para>
308 In general, all layers have a similar structure.
309 They all contain a licensing file
310 (e.g. <filename>COPYING</filename>) if the layer is to be
311 distributed, a <filename>README</filename> file as good practice
312 and especially if the layer is to be distributed, a
313 configuration directory, and recipe directories.
314 </para>
315
316 <para>
317 The Yocto Project has many layers that can be used.
318 You can see a web-interface listing of them on the
319 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
320 page.
321 The layers are shown at the bottom categorized under
322 "Yocto Metadata Layers."
323 These layers are fundamentally a subset of the
324 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Metadata Index</ulink>,
325 which lists all layers provided by the OpenEmbedded community.
326 <note>
327 Layers exist in the Yocto Project Source Repositories that
328 cannot be found in the OpenEmbedded Metadata Index.
329 These layers are either deprecated or experimental in nature.
330 </note>
331 </para>
332
333 <para>
334 BitBake uses the <filename>conf/bblayers.conf</filename> file,
335 which is part of the user configuration, to find what layers it
336 should be using as part of the build.
337 </para>
338
339 <para>
340 For more information on layers, see the
341 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
342 section in the Yocto Project Development Manual.
343 </para>
344
345 <section id="distro-layer">
346 <title>Distro Layer</title>
347
348 <para>
349 The distribution layer provides policy configurations for your
350 distribution.
351 Best practices dictate that you isolate these types of
352 configurations into their own layer.
353 Settings you provide in
354 <filename>conf/distro/&lt;distro&gt;.conf</filename> override
355 similar
356 settings that BitBake finds in your
357 <filename>conf/local.conf</filename> file in the Build
358 Directory.
359 </para>
360
361 <para>
362 The following list provides some explanation and references
363 for what you typically find in the distribution layer:
364 <itemizedlist>
365 <listitem><para><emphasis>classes:</emphasis>
366 Class files (<filename>.bbclass</filename>) hold
367 common functionality that can be shared among
368 recipes in the distribution.
369 When your recipes inherit a class, they take on the
370 settings and functions for that class.
371 You can read more about class files in the
372 "<link linkend='ref-classes'>Classes</link>" section.
373 </para></listitem>
374 <listitem><para><emphasis>conf:</emphasis>
375 This area holds configuration files for the
376 layer (<filename>conf/layer.conf</filename>),
377 the distribution
378 (<filename>conf/distro/&lt;distro&gt;.conf</filename>),
379 and any distribution-wide include files.
380 </para></listitem>
381 <listitem><para><emphasis>recipes-*:</emphasis>
382 Recipes and append files that affect common
383 functionality across the distribution.
384 This area could include recipes and append files
385 to add distribution-specific configuration,
386 initialization scripts, custom image recipes,
387 and so forth.</para></listitem>
388 </itemizedlist>
389 </para>
390 </section>
391
392 <section id="bsp-layer">
393 <title>BSP Layer</title>
394
395 <para>
396 The BSP Layer provides machine configurations.
397 Everything in this layer is specific to the machine for which
398 you are building the image or the SDK.
399 A common structure or form is defined for BSP layers.
400 You can learn more about this structure in the
401 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
402 <note>
403 In order for a BSP layer to be considered compliant with the
404 Yocto Project, it must meet some structural requirements.
405 </note>
406 </para>
407
408 <para>
409 The BSP Layer's configuration directory contains
410 configuration files for the machine
411 (<filename>conf/machine/&lt;machine&gt;.conf</filename>) and,
412 of course, the layer (<filename>conf/layer.conf</filename>).
413 </para>
414
415 <para>
416 The remainder of the layer is dedicated to specific recipes
417 by function: <filename>recipes-bsp</filename>,
418 <filename>recipes-core</filename>,
419 <filename>recipes-graphics</filename>, and
420 <filename>recipes-kernel</filename>.
421 Metadata can exist for multiple formfactors, graphics
422 support systems, and so forth.
423 <note>
424 While the figure shows several <filename>recipes-*</filename>
425 directories, not all these directories appear in all
426 BSP layers.
427 </note>
428 </para>
429 </section>
430
431 <section id="software-layer">
432 <title>Software Layer</title>
433
434 <para>
435 The software layer provides the Metadata for additional
436 software packages used during the build.
437 This layer does not include Metadata that is specific to the
438 distribution or the machine, which are found in their
439 respective layers.
440 </para>
441
442 <para>
443 This layer contains any new recipes that your project needs
444 in the form of recipe files.
445 </para>
446 </section>
447 </section>
448
449 <section id="sources-dev-environment">
450 <title>Sources</title>
451
452 <para>
453 In order for the OpenEmbedded build system to create an image or
454 any target, it must be able to access source files.
455 The
456 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
457 represents source files using the "Upstream Project Releases",
458 "Local Projects", and "SCMs (optional)" boxes.
459 The figure represents mirrors, which also play a role in locating
460 source files, with the "Source Mirror(s)" box.
461 </para>
462
463 <para>
464 The method by which source files are ultimately organized is
465 a function of the project.
466 For example, for released software, projects tend to use tarballs
467 or other archived files that can capture the state of a release
468 guaranteeing that it is statically represented.
469 On the other hand, for a project that is more dynamic or
470 experimental in nature, a project might keep source files in a
471 repository controlled by a Source Control Manager (SCM) such as
472 Git.
473 Pulling source from a repository allows you to control
474 the point in the repository (the revision) from which you want to
475 build software.
476 Finally, a combination of the two might exist, which would give the
477 consumer a choice when deciding where to get source files.
478 </para>
479
480 <para>
481 BitBake uses the
482 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
483 variable to point to source files regardless of their location.
484 Each recipe must have a <filename>SRC_URI</filename> variable
485 that points to the source.
486 </para>
487
488 <para>
489 Another area that plays a significant role in where source files
490 come from is pointed to by the
491 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
492 variable.
493 This area is a cache that can hold previously downloaded source.
494 You can also instruct the OpenEmbedded build system to create
495 tarballs from Git repositories, which is not the default behavior,
496 and store them in the <filename>DL_DIR</filename> by using the
497 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
498 variable.
499 </para>
500
501 <para>
502 Judicious use of a <filename>DL_DIR</filename> directory can
503 save the build system a trip across the Internet when looking
504 for files.
505 A good method for using a download directory is to have
506 <filename>DL_DIR</filename> point to an area outside of your
507 Build Directory.
508 Doing so allows you to safely delete the Build Directory
509 if needed without fear of removing any downloaded source file.
510 </para>
511
512 <para>
513 The remainder of this section provides a deeper look into the
514 source files and the mirrors.
515 Here is a more detailed look at the source file area of the
516 base figure:
517 <imagedata fileref="figures/source-input.png" align="center" width="7in" depth="7.5in" />
518 </para>
519
520 <section id='upstream-project-releases'>
521 <title>Upstream Project Releases</title>
522
523 <para>
524 Upstream project releases exist anywhere in the form of an
525 archived file (e.g. tarball or zip file).
526 These files correspond to individual recipes.
527 For example, the figure uses specific releases each for
528 BusyBox, Qt, and Dbus.
529 An archive file can be for any released product that can be
530 built using a recipe.
531 </para>
532 </section>
533
534 <section id='local-projects'>
535 <title>Local Projects</title>
536
537 <para>
538 Local projects are custom bits of software the user provides.
539 These bits reside somewhere local to a project - perhaps
540 a directory into which the user checks in items (e.g.
541 a local directory containing a development source tree
542 used by the group).
543 </para>
544
545 <para>
546 The canonical method through which to include a local project
547 is to use the
548 <link linkend='ref-classes-externalsrc'><filename>externalsrc</filename></link>
549 class to include that local project.
550 You use either the <filename>local.conf</filename> or a
551 recipe's append file to override or set the
552 recipe to point to the local directory on your disk to pull
553 in the whole source tree.
554 </para>
555
556 <para>
557 For information on how to use the
558 <filename>externalsrc</filename> class, see the
559 "<link linkend='ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></link>"
560 section.
561 </para>
562 </section>
563
564 <section id='scms'>
565 <title>Source Control Managers (Optional)</title>
566
567 <para>
568 Another place the build system can get source files from is
569 through an SCM such as Git or Subversion.
570 In this case, a repository is cloned or checked out.
571 The <filename>do_fetch</filename> task inside BitBake uses
572 the <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
573 variable and the argument's prefix to determine the correct
574 fetcher module.
575 </para>
576
577 <note>
578 For information on how to have the OpenEmbedded build system
579 generate tarballs for Git repositories and place them in the
580 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
581 directory, see the
582 <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
583 variable.
584 </note>
585
586 <para>
587 When fetching a repository, BitBake uses the
588 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
589 variable to determine the specific revision from which to
590 build.
591 </para>
592 </section>
593
594 <section id='source-mirrors'>
595 <title>Source Mirror(s)</title>
596
597 <para>
598 Two kinds of mirrors exist: pre-mirrors and regular mirrors.
599 The <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
600 and
601 <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
602 variables point to these, respectively.
603 BitBake checks pre-mirrors before looking upstream for any
604 source files.
605 Pre-mirrors are appropriate when you have a shared directory
606 that is not a directory defined by the
607 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
608 variable.
609 A Pre-mirror typically points to a shared directory that is
610 local to your organization.
611 </para>
612
613 <para>
614 Regular mirrors can be any site across the Internet that is
615 used as an alternative location for source code should the
616 primary site not be functioning for some reason or another.
617 </para>
618 </section>
619 </section>
620
621 <section id="package-feeds-dev-environment">
622 <title>Package Feeds</title>
623
624 <para>
625 When the OpenEmbedded build system generates an image or an SDK,
626 it gets the packages from a package feed area located in the
627 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
628 The
629 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
630 shows this package feeds area in the upper-right corner.
631 </para>
632
633 <para>
634 This section looks a little closer into the package feeds area used
635 by the build system.
636 Here is a more detailed look at the area:
637 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
638 </para>
639
640 <para>
641 Package feeds are an intermediary step in the build process.
642 BitBake generates packages whose types are defined by the
643 <link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>
644 variable.
645 Before placing the packages into package feeds,
646 the build process validates them with generated output quality
647 assurance checks through the
648 <link linkend='ref-classes-insane'><filename>insane</filename></link>
649 class.
650 </para>
651
652 <para>
653 The package feed area resides in
654 <filename>tmp/deploy</filename> of the Build Directory.
655 Folders are created that correspond to the package type
656 (IPK, DEB, or RPM) created.
657 Further organization is derived through the value of the
658 <link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link>
659 variable for each package.
660 For example, packages can exist for the i586 or qemux86
661 architectures.
662 The package files themselves reside within the appropriate
663 architecture folder.
664 </para>
665
666 <para>
667 BitBake uses the <filename>do_package_write_*</filename> task to
668 place generated packages into the package holding area (e.g.
669 <filename>do_package_write_ipk</filename> for IPK packages).
670 </para>
671 </section>
672
673 <section id='bitbake-dev-environment'>
674 <title>BitBake</title>
675
676 <para>
677 The OpenEmbedded build system uses
678 <ulink url='&YOCTO_DOCS_DEV_URL;#bitbake-term'>BitBake</ulink>
679 to produce images.
680 You can see from the
681 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
682 the BitBake area consists of several functional areas.
683 This section takes a closer look at each of those areas.
684 </para>
685
686 <para>
687 Separate documentation exists for the BitBake tool.
688 See the
689 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>
690 for reference material on BitBake.
691 </para>
692
693 <section id='source-fetching-dev-environment'>
694 <title>Source Fetching</title>
695
696 <para>
697 The first stages of building a recipe are to fetch and unpack
698 the source code:
699 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
700 </para>
701
702 <para>
703 The <filename>do_fetch</filename> and
704 <filename>do_unpack</filename> tasks fetch the source files
705 and unpack them into the work directory.
706 By default, everything is accomplished in the
707 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>,
708 which has a defined structure.
709 For additional general information on the Build Directory,
710 see the
711 "<link linkend='structure-core-build'><filename>build/</filename></link>"
712 section.
713 </para>
714
715 <para>
716 Unpacked source files are pointed to by the
717 <link linkend='var-S'><filename>S</filename></link> variable.
718 Each recipe has an area in the Build Directory where the
719 unpacked source code resides.
720 The name of that directory for any given recipe is defined from
721 several different variables.
722 You can see the variables that define these directories
723 by looking at the figure:
724 <itemizedlist>
725 <listitem><para><link linkend='var-TMPDIR'><filename>TMPDIR</filename></link> -
726 The base directory where the OpenEmbedded build system
727 performs all its work during the build.
728 </para></listitem>
729 <listitem><para><link linkend='var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></link> -
730 The architecture of the built package or packages.
731 </para></listitem>
732 <listitem><para><link linkend='var-TARGET_OS'><filename>TARGET_OS</filename></link> -
733 The operating system of the target device.
734 </para></listitem>
735 <listitem><para><link linkend='var-PN'><filename>PN</filename></link> -
736 The name of the built package.
737 </para></listitem>
738 <listitem><para><link linkend='var-PV'><filename>PV</filename></link> -
739 The version of the recipe used to build the package.
740 </para></listitem>
741 <listitem><para><link linkend='var-PR'><filename>PR</filename></link> -
742 The revision of the recipe used to build the package.
743 </para></listitem>
744 <listitem><para><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link> -
745 The location within <filename>TMPDIR</filename> where
746 a specific package is built.
747 </para></listitem>
748 <listitem><para><link linkend='var-S'><filename>S</filename></link> -
749 Contains the unpacked source files for a given recipe.
750 </para></listitem>
751 </itemizedlist>
752 </para>
753 </section>
754
755 <section id='patching-dev-environment'>
756 <title>Patching</title>
757
758 <para>
759 Once source code is fetched and unpacked, BitBake locates
760 patch files and applies them to the source files:
761 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
762 </para>
763
764 <para>
765 The <filename>do_patch</filename> task processes recipes by
766 using the
767 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
768 variable to locate applicable patch files, which by default
769 are <filename>*.patch</filename> or
770 <filename>*.diff</filename> files, or any file if
771 "apply=yes" is specified for the file in
772 <filename>SRC_URI</filename>.
773 </para>
774
775 <para>
776 BitBake finds and applies multiple patches for a single recipe
777 in the order in which it finds the patches.
778 Patches are applied to the recipe's source files located in the
779 <link linkend='var-S'><filename>S</filename></link> directory.
780 </para>
781
782 <para>
783 For more information on how the source directories are
784 created, see the
785 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
786 section.
787 </para>
788 </section>
789
790 <section id='configuration-and-compilation-dev-environment'>
791 <title>Configuration and Compilation</title>
792
793 <para>
794 After source code is patched, BitBake executes tasks that
795 configure and compile the source code:
796 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
797 </para>
798
799 <para>
800 This step in the build process consists of three tasks:
801 <itemizedlist>
802 <listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
803 This task configures the source by enabling and
804 disabling any build-time and configuration options for
805 the software being built.
806 Configurations can come from the recipe itself as well
807 as from an inherited class.
808 Additionally, the software itself might configure itself
809 depending on the target for which it is being built.
810 </para>
811
812 <para>The configurations handled by the
813 <filename>do_configure</filename> task are specific
814 to source code configuration for the source code
815 being built by the recipe.</para>
816
817 <para>If you are using the
818 <link linkend='ref-classes-autotools'><filename>autotools</filename></link>
819 class,
820 you can add additional configuration options by using
821 the <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
822 variable.
823 For information on how this variable works within
824 that class, see the
825 <filename>meta/classes/autotools.bbclass</filename> file.
826 </para></listitem>
827 <listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
828 Once a configuration task has been satisfied, BitBake
829 compiles the source using the
830 <filename>do_compile</filename> task.
831 Compilation occurs in the directory pointed to by the
832 <link linkend='var-B'><filename>B</filename></link>
833 variable.
834 Realize that the <filename>B</filename> directory is, by
835 default, the same as the
836 <link linkend='var-S'><filename>S</filename></link>
837 directory.</para></listitem>
838 <listitem><para><emphasis><filename>do_install</filename>:</emphasis>
839 Once compilation is done, BitBake executes the
840 <filename>do_install</filename> task.
841 This task copies files from the <filename>B</filename>
842 directory and places them in a holding area pointed to
843 by the
844 <link linkend='var-D'><filename>D</filename></link>
845 variable.</para></listitem>
846 </itemizedlist>
847 </para>
848 </section>
849
850 <section id='package-splitting-dev-environment'>
851 <title>Package Splitting</title>
852
853 <para>
854 After source code is configured and compiled, the
855 OpenEmbedded build system analyzes
856 the results and splits the output into packages:
857 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
858 </para>
859
860 <para>
861 The <filename>do_package</filename> and
862 <filename>do_packagedata</filename> tasks combine to analyze
863 the files found in the
864 <link linkend='var-D'><filename>D</filename></link> directory
865 and split them into subsets based on available packages and
866 files.
867 The analyzing process involves the following as well as other
868 items: splitting out debugging symbols,
869 looking at shared library dependencies between packages,
870 and looking at package relationships.
871 The <filename>do_packagedata</filename> task creates package
872 metadata based on the analysis such that the
873 OpenEmbedded build system can generate the final packages.
874 Working, staged, and intermediate results of the analysis
875 and package splitting process use these areas:
876 <itemizedlist>
877 <listitem><para><link linkend='var-PKGD'><filename>PKGD</filename></link> -
878 The destination directory for packages before they are
879 split.
880 </para></listitem>
881 <listitem><para><link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link> -
882 A shared, global-state directory that holds data
883 generated during the packaging process.
884 </para></listitem>
885 <listitem><para><link linkend='var-PKGDESTWORK'><filename>PKGDESTWORK</filename></link> -
886 A temporary work area used by the
887 <filename>do_package</filename> task.
888 </para></listitem>
889 <listitem><para><link linkend='var-PKGDEST'><filename>PKGDEST</filename></link> -
890 The parent directory for packages after they have
891 been split.
892 </para></listitem>
893 </itemizedlist>
894 The <link linkend='var-FILES'><filename>FILES</filename></link>
895 variable defines the files that go into each package in
896 <link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>.
897 If you want details on how this is accomplished, you can
898 look at the
899 <link linkend='ref-classes-package'><filename>package</filename></link>
900 class.
901 </para>
902
903 <para>
904 Depending on the type of packages being created (RPM, DEB, or
905 IPK), the <filename>do_package_write_*</filename> task
906 creates the actual packages and places them in the
907 Package Feed area, which is
908 <filename>${TMPDIR}/deploy</filename>.
909 You can see the
910 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
911 section for more detail on that part of the build process.
912 <note>
913 Support for creating feeds directly from the
914 <filename>deploy/*</filename> directories does not exist.
915 Creating such feeds usually requires some kind of feed
916 maintenance mechanism that would upload the new packages
917 into an official package feed (e.g. the
918 Ångström distribution).
919 This functionality is highly distribution-specific
920 and thus is not provided out of the box.
921 </note>
922 </para>
923 </section>
924
925 <section id='image-generation-dev-environment'>
926 <title>Image Generation</title>
927
928 <para>
929 Once packages are split and stored in the Package Feeds area,
930 the OpenEmbedded build system uses BitBake to generate the
931 root filesystem image:
932 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
933 </para>
934
935 <para>
936 The image generation process consists of several stages and
937 depends on many variables.
938 The <filename>do_rootfs</filename> task uses these key variables
939 to help create the list of packages to actually install:
940 <itemizedlist>
941 <listitem><para><link linkend='var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></link>:
942 Lists out the base set of packages to install from
943 the Package Feeds area.</para></listitem>
944 <listitem><para><link linkend='var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></link>:
945 Specifies packages that should not be installed.
946 </para></listitem>
947 <listitem><para><link linkend='var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></link>:
948 Specifies features to include in the image.
949 Most of these features map to additional packages for
950 installation.</para></listitem>
951 <listitem><para><link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>:
952 Specifies the package backend to use and consequently
953 helps determine where to locate packages within the
954 Package Feeds area.</para></listitem>
955 <listitem><para><link linkend='var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></link>:
956 Determines the language(s) for which additional
957 language support packages are installed.
958 </para></listitem>
959 </itemizedlist>
960 </para>
961
962 <para>
963 Package installation is under control of the package manager
964 (e.g. smart/rpm, opkg, or apt/dpkg) regardless of whether or
965 not package management is enabled for the target.
966 At the end of the process, if package management is not
967 enabled for the target, the package manager's data files
968 are deleted from the root filesystem.
969 </para>
970
971 <para>
972 During image generation, the build system attempts to run
973 all post-installation scripts.
974 Any that fail to run on the build host are run on the
975 target when the target system is first booted.
976 If you are using a
977 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
978 all the post installation scripts must succeed during the
979 package installation phase since the root filesystem is
980 read-only.
981 </para>
982
983 <para>
984 During Optimization, optimizing processes are run across
985 the image.
986 These processes include <filename>mklibs</filename> and
987 <filename>prelink</filename>.
988 The <filename>mklibs</filename> process optimizes the size
989 of the libraries.
990 A <filename>prelink</filename> process optimizes the dynamic
991 linking of shared libraries to reduce start up time of
992 executables.
993 </para>
994
995 <para>
996 Along with writing out the root filesystem image, the
997 <filename>do_rootfs</filename> task creates a manifest file
998 (<filename>.manifest</filename>) in the same directory as
999 the root filesystem image that lists out, line-by-line, the
1000 installed packages.
1001 This manifest file is useful for the
1002 <link linkend='ref-classes-testimage'><filename>testimage</filename></link>
1003 class, for example, to determine whether or not to run
1004 specific tests.
1005 See the
1006 <link linkend='var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></link>
1007 variable for additional information.
1008 </para>
1009
1010 <para>
1011 Part of the image generation process includes compressing the
1012 root filesystem image.
1013 Compression is accomplished through several optimization
1014 routines designed to reduce the overall size of the image.
1015 </para>
1016
1017 <para>
1018 After the root filesystem has been constructed, the image
1019 generation process turns everything into an image file or
1020 a set of image files.
1021 The formats used for the root filesystem depend on the
1022 <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
1023 variable.
1024 </para>
1025
1026 <note>
1027 The entire image generation process is run under Pseudo.
1028 Running under Pseudo ensures that the files in the root
1029 filesystem have correct ownership.
1030 </note>
1031 </section>
1032
1033 <section id='sdk-generation-dev-environment'>
1034 <title>SDK Generation</title>
1035
1036 <para>
1037 The OpenEmbedded build system uses BitBake to generate the
1038 Software Development Kit (SDK) installer script:
1039 <imagedata fileref="figures/sdk-generation.png" align="center" width="6in" depth="7in" />
1040 </para>
1041
1042 <note>
1043 For more information on the cross-development toolchain
1044 generation, see the
1045 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1046 section.
1047 For information on advantages gained when building a
1048 cross-development toolchain using the
1049 <filename>do_populate_sdk</filename> task, see the
1050 "<ulink url='&YOCTO_DOCS_ADT_URL;#optionally-building-a-toolchain-installer'>Optionally Building a Toolchain Installer</ulink>"
1051 section in the Yocto Project Application Developer's Guide.
1052 </note>
1053
1054 <para>
1055 Like image generation, the SDK script process consists of
1056 several stages and depends on many variables.
1057 The <filename>do_populate_sdk</filename> task uses these
1058 key variables to help create the list of packages to actually
1059 install.
1060 For information on the variables listed in the figure, see the
1061 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1062 section.
1063 </para>
1064
1065 <para>
1066 The <filename>do_populate_sdk</filename> task handles two
1067 parts: a target part and a host part.
1068 The target part is the part built for the target hardware and
1069 includes libraries and headers.
1070 The host part is the part of the SDK that runs on the
1071 <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>.
1072 </para>
1073
1074 <para>
1075 Once both parts are constructed, the
1076 <filename>do_populate_sdk</filename> task performs some cleanup
1077 on both parts.
1078 After the cleanup, the task creates a cross-development
1079 environment setup script and any configuration files that
1080 might be needed.
1081 </para>
1082
1083 <para>
1084 The final output of the task is the Cross-development
1085 toolchain installation script (<filename>.sh</filename> file),
1086 which includes the environment setup script.
1087 </para>
1088 </section>
1089 </section>
1090
1091 <section id='images-dev-environment'>
1092 <title>Images</title>
1093
1094 <para>
1095 The images produced by the OpenEmbedded build system
1096 are compressed forms of the
1097 root filesystem that are ready to boot on a target device.
1098 You can see from the
1099 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
1100 that BitBake output, in part, consists of images.
1101 This section is going to look more closely at this output:
1102 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1103 </para>
1104
1105 <para>
1106 For a list of example images that the Yocto Project provides,
1107 see the
1108 "<link linkend='ref-images'>Images</link>" chapter.
1109 </para>
1110
1111 <para>
1112 Images are written out to the
1113 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
1114 inside the <filename>tmp/deploy/images/&lt;machine&gt;/</filename>
1115 folder as shown in the figure.
1116 This folder contains any files expected to be loaded on the
1117 target device.
1118 The
1119 <link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>
1120 variable points to the <filename>deploy</filename> directory,
1121 while the
1122 <link linkend='var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></link>
1123 variable points to the appropriate directory containing images for
1124 the current configuration.
1125 <itemizedlist>
1126 <listitem><para><filename>&lt;kernel-image&gt;</filename>:
1127 A kernel binary file.
1128 The <link linkend='var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></link>
1129 variable setting determines the naming scheme for the
1130 kernel image file.
1131 Depending on that variable, the file could begin with
1132 a variety of naming strings.
1133 The <filename>deploy/images/&lt;machine&gt;</filename>
1134 directory can contain multiple image files for the
1135 machine.</para></listitem>
1136 <listitem><para><filename>&lt;root-filesystem-image&gt;</filename>:
1137 Root filesystems for the target device (e.g.
1138 <filename>*.ext3</filename> or <filename>*.bz2</filename>
1139 files).
1140 The <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>
1141 variable setting determines the root filesystem image
1142 type.
1143 The <filename>deploy/images/&lt;machine&gt;</filename>
1144 directory can contain multiple root filesystems for the
1145 machine.</para></listitem>
1146 <listitem><para><filename>&lt;kernel-modules&gt;</filename>:
1147 Tarballs that contain all the modules built for the kernel.
1148 Kernel module tarballs exist for legacy purposes and
1149 can be suppressed by setting the
1150 <link linkend='var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></link>
1151 variable to "0".
1152 The <filename>deploy/images/&lt;machine&gt;</filename>
1153 directory can contain multiple kernel module tarballs
1154 for the machine.</para></listitem>
1155 <listitem><para><filename>&lt;bootloaders&gt;</filename>:
1156 Bootloaders supporting the image, if applicable to the
1157 target machine.
1158 The <filename>deploy/images/&lt;machine&gt;</filename>
1159 directory can contain multiple bootloaders for the
1160 machine.</para></listitem>
1161 <listitem><para><filename>&lt;symlinks&gt;</filename>:
1162 The <filename>deploy/images/&lt;machine&gt;</filename>
1163 folder contains
1164 a symbolic link that points to the most recently built file
1165 for each machine.
1166 These links might be useful for external scripts that
1167 need to obtain the latest version of each file.
1168 </para></listitem>
1169 </itemizedlist>
1170 </para>
1171 </section>
1172
1173 <section id='sdk-dev-environment'>
1174 <title>Application Development SDK</title>
1175
1176 <para>
1177 In the
1178 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
1179 the output labeled "Application Development SDK" represents an
1180 SDK.
1181 This section is going to take a closer look at this output:
1182 <imagedata fileref="figures/sdk.png" align="center" width="5in" depth="4in" />
1183 </para>
1184
1185 <para>
1186 The specific form of this output is a self-extracting
1187 SDK installer (<filename>*.sh</filename>) that, when run,
1188 installs the SDK, which consists of a cross-development
1189 toolchain, a set of libraries and headers, and an SDK
1190 environment setup script.
1191 Running this installer essentially sets up your
1192 cross-development environment.
1193 You can think of the cross-toolchain as the "host"
1194 part because it runs on the SDK machine.
1195 You can think of the libraries and headers as the "target"
1196 part because they are built for the target hardware.
1197 The setup script is added so that you can initialize the
1198 environment before using the tools.
1199 </para>
1200
1201 <note>
1202 <para>
1203 The Yocto Project supports several methods by which you can
1204 set up this cross-development environment.
1205 These methods include downloading pre-built SDK installers,
1206 building and installing your own SDK installer, or running
1207 an Application Development Toolkit (ADT) installer to
1208 install not just cross-development toolchains
1209 but also additional tools to help in this type of
1210 development.
1211 </para>
1212
1213 <para>
1214 For background information on cross-development toolchains
1215 in the Yocto Project development environment, see the
1216 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1217 section.
1218 For information on setting up a cross-development
1219 environment, see the
1220 "<ulink url='&YOCTO_DOCS_ADT_URL;#installing-the-adt'>Installing the ADT and Toolchains</ulink>"
1221 section in the Yocto Project Application Developer's Guide.
1222 </para>
1223 </note>
1224
1225 <para>
1226 Once built, the SDK installers are written out to the
1227 <filename>deploy/sdk</filename> folder inside the
1228 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
1229 as shown in the figure at the beginning of this section.
1230 Several variables exist that help configure these files:
1231 <itemizedlist>
1232 <listitem><para><link linkend='var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></link>:
1233 Points to the <filename>deploy</filename>
1234 directory.</para></listitem>
1235 <listitem><para><link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>:
1236 Specifies the architecture of the machine
1237 on which the cross-development tools are run to
1238 create packages for the target hardware.
1239 </para></listitem>
1240 <listitem><para><link linkend='var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></link>:
1241 Lists the features to include in the "target" part
1242 of the SDK.
1243 </para></listitem>
1244 <listitem><para><link linkend='var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></link>:
1245 Lists packages that make up the host
1246 part of the SDK (i.e. the part that runs on
1247 the <filename>SDKMACHINE</filename>).
1248 When you use
1249 <filename>bitbake -c populate_sdk &lt;imagename&gt;</filename>
1250 to create the SDK, a set of default packages
1251 apply.
1252 This variable allows you to add more packages.
1253 </para></listitem>
1254 <listitem><para><link linkend='var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></link>:
1255 Lists packages that make up the target part
1256 of the SDK (i.e. the part built for the
1257 target hardware).
1258 </para></listitem>
1259 <listitem><para><link linkend='var-SDKPATH'><filename>SDKPATH</filename></link>:
1260 Defines the default SDK installation path offered by the
1261 installation script.
1262 </para></listitem>
1263 </itemizedlist>
1264 </para>
1265 </section>
1266
1267</chapter>
1268<!--
1269vim: expandtab tw=80 ts=4
1270-->