diff options
Diffstat (limited to 'documentation/overview-manual/overview-manual-concepts.xml')
-rw-r--r-- | documentation/overview-manual/overview-manual-concepts.xml | 3235 |
1 files changed, 0 insertions, 3235 deletions
diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml deleted file mode 100644 index 58b64bd269..0000000000 --- a/documentation/overview-manual/overview-manual-concepts.xml +++ /dev/null | |||
@@ -1,3235 +0,0 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | ||
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | ||
4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
5 | |||
6 | <chapter id=' overview-manual-concepts'> | ||
7 | <title>Yocto Project Concepts</title> | ||
8 | |||
9 | <para> | ||
10 | This chapter provides explanations for Yocto Project concepts that | ||
11 | go beyond the surface of "how-to" information and reference (or | ||
12 | look-up) material. | ||
13 | Concepts such as components, the | ||
14 | <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> | ||
15 | workflow, cross-development toolchains, shared state cache, and so | ||
16 | forth are explained. | ||
17 | </para> | ||
18 | |||
19 | <section id='yocto-project-components'> | ||
20 | <title>Yocto Project Components</title> | ||
21 | |||
22 | <para> | ||
23 | The | ||
24 | <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> | ||
25 | task executor together with various types of configuration files | ||
26 | form the | ||
27 | <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>. | ||
28 | This section overviews these components by describing their use and | ||
29 | how they interact. | ||
30 | </para> | ||
31 | |||
32 | <para> | ||
33 | BitBake handles the parsing and execution of the data files. | ||
34 | The data itself is of various types: | ||
35 | <itemizedlist> | ||
36 | <listitem><para> | ||
37 | <emphasis>Recipes:</emphasis> | ||
38 | Provides details about particular pieces of software. | ||
39 | </para></listitem> | ||
40 | <listitem><para> | ||
41 | <emphasis>Class Data:</emphasis> | ||
42 | Abstracts common build information (e.g. how to build a | ||
43 | Linux kernel). | ||
44 | </para></listitem> | ||
45 | <listitem><para> | ||
46 | <emphasis>Configuration Data:</emphasis> | ||
47 | Defines machine-specific settings, policy decisions, and | ||
48 | so forth. | ||
49 | Configuration data acts as the glue to bind everything | ||
50 | together. | ||
51 | </para></listitem> | ||
52 | </itemizedlist> | ||
53 | </para> | ||
54 | |||
55 | <para> | ||
56 | BitBake knows how to combine multiple data sources together and | ||
57 | refers to each data source as a layer. | ||
58 | For information on layers, see the | ||
59 | "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" | ||
60 | section of the Yocto Project Development Tasks Manual. | ||
61 | </para> | ||
62 | |||
63 | <para> | ||
64 | Following are some brief details on these core components. | ||
65 | For additional information on how these components interact during | ||
66 | a build, see the | ||
67 | "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>" | ||
68 | section. | ||
69 | </para> | ||
70 | |||
71 | <section id='usingpoky-components-bitbake'> | ||
72 | <title>BitBake</title> | ||
73 | |||
74 | <para> | ||
75 | BitBake is the tool at the heart of the | ||
76 | <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> | ||
77 | and is responsible for parsing the | ||
78 | <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>, | ||
79 | generating a list of tasks from it, and then executing those | ||
80 | tasks. | ||
81 | </para> | ||
82 | |||
83 | <para> | ||
84 | This section briefly introduces BitBake. | ||
85 | If you want more information on BitBake, see the | ||
86 | <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>. | ||
87 | </para> | ||
88 | |||
89 | <para> | ||
90 | To see a list of the options BitBake supports, use either of | ||
91 | the following commands: | ||
92 | <literallayout class='monospaced'> | ||
93 | $ bitbake -h | ||
94 | $ bitbake --help | ||
95 | </literallayout> | ||
96 | </para> | ||
97 | |||
98 | <para> | ||
99 | The most common usage for BitBake is | ||
100 | <filename>bitbake <replaceable>packagename</replaceable></filename>, | ||
101 | where <filename>packagename</filename> is the name of the | ||
102 | package you want to build (referred to as the "target"). | ||
103 | The target often equates to the first part of a recipe's | ||
104 | filename (e.g. "foo" for a recipe named | ||
105 | <filename>foo_1.3.0-r0.bb</filename>). | ||
106 | So, to process the | ||
107 | <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you | ||
108 | might type the following: | ||
109 | <literallayout class='monospaced'> | ||
110 | $ bitbake matchbox-desktop | ||
111 | </literallayout> | ||
112 | Several different versions of | ||
113 | <filename>matchbox-desktop</filename> might exist. | ||
114 | BitBake chooses the one selected by the distribution | ||
115 | configuration. | ||
116 | You can get more details about how BitBake chooses between | ||
117 | different target versions and providers in the | ||
118 | "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>" | ||
119 | section of the BitBake User Manual. | ||
120 | </para> | ||
121 | |||
122 | <para> | ||
123 | BitBake also tries to execute any dependent tasks first. | ||
124 | So for example, before building | ||
125 | <filename>matchbox-desktop</filename>, BitBake would build a | ||
126 | cross compiler and <filename>glibc</filename> if they had not | ||
127 | already been built. | ||
128 | </para> | ||
129 | |||
130 | <para> | ||
131 | A useful BitBake option to consider is the | ||
132 | <filename>-k</filename> or <filename>--continue</filename> | ||
133 | option. | ||
134 | This option instructs BitBake to try and continue processing | ||
135 | the job as long as possible even after encountering an error. | ||
136 | When an error occurs, the target that failed and those that | ||
137 | depend on it cannot be remade. | ||
138 | However, when you use this option other dependencies can | ||
139 | still be processed. | ||
140 | </para> | ||
141 | </section> | ||
142 | |||
143 | <section id='overview-components-recipes'> | ||
144 | <title>Recipes</title> | ||
145 | |||
146 | <para> | ||
147 | Files that have the <filename>.bb</filename> suffix are | ||
148 | "recipes" files. | ||
149 | In general, a recipe contains information about a single piece | ||
150 | of software. | ||
151 | This information includes the location from which to download | ||
152 | the unaltered source, any source patches to be applied to that | ||
153 | source (if needed), which special configuration options to | ||
154 | apply, how to compile the source files, and how to package the | ||
155 | compiled output. | ||
156 | </para> | ||
157 | |||
158 | <para> | ||
159 | The term "package" is sometimes used to refer to recipes. | ||
160 | However, since the word "package" is used for the packaged | ||
161 | output from the OpenEmbedded build system (i.e. | ||
162 | <filename>.ipk</filename> or <filename>.deb</filename> files), | ||
163 | this document avoids using the term "package" when referring | ||
164 | to recipes. | ||
165 | </para> | ||
166 | </section> | ||
167 | |||
168 | <section id='overview-components-classes'> | ||
169 | <title>Classes</title> | ||
170 | |||
171 | <para> | ||
172 | Class files (<filename>.bbclass</filename>) contain information | ||
173 | that is useful to share between recipes files. | ||
174 | An example is the | ||
175 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> | ||
176 | class, which contains common settings for any application that | ||
177 | Autotools uses. | ||
178 | The | ||
179 | "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>" | ||
180 | chapter in the Yocto Project Reference Manual provides | ||
181 | details about classes and how to use them. | ||
182 | </para> | ||
183 | </section> | ||
184 | |||
185 | <section id='overview-components-configurations'> | ||
186 | <title>Configurations</title> | ||
187 | |||
188 | <para> | ||
189 | The configuration files (<filename>.conf</filename>) define | ||
190 | various configuration variables that govern the OpenEmbedded | ||
191 | build process. | ||
192 | These files fall into several areas that define machine | ||
193 | configuration options, distribution configuration options, | ||
194 | compiler tuning options, general common configuration options, | ||
195 | and user configuration options in | ||
196 | <filename>conf/local.conf</filename>, which is found in the | ||
197 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. | ||
198 | </para> | ||
199 | </section> | ||
200 | </section> | ||
201 | |||
202 | <section id='overview-layers'> | ||
203 | <title>Layers</title> | ||
204 | |||
205 | <para> | ||
206 | Layers are repositories that contain related metadata (i.e. | ||
207 | sets of instructions) that tell the OpenEmbedded build system how | ||
208 | to build a target. | ||
209 | Yocto Project's | ||
210 | <link linkend='the-yocto-project-layer-model'>layer model</link> | ||
211 | facilitates collaboration, sharing, customization, and reuse | ||
212 | within the Yocto Project development environment. | ||
213 | Layers logically separate information for your project. | ||
214 | For example, you can use a layer to hold all the configurations | ||
215 | for a particular piece of hardware. | ||
216 | Isolating hardware-specific configurations allows you to share | ||
217 | other metadata by using a different layer where that metadata | ||
218 | might be common across several pieces of hardware. | ||
219 | </para> | ||
220 | |||
221 | <para> | ||
222 | Many layers exist that work in the Yocto Project development | ||
223 | environment. | ||
224 | The | ||
225 | <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project Curated Layer Index</ulink> | ||
226 | and | ||
227 | <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink> | ||
228 | both contain layers from which you can use or leverage. | ||
229 | </para> | ||
230 | |||
231 | <para> | ||
232 | By convention, layers in the Yocto Project follow a specific form. | ||
233 | Conforming to a known structure allows BitBake to make assumptions | ||
234 | during builds on where to find types of metadata. | ||
235 | You can find procedures and learn about tools (i.e. | ||
236 | <filename>bitbake-layers</filename>) for creating layers suitable | ||
237 | for the Yocto Project in the | ||
238 | "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" | ||
239 | section of the Yocto Project Development Tasks Manual. | ||
240 | </para> | ||
241 | </section> | ||
242 | |||
243 | <section id="openembedded-build-system-build-concepts"> | ||
244 | <title>OpenEmbedded Build System Concepts</title> | ||
245 | |||
246 | <para> | ||
247 | This section takes a more detailed look inside the build | ||
248 | process used by the | ||
249 | <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>, | ||
250 | which is the build system specific to the Yocto Project. | ||
251 | At the heart of the build system is BitBake, the task executor. | ||
252 | </para> | ||
253 | |||
254 | <para> | ||
255 | The following diagram represents the high-level workflow of a | ||
256 | build. | ||
257 | The remainder of this section expands on the fundamental input, | ||
258 | output, process, and metadata logical blocks that make up the | ||
259 | workflow. | ||
260 | </para> | ||
261 | |||
262 | <para id='general-workflow-figure'> | ||
263 | <imagedata fileref="figures/YP-flow-diagram.png" format="PNG" align='center' width="8in"/> | ||
264 | </para> | ||
265 | |||
266 | <para> | ||
267 | In general, the build's workflow consists of several functional | ||
268 | areas: | ||
269 | <itemizedlist> | ||
270 | <listitem><para> | ||
271 | <emphasis>User Configuration:</emphasis> | ||
272 | metadata you can use to control the build process. | ||
273 | </para></listitem> | ||
274 | <listitem><para> | ||
275 | <emphasis>Metadata Layers:</emphasis> | ||
276 | Various layers that provide software, machine, and | ||
277 | distro metadata. | ||
278 | </para></listitem> | ||
279 | <listitem><para> | ||
280 | <emphasis>Source Files:</emphasis> | ||
281 | Upstream releases, local projects, and SCMs. | ||
282 | </para></listitem> | ||
283 | <listitem><para> | ||
284 | <emphasis>Build System:</emphasis> | ||
285 | Processes under the control of | ||
286 | <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>. | ||
287 | This block expands on how BitBake fetches source, applies | ||
288 | patches, completes compilation, analyzes output for package | ||
289 | generation, creates and tests packages, generates images, | ||
290 | and generates cross-development tools. | ||
291 | </para></listitem> | ||
292 | <listitem><para> | ||
293 | <emphasis>Package Feeds:</emphasis> | ||
294 | Directories containing output packages (RPM, DEB or IPK), | ||
295 | which are subsequently used in the construction of an | ||
296 | image or Software Development Kit (SDK), produced by the | ||
297 | build system. | ||
298 | These feeds can also be copied and shared using a web | ||
299 | server or other means to facilitate extending or updating | ||
300 | existing images on devices at runtime if runtime package | ||
301 | management is enabled. | ||
302 | </para></listitem> | ||
303 | <listitem><para> | ||
304 | <emphasis>Images:</emphasis> | ||
305 | Images produced by the workflow. | ||
306 | </para></listitem> | ||
307 | <listitem><para> | ||
308 | <emphasis>Application Development SDK:</emphasis> | ||
309 | Cross-development tools that are produced along with | ||
310 | an image or separately with BitBake. | ||
311 | </para></listitem> | ||
312 | </itemizedlist> | ||
313 | </para> | ||
314 | |||
315 | <section id="user-configuration"> | ||
316 | <title>User Configuration</title> | ||
317 | |||
318 | <para> | ||
319 | User configuration helps define the build. | ||
320 | Through user configuration, you can tell BitBake the | ||
321 | target architecture for which you are building the image, | ||
322 | where to store downloaded source, and other build properties. | ||
323 | </para> | ||
324 | |||
325 | <para> | ||
326 | The following figure shows an expanded representation of the | ||
327 | "User Configuration" box of the | ||
328 | <link linkend='general-workflow-figure'>general workflow figure</link>: | ||
329 | </para> | ||
330 | |||
331 | <para> | ||
332 | <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" /> | ||
333 | </para> | ||
334 | |||
335 | <para> | ||
336 | BitBake needs some basic configuration files in order to | ||
337 | complete a build. | ||
338 | These files are <filename>*.conf</filename> files. | ||
339 | The minimally necessary ones reside as example files in the | ||
340 | <filename>build/conf</filename> directory of the | ||
341 | <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. | ||
342 | For simplicity, this section refers to the Source Directory as | ||
343 | the "Poky Directory." | ||
344 | </para> | ||
345 | |||
346 | <para> | ||
347 | When you clone the | ||
348 | <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink> | ||
349 | Git repository or you download and unpack a Yocto Project | ||
350 | release, you can set up the Source Directory to be named | ||
351 | anything you want. | ||
352 | For this discussion, the cloned repository uses the default | ||
353 | name <filename>poky</filename>. | ||
354 | <note> | ||
355 | The Poky repository is primarily an aggregation of existing | ||
356 | repositories. | ||
357 | It is not a canonical upstream source. | ||
358 | </note> | ||
359 | </para> | ||
360 | |||
361 | <para> | ||
362 | The <filename>meta-poky</filename> layer inside Poky contains | ||
363 | a <filename>conf</filename> directory that has example | ||
364 | configuration files. | ||
365 | These example files are used as a basis for creating actual | ||
366 | configuration files when you source | ||
367 | <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>, | ||
368 | which is the build environment script. | ||
369 | </para> | ||
370 | |||
371 | <para> | ||
372 | Sourcing the build environment script creates a | ||
373 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> | ||
374 | if one does not already exist. | ||
375 | BitBake uses the Build Directory for all its work during | ||
376 | builds. | ||
377 | The Build Directory has a <filename>conf</filename> directory | ||
378 | that contains default versions of your | ||
379 | <filename>local.conf</filename> and | ||
380 | <filename>bblayers.conf</filename> configuration files. | ||
381 | These default configuration files are created only if versions | ||
382 | do not already exist in the Build Directory at the time you | ||
383 | source the build environment setup script. | ||
384 | </para> | ||
385 | |||
386 | <para> | ||
387 | Because the Poky repository is fundamentally an aggregation of | ||
388 | existing repositories, some users might be familiar with | ||
389 | running the <filename>&OE_INIT_FILE;</filename> script | ||
390 | in the context of separate | ||
391 | <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink> | ||
392 | and BitBake repositories rather than a single Poky repository. | ||
393 | This discussion assumes the script is executed from | ||
394 | within a cloned or unpacked version of Poky. | ||
395 | </para> | ||
396 | |||
397 | <para> | ||
398 | Depending on where the script is sourced, different | ||
399 | sub-scripts are called to set up the Build Directory | ||
400 | (Yocto or OpenEmbedded). | ||
401 | Specifically, the script | ||
402 | <filename>scripts/oe-setup-builddir</filename> inside the | ||
403 | poky directory sets up the Build Directory and seeds the | ||
404 | directory (if necessary) with configuration files appropriate | ||
405 | for the Yocto Project development environment. | ||
406 | <note> | ||
407 | The <filename>scripts/oe-setup-builddir</filename> script | ||
408 | uses the <filename>$TEMPLATECONF</filename> variable to | ||
409 | determine which sample configuration files to locate. | ||
410 | </note> | ||
411 | </para> | ||
412 | |||
413 | <para> | ||
414 | The <filename>local.conf</filename> file provides many | ||
415 | basic variables that define a build environment. | ||
416 | Here is a list of a few. | ||
417 | To see the default configurations in a | ||
418 | <filename>local.conf</filename> file created by the build | ||
419 | environment script, see the | ||
420 | <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample'><filename>local.conf.sample</filename></ulink> | ||
421 | in the <filename>meta-poky</filename> layer: | ||
422 | <itemizedlist> | ||
423 | <listitem><para> | ||
424 | <emphasis>Target Machine Selection:</emphasis> | ||
425 | Controlled by the | ||
426 | <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> | ||
427 | variable. | ||
428 | </para></listitem> | ||
429 | <listitem><para> | ||
430 | <emphasis>Download Directory:</emphasis> | ||
431 | Controlled by the | ||
432 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> | ||
433 | variable. | ||
434 | </para></listitem> | ||
435 | <listitem><para> | ||
436 | <emphasis>Shared State Directory:</emphasis> | ||
437 | Controlled by the | ||
438 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> | ||
439 | variable. | ||
440 | </para></listitem> | ||
441 | <listitem><para> | ||
442 | <emphasis>Build Output:</emphasis> | ||
443 | Controlled by the | ||
444 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> | ||
445 | variable. | ||
446 | </para></listitem> | ||
447 | <listitem><para> | ||
448 | <emphasis>Distribution Policy:</emphasis> | ||
449 | Controlled by the | ||
450 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> | ||
451 | variable. | ||
452 | </para></listitem> | ||
453 | <listitem><para> | ||
454 | <emphasis>Packaging Format:</emphasis> | ||
455 | Controlled by the | ||
456 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> | ||
457 | variable. | ||
458 | </para></listitem> | ||
459 | <listitem><para> | ||
460 | <emphasis>SDK Target Architecture:</emphasis> | ||
461 | Controlled by the | ||
462 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink> | ||
463 | variable. | ||
464 | </para></listitem> | ||
465 | <listitem><para> | ||
466 | <emphasis>Extra Image Packages:</emphasis> | ||
467 | Controlled by the | ||
468 | <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> | ||
469 | variable. | ||
470 | </para></listitem> | ||
471 | </itemizedlist> | ||
472 | <note> | ||
473 | Configurations set in the | ||
474 | <filename>conf/local.conf</filename> file can also be set | ||
475 | in the <filename>conf/site.conf</filename> and | ||
476 | <filename>conf/auto.conf</filename> configuration files. | ||
477 | </note> | ||
478 | </para> | ||
479 | |||
480 | <para> | ||
481 | The <filename>bblayers.conf</filename> file tells BitBake what | ||
482 | layers you want considered during the build. | ||
483 | By default, the layers listed in this file include layers | ||
484 | minimally needed by the build system. | ||
485 | However, you must manually add any custom layers you have | ||
486 | created. | ||
487 | You can find more information on working with the | ||
488 | <filename>bblayers.conf</filename> file in the | ||
489 | "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>" | ||
490 | section in the Yocto Project Development Tasks Manual. | ||
491 | </para> | ||
492 | |||
493 | <para> | ||
494 | The files <filename>site.conf</filename> and | ||
495 | <filename>auto.conf</filename> are not created by the | ||
496 | environment initialization script. | ||
497 | If you want the <filename>site.conf</filename> file, you | ||
498 | need to create that yourself. | ||
499 | The <filename>auto.conf</filename> file is typically created by | ||
500 | an autobuilder: | ||
501 | <itemizedlist> | ||
502 | <listitem><para> | ||
503 | <emphasis><filename>site.conf</filename>:</emphasis> | ||
504 | You can use the <filename>conf/site.conf</filename> | ||
505 | configuration file to configure multiple | ||
506 | build directories. | ||
507 | For example, suppose you had several build environments | ||
508 | and they shared some common features. | ||
509 | You can set these default build properties here. | ||
510 | A good example is perhaps the packaging format to use | ||
511 | through the | ||
512 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> | ||
513 | variable.</para> | ||
514 | |||
515 | <para>One useful scenario for using the | ||
516 | <filename>conf/site.conf</filename> file is to extend | ||
517 | your | ||
518 | <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> | ||
519 | variable to include the path to a | ||
520 | <filename>conf/site.conf</filename>. | ||
521 | Then, when BitBake looks for Metadata using | ||
522 | <filename>BBPATH</filename>, it finds the | ||
523 | <filename>conf/site.conf</filename> file and applies | ||
524 | your common configurations found in the file. | ||
525 | To override configurations in a particular build | ||
526 | directory, alter the similar configurations within | ||
527 | that build directory's | ||
528 | <filename>conf/local.conf</filename> file. | ||
529 | </para></listitem> | ||
530 | <listitem><para> | ||
531 | <emphasis><filename>auto.conf</filename>:</emphasis> | ||
532 | The file is usually created and written to by | ||
533 | an autobuilder. | ||
534 | The settings put into the file are typically the | ||
535 | same as you would find in the | ||
536 | <filename>conf/local.conf</filename> or the | ||
537 | <filename>conf/site.conf</filename> files. | ||
538 | </para></listitem> | ||
539 | </itemizedlist> | ||
540 | </para> | ||
541 | |||
542 | <para> | ||
543 | You can edit all configuration files to further define | ||
544 | any particular build environment. | ||
545 | This process is represented by the "User Configuration Edits" | ||
546 | box in the figure. | ||
547 | </para> | ||
548 | |||
549 | <para> | ||
550 | When you launch your build with the | ||
551 | <filename>bitbake <replaceable>target</replaceable></filename> | ||
552 | command, BitBake sorts out the configurations to ultimately | ||
553 | define your build environment. | ||
554 | It is important to understand that the | ||
555 | <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> | ||
556 | reads the configuration files in a specific order: | ||
557 | <filename>site.conf</filename>, <filename>auto.conf</filename>, | ||
558 | and <filename>local.conf</filename>. | ||
559 | And, the build system applies the normal assignment statement | ||
560 | rules as described in the | ||
561 | "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>" | ||
562 | chapter of the BitBake User Manual. | ||
563 | Because the files are parsed in a specific order, variable | ||
564 | assignments for the same variable could be affected. | ||
565 | For example, if the <filename>auto.conf</filename> file and | ||
566 | the <filename>local.conf</filename> set | ||
567 | <replaceable>variable1</replaceable> to different values, | ||
568 | because the build system parses <filename>local.conf</filename> | ||
569 | after <filename>auto.conf</filename>, | ||
570 | <replaceable>variable1</replaceable> is assigned the value from | ||
571 | the <filename>local.conf</filename> file. | ||
572 | </para> | ||
573 | </section> | ||
574 | |||
575 | <section id="metadata-machine-configuration-and-policy-configuration"> | ||
576 | <title>Metadata, Machine Configuration, and Policy Configuration</title> | ||
577 | |||
578 | <para> | ||
579 | The previous section described the user configurations that | ||
580 | define BitBake's global behavior. | ||
581 | This section takes a closer look at the layers the build system | ||
582 | uses to further control the build. | ||
583 | These layers provide Metadata for the software, machine, and | ||
584 | policies. | ||
585 | </para> | ||
586 | |||
587 | <para> | ||
588 | In general, three types of layer input exists. | ||
589 | You can see them below the "User Configuration" box in the | ||
590 | <link linkend='general-workflow-figure'>general workflow figure</link>: | ||
591 | <itemizedlist> | ||
592 | <listitem><para> | ||
593 | <emphasis>Metadata (<filename>.bb</filename> + Patches):</emphasis> | ||
594 | Software layers containing user-supplied recipe files, | ||
595 | patches, and append files. | ||
596 | A good example of a software layer might be the | ||
597 | <ulink url='https://github.com/meta-qt5/meta-qt5'><filename>meta-qt5</filename></ulink> | ||
598 | layer from the | ||
599 | <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>. | ||
600 | This layer is for version 5.0 of the popular | ||
601 | <ulink url='https://wiki.qt.io/About_Qt'>Qt</ulink> | ||
602 | cross-platform application development framework for | ||
603 | desktop, embedded and mobile. | ||
604 | </para></listitem> | ||
605 | <listitem><para> | ||
606 | <emphasis>Machine BSP Configuration:</emphasis> | ||
607 | Board Support Package (BSP) layers (i.e. "BSP Layer" | ||
608 | in the following figure) providing machine-specific | ||
609 | configurations. | ||
610 | This type of information is specific to a particular | ||
611 | target architecture. | ||
612 | A good example of a BSP layer from the | ||
613 | <link linkend='gs-reference-distribution-poky'>Poky Reference Distribution</link> | ||
614 | is the | ||
615 | <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink> | ||
616 | layer. | ||
617 | </para></listitem> | ||
618 | <listitem><para> | ||
619 | <emphasis>Policy Configuration:</emphasis> | ||
620 | Distribution Layers (i.e. "Distro Layer" in the | ||
621 | following figure) providing top-level or general | ||
622 | policies for the images or SDKs being built for a | ||
623 | particular distribution. | ||
624 | For example, in the Poky Reference Distribution the | ||
625 | distro layer is the | ||
626 | <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink> | ||
627 | layer. | ||
628 | Within the distro layer is a | ||
629 | <filename>conf/distro</filename> directory that | ||
630 | contains distro configuration files (e.g. | ||
631 | <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink> | ||
632 | that contain many policy configurations for the | ||
633 | Poky distribution. | ||
634 | </para></listitem> | ||
635 | </itemizedlist> | ||
636 | </para> | ||
637 | |||
638 | <para> | ||
639 | The following figure shows an expanded representation of | ||
640 | these three layers from the | ||
641 | <link linkend='general-workflow-figure'>general workflow figure</link>: | ||
642 | </para> | ||
643 | |||
644 | <para> | ||
645 | <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="8in" /> | ||
646 | </para> | ||
647 | |||
648 | <para> | ||
649 | In general, all layers have a similar structure. | ||
650 | They all contain a licensing file | ||
651 | (e.g. <filename>COPYING.MIT</filename>) if the layer is to be | ||
652 | distributed, a <filename>README</filename> file as good | ||
653 | practice and especially if the layer is to be distributed, a | ||
654 | configuration directory, and recipe directories. | ||
655 | You can learn about the general structure for layers used with | ||
656 | the Yocto Project in the | ||
657 | "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-your-own-layer'>Creating Your Own Layer</ulink>" | ||
658 | section in the Yocto Project Development Tasks Manual. | ||
659 | For a general discussion on layers and the many layers from | ||
660 | which you can draw, see the | ||
661 | "<link linkend='overview-layers'>Layers</link>" and | ||
662 | "<link linkend='the-yocto-project-layer-model'>The Yocto Project Layer Model</link>" | ||
663 | sections both earlier in this manual. | ||
664 | </para> | ||
665 | |||
666 | <para> | ||
667 | If you explored the previous links, you discovered some | ||
668 | areas where many layers that work with the Yocto Project | ||
669 | exist. | ||
670 | The | ||
671 | <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink> | ||
672 | also shows layers categorized under "Yocto Metadata Layers." | ||
673 | <note> | ||
674 | Layers exist in the Yocto Project Source Repositories that | ||
675 | cannot be found in the OpenEmbedded Layer Index. | ||
676 | These layers are either deprecated or experimental | ||
677 | in nature. | ||
678 | </note> | ||
679 | </para> | ||
680 | |||
681 | <para> | ||
682 | BitBake uses the <filename>conf/bblayers.conf</filename> file, | ||
683 | which is part of the user configuration, to find what layers it | ||
684 | should be using as part of the build. | ||
685 | </para> | ||
686 | |||
687 | <section id="distro-layer"> | ||
688 | <title>Distro Layer</title> | ||
689 | |||
690 | <para> | ||
691 | The distribution layer provides policy configurations for | ||
692 | your distribution. | ||
693 | Best practices dictate that you isolate these types of | ||
694 | configurations into their own layer. | ||
695 | Settings you provide in | ||
696 | <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override | ||
697 | similar settings that BitBake finds in your | ||
698 | <filename>conf/local.conf</filename> file in the Build | ||
699 | Directory. | ||
700 | </para> | ||
701 | |||
702 | <para> | ||
703 | The following list provides some explanation and references | ||
704 | for what you typically find in the distribution layer: | ||
705 | <itemizedlist> | ||
706 | <listitem><para> | ||
707 | <emphasis>classes:</emphasis> | ||
708 | Class files (<filename>.bbclass</filename>) hold | ||
709 | common functionality that can be shared among | ||
710 | recipes in the distribution. | ||
711 | When your recipes inherit a class, they take on the | ||
712 | settings and functions for that class. | ||
713 | You can read more about class files in the | ||
714 | "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>" | ||
715 | chapter of the Yocto Reference Manual. | ||
716 | </para></listitem> | ||
717 | <listitem><para> | ||
718 | <emphasis>conf:</emphasis> | ||
719 | This area holds configuration files for the | ||
720 | layer (<filename>conf/layer.conf</filename>), | ||
721 | the distribution | ||
722 | (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>), | ||
723 | and any distribution-wide include files. | ||
724 | </para></listitem> | ||
725 | <listitem><para> | ||
726 | <emphasis>recipes-*:</emphasis> | ||
727 | Recipes and append files that affect common | ||
728 | functionality across the distribution. | ||
729 | This area could include recipes and append files | ||
730 | to add distribution-specific configuration, | ||
731 | initialization scripts, custom image recipes, | ||
732 | and so forth. | ||
733 | Examples of <filename>recipes-*</filename> | ||
734 | directories are <filename>recipes-core</filename> | ||
735 | and <filename>recipes-extra</filename>. | ||
736 | Hierarchy and contents within a | ||
737 | <filename>recipes-*</filename> directory can vary. | ||
738 | Generally, these directories contain recipe files | ||
739 | (<filename>*.bb</filename>), recipe append files | ||
740 | (<filename>*.bbappend</filename>), directories | ||
741 | that are distro-specific for configuration files, | ||
742 | and so forth. | ||
743 | </para></listitem> | ||
744 | </itemizedlist> | ||
745 | </para> | ||
746 | </section> | ||
747 | |||
748 | <section id="bsp-layer"> | ||
749 | <title>BSP Layer</title> | ||
750 | |||
751 | <para> | ||
752 | The BSP Layer provides machine configurations that | ||
753 | target specific hardware. | ||
754 | Everything in this layer is specific to the machine for | ||
755 | which you are building the image or the SDK. | ||
756 | A common structure or form is defined for BSP layers. | ||
757 | You can learn more about this structure in the | ||
758 | <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>. | ||
759 | <note> | ||
760 | In order for a BSP layer to be considered compliant | ||
761 | with the Yocto Project, it must meet some structural | ||
762 | requirements. | ||
763 | </note> | ||
764 | </para> | ||
765 | |||
766 | <para> | ||
767 | The BSP Layer's configuration directory contains | ||
768 | configuration files for the machine | ||
769 | (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>) | ||
770 | and, of course, the layer | ||
771 | (<filename>conf/layer.conf</filename>). | ||
772 | </para> | ||
773 | |||
774 | <para> | ||
775 | The remainder of the layer is dedicated to specific recipes | ||
776 | by function: <filename>recipes-bsp</filename>, | ||
777 | <filename>recipes-core</filename>, | ||
778 | <filename>recipes-graphics</filename>, | ||
779 | <filename>recipes-kernel</filename>, and so forth. | ||
780 | Metadata can exist for multiple formfactors, graphics | ||
781 | support systems, and so forth. | ||
782 | <note> | ||
783 | While the figure shows several | ||
784 | <filename>recipes-*</filename> directories, not all | ||
785 | these directories appear in all BSP layers. | ||
786 | </note> | ||
787 | </para> | ||
788 | </section> | ||
789 | |||
790 | <section id="software-layer"> | ||
791 | <title>Software Layer</title> | ||
792 | |||
793 | <para> | ||
794 | The software layer provides the Metadata for additional | ||
795 | software packages used during the build. | ||
796 | This layer does not include Metadata that is specific to | ||
797 | the distribution or the machine, which are found in their | ||
798 | respective layers. | ||
799 | </para> | ||
800 | |||
801 | <para> | ||
802 | This layer contains any recipes, append files, and | ||
803 | patches, that your project needs. | ||
804 | </para> | ||
805 | </section> | ||
806 | </section> | ||
807 | |||
808 | <section id="sources-dev-environment"> | ||
809 | <title>Sources</title> | ||
810 | |||
811 | <para> | ||
812 | In order for the OpenEmbedded build system to create an | ||
813 | image or any target, it must be able to access source files. | ||
814 | The | ||
815 | <link linkend='general-workflow-figure'>general workflow figure</link> | ||
816 | represents source files using the "Upstream Project Releases", | ||
817 | "Local Projects", and "SCMs (optional)" boxes. | ||
818 | The figure represents mirrors, which also play a role in | ||
819 | locating source files, with the "Source Materials" box. | ||
820 | </para> | ||
821 | |||
822 | <para> | ||
823 | The method by which source files are ultimately organized is | ||
824 | a function of the project. | ||
825 | For example, for released software, projects tend to use | ||
826 | tarballs or other archived files that can capture the | ||
827 | state of a release guaranteeing that it is statically | ||
828 | represented. | ||
829 | On the other hand, for a project that is more dynamic or | ||
830 | experimental in nature, a project might keep source files in a | ||
831 | repository controlled by a Source Control Manager (SCM) such as | ||
832 | Git. | ||
833 | Pulling source from a repository allows you to control | ||
834 | the point in the repository (the revision) from which you | ||
835 | want to build software. | ||
836 | Finally, a combination of the two might exist, which would | ||
837 | give the consumer a choice when deciding where to get | ||
838 | source files. | ||
839 | </para> | ||
840 | |||
841 | <para> | ||
842 | BitBake uses the | ||
843 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> | ||
844 | variable to point to source files regardless of their location. | ||
845 | Each recipe must have a <filename>SRC_URI</filename> variable | ||
846 | that points to the source. | ||
847 | </para> | ||
848 | |||
849 | <para> | ||
850 | Another area that plays a significant role in where source | ||
851 | files come from is pointed to by the | ||
852 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> | ||
853 | variable. | ||
854 | This area is a cache that can hold previously downloaded | ||
855 | source. | ||
856 | You can also instruct the OpenEmbedded build system to create | ||
857 | tarballs from Git repositories, which is not the default | ||
858 | behavior, and store them in the <filename>DL_DIR</filename> | ||
859 | by using the | ||
860 | <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> | ||
861 | variable. | ||
862 | </para> | ||
863 | |||
864 | <para> | ||
865 | Judicious use of a <filename>DL_DIR</filename> directory can | ||
866 | save the build system a trip across the Internet when looking | ||
867 | for files. | ||
868 | A good method for using a download directory is to have | ||
869 | <filename>DL_DIR</filename> point to an area outside of your | ||
870 | Build Directory. | ||
871 | Doing so allows you to safely delete the Build Directory | ||
872 | if needed without fear of removing any downloaded source file. | ||
873 | </para> | ||
874 | |||
875 | <para> | ||
876 | The remainder of this section provides a deeper look into the | ||
877 | source files and the mirrors. | ||
878 | Here is a more detailed look at the source file area of the | ||
879 | <link linkend='general-workflow-figure'>general workflow figure</link>: | ||
880 | </para> | ||
881 | |||
882 | <para> | ||
883 | <imagedata fileref="figures/source-input.png" width="6in" depth="6in" align="center" /> | ||
884 | </para> | ||
885 | |||
886 | <section id='upstream-project-releases'> | ||
887 | <title>Upstream Project Releases</title> | ||
888 | |||
889 | <para> | ||
890 | Upstream project releases exist anywhere in the form of an | ||
891 | archived file (e.g. tarball or zip file). | ||
892 | These files correspond to individual recipes. | ||
893 | For example, the figure uses specific releases each for | ||
894 | BusyBox, Qt, and Dbus. | ||
895 | An archive file can be for any released product that can be | ||
896 | built using a recipe. | ||
897 | </para> | ||
898 | </section> | ||
899 | |||
900 | <section id='local-projects'> | ||
901 | <title>Local Projects</title> | ||
902 | |||
903 | <para> | ||
904 | Local projects are custom bits of software the user | ||
905 | provides. | ||
906 | These bits reside somewhere local to a project - perhaps | ||
907 | a directory into which the user checks in items (e.g. | ||
908 | a local directory containing a development source tree | ||
909 | used by the group). | ||
910 | </para> | ||
911 | |||
912 | <para> | ||
913 | The canonical method through which to include a local | ||
914 | project is to use the | ||
915 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> | ||
916 | class to include that local project. | ||
917 | You use either the <filename>local.conf</filename> or a | ||
918 | recipe's append file to override or set the | ||
919 | recipe to point to the local directory on your disk to pull | ||
920 | in the whole source tree. | ||
921 | </para> | ||
922 | </section> | ||
923 | |||
924 | <section id='scms'> | ||
925 | <title>Source Control Managers (Optional)</title> | ||
926 | |||
927 | <para> | ||
928 | Another place from which the build system can get source | ||
929 | files is with | ||
930 | <ulink url='&YOCTO_DOCS_BB_URL;#bb-fetchers'>fetchers</ulink> | ||
931 | employing various Source Control Managers (SCMs) such as | ||
932 | Git or Subversion. | ||
933 | In such cases, a repository is cloned or checked out. | ||
934 | The | ||
935 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> | ||
936 | task inside BitBake uses | ||
937 | the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> | ||
938 | variable and the argument's prefix to determine the correct | ||
939 | fetcher module. | ||
940 | <note> | ||
941 | For information on how to have the OpenEmbedded build | ||
942 | system generate tarballs for Git repositories and place | ||
943 | them in the | ||
944 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> | ||
945 | directory, see the | ||
946 | <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> | ||
947 | variable in the Yocto Project Reference Manual. | ||
948 | </note> | ||
949 | </para> | ||
950 | |||
951 | <para> | ||
952 | When fetching a repository, BitBake uses the | ||
953 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> | ||
954 | variable to determine the specific revision from which to | ||
955 | build. | ||
956 | </para> | ||
957 | </section> | ||
958 | |||
959 | <section id='source-mirrors'> | ||
960 | <title>Source Mirror(s)</title> | ||
961 | |||
962 | <para> | ||
963 | Two kinds of mirrors exist: pre-mirrors and regular | ||
964 | mirrors. | ||
965 | The | ||
966 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink> | ||
967 | and | ||
968 | <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink> | ||
969 | variables point to these, respectively. | ||
970 | BitBake checks pre-mirrors before looking upstream for any | ||
971 | source files. | ||
972 | Pre-mirrors are appropriate when you have a shared | ||
973 | directory that is not a directory defined by the | ||
974 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> | ||
975 | variable. | ||
976 | A Pre-mirror typically points to a shared directory that is | ||
977 | local to your organization. | ||
978 | </para> | ||
979 | |||
980 | <para> | ||
981 | Regular mirrors can be any site across the Internet | ||
982 | that is used as an alternative location for source | ||
983 | code should the primary site not be functioning for | ||
984 | some reason or another. | ||
985 | </para> | ||
986 | </section> | ||
987 | </section> | ||
988 | |||
989 | <section id="package-feeds-dev-environment"> | ||
990 | <title>Package Feeds</title> | ||
991 | |||
992 | <para> | ||
993 | When the OpenEmbedded build system generates an image or an | ||
994 | SDK, it gets the packages from a package feed area located | ||
995 | in the | ||
996 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. | ||
997 | The | ||
998 | <link linkend='general-workflow-figure'>general workflow figure</link> | ||
999 | shows this package feeds area in the upper-right corner. | ||
1000 | </para> | ||
1001 | |||
1002 | <para> | ||
1003 | This section looks a little closer into the package feeds | ||
1004 | area used by the build system. | ||
1005 | Here is a more detailed look at the area: | ||
1006 | <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" /> | ||
1007 | </para> | ||
1008 | |||
1009 | <para> | ||
1010 | Package feeds are an intermediary step in the build process. | ||
1011 | The OpenEmbedded build system provides classes to generate | ||
1012 | different package types, and you specify which classes to | ||
1013 | enable through the | ||
1014 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> | ||
1015 | variable. | ||
1016 | Before placing the packages into package feeds, | ||
1017 | the build process validates them with generated output quality | ||
1018 | assurance checks through the | ||
1019 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink> | ||
1020 | class. | ||
1021 | </para> | ||
1022 | |||
1023 | <para> | ||
1024 | The package feed area resides in the Build Directory. | ||
1025 | The directory the build system uses to temporarily store | ||
1026 | packages is determined by a combination of variables and the | ||
1027 | particular package manager in use. | ||
1028 | See the "Package Feeds" box in the illustration and note the | ||
1029 | information to the right of that area. | ||
1030 | In particular, the following defines where package files are | ||
1031 | kept: | ||
1032 | <itemizedlist> | ||
1033 | <listitem><para> | ||
1034 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: | ||
1035 | Defined as <filename>tmp/deploy</filename> in the Build | ||
1036 | Directory. | ||
1037 | </para></listitem> | ||
1038 | <listitem><para> | ||
1039 | <filename>DEPLOY_DIR_*</filename>: | ||
1040 | Depending on the package manager used, the package type | ||
1041 | sub-folder. | ||
1042 | Given RPM, IPK, or DEB packaging and tarball creation, | ||
1043 | the | ||
1044 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>, | ||
1045 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>, | ||
1046 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>, | ||
1047 | or | ||
1048 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>, | ||
1049 | variables are used, respectively. | ||
1050 | </para></listitem> | ||
1051 | <listitem><para> | ||
1052 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>: | ||
1053 | Defines architecture-specific sub-folders. | ||
1054 | For example, packages could exist for the i586 or | ||
1055 | qemux86 architectures. | ||
1056 | </para></listitem> | ||
1057 | </itemizedlist> | ||
1058 | </para> | ||
1059 | |||
1060 | <para> | ||
1061 | BitBake uses the | ||
1062 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> | ||
1063 | tasks to generate packages and place them into the package | ||
1064 | holding area (e.g. <filename>do_package_write_ipk</filename> | ||
1065 | for IPK packages). | ||
1066 | See the | ||
1067 | "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>", | ||
1068 | "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>", | ||
1069 | "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>", | ||
1070 | and | ||
1071 | "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>" | ||
1072 | sections in the Yocto Project Reference Manual | ||
1073 | for additional information. | ||
1074 | As an example, consider a scenario where an IPK packaging | ||
1075 | manager is being used and package architecture support for | ||
1076 | both i586 and qemux86 exist. | ||
1077 | Packages for the i586 architecture are placed in | ||
1078 | <filename>build/tmp/deploy/ipk/i586</filename>, while packages | ||
1079 | for the qemux86 architecture are placed in | ||
1080 | <filename>build/tmp/deploy/ipk/qemux86</filename>. | ||
1081 | </para> | ||
1082 | </section> | ||
1083 | |||
1084 | <section id='bitbake-dev-environment'> | ||
1085 | <title>BitBake</title> | ||
1086 | |||
1087 | <para> | ||
1088 | The OpenEmbedded build system uses | ||
1089 | <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> | ||
1090 | to produce images and Software Development Kits (SDKs). | ||
1091 | You can see from the | ||
1092 | <link linkend='general-workflow-figure'>general workflow figure</link>, | ||
1093 | the BitBake area consists of several functional areas. | ||
1094 | This section takes a closer look at each of those areas. | ||
1095 | <note> | ||
1096 | Separate documentation exists for the BitBake tool. | ||
1097 | See the | ||
1098 | <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink> | ||
1099 | for reference material on BitBake. | ||
1100 | </note> | ||
1101 | </para> | ||
1102 | |||
1103 | <section id='source-fetching-dev-environment'> | ||
1104 | <title>Source Fetching</title> | ||
1105 | |||
1106 | <para> | ||
1107 | The first stages of building a recipe are to fetch and | ||
1108 | unpack the source code: | ||
1109 | <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" /> | ||
1110 | </para> | ||
1111 | |||
1112 | <para> | ||
1113 | The | ||
1114 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> | ||
1115 | and | ||
1116 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> | ||
1117 | tasks fetch the source files and unpack them into the | ||
1118 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. | ||
1119 | <note> | ||
1120 | For every local file (e.g. <filename>file://</filename>) | ||
1121 | that is part of a recipe's | ||
1122 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> | ||
1123 | statement, the OpenEmbedded build system takes a | ||
1124 | checksum of the file for the recipe and inserts the | ||
1125 | checksum into the signature for the | ||
1126 | <filename>do_fetch</filename> task. | ||
1127 | If any local file has been modified, the | ||
1128 | <filename>do_fetch</filename> task and all tasks that | ||
1129 | depend on it are re-executed. | ||
1130 | </note> | ||
1131 | By default, everything is accomplished in the Build | ||
1132 | Directory, which has a defined structure. | ||
1133 | For additional general information on the Build Directory, | ||
1134 | see the | ||
1135 | "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>" | ||
1136 | section in the Yocto Project Reference Manual. | ||
1137 | </para> | ||
1138 | |||
1139 | <para> | ||
1140 | Each recipe has an area in the Build Directory where the | ||
1141 | unpacked source code resides. | ||
1142 | The | ||
1143 | <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> | ||
1144 | variable points to this area for a recipe's unpacked source | ||
1145 | code. | ||
1146 | The name of that directory for any given recipe is defined | ||
1147 | from several different variables. | ||
1148 | The preceding figure and the following list describe | ||
1149 | the Build Directory's hierarchy: | ||
1150 | <itemizedlist> | ||
1151 | <listitem><para> | ||
1152 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>: | ||
1153 | The base directory where the OpenEmbedded build | ||
1154 | system performs all its work during the build. | ||
1155 | The default base directory is the | ||
1156 | <filename>tmp</filename> directory. | ||
1157 | </para></listitem> | ||
1158 | <listitem><para> | ||
1159 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>: | ||
1160 | The architecture of the built package or packages. | ||
1161 | Depending on the eventual destination of the | ||
1162 | package or packages (i.e. machine architecture, | ||
1163 | <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>, | ||
1164 | SDK, or specific machine), | ||
1165 | <filename>PACKAGE_ARCH</filename> varies. | ||
1166 | See the variable's description for details. | ||
1167 | </para></listitem> | ||
1168 | <listitem><para> | ||
1169 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>: | ||
1170 | The operating system of the target device. | ||
1171 | A typical value would be "linux" (e.g. | ||
1172 | "qemux86-poky-linux"). | ||
1173 | </para></listitem> | ||
1174 | <listitem><para> | ||
1175 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>: | ||
1176 | The name of the recipe used to build the package. | ||
1177 | This variable can have multiple meanings. | ||
1178 | However, when used in the context of input files, | ||
1179 | <filename>PN</filename> represents the the name | ||
1180 | of the recipe. | ||
1181 | </para></listitem> | ||
1182 | <listitem><para> | ||
1183 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>: | ||
1184 | The location where the OpenEmbedded build system | ||
1185 | builds a recipe (i.e. does the work to create the | ||
1186 | package). | ||
1187 | <itemizedlist> | ||
1188 | <listitem><para> | ||
1189 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: | ||
1190 | The version of the recipe used to build the | ||
1191 | package. | ||
1192 | </para></listitem> | ||
1193 | <listitem><para> | ||
1194 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>: | ||
1195 | The revision of the recipe used to build the | ||
1196 | package. | ||
1197 | </para></listitem> | ||
1198 | </itemizedlist> | ||
1199 | </para></listitem> | ||
1200 | <listitem><para> | ||
1201 | <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>: | ||
1202 | Contains the unpacked source files for a given | ||
1203 | recipe. | ||
1204 | <itemizedlist> | ||
1205 | <listitem><para> | ||
1206 | <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>: | ||
1207 | The name of the recipe used to build the | ||
1208 | package. | ||
1209 | The <filename>BPN</filename> variable is | ||
1210 | a version of the <filename>PN</filename> | ||
1211 | variable but with common prefixes and | ||
1212 | suffixes removed. | ||
1213 | </para></listitem> | ||
1214 | <listitem><para> | ||
1215 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: | ||
1216 | The version of the recipe used to build the | ||
1217 | package. | ||
1218 | </para></listitem> | ||
1219 | </itemizedlist> | ||
1220 | </para></listitem> | ||
1221 | </itemizedlist> | ||
1222 | <note> | ||
1223 | In the previous figure, notice that two sample | ||
1224 | hierarchies exist: one based on package architecture (i.e. | ||
1225 | <filename>PACKAGE_ARCH</filename>) and one based on a | ||
1226 | machine (i.e. <filename>MACHINE</filename>). | ||
1227 | The underlying structures are identical. | ||
1228 | The differentiator being what the OpenEmbedded build | ||
1229 | system is using as a build target (e.g. general | ||
1230 | architecture, a build host, an SDK, or a specific | ||
1231 | machine). | ||
1232 | </note> | ||
1233 | </para> | ||
1234 | </section> | ||
1235 | |||
1236 | <section id='patching-dev-environment'> | ||
1237 | <title>Patching</title> | ||
1238 | |||
1239 | <para> | ||
1240 | Once source code is fetched and unpacked, BitBake locates | ||
1241 | patch files and applies them to the source files: | ||
1242 | <imagedata fileref="figures/patching.png" align="center" width="7in" depth="6in" /> | ||
1243 | </para> | ||
1244 | |||
1245 | <para> | ||
1246 | The | ||
1247 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> | ||
1248 | task uses a recipe's | ||
1249 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> | ||
1250 | statements and the | ||
1251 | <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> | ||
1252 | variable to locate applicable patch files. | ||
1253 | </para> | ||
1254 | |||
1255 | <para> | ||
1256 | Default processing for patch files assumes the files have | ||
1257 | either <filename>*.patch</filename> or | ||
1258 | <filename>*.diff</filename> file types. | ||
1259 | You can use <filename>SRC_URI</filename> parameters to | ||
1260 | change the way the build system recognizes patch files. | ||
1261 | See the | ||
1262 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> | ||
1263 | task for more information. | ||
1264 | </para> | ||
1265 | |||
1266 | <para> | ||
1267 | BitBake finds and applies multiple patches for a single | ||
1268 | recipe in the order in which it locates the patches. | ||
1269 | The <filename>FILESPATH</filename> variable defines the | ||
1270 | default set of directories that the build system uses to | ||
1271 | search for patch files. | ||
1272 | Once found, patches are applied to the recipe's source | ||
1273 | files, which are located in the | ||
1274 | <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> | ||
1275 | directory. | ||
1276 | </para> | ||
1277 | |||
1278 | <para> | ||
1279 | For more information on how the source directories are | ||
1280 | created, see the | ||
1281 | "<link linkend='source-fetching-dev-environment'>Source Fetching</link>" | ||
1282 | section. | ||
1283 | For more information on how to create patches and how the | ||
1284 | build system processes patches, see the | ||
1285 | "<ulink url='&YOCTO_DOCS_DEV_URL;#new-recipe-patching-code'>Patching Code</ulink>" | ||
1286 | section in the Yocto Project Development Tasks Manual. | ||
1287 | You can also see the | ||
1288 | "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-modify-to-modify-the-source-of-an-existing-component'>Use <filename>devtool modify</filename> to Modify the Source of an Existing Component</ulink>" | ||
1289 | section in the Yocto Project Application Development and | ||
1290 | the Extensible Software Development Kit (SDK) manual and | ||
1291 | the | ||
1292 | "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</ulink>" | ||
1293 | section in the Yocto Project Linux Kernel Development | ||
1294 | Manual. | ||
1295 | </para> | ||
1296 | </section> | ||
1297 | |||
1298 | <section id='configuration-compilation-and-staging-dev-environment'> | ||
1299 | <title>Configuration, Compilation, and Staging</title> | ||
1300 | |||
1301 | <para> | ||
1302 | After source code is patched, BitBake executes tasks that | ||
1303 | configure and compile the source code. | ||
1304 | Once compilation occurs, the files are copied to a holding | ||
1305 | area (staged) in preparation for packaging: | ||
1306 | <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" /> | ||
1307 | </para> | ||
1308 | |||
1309 | <para> | ||
1310 | This step in the build process consists of the following | ||
1311 | tasks: | ||
1312 | <itemizedlist> | ||
1313 | <listitem><para> | ||
1314 | <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>: | ||
1315 | This task sets up the two sysroots in | ||
1316 | <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename> | ||
1317 | (i.e. <filename>recipe-sysroot</filename> and | ||
1318 | <filename>recipe-sysroot-native</filename>) so that | ||
1319 | during the packaging phase the sysroots can contain | ||
1320 | the contents of the | ||
1321 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> | ||
1322 | tasks of the recipes on which the recipe | ||
1323 | containing the tasks depends. | ||
1324 | A sysroot exists for both the target and for the | ||
1325 | native binaries, which run on the host system. | ||
1326 | </para></listitem> | ||
1327 | <listitem><para> | ||
1328 | <emphasis><filename>do_configure</filename></emphasis>: | ||
1329 | This task configures the source by enabling and | ||
1330 | disabling any build-time and configuration options | ||
1331 | for the software being built. | ||
1332 | Configurations can come from the recipe itself as | ||
1333 | well as from an inherited class. | ||
1334 | Additionally, the software itself might configure | ||
1335 | itself depending on the target for which it is | ||
1336 | being built.</para> | ||
1337 | |||
1338 | <para>The configurations handled by the | ||
1339 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> | ||
1340 | task are specific to configurations for the source | ||
1341 | code being built by the recipe.</para> | ||
1342 | |||
1343 | <para>If you are using the | ||
1344 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> | ||
1345 | class, you can add additional configuration options | ||
1346 | by using the | ||
1347 | <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink> | ||
1348 | or | ||
1349 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink> | ||
1350 | variables. | ||
1351 | For information on how this variable works within | ||
1352 | that class, see the | ||
1353 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> | ||
1354 | class | ||
1355 | <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass'>here</ulink>. | ||
1356 | </para></listitem> | ||
1357 | <listitem><para> | ||
1358 | <emphasis><filename>do_compile</filename></emphasis>: | ||
1359 | Once a configuration task has been satisfied, | ||
1360 | BitBake compiles the source using the | ||
1361 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> | ||
1362 | task. | ||
1363 | Compilation occurs in the directory pointed to by | ||
1364 | the | ||
1365 | <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink> | ||
1366 | variable. | ||
1367 | Realize that the <filename>B</filename> directory | ||
1368 | is, by default, the same as the | ||
1369 | <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> | ||
1370 | directory. | ||
1371 | </para></listitem> | ||
1372 | <listitem><para> | ||
1373 | <emphasis><filename>do_install</filename></emphasis>: | ||
1374 | After compilation completes, BitBake executes the | ||
1375 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> | ||
1376 | task. | ||
1377 | This task copies files from the | ||
1378 | <filename>B</filename> directory and places them | ||
1379 | in a holding area pointed to by the | ||
1380 | <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> | ||
1381 | variable. | ||
1382 | Packaging occurs later using files from this | ||
1383 | holding directory. | ||
1384 | </para></listitem> | ||
1385 | </itemizedlist> | ||
1386 | </para> | ||
1387 | </section> | ||
1388 | |||
1389 | <section id='package-splitting-dev-environment'> | ||
1390 | <title>Package Splitting</title> | ||
1391 | |||
1392 | <para> | ||
1393 | After source code is configured, compiled, and staged, the | ||
1394 | build system analyzes the results and splits the output | ||
1395 | into packages: | ||
1396 | <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" /> | ||
1397 | </para> | ||
1398 | |||
1399 | <para> | ||
1400 | The | ||
1401 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> | ||
1402 | and | ||
1403 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink> | ||
1404 | tasks combine to analyze the files found in the | ||
1405 | <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> | ||
1406 | directory and split them into subsets based on available | ||
1407 | packages and files. | ||
1408 | Analysis involves the following as well as other items: | ||
1409 | splitting out debugging symbols, looking at shared library | ||
1410 | dependencies between packages, and looking at package | ||
1411 | relationships. | ||
1412 | </para> | ||
1413 | |||
1414 | <para> | ||
1415 | The <filename>do_packagedata</filename> task creates | ||
1416 | package metadata based on the analysis such that the | ||
1417 | build system can generate the final packages. | ||
1418 | The | ||
1419 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> | ||
1420 | task stages (copies) a subset of the files installed by | ||
1421 | the | ||
1422 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> | ||
1423 | task into the appropriate sysroot. | ||
1424 | Working, staged, and intermediate results of the analysis | ||
1425 | and package splitting process use several areas: | ||
1426 | <itemizedlist> | ||
1427 | <listitem><para> | ||
1428 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>: | ||
1429 | The destination directory | ||
1430 | (i.e. <filename>package</filename>) for packages | ||
1431 | before they are split into individual packages. | ||
1432 | </para></listitem> | ||
1433 | <listitem><para> | ||
1434 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>: | ||
1435 | A temporary work area (i.e. | ||
1436 | <filename>pkgdata</filename>) used by the | ||
1437 | <filename>do_package</filename> task to save | ||
1438 | package metadata. | ||
1439 | </para></listitem> | ||
1440 | <listitem><para> | ||
1441 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>: | ||
1442 | The parent directory (i.e. | ||
1443 | <filename>packages-split</filename>) for packages | ||
1444 | after they have been split. | ||
1445 | </para></listitem> | ||
1446 | <listitem><para> | ||
1447 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>: | ||
1448 | A shared, global-state directory that holds | ||
1449 | packaging metadata generated during the packaging | ||
1450 | process. | ||
1451 | The packaging process copies metadata from | ||
1452 | <filename>PKGDESTWORK</filename> to the | ||
1453 | <filename>PKGDATA_DIR</filename> area where it | ||
1454 | becomes globally available. | ||
1455 | </para></listitem> | ||
1456 | <listitem><para> | ||
1457 | <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></ulink>: | ||
1458 | The path for the sysroot for the system on which | ||
1459 | a component is built to run (i.e. | ||
1460 | <filename>recipe-sysroot</filename>). | ||
1461 | </para></listitem> | ||
1462 | <listitem><para> | ||
1463 | <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_NATIVE'><filename>STAGING_DIR_NATIVE</filename></ulink>: | ||
1464 | The path for the sysroot used when building | ||
1465 | components for the build host (i.e. | ||
1466 | <filename>recipe-sysroot-native</filename>). | ||
1467 | </para></listitem> | ||
1468 | <listitem><para> | ||
1469 | <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_TARGET'><filename>STAGING_DIR_TARGET</filename></ulink>: | ||
1470 | The path for the sysroot used when a component that | ||
1471 | is built to execute on a system and it generates | ||
1472 | code for yet another machine (e.g. cross-canadian | ||
1473 | recipes). | ||
1474 | </para></listitem> | ||
1475 | </itemizedlist> | ||
1476 | The | ||
1477 | <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink> | ||
1478 | variable defines the files that go into each package in | ||
1479 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>. | ||
1480 | If you want details on how this is accomplished, you can | ||
1481 | look at | ||
1482 | <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass'><filename>package.bbclass</filename></ulink>. | ||
1483 | </para> | ||
1484 | |||
1485 | <para> | ||
1486 | Depending on the type of packages being created (RPM, DEB, | ||
1487 | or IPK), the | ||
1488 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> | ||
1489 | task creates the actual packages and places them in the | ||
1490 | Package Feed area, which is | ||
1491 | <filename>${TMPDIR}/deploy</filename>. | ||
1492 | You can see the | ||
1493 | "<link linkend='package-feeds-dev-environment'>Package Feeds</link>" | ||
1494 | section for more detail on that part of the build process. | ||
1495 | <note> | ||
1496 | Support for creating feeds directly from the | ||
1497 | <filename>deploy/*</filename> directories does not | ||
1498 | exist. | ||
1499 | Creating such feeds usually requires some kind of feed | ||
1500 | maintenance mechanism that would upload the new | ||
1501 | packages into an official package feed (e.g. the | ||
1502 | Ångström distribution). | ||
1503 | This functionality is highly distribution-specific | ||
1504 | and thus is not provided out of the box. | ||
1505 | </note> | ||
1506 | </para> | ||
1507 | </section> | ||
1508 | |||
1509 | <section id='image-generation-dev-environment'> | ||
1510 | <title>Image Generation</title> | ||
1511 | |||
1512 | <para> | ||
1513 | Once packages are split and stored in the Package Feeds | ||
1514 | area, the build system uses BitBake to generate the root | ||
1515 | filesystem image: | ||
1516 | <imagedata fileref="figures/image-generation.png" align="center" width="7.5in" depth="7.5in" /> | ||
1517 | </para> | ||
1518 | |||
1519 | <para> | ||
1520 | The image generation process consists of several stages and | ||
1521 | depends on several tasks and variables. | ||
1522 | The | ||
1523 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink> | ||
1524 | task creates the root filesystem (file and directory | ||
1525 | structure) for an image. | ||
1526 | This task uses several key variables to help create the | ||
1527 | list of packages to actually install: | ||
1528 | <itemizedlist> | ||
1529 | <listitem><para> | ||
1530 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>: | ||
1531 | Lists out the base set of packages from which to | ||
1532 | install from the Package Feeds area. | ||
1533 | </para></listitem> | ||
1534 | <listitem><para> | ||
1535 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>: | ||
1536 | Specifies packages that should not be installed | ||
1537 | into the image. | ||
1538 | </para></listitem> | ||
1539 | <listitem><para> | ||
1540 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>: | ||
1541 | Specifies features to include in the image. | ||
1542 | Most of these features map to additional packages | ||
1543 | for installation. | ||
1544 | </para></listitem> | ||
1545 | <listitem><para> | ||
1546 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>: | ||
1547 | Specifies the package backend (e.g. RPM, DEB, or | ||
1548 | IPK) to use and consequently helps determine where | ||
1549 | to locate packages within the Package Feeds area. | ||
1550 | </para></listitem> | ||
1551 | <listitem><para> | ||
1552 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>: | ||
1553 | Determines the language(s) for which additional | ||
1554 | language support packages are installed. | ||
1555 | </para></listitem> | ||
1556 | <listitem><para> | ||
1557 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>: | ||
1558 | The final list of packages passed to the package | ||
1559 | manager for installation into the image. | ||
1560 | </para></listitem> | ||
1561 | </itemizedlist> | ||
1562 | </para> | ||
1563 | |||
1564 | <para> | ||
1565 | With | ||
1566 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink> | ||
1567 | pointing to the location of the filesystem under | ||
1568 | construction and the <filename>PACKAGE_INSTALL</filename> | ||
1569 | variable providing the final list of packages to install, | ||
1570 | the root file system is created. | ||
1571 | </para> | ||
1572 | |||
1573 | <para> | ||
1574 | Package installation is under control of the package | ||
1575 | manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of | ||
1576 | whether or not package management is enabled for the | ||
1577 | target. | ||
1578 | At the end of the process, if package management is not | ||
1579 | enabled for the target, the package manager's data files | ||
1580 | are deleted from the root filesystem. | ||
1581 | As part of the final stage of package installation, | ||
1582 | post installation scripts that are part of the packages | ||
1583 | are run. | ||
1584 | Any scripts that fail to run on the build host are run on | ||
1585 | the target when the target system is first booted. | ||
1586 | If you are using a | ||
1587 | <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>, | ||
1588 | all the post installation scripts must succeed on the | ||
1589 | build host during the package installation phase since the | ||
1590 | root filesystem on the target is read-only. | ||
1591 | </para> | ||
1592 | |||
1593 | <para> | ||
1594 | The final stages of the <filename>do_rootfs</filename> task | ||
1595 | handle post processing. | ||
1596 | Post processing includes creation of a manifest file and | ||
1597 | optimizations. | ||
1598 | </para> | ||
1599 | |||
1600 | <para> | ||
1601 | The manifest file (<filename>.manifest</filename>) resides | ||
1602 | in the same directory as the root filesystem image. | ||
1603 | This file lists out, line-by-line, the installed packages. | ||
1604 | The manifest file is useful for the | ||
1605 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink> | ||
1606 | class, for example, to determine whether or not to run | ||
1607 | specific tests. | ||
1608 | See the | ||
1609 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink> | ||
1610 | variable for additional information. | ||
1611 | </para> | ||
1612 | |||
1613 | <para> | ||
1614 | Optimizing processes that are run across the image include | ||
1615 | <filename>mklibs</filename>, <filename>prelink</filename>, | ||
1616 | and any other post-processing commands as defined by the | ||
1617 | <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink> | ||
1618 | variable. | ||
1619 | The <filename>mklibs</filename> process optimizes the size | ||
1620 | of the libraries, while the <filename>prelink</filename> | ||
1621 | process optimizes the dynamic linking of shared libraries | ||
1622 | to reduce start up time of executables. | ||
1623 | </para> | ||
1624 | |||
1625 | <para> | ||
1626 | After the root filesystem is built, processing begins on | ||
1627 | the image through the | ||
1628 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink> | ||
1629 | task. | ||
1630 | The build system runs any pre-processing commands as | ||
1631 | defined by the | ||
1632 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink> | ||
1633 | variable. | ||
1634 | This variable specifies a list of functions to call before | ||
1635 | the build system creates the final image output files. | ||
1636 | </para> | ||
1637 | |||
1638 | <para> | ||
1639 | The build system dynamically creates | ||
1640 | <filename>do_image_*</filename> tasks as needed, based | ||
1641 | on the image types specified in the | ||
1642 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> | ||
1643 | variable. | ||
1644 | The process turns everything into an image file or a set of | ||
1645 | image files and can compress the root filesystem image to | ||
1646 | reduce the overall size of the image. | ||
1647 | The formats used for the root filesystem depend on the | ||
1648 | <filename>IMAGE_FSTYPES</filename> variable. | ||
1649 | Compression depends on whether the formats support | ||
1650 | compression. | ||
1651 | </para> | ||
1652 | |||
1653 | <para> | ||
1654 | As an example, a dynamically created task when creating a | ||
1655 | particular image <replaceable>type</replaceable> would | ||
1656 | take the following form: | ||
1657 | <literallayout class='monospaced'> | ||
1658 | do_image_<replaceable>type</replaceable> | ||
1659 | </literallayout> | ||
1660 | So, if the <replaceable>type</replaceable> as specified by | ||
1661 | the <filename>IMAGE_FSTYPES</filename> were | ||
1662 | <filename>ext4</filename>, the dynamically generated task | ||
1663 | would be as follows: | ||
1664 | <literallayout class='monospaced'> | ||
1665 | do_image_ext4 | ||
1666 | </literallayout> | ||
1667 | </para> | ||
1668 | |||
1669 | <para> | ||
1670 | The final task involved in image creation is the | ||
1671 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink> | ||
1672 | task. | ||
1673 | This task completes the image by applying any image | ||
1674 | post processing as defined through the | ||
1675 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink> | ||
1676 | variable. | ||
1677 | The variable specifies a list of functions to call once the | ||
1678 | build system has created the final image output files. | ||
1679 | <note> | ||
1680 | The entire image generation process is run under | ||
1681 | <link linkend='fakeroot-and-pseudo'>Pseudo</link>. | ||
1682 | Running under Pseudo ensures that the files in the | ||
1683 | root filesystem have correct ownership. | ||
1684 | </note> | ||
1685 | </para> | ||
1686 | </section> | ||
1687 | |||
1688 | <section id='sdk-generation-dev-environment'> | ||
1689 | <title>SDK Generation</title> | ||
1690 | |||
1691 | <para> | ||
1692 | The OpenEmbedded build system uses BitBake to generate the | ||
1693 | Software Development Kit (SDK) installer scripts for both | ||
1694 | the standard SDK and the extensible SDK (eSDK): | ||
1695 | </para> | ||
1696 | |||
1697 | <para> | ||
1698 | <imagedata fileref="figures/sdk-generation.png" width="9in" align="center" /> | ||
1699 | <note> | ||
1700 | For more information on the cross-development toolchain | ||
1701 | generation, see the | ||
1702 | "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>" | ||
1703 | section. | ||
1704 | For information on advantages gained when building a | ||
1705 | cross-development toolchain using the | ||
1706 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink> | ||
1707 | task, see the | ||
1708 | "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>" | ||
1709 | section in the Yocto Project Application Development | ||
1710 | and the Extensible Software Development Kit (eSDK) | ||
1711 | manual. | ||
1712 | </note> | ||
1713 | </para> | ||
1714 | |||
1715 | <para> | ||
1716 | Like image generation, the SDK script process consists of | ||
1717 | several stages and depends on many variables. | ||
1718 | The | ||
1719 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink> | ||
1720 | and | ||
1721 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk_ext'><filename>do_populate_sdk_ext</filename></ulink> | ||
1722 | tasks use these key variables to help create the list of | ||
1723 | packages to actually install. | ||
1724 | For information on the variables listed in the figure, | ||
1725 | see the | ||
1726 | "<link linkend='sdk-dev-environment'>Application Development SDK</link>" | ||
1727 | section. | ||
1728 | </para> | ||
1729 | |||
1730 | <para> | ||
1731 | The <filename>do_populate_sdk</filename> task helps create | ||
1732 | the standard SDK and handles two parts: a target part and a | ||
1733 | host part. | ||
1734 | The target part is the part built for the target hardware | ||
1735 | and includes libraries and headers. | ||
1736 | The host part is the part of the SDK that runs on the | ||
1737 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>. | ||
1738 | </para> | ||
1739 | |||
1740 | <para> | ||
1741 | The <filename>do_populate_sdk_ext</filename> task helps | ||
1742 | create the extensible SDK and handles host and target parts | ||
1743 | differently than its counter part does for the standard SDK. | ||
1744 | For the extensible SDK, the task encapsulates the build | ||
1745 | system, which includes everything needed (host and target) | ||
1746 | for the SDK. | ||
1747 | </para> | ||
1748 | |||
1749 | <para> | ||
1750 | Regardless of the type of SDK being constructed, the | ||
1751 | tasks perform some cleanup after which a cross-development | ||
1752 | environment setup script and any needed configuration files | ||
1753 | are created. | ||
1754 | The final output is the Cross-development | ||
1755 | toolchain installation script (<filename>.sh</filename> | ||
1756 | file), which includes the environment setup script. | ||
1757 | </para> | ||
1758 | </section> | ||
1759 | |||
1760 | <section id='stamp-files-and-the-rerunning-of-tasks'> | ||
1761 | <title>Stamp Files and the Rerunning of Tasks</title> | ||
1762 | |||
1763 | <para> | ||
1764 | For each task that completes successfully, BitBake writes a | ||
1765 | stamp file into the | ||
1766 | <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink> | ||
1767 | directory. | ||
1768 | The beginning of the stamp file's filename is determined | ||
1769 | by the | ||
1770 | <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink> | ||
1771 | variable, and the end of the name consists of the task's | ||
1772 | name and current | ||
1773 | <link linkend='overview-checksums'>input checksum</link>. | ||
1774 | <note> | ||
1775 | This naming scheme assumes that | ||
1776 | <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink> | ||
1777 | is "OEBasicHash", which is almost always the case in | ||
1778 | current OpenEmbedded. | ||
1779 | </note> | ||
1780 | To determine if a task needs to be rerun, BitBake checks | ||
1781 | if a stamp file with a matching input checksum exists | ||
1782 | for the task. | ||
1783 | If such a stamp file exists, the task's output is | ||
1784 | assumed to exist and still be valid. | ||
1785 | If the file does not exist, the task is rerun. | ||
1786 | <note> | ||
1787 | <para>The stamp mechanism is more general than the | ||
1788 | shared state (sstate) cache mechanism described in the | ||
1789 | "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>" | ||
1790 | section. | ||
1791 | BitBake avoids rerunning any task that has a valid | ||
1792 | stamp file, not just tasks that can be accelerated | ||
1793 | through the sstate cache.</para> | ||
1794 | |||
1795 | <para>However, you should realize that stamp files only | ||
1796 | serve as a marker that some work has been done and that | ||
1797 | these files do not record task output. | ||
1798 | The actual task output would usually be somewhere in | ||
1799 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> | ||
1800 | (e.g. in some recipe's | ||
1801 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.) | ||
1802 | What the sstate cache mechanism adds is a way to cache | ||
1803 | task output that can then be shared between build | ||
1804 | machines.</para> | ||
1805 | </note> | ||
1806 | Since <filename>STAMPS_DIR</filename> is usually a | ||
1807 | subdirectory of <filename>TMPDIR</filename>, removing | ||
1808 | <filename>TMPDIR</filename> will also remove | ||
1809 | <filename>STAMPS_DIR</filename>, which means tasks will | ||
1810 | properly be rerun to repopulate | ||
1811 | <filename>TMPDIR</filename>. | ||
1812 | </para> | ||
1813 | |||
1814 | <para> | ||
1815 | If you want some task to always be considered "out of | ||
1816 | date", you can mark it with the | ||
1817 | <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink> | ||
1818 | varflag. | ||
1819 | If some other task depends on such a task, then that | ||
1820 | task will also always be considered out of date, which | ||
1821 | might not be what you want. | ||
1822 | </para> | ||
1823 | |||
1824 | <para> | ||
1825 | For details on how to view information about a task's | ||
1826 | signature, see the | ||
1827 | "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>" | ||
1828 | section in the Yocto Project Development Tasks Manual. | ||
1829 | </para> | ||
1830 | </section> | ||
1831 | |||
1832 | <section id='setscene-tasks-and-shared-state'> | ||
1833 | <title>Setscene Tasks and Shared State</title> | ||
1834 | |||
1835 | <para> | ||
1836 | The description of tasks so far assumes that BitBake needs | ||
1837 | to build everything and no available prebuilt objects | ||
1838 | exist. | ||
1839 | BitBake does support skipping tasks if prebuilt objects are | ||
1840 | available. | ||
1841 | These objects are usually made available in the form of a | ||
1842 | shared state (sstate) cache. | ||
1843 | <note> | ||
1844 | For information on variables affecting sstate, see the | ||
1845 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> | ||
1846 | and | ||
1847 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink> | ||
1848 | variables. | ||
1849 | </note> | ||
1850 | </para> | ||
1851 | |||
1852 | <para> | ||
1853 | The idea of a setscene task (i.e | ||
1854 | <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>) | ||
1855 | is a version of the task where | ||
1856 | instead of building something, BitBake can skip to the end | ||
1857 | result and simply place a set of files into specific | ||
1858 | locations as needed. | ||
1859 | In some cases, it makes sense to have a setscene task | ||
1860 | variant (e.g. generating package files in the | ||
1861 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> | ||
1862 | task). | ||
1863 | In other cases, it does not make sense (e.g. a | ||
1864 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> | ||
1865 | task or a | ||
1866 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> | ||
1867 | task) since the work involved would be equal to or greater | ||
1868 | than the underlying task. | ||
1869 | </para> | ||
1870 | |||
1871 | <para> | ||
1872 | In the build system, the common tasks that have setscene | ||
1873 | variants are | ||
1874 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>, | ||
1875 | <filename>do_package_write_*</filename>, | ||
1876 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>, | ||
1877 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>, | ||
1878 | and | ||
1879 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>. | ||
1880 | Notice that these tasks represent most of the tasks whose | ||
1881 | output is an end result. | ||
1882 | </para> | ||
1883 | |||
1884 | <para> | ||
1885 | The build system has knowledge of the relationship between | ||
1886 | these tasks and other preceding tasks. | ||
1887 | For example, if BitBake runs | ||
1888 | <filename>do_populate_sysroot_setscene</filename> for | ||
1889 | something, it does not make sense to run any of the | ||
1890 | <filename>do_fetch</filename>, | ||
1891 | <filename>do_unpack</filename>, | ||
1892 | <filename>do_patch</filename>, | ||
1893 | <filename>do_configure</filename>, | ||
1894 | <filename>do_compile</filename>, and | ||
1895 | <filename>do_install</filename> tasks. | ||
1896 | However, if <filename>do_package</filename> needs to be | ||
1897 | run, BitBake needs to run those other tasks. | ||
1898 | </para> | ||
1899 | |||
1900 | <para> | ||
1901 | It becomes more complicated if everything can come | ||
1902 | from an sstate cache because some objects are simply | ||
1903 | not required at all. | ||
1904 | For example, you do not need a compiler or native tools, | ||
1905 | such as quilt, if nothing exists to compile or patch. | ||
1906 | If the <filename>do_package_write_*</filename> packages | ||
1907 | are available from sstate, BitBake does not need the | ||
1908 | <filename>do_package</filename> task data. | ||
1909 | </para> | ||
1910 | |||
1911 | <para> | ||
1912 | To handle all these complexities, BitBake runs in two | ||
1913 | phases. | ||
1914 | The first is the "setscene" stage. | ||
1915 | During this stage, BitBake first checks the sstate cache | ||
1916 | for any targets it is planning to build. | ||
1917 | BitBake does a fast check to see if the object exists | ||
1918 | rather than a complete download. | ||
1919 | If nothing exists, the second phase, which is the setscene | ||
1920 | stage, completes and the main build proceeds. | ||
1921 | </para> | ||
1922 | |||
1923 | <para> | ||
1924 | If objects are found in the sstate cache, the build system | ||
1925 | works backwards from the end targets specified by the user. | ||
1926 | For example, if an image is being built, the build system | ||
1927 | first looks for the packages needed for that image and the | ||
1928 | tools needed to construct an image. | ||
1929 | If those are available, the compiler is not needed. | ||
1930 | Thus, the compiler is not even downloaded. | ||
1931 | If something was found to be unavailable, or the | ||
1932 | download or setscene task fails, the build system then | ||
1933 | tries to install dependencies, such as the compiler, from | ||
1934 | the cache. | ||
1935 | </para> | ||
1936 | |||
1937 | <para> | ||
1938 | The availability of objects in the sstate cache is | ||
1939 | handled by the function specified by the | ||
1940 | <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink> | ||
1941 | variable and returns a list of available objects. | ||
1942 | The function specified by the | ||
1943 | <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink> | ||
1944 | variable is the function that determines whether a given | ||
1945 | dependency needs to be followed, and whether for any given | ||
1946 | relationship the function needs to be passed. | ||
1947 | The function returns a True or False value. | ||
1948 | </para> | ||
1949 | </section> | ||
1950 | </section> | ||
1951 | |||
1952 | <section id='images-dev-environment'> | ||
1953 | <title>Images</title> | ||
1954 | |||
1955 | <para> | ||
1956 | The images produced by the build system are compressed forms | ||
1957 | of the root filesystem and are ready to boot on a target | ||
1958 | device. | ||
1959 | You can see from the | ||
1960 | <link linkend='general-workflow-figure'>general workflow figure</link> | ||
1961 | that BitBake output, in part, consists of images. | ||
1962 | This section takes a closer look at this output: | ||
1963 | <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" /> | ||
1964 | </para> | ||
1965 | |||
1966 | <note> | ||
1967 | For a list of example images that the Yocto Project provides, | ||
1968 | see the | ||
1969 | "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" | ||
1970 | chapter in the Yocto Project Reference Manual. | ||
1971 | </note> | ||
1972 | |||
1973 | <para> | ||
1974 | The build process writes images out to the | ||
1975 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> | ||
1976 | inside the | ||
1977 | <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename> | ||
1978 | folder as shown in the figure. | ||
1979 | This folder contains any files expected to be loaded on the | ||
1980 | target device. | ||
1981 | The | ||
1982 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink> | ||
1983 | variable points to the <filename>deploy</filename> directory, | ||
1984 | while the | ||
1985 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink> | ||
1986 | variable points to the appropriate directory containing images | ||
1987 | for the current configuration. | ||
1988 | <itemizedlist> | ||
1989 | <listitem><para> | ||
1990 | <replaceable>kernel-image</replaceable>: | ||
1991 | A kernel binary file. | ||
1992 | The | ||
1993 | <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink> | ||
1994 | variable determines the naming scheme for the | ||
1995 | kernel image file. | ||
1996 | Depending on this variable, the file could begin with | ||
1997 | a variety of naming strings. | ||
1998 | The | ||
1999 | <filename>deploy/images/</filename><replaceable>machine</replaceable> | ||
2000 | directory can contain multiple image files for the | ||
2001 | machine. | ||
2002 | </para></listitem> | ||
2003 | <listitem><para> | ||
2004 | <replaceable>root-filesystem-image</replaceable>: | ||
2005 | Root filesystems for the target device (e.g. | ||
2006 | <filename>*.ext3</filename> or | ||
2007 | <filename>*.bz2</filename> files). | ||
2008 | The | ||
2009 | <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> | ||
2010 | variable determines the root filesystem image type. | ||
2011 | The | ||
2012 | <filename>deploy/images/</filename><replaceable>machine</replaceable> | ||
2013 | directory can contain multiple root filesystems for the | ||
2014 | machine. | ||
2015 | </para></listitem> | ||
2016 | <listitem><para> | ||
2017 | <replaceable>kernel-modules</replaceable>: | ||
2018 | Tarballs that contain all the modules built for the | ||
2019 | kernel. | ||
2020 | Kernel module tarballs exist for legacy purposes and | ||
2021 | can be suppressed by setting the | ||
2022 | <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink> | ||
2023 | variable to "0". | ||
2024 | The | ||
2025 | <filename>deploy/images/</filename><replaceable>machine</replaceable> | ||
2026 | directory can contain multiple kernel module tarballs | ||
2027 | for the machine. | ||
2028 | </para></listitem> | ||
2029 | <listitem><para> | ||
2030 | <replaceable>bootloaders</replaceable>: | ||
2031 | If applicable to the target machine, bootloaders | ||
2032 | supporting the image. | ||
2033 | The <filename>deploy/images/</filename><replaceable>machine</replaceable> | ||
2034 | directory can contain multiple bootloaders for the | ||
2035 | machine. | ||
2036 | </para></listitem> | ||
2037 | <listitem><para> | ||
2038 | <replaceable>symlinks</replaceable>: | ||
2039 | The | ||
2040 | <filename>deploy/images/</filename><replaceable>machine</replaceable> | ||
2041 | folder contains a symbolic link that points to the | ||
2042 | most recently built file for each machine. | ||
2043 | These links might be useful for external scripts that | ||
2044 | need to obtain the latest version of each file. | ||
2045 | </para></listitem> | ||
2046 | </itemizedlist> | ||
2047 | </para> | ||
2048 | </section> | ||
2049 | |||
2050 | <section id='sdk-dev-environment'> | ||
2051 | <title>Application Development SDK</title> | ||
2052 | |||
2053 | <para> | ||
2054 | In the | ||
2055 | <link linkend='general-workflow-figure'>general workflow figure</link>, | ||
2056 | the output labeled "Application Development SDK" represents an | ||
2057 | SDK. | ||
2058 | The SDK generation process differs depending on whether you | ||
2059 | build an extensible SDK (e.g. | ||
2060 | <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>) | ||
2061 | or a standard SDK (e.g. | ||
2062 | <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>). | ||
2063 | This section takes a closer look at this output: | ||
2064 | <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" /> | ||
2065 | </para> | ||
2066 | |||
2067 | <para> | ||
2068 | The specific form of this output is a set of files that | ||
2069 | includes a self-extracting SDK installer | ||
2070 | (<filename>*.sh</filename>), host and target manifest files, | ||
2071 | and files used for SDK testing. | ||
2072 | When the SDK installer file is run, it installs the SDK. | ||
2073 | The SDK consists of a cross-development toolchain, a set of | ||
2074 | libraries and headers, and an SDK environment setup script. | ||
2075 | Running this installer essentially sets up your | ||
2076 | cross-development environment. | ||
2077 | You can think of the cross-toolchain as the "host" | ||
2078 | part because it runs on the SDK machine. | ||
2079 | You can think of the libraries and headers as the "target" | ||
2080 | part because they are built for the target hardware. | ||
2081 | The environment setup script is added so that you can | ||
2082 | initialize the environment before using the tools. | ||
2083 | </para> | ||
2084 | |||
2085 | <note><title>Notes</title> | ||
2086 | <itemizedlist> | ||
2087 | <listitem><para> | ||
2088 | The Yocto Project supports several methods by which | ||
2089 | you can set up this cross-development environment. | ||
2090 | These methods include downloading pre-built SDK | ||
2091 | installers or building and installing your own SDK | ||
2092 | installer. | ||
2093 | </para></listitem> | ||
2094 | <listitem><para> | ||
2095 | For background information on cross-development | ||
2096 | toolchains in the Yocto Project development | ||
2097 | environment, see the | ||
2098 | "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>" | ||
2099 | section. | ||
2100 | </para></listitem> | ||
2101 | <listitem><para> | ||
2102 | For information on setting up a cross-development | ||
2103 | environment, see the | ||
2104 | <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> | ||
2105 | manual. | ||
2106 | </para></listitem> | ||
2107 | </itemizedlist> | ||
2108 | </note> | ||
2109 | |||
2110 | <para> | ||
2111 | All the output files for an SDK are written to the | ||
2112 | <filename>deploy/sdk</filename> folder inside the | ||
2113 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> | ||
2114 | as shown in the previous figure. | ||
2115 | Depending on the type of SDK, several variables exist that help | ||
2116 | configure these files. | ||
2117 | The following list shows the variables associated with an | ||
2118 | extensible SDK: | ||
2119 | <itemizedlist> | ||
2120 | <listitem><para> | ||
2121 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: | ||
2122 | Points to the <filename>deploy</filename> directory. | ||
2123 | </para></listitem> | ||
2124 | <listitem><para> | ||
2125 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>: | ||
2126 | Controls whether or not shared state artifacts are | ||
2127 | copied into the extensible SDK. | ||
2128 | By default, all required shared state artifacts are | ||
2129 | copied into the SDK. | ||
2130 | </para></listitem> | ||
2131 | <listitem><para> | ||
2132 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>: | ||
2133 | Specifies whether or not packagedata is included in the | ||
2134 | extensible SDK for all recipes in the "world" target. | ||
2135 | </para></listitem> | ||
2136 | <listitem><para> | ||
2137 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>: | ||
2138 | Specifies whether or not the toolchain is included | ||
2139 | when building the extensible SDK. | ||
2140 | </para></listitem> | ||
2141 | <listitem><para> | ||
2142 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>: | ||
2143 | A list of variables allowed through from the build | ||
2144 | system configuration into the extensible SDK | ||
2145 | configuration. | ||
2146 | </para></listitem> | ||
2147 | <listitem><para> | ||
2148 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>: | ||
2149 | A list of variables not allowed through from the build | ||
2150 | system configuration into the extensible SDK | ||
2151 | configuration. | ||
2152 | </para></listitem> | ||
2153 | <listitem><para> | ||
2154 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>: | ||
2155 | A list of classes to remove from the | ||
2156 | <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink> | ||
2157 | value globally within the extensible SDK configuration. | ||
2158 | </para></listitem> | ||
2159 | </itemizedlist> | ||
2160 | This next list, shows the variables associated with a standard | ||
2161 | SDK: | ||
2162 | <itemizedlist> | ||
2163 | <listitem><para> | ||
2164 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: | ||
2165 | Points to the <filename>deploy</filename> directory. | ||
2166 | </para></listitem> | ||
2167 | <listitem><para> | ||
2168 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>: | ||
2169 | Specifies the architecture of the machine on which the | ||
2170 | cross-development tools are run to create packages for | ||
2171 | the target hardware. | ||
2172 | </para></listitem> | ||
2173 | <listitem><para> | ||
2174 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>: | ||
2175 | Lists the features to include in the "target" part | ||
2176 | of the SDK. | ||
2177 | </para></listitem> | ||
2178 | <listitem><para> | ||
2179 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>: | ||
2180 | Lists packages that make up the host part of the SDK | ||
2181 | (i.e. the part that runs on the | ||
2182 | <filename>SDKMACHINE</filename>). | ||
2183 | When you use | ||
2184 | <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename> | ||
2185 | to create the SDK, a set of default packages apply. | ||
2186 | This variable allows you to add more packages. | ||
2187 | </para></listitem> | ||
2188 | <listitem><para> | ||
2189 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>: | ||
2190 | Lists packages that make up the target part of the SDK | ||
2191 | (i.e. the part built for the target hardware). | ||
2192 | </para></listitem> | ||
2193 | <listitem><para> | ||
2194 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>: | ||
2195 | Defines the default SDK installation path offered by | ||
2196 | the installation script. | ||
2197 | </para></listitem> | ||
2198 | <listitem><para> | ||
2199 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_HOST_MANIFEST'><filename>SDK_HOST_MANIFEST</filename></ulink>: | ||
2200 | Lists all the installed packages that make up the host | ||
2201 | part of the SDK. | ||
2202 | This variable also plays a minor role for extensible | ||
2203 | SDK development as well. | ||
2204 | However, it is mainly used for the standard SDK. | ||
2205 | </para></listitem> | ||
2206 | <listitem><para> | ||
2207 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_TARGET_MANIFEST'><filename>SDK_TARGET_MANIFEST</filename></ulink>: | ||
2208 | Lists all the installed packages that make up the | ||
2209 | target part of the SDK. | ||
2210 | This variable also plays a minor role for extensible | ||
2211 | SDK development as well. | ||
2212 | However, it is mainly used for the standard SDK. | ||
2213 | </para></listitem> | ||
2214 | </itemizedlist> | ||
2215 | </para> | ||
2216 | </section> | ||
2217 | </section> | ||
2218 | |||
2219 | <section id="cross-development-toolchain-generation"> | ||
2220 | <title>Cross-Development Toolchain Generation</title> | ||
2221 | |||
2222 | <para> | ||
2223 | The Yocto Project does most of the work for you when it comes to | ||
2224 | creating | ||
2225 | <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>. | ||
2226 | This section provides some technical background on how | ||
2227 | cross-development toolchains are created and used. | ||
2228 | For more information on toolchains, you can also see the | ||
2229 | <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> | ||
2230 | manual. | ||
2231 | </para> | ||
2232 | |||
2233 | <para> | ||
2234 | In the Yocto Project development environment, cross-development | ||
2235 | toolchains are used to build images and applications that run | ||
2236 | on the target hardware. | ||
2237 | With just a few commands, the OpenEmbedded build system creates | ||
2238 | these necessary toolchains for you. | ||
2239 | </para> | ||
2240 | |||
2241 | <para> | ||
2242 | The following figure shows a high-level build environment regarding | ||
2243 | toolchain construction and use. | ||
2244 | </para> | ||
2245 | |||
2246 | <para> | ||
2247 | <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" /> | ||
2248 | </para> | ||
2249 | |||
2250 | <para> | ||
2251 | Most of the work occurs on the Build Host. | ||
2252 | This is the machine used to build images and generally work within | ||
2253 | the the Yocto Project environment. | ||
2254 | When you run | ||
2255 | <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> | ||
2256 | to create an image, the OpenEmbedded build system | ||
2257 | uses the host <filename>gcc</filename> compiler to bootstrap a | ||
2258 | cross-compiler named <filename>gcc-cross</filename>. | ||
2259 | The <filename>gcc-cross</filename> compiler is what BitBake uses to | ||
2260 | compile source files when creating the target image. | ||
2261 | You can think of <filename>gcc-cross</filename> simply as an | ||
2262 | automatically generated cross-compiler that is used internally | ||
2263 | within BitBake only. | ||
2264 | <note> | ||
2265 | The extensible SDK does not use | ||
2266 | <filename>gcc-cross-canadian</filename> since this SDK | ||
2267 | ships a copy of the OpenEmbedded build system and the sysroot | ||
2268 | within it contains <filename>gcc-cross</filename>. | ||
2269 | </note> | ||
2270 | </para> | ||
2271 | |||
2272 | <para> | ||
2273 | The chain of events that occurs when <filename>gcc-cross</filename> is | ||
2274 | bootstrapped is as follows: | ||
2275 | <literallayout class='monospaced'> | ||
2276 | gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime | ||
2277 | </literallayout> | ||
2278 | <itemizedlist> | ||
2279 | <listitem><para> | ||
2280 | <filename>gcc</filename>: | ||
2281 | The build host's GNU Compiler Collection (GCC). | ||
2282 | </para></listitem> | ||
2283 | <listitem><para> | ||
2284 | <filename>binutils-cross</filename>: | ||
2285 | The bare minimum binary utilities needed in order to run | ||
2286 | the <filename>gcc-cross-initial</filename> phase of the | ||
2287 | bootstrap operation. | ||
2288 | </para></listitem> | ||
2289 | <listitem><para> | ||
2290 | <filename>gcc-cross-initial</filename>: | ||
2291 | An early stage of the bootstrap process for creating | ||
2292 | the cross-compiler. | ||
2293 | This stage builds enough of the <filename>gcc-cross</filename>, | ||
2294 | the C library, and other pieces needed to finish building the | ||
2295 | final cross-compiler in later stages. | ||
2296 | This tool is a "native" package (i.e. it is designed to run on | ||
2297 | the build host). | ||
2298 | </para></listitem> | ||
2299 | <listitem><para> | ||
2300 | <filename>linux-libc-headers</filename>: | ||
2301 | Headers needed for the cross-compiler. | ||
2302 | </para></listitem> | ||
2303 | <listitem><para> | ||
2304 | <filename>glibc-initial</filename>: | ||
2305 | An initial version of the Embedded GNU C Library | ||
2306 | (GLIBC) needed to bootstrap <filename>glibc</filename>. | ||
2307 | </para></listitem> | ||
2308 | <listitem><para> | ||
2309 | <filename>glibc</filename>: | ||
2310 | The GNU C Library. | ||
2311 | </para></listitem> | ||
2312 | <listitem><para> | ||
2313 | <filename>gcc-cross</filename>: | ||
2314 | The final stage of the bootstrap process for the | ||
2315 | cross-compiler. | ||
2316 | This stage results in the actual cross-compiler that | ||
2317 | BitBake uses when it builds an image for a targeted | ||
2318 | device. | ||
2319 | <note> | ||
2320 | If you are replacing this cross compiler toolchain | ||
2321 | with a custom version, you must replace | ||
2322 | <filename>gcc-cross</filename>. | ||
2323 | </note> | ||
2324 | This tool is also a "native" package (i.e. it is | ||
2325 | designed to run on the build host). | ||
2326 | </para></listitem> | ||
2327 | <listitem><para> | ||
2328 | <filename>gcc-runtime</filename>: | ||
2329 | Runtime libraries resulting from the toolchain bootstrapping | ||
2330 | process. | ||
2331 | This tool produces a binary that consists of the | ||
2332 | runtime libraries need for the targeted device. | ||
2333 | </para></listitem> | ||
2334 | </itemizedlist> | ||
2335 | </para> | ||
2336 | |||
2337 | <para> | ||
2338 | You can use the OpenEmbedded build system to build an installer for | ||
2339 | the relocatable SDK used to develop applications. | ||
2340 | When you run the installer, it installs the toolchain, which | ||
2341 | contains the development tools (e.g., | ||
2342 | <filename>gcc-cross-canadian</filename>, | ||
2343 | <filename>binutils-cross-canadian</filename>, and other | ||
2344 | <filename>nativesdk-*</filename> tools), | ||
2345 | which are tools native to the SDK (i.e. native to | ||
2346 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>), | ||
2347 | you need to cross-compile and test your software. | ||
2348 | The figure shows the commands you use to easily build out this | ||
2349 | toolchain. | ||
2350 | This cross-development toolchain is built to execute on the | ||
2351 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>, | ||
2352 | which might or might not be the same | ||
2353 | machine as the Build Host. | ||
2354 | <note> | ||
2355 | If your target architecture is supported by the Yocto Project, | ||
2356 | you can take advantage of pre-built images that ship with the | ||
2357 | Yocto Project and already contain cross-development toolchain | ||
2358 | installers. | ||
2359 | </note> | ||
2360 | </para> | ||
2361 | |||
2362 | <para> | ||
2363 | Here is the bootstrap process for the relocatable toolchain: | ||
2364 | <literallayout class='monospaced'> | ||
2365 | gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> | ||
2366 | glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian | ||
2367 | </literallayout> | ||
2368 | <itemizedlist> | ||
2369 | <listitem><para> | ||
2370 | <filename>gcc</filename>: | ||
2371 | The build host's GNU Compiler Collection (GCC). | ||
2372 | </para></listitem> | ||
2373 | <listitem><para> | ||
2374 | <filename>binutils-crosssdk</filename>: | ||
2375 | The bare minimum binary utilities needed in order to run | ||
2376 | the <filename>gcc-crosssdk-initial</filename> phase of the | ||
2377 | bootstrap operation. | ||
2378 | </para></listitem> | ||
2379 | <listitem><para> | ||
2380 | <filename>gcc-crosssdk-initial</filename>: | ||
2381 | An early stage of the bootstrap process for creating | ||
2382 | the cross-compiler. | ||
2383 | This stage builds enough of the | ||
2384 | <filename>gcc-crosssdk</filename> and supporting pieces so that | ||
2385 | the final stage of the bootstrap process can produce the | ||
2386 | finished cross-compiler. | ||
2387 | This tool is a "native" binary that runs on the build host. | ||
2388 | </para></listitem> | ||
2389 | <listitem><para> | ||
2390 | <filename>linux-libc-headers</filename>: | ||
2391 | Headers needed for the cross-compiler. | ||
2392 | </para></listitem> | ||
2393 | <listitem><para> | ||
2394 | <filename>glibc-initial</filename>: | ||
2395 | An initial version of the Embedded GLIBC needed to bootstrap | ||
2396 | <filename>nativesdk-glibc</filename>. | ||
2397 | </para></listitem> | ||
2398 | <listitem><para> | ||
2399 | <filename>nativesdk-glibc</filename>: | ||
2400 | The Embedded GLIBC needed to bootstrap the | ||
2401 | <filename>gcc-crosssdk</filename>. | ||
2402 | </para></listitem> | ||
2403 | <listitem><para> | ||
2404 | <filename>gcc-crosssdk</filename>: | ||
2405 | The final stage of the bootstrap process for the | ||
2406 | relocatable cross-compiler. | ||
2407 | The <filename>gcc-crosssdk</filename> is a transitory | ||
2408 | compiler and never leaves the build host. | ||
2409 | Its purpose is to help in the bootstrap process to create | ||
2410 | the eventual <filename>gcc-cross-canadian</filename> | ||
2411 | compiler, which is relocatable. | ||
2412 | This tool is also a "native" package (i.e. it is | ||
2413 | designed to run on the build host). | ||
2414 | </para></listitem> | ||
2415 | <listitem><para> | ||
2416 | <filename>gcc-cross-canadian</filename>: | ||
2417 | The final relocatable cross-compiler. | ||
2418 | When run on the | ||
2419 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>, | ||
2420 | this tool | ||
2421 | produces executable code that runs on the target device. | ||
2422 | Only one cross-canadian compiler is produced per architecture | ||
2423 | since they can be targeted at different processor optimizations | ||
2424 | using configurations passed to the compiler through the | ||
2425 | compile commands. | ||
2426 | This circumvents the need for multiple compilers and thus | ||
2427 | reduces the size of the toolchains. | ||
2428 | </para></listitem> | ||
2429 | </itemizedlist> | ||
2430 | </para> | ||
2431 | |||
2432 | <note> | ||
2433 | For information on advantages gained when building a | ||
2434 | cross-development toolchain installer, see the | ||
2435 | "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>" | ||
2436 | appendix in the Yocto Project Application Development and the | ||
2437 | Extensible Software Development Kit (eSDK) manual. | ||
2438 | </note> | ||
2439 | </section> | ||
2440 | |||
2441 | <section id="shared-state-cache"> | ||
2442 | <title>Shared State Cache</title> | ||
2443 | |||
2444 | <para> | ||
2445 | By design, the OpenEmbedded build system builds everything from | ||
2446 | scratch unless | ||
2447 | <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> | ||
2448 | can determine that parts do not need to be rebuilt. | ||
2449 | Fundamentally, building from scratch is attractive as it means all | ||
2450 | parts are built fresh and no possibility of stale data exists that | ||
2451 | can cause problems. | ||
2452 | When developers hit problems, they typically default back to | ||
2453 | building from scratch so they have a know state from the | ||
2454 | start. | ||
2455 | </para> | ||
2456 | |||
2457 | <para> | ||
2458 | Building an image from scratch is both an advantage and a | ||
2459 | disadvantage to the process. | ||
2460 | As mentioned in the previous paragraph, building from scratch | ||
2461 | ensures that everything is current and starts from a known state. | ||
2462 | However, building from scratch also takes much longer as it | ||
2463 | generally means rebuilding things that do not necessarily need | ||
2464 | to be rebuilt. | ||
2465 | </para> | ||
2466 | |||
2467 | <para> | ||
2468 | The Yocto Project implements shared state code that supports | ||
2469 | incremental builds. | ||
2470 | The implementation of the shared state code answers the following | ||
2471 | questions that were fundamental roadblocks within the OpenEmbedded | ||
2472 | incremental build support system: | ||
2473 | <itemizedlist> | ||
2474 | <listitem><para> | ||
2475 | What pieces of the system have changed and what pieces have | ||
2476 | not changed? | ||
2477 | </para></listitem> | ||
2478 | <listitem><para> | ||
2479 | How are changed pieces of software removed and replaced? | ||
2480 | </para></listitem> | ||
2481 | <listitem><para> | ||
2482 | How are pre-built components that do not need to be rebuilt | ||
2483 | from scratch used when they are available? | ||
2484 | </para></listitem> | ||
2485 | </itemizedlist> | ||
2486 | </para> | ||
2487 | |||
2488 | <para> | ||
2489 | For the first question, the build system detects changes in the | ||
2490 | "inputs" to a given task by creating a checksum (or signature) of | ||
2491 | the task's inputs. | ||
2492 | If the checksum changes, the system assumes the inputs have changed | ||
2493 | and the task needs to be rerun. | ||
2494 | For the second question, the shared state (sstate) code tracks | ||
2495 | which tasks add which output to the build process. | ||
2496 | This means the output from a given task can be removed, upgraded | ||
2497 | or otherwise manipulated. | ||
2498 | The third question is partly addressed by the solution for the | ||
2499 | second question assuming the build system can fetch the sstate | ||
2500 | objects from remote locations and install them if they are deemed | ||
2501 | to be valid. | ||
2502 | <note><title>Notes</title> | ||
2503 | <itemizedlist> | ||
2504 | <listitem><para> | ||
2505 | The build system does not maintain | ||
2506 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> | ||
2507 | information as part of the shared state packages. | ||
2508 | Consequently, considerations exist that affect | ||
2509 | maintaining shared state feeds. | ||
2510 | For information on how the build system works with | ||
2511 | packages and can track incrementing | ||
2512 | <filename>PR</filename> information, see the | ||
2513 | "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>" | ||
2514 | section in the Yocto Project Development Tasks Manual. | ||
2515 | </para></listitem> | ||
2516 | <listitem><para> | ||
2517 | The code in the build system that supports incremental | ||
2518 | builds is not simple code. | ||
2519 | For techniques that help you work around issues related | ||
2520 | to shared state code, see the | ||
2521 | "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'>Viewing Metadata Used to Create the Input Signature of a Shared State Task</ulink>" | ||
2522 | and | ||
2523 | "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-invalidating-shared-state-to-force-a-task-to-run'>Invalidating Shared State to Force a Task to Run</ulink>" | ||
2524 | sections both in the Yocto Project Development Tasks | ||
2525 | Manual. | ||
2526 | </para></listitem> | ||
2527 | </itemizedlist> | ||
2528 | </note> | ||
2529 | </para> | ||
2530 | |||
2531 | <para> | ||
2532 | The rest of this section goes into detail about the overall | ||
2533 | incremental build architecture, the checksums (signatures), and | ||
2534 | shared state. | ||
2535 | </para> | ||
2536 | |||
2537 | <section id='concepts-overall-architecture'> | ||
2538 | <title>Overall Architecture</title> | ||
2539 | |||
2540 | <para> | ||
2541 | When determining what parts of the system need to be built, | ||
2542 | BitBake works on a per-task basis rather than a per-recipe | ||
2543 | basis. | ||
2544 | You might wonder why using a per-task basis is preferred over | ||
2545 | a per-recipe basis. | ||
2546 | To help explain, consider having the IPK packaging backend | ||
2547 | enabled and then switching to DEB. | ||
2548 | In this case, the | ||
2549 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> | ||
2550 | and | ||
2551 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> | ||
2552 | task outputs are still valid. | ||
2553 | However, with a per-recipe approach, the build would not | ||
2554 | include the <filename>.deb</filename> files. | ||
2555 | Consequently, you would have to invalidate the whole build and | ||
2556 | rerun it. | ||
2557 | Rerunning everything is not the best solution. | ||
2558 | Also, in this case, the core must be "taught" much about | ||
2559 | specific tasks. | ||
2560 | This methodology does not scale well and does not allow users | ||
2561 | to easily add new tasks in layers or as external recipes | ||
2562 | without touching the packaged-staging core. | ||
2563 | </para> | ||
2564 | </section> | ||
2565 | |||
2566 | <section id='overview-checksums'> | ||
2567 | <title>Checksums (Signatures)</title> | ||
2568 | |||
2569 | <para> | ||
2570 | The shared state code uses a checksum, which is a unique | ||
2571 | signature of a task's inputs, to determine if a task needs to | ||
2572 | be run again. | ||
2573 | Because it is a change in a task's inputs that triggers a | ||
2574 | rerun, the process needs to detect all the inputs to a given | ||
2575 | task. | ||
2576 | For shell tasks, this turns out to be fairly easy because | ||
2577 | the build process generates a "run" shell script for each task | ||
2578 | and it is possible to create a checksum that gives you a good | ||
2579 | idea of when the task's data changes. | ||
2580 | </para> | ||
2581 | |||
2582 | <para> | ||
2583 | To complicate the problem, there are things that should not be | ||
2584 | included in the checksum. | ||
2585 | First, there is the actual specific build path of a given | ||
2586 | task - the | ||
2587 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. | ||
2588 | It does not matter if the work directory changes because it | ||
2589 | should not affect the output for target packages. | ||
2590 | Also, the build process has the objective of making native | ||
2591 | or cross packages relocatable. | ||
2592 | <note> | ||
2593 | Both native and cross packages run on the | ||
2594 | <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>. | ||
2595 | However, cross packages generate output for the target | ||
2596 | architecture. | ||
2597 | </note> | ||
2598 | The checksum therefore needs to exclude | ||
2599 | <filename>WORKDIR</filename>. | ||
2600 | The simplistic approach for excluding the work directory is to | ||
2601 | set <filename>WORKDIR</filename> to some fixed value and | ||
2602 | create the checksum for the "run" script. | ||
2603 | </para> | ||
2604 | |||
2605 | <para> | ||
2606 | Another problem results from the "run" scripts containing | ||
2607 | functions that might or might not get called. | ||
2608 | The incremental build solution contains code that figures out | ||
2609 | dependencies between shell functions. | ||
2610 | This code is used to prune the "run" scripts down to the | ||
2611 | minimum set, thereby alleviating this problem and making the | ||
2612 | "run" scripts much more readable as a bonus. | ||
2613 | </para> | ||
2614 | |||
2615 | <para> | ||
2616 | So far, solutions for shell scripts exist. | ||
2617 | What about Python tasks? | ||
2618 | The same approach applies even though these tasks are more | ||
2619 | difficult. | ||
2620 | The process needs to figure out what variables a Python | ||
2621 | function accesses and what functions it calls. | ||
2622 | Again, the incremental build solution contains code that first | ||
2623 | figures out the variable and function dependencies, and then | ||
2624 | creates a checksum for the data used as the input to the task. | ||
2625 | </para> | ||
2626 | |||
2627 | <para> | ||
2628 | Like the <filename>WORKDIR</filename> case, situations exist | ||
2629 | where dependencies should be ignored. | ||
2630 | For these situations, you can instruct the build process to | ||
2631 | ignore a dependency by using a line like the following: | ||
2632 | <literallayout class='monospaced'> | ||
2633 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
2634 | </literallayout> | ||
2635 | This example ensures that the | ||
2636 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink> | ||
2637 | variable does not depend on the value of | ||
2638 | <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, | ||
2639 | even if it does reference it. | ||
2640 | </para> | ||
2641 | |||
2642 | <para> | ||
2643 | Equally, there are cases where you need to add dependencies | ||
2644 | BitBake is not able to find. | ||
2645 | You can accomplish this by using a line like the following: | ||
2646 | <literallayout class='monospaced'> | ||
2647 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
2648 | </literallayout> | ||
2649 | This example explicitly adds the <filename>MACHINE</filename> | ||
2650 | variable as a dependency for | ||
2651 | <filename>PACKAGE_ARCHS</filename>. | ||
2652 | </para> | ||
2653 | |||
2654 | <para> | ||
2655 | As an example, consider a case with in-line Python where | ||
2656 | BitBake is not able to figure out dependencies. | ||
2657 | When running in debug mode (i.e. using | ||
2658 | <filename>-DDD</filename>), BitBake produces output when it | ||
2659 | discovers something for which it cannot figure out dependencies. | ||
2660 | The Yocto Project team has currently not managed to cover | ||
2661 | those dependencies in detail and is aware of the need to fix | ||
2662 | this situation. | ||
2663 | </para> | ||
2664 | |||
2665 | <para> | ||
2666 | Thus far, this section has limited discussion to the direct | ||
2667 | inputs into a task. | ||
2668 | Information based on direct inputs is referred to as the | ||
2669 | "basehash" in the code. | ||
2670 | However, the question of a task's indirect inputs still | ||
2671 | exits - items already built and present in the | ||
2672 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. | ||
2673 | The checksum (or signature) for a particular task needs to add | ||
2674 | the hashes of all the tasks on which the particular task | ||
2675 | depends. | ||
2676 | Choosing which dependencies to add is a policy decision. | ||
2677 | However, the effect is to generate a master checksum that | ||
2678 | combines the basehash and the hashes of the task's | ||
2679 | dependencies. | ||
2680 | </para> | ||
2681 | |||
2682 | <para> | ||
2683 | At the code level, a variety of ways exist by which both the | ||
2684 | basehash and the dependent task hashes can be influenced. | ||
2685 | Within the BitBake configuration file, you can give BitBake | ||
2686 | some extra information to help it construct the basehash. | ||
2687 | The following statement effectively results in a list of | ||
2688 | global variable dependency excludes (i.e. variables never | ||
2689 | included in any checksum): | ||
2690 | <literallayout class='monospaced'> | ||
2691 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ | ||
2692 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ | ||
2693 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ | ||
2694 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ | ||
2695 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
2696 | </literallayout> | ||
2697 | The previous example excludes | ||
2698 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> | ||
2699 | since that variable is actually constructed as a path within | ||
2700 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>, | ||
2701 | which is on the whitelist. | ||
2702 | </para> | ||
2703 | |||
2704 | <para> | ||
2705 | The rules for deciding which hashes of dependent tasks to | ||
2706 | include through dependency chains are more complex and are | ||
2707 | generally accomplished with a Python function. | ||
2708 | The code in <filename>meta/lib/oe/sstatesig.py</filename> shows | ||
2709 | two examples of this and also illustrates how you can insert | ||
2710 | your own policy into the system if so desired. | ||
2711 | This file defines the two basic signature generators | ||
2712 | <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink> | ||
2713 | uses: "OEBasic" and "OEBasicHash". | ||
2714 | By default, a dummy "noop" signature handler is enabled | ||
2715 | in BitBake. | ||
2716 | This means that behavior is unchanged from previous versions. | ||
2717 | OE-Core uses the "OEBasicHash" signature handler by default | ||
2718 | through this setting in the <filename>bitbake.conf</filename> | ||
2719 | file: | ||
2720 | <literallayout class='monospaced'> | ||
2721 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
2722 | </literallayout> | ||
2723 | The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> | ||
2724 | is the same as the "OEBasic" version but adds the task hash to | ||
2725 | the | ||
2726 | <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp files</link>. | ||
2727 | This results in any metadata change that changes the task hash, | ||
2728 | automatically causing the task to be run again. | ||
2729 | This removes the need to bump | ||
2730 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> | ||
2731 | values, and changes to metadata automatically ripple across | ||
2732 | the build. | ||
2733 | </para> | ||
2734 | |||
2735 | <para> | ||
2736 | It is also worth noting that the end result of these | ||
2737 | signature generators is to make some dependency and hash | ||
2738 | information available to the build. | ||
2739 | This information includes: | ||
2740 | <itemizedlist> | ||
2741 | <listitem><para> | ||
2742 | <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: | ||
2743 | The base hashes for each task in the recipe. | ||
2744 | </para></listitem> | ||
2745 | <listitem><para> | ||
2746 | <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: | ||
2747 | The base hashes for each dependent task. | ||
2748 | </para></listitem> | ||
2749 | <listitem><para> | ||
2750 | <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: | ||
2751 | The task dependencies for each task. | ||
2752 | </para></listitem> | ||
2753 | <listitem><para> | ||
2754 | <filename>BB_TASKHASH</filename>: | ||
2755 | The hash of the currently running task. | ||
2756 | </para></listitem> | ||
2757 | </itemizedlist> | ||
2758 | </para> | ||
2759 | </section> | ||
2760 | |||
2761 | <section id='shared-state'> | ||
2762 | <title>Shared State</title> | ||
2763 | |||
2764 | <para> | ||
2765 | Checksums and dependencies, as discussed in the previous | ||
2766 | section, solve half the problem of supporting a shared state. | ||
2767 | The other half of the problem is being able to use checksum | ||
2768 | information during the build and being able to reuse or rebuild | ||
2769 | specific components. | ||
2770 | </para> | ||
2771 | |||
2772 | <para> | ||
2773 | The | ||
2774 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> | ||
2775 | class is a relatively generic implementation of how to | ||
2776 | "capture" a snapshot of a given task. | ||
2777 | The idea is that the build process does not care about the | ||
2778 | source of a task's output. | ||
2779 | Output could be freshly built or it could be downloaded and | ||
2780 | unpacked from somewhere. | ||
2781 | In other words, the build process does not need to worry about | ||
2782 | its origin. | ||
2783 | </para> | ||
2784 | |||
2785 | <para> | ||
2786 | Two types of output exist. | ||
2787 | One type is just about creating a directory in | ||
2788 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. | ||
2789 | A good example is the output of either | ||
2790 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> | ||
2791 | or | ||
2792 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>. | ||
2793 | The other type of output occurs when a set of data is merged | ||
2794 | into a shared directory tree such as the sysroot. | ||
2795 | </para> | ||
2796 | |||
2797 | <para> | ||
2798 | The Yocto Project team has tried to keep the details of the | ||
2799 | implementation hidden in <filename>sstate</filename> class. | ||
2800 | From a user's perspective, adding shared state wrapping to a | ||
2801 | task is as simple as this | ||
2802 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> | ||
2803 | example taken from the | ||
2804 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink> | ||
2805 | class: | ||
2806 | <literallayout class='monospaced'> | ||
2807 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" | ||
2808 | SSTATETASKS += "do_deploy" | ||
2809 | do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" | ||
2810 | do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" | ||
2811 | |||
2812 | python do_deploy_setscene () { | ||
2813 | sstate_setscene(d) | ||
2814 | } | ||
2815 | addtask do_deploy_setscene | ||
2816 | do_deploy[dirs] = "${DEPLOYDIR} ${B}" | ||
2817 | do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" | ||
2818 | </literallayout> | ||
2819 | The following list explains the previous example: | ||
2820 | <itemizedlist> | ||
2821 | <listitem><para> | ||
2822 | Adding "do_deploy" to <filename>SSTATETASKS</filename> | ||
2823 | adds some required sstate-related processing, which is | ||
2824 | implemented in the | ||
2825 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> | ||
2826 | class, to before and after the | ||
2827 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> | ||
2828 | task. | ||
2829 | </para></listitem> | ||
2830 | <listitem><para> | ||
2831 | The | ||
2832 | <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename> | ||
2833 | declares that <filename>do_deploy</filename> places its | ||
2834 | output in <filename>${DEPLOYDIR}</filename> when run | ||
2835 | normally (i.e. when not using the sstate cache). | ||
2836 | This output becomes the input to the shared state cache. | ||
2837 | </para></listitem> | ||
2838 | <listitem><para> | ||
2839 | The | ||
2840 | <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename> | ||
2841 | line causes the contents of the shared state cache to be | ||
2842 | copied to <filename>${DEPLOY_DIR_IMAGE}</filename>. | ||
2843 | <note> | ||
2844 | If <filename>do_deploy</filename> is not already in | ||
2845 | the shared state cache or if its input checksum | ||
2846 | (signature) has changed from when the output was | ||
2847 | cached, the task runs to populate the shared | ||
2848 | state cache, after which the contents of the shared | ||
2849 | state cache is copied to | ||
2850 | <filename>${DEPLOY_DIR_IMAGE}</filename>. | ||
2851 | If <filename>do_deploy</filename> is in the shared | ||
2852 | state cache and its signature indicates that the | ||
2853 | cached output is still valid (i.e. if no | ||
2854 | relevant task inputs have changed), then the | ||
2855 | contents of the shared state cache copies | ||
2856 | directly to | ||
2857 | <filename>${DEPLOY_DIR_IMAGE}</filename> by the | ||
2858 | <filename>do_deploy_setscene</filename> task | ||
2859 | instead, skipping the | ||
2860 | <filename>do_deploy</filename> task. | ||
2861 | </note> | ||
2862 | </para></listitem> | ||
2863 | <listitem><para> | ||
2864 | The following task definition is glue logic needed to | ||
2865 | make the previous settings effective: | ||
2866 | <literallayout class='monospaced'> | ||
2867 | python do_deploy_setscene () { | ||
2868 | sstate_setscene(d) | ||
2869 | } | ||
2870 | addtask do_deploy_setscene | ||
2871 | </literallayout> | ||
2872 | <filename>sstate_setscene()</filename> takes the flags | ||
2873 | above as input and accelerates the | ||
2874 | <filename>do_deploy</filename> task through the | ||
2875 | shared state cache if possible. | ||
2876 | If the task was accelerated, | ||
2877 | <filename>sstate_setscene()</filename> returns True. | ||
2878 | Otherwise, it returns False, and the normal | ||
2879 | <filename>do_deploy</filename> task runs. | ||
2880 | For more information, see the | ||
2881 | "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>" | ||
2882 | section in the BitBake User Manual. | ||
2883 | </para></listitem> | ||
2884 | <listitem><para> | ||
2885 | The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename> | ||
2886 | line creates <filename>${DEPLOYDIR}</filename> and | ||
2887 | <filename>${B}</filename> before the | ||
2888 | <filename>do_deploy</filename> task runs, and also sets | ||
2889 | the current working directory of | ||
2890 | <filename>do_deploy</filename> to | ||
2891 | <filename>${B}</filename>. | ||
2892 | For more information, see the | ||
2893 | "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" | ||
2894 | section in the BitBake User Manual. | ||
2895 | <note> | ||
2896 | In cases where | ||
2897 | <filename>sstate-inputdirs</filename> and | ||
2898 | <filename>sstate-outputdirs</filename> would be the | ||
2899 | same, you can use | ||
2900 | <filename>sstate-plaindirs</filename>. | ||
2901 | For example, to preserve the | ||
2902 | <filename>${PKGD}</filename> and | ||
2903 | <filename>${PKGDEST}</filename> output from the | ||
2904 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> | ||
2905 | task, use the following: | ||
2906 | <literallayout class='monospaced'> | ||
2907 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | ||
2908 | </literallayout> | ||
2909 | </note> | ||
2910 | </para></listitem> | ||
2911 | <listitem><para> | ||
2912 | The <filename>do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"</filename> | ||
2913 | line appends extra metadata to the | ||
2914 | <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp file</link>. | ||
2915 | In this case, the metadata makes the task specific | ||
2916 | to a machine's architecture. | ||
2917 | See | ||
2918 | "<ulink url='&YOCTO_DOCS_BB_URL;#ref-bitbake-tasklist'>The Task List</ulink>" | ||
2919 | section in the BitBake User Manual for more | ||
2920 | information on the <filename>stamp-extra-info</filename> | ||
2921 | flag. | ||
2922 | </para></listitem> | ||
2923 | <listitem><para> | ||
2924 | <filename>sstate-inputdirs</filename> and | ||
2925 | <filename>sstate-outputdirs</filename> can also be used | ||
2926 | with multiple directories. | ||
2927 | For example, the following declares | ||
2928 | <filename>PKGDESTWORK</filename> and | ||
2929 | <filename>SHLIBWORK</filename> as shared state | ||
2930 | input directories, which populates the shared state | ||
2931 | cache, and <filename>PKGDATA_DIR</filename> and | ||
2932 | <filename>SHLIBSDIR</filename> as the corresponding | ||
2933 | shared state output directories: | ||
2934 | <literallayout class='monospaced'> | ||
2935 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" | ||
2936 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" | ||
2937 | </literallayout> | ||
2938 | </para></listitem> | ||
2939 | <listitem><para> | ||
2940 | These methods also include the ability to take a | ||
2941 | lockfile when manipulating shared state directory | ||
2942 | structures, for cases where file additions or removals | ||
2943 | are sensitive: | ||
2944 | <literallayout class='monospaced'> | ||
2945 | do_package[sstate-lockfile] = "${PACKAGELOCK}" | ||
2946 | </literallayout> | ||
2947 | </para></listitem> | ||
2948 | </itemizedlist> | ||
2949 | </para> | ||
2950 | |||
2951 | <para> | ||
2952 | Behind the scenes, the shared state code works by looking in | ||
2953 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> | ||
2954 | and | ||
2955 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink> | ||
2956 | for shared state files. | ||
2957 | Here is an example: | ||
2958 | <literallayout class='monospaced'> | ||
2959 | SSTATE_MIRRORS ?= "\ | ||
2960 | file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ | ||
2961 | file://.* file:///some/local/dir/sstate/PATH" | ||
2962 | </literallayout> | ||
2963 | <note> | ||
2964 | The shared state directory | ||
2965 | (<filename>SSTATE_DIR</filename>) is organized into | ||
2966 | two-character subdirectories, where the subdirectory | ||
2967 | names are based on the first two characters of the hash. | ||
2968 | If the shared state directory structure for a mirror has the | ||
2969 | same structure as <filename>SSTATE_DIR</filename>, you must | ||
2970 | specify "PATH" as part of the URI to enable the build system | ||
2971 | to map to the appropriate subdirectory. | ||
2972 | </note> | ||
2973 | </para> | ||
2974 | |||
2975 | <para> | ||
2976 | The shared state package validity can be detected just by | ||
2977 | looking at the filename since the filename contains the task | ||
2978 | checksum (or signature) as described earlier in this section. | ||
2979 | If a valid shared state package is found, the build process | ||
2980 | downloads it and uses it to accelerate the task. | ||
2981 | </para> | ||
2982 | |||
2983 | <para> | ||
2984 | The build processes use the <filename>*_setscene</filename> | ||
2985 | tasks for the task acceleration phase. | ||
2986 | BitBake goes through this phase before the main execution | ||
2987 | code and tries to accelerate any tasks for which it can find | ||
2988 | shared state packages. | ||
2989 | If a shared state package for a task is available, the | ||
2990 | shared state package is used. | ||
2991 | This means the task and any tasks on which it is dependent | ||
2992 | are not executed. | ||
2993 | </para> | ||
2994 | |||
2995 | <para> | ||
2996 | As a real world example, the aim is when building an IPK-based | ||
2997 | image, only the | ||
2998 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink> | ||
2999 | tasks would have their shared state packages fetched and | ||
3000 | extracted. | ||
3001 | Since the sysroot is not used, it would never get extracted. | ||
3002 | This is another reason why a task-based approach is preferred | ||
3003 | over a recipe-based approach, which would have to install the | ||
3004 | output from every task. | ||
3005 | </para> | ||
3006 | </section> | ||
3007 | </section> | ||
3008 | |||
3009 | <section id='automatically-added-runtime-dependencies'> | ||
3010 | <title>Automatically Added Runtime Dependencies</title> | ||
3011 | |||
3012 | <para> | ||
3013 | The OpenEmbedded build system automatically adds common types of | ||
3014 | runtime dependencies between packages, which means that you do not | ||
3015 | need to explicitly declare the packages using | ||
3016 | <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>. | ||
3017 | Three automatic mechanisms exist (<filename>shlibdeps</filename>, | ||
3018 | <filename>pcdeps</filename>, and <filename>depchains</filename>) | ||
3019 | that handle shared libraries, package configuration (pkg-config) | ||
3020 | modules, and <filename>-dev</filename> and | ||
3021 | <filename>-dbg</filename> packages, respectively. | ||
3022 | For other types of runtime dependencies, you must manually declare | ||
3023 | the dependencies. | ||
3024 | <itemizedlist> | ||
3025 | <listitem><para> | ||
3026 | <filename>shlibdeps</filename>: | ||
3027 | During the | ||
3028 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> | ||
3029 | task of each recipe, all shared libraries installed by the | ||
3030 | recipe are located. | ||
3031 | For each shared library, the package that contains the | ||
3032 | shared library is registered as providing the shared | ||
3033 | library. | ||
3034 | More specifically, the package is registered as providing | ||
3035 | the | ||
3036 | <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink> | ||
3037 | of the library. | ||
3038 | The resulting shared-library-to-package mapping | ||
3039 | is saved globally in | ||
3040 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> | ||
3041 | by the | ||
3042 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink> | ||
3043 | task.</para> | ||
3044 | |||
3045 | <para>Simultaneously, all executables and shared libraries | ||
3046 | installed by the recipe are inspected to see what shared | ||
3047 | libraries they link against. | ||
3048 | For each shared library dependency that is found, | ||
3049 | <filename>PKGDATA_DIR</filename> is queried to | ||
3050 | see if some package (likely from a different recipe) | ||
3051 | contains the shared library. | ||
3052 | If such a package is found, a runtime dependency is added | ||
3053 | from the package that depends on the shared library to the | ||
3054 | package that contains the library.</para> | ||
3055 | |||
3056 | <para>The automatically added runtime dependency also | ||
3057 | includes a version restriction. | ||
3058 | This version restriction specifies that at least the | ||
3059 | current version of the package that provides the shared | ||
3060 | library must be used, as if | ||
3061 | "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)" | ||
3062 | had been added to <filename>RDEPENDS</filename>. | ||
3063 | This forces an upgrade of the package containing the shared | ||
3064 | library when installing the package that depends on the | ||
3065 | library, if needed.</para> | ||
3066 | |||
3067 | <para>If you want to avoid a package being registered as | ||
3068 | providing a particular shared library (e.g. because the library | ||
3069 | is for internal use only), then add the library to | ||
3070 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink> | ||
3071 | inside the package's recipe. | ||
3072 | </para></listitem> | ||
3073 | <listitem><para> | ||
3074 | <filename>pcdeps</filename>: | ||
3075 | During the <filename>do_package</filename> task of each | ||
3076 | recipe, all pkg-config modules | ||
3077 | (<filename>*.pc</filename> files) installed by the recipe | ||
3078 | are located. | ||
3079 | For each module, the package that contains the module is | ||
3080 | registered as providing the module. | ||
3081 | The resulting module-to-package mapping is saved globally in | ||
3082 | <filename>PKGDATA_DIR</filename> by the | ||
3083 | <filename>do_packagedata</filename> task.</para> | ||
3084 | |||
3085 | <para>Simultaneously, all pkg-config modules installed by | ||
3086 | the recipe are inspected to see what other pkg-config | ||
3087 | modules they depend on. | ||
3088 | A module is seen as depending on another module if it | ||
3089 | contains a "Requires:" line that specifies the other module. | ||
3090 | For each module dependency, | ||
3091 | <filename>PKGDATA_DIR</filename> is queried to see if some | ||
3092 | package contains the module. | ||
3093 | If such a package is found, a runtime dependency is added | ||
3094 | from the package that depends on the module to the package | ||
3095 | that contains the module. | ||
3096 | <note> | ||
3097 | The <filename>pcdeps</filename> mechanism most often | ||
3098 | infers dependencies between <filename>-dev</filename> | ||
3099 | packages. | ||
3100 | </note> | ||
3101 | </para></listitem> | ||
3102 | <listitem><para> | ||
3103 | <filename>depchains</filename>: | ||
3104 | If a package <filename>foo</filename> depends on a package | ||
3105 | <filename>bar</filename>, then <filename>foo-dev</filename> | ||
3106 | and <filename>foo-dbg</filename> are also made to depend on | ||
3107 | <filename>bar-dev</filename> and | ||
3108 | <filename>bar-dbg</filename>, respectively. | ||
3109 | Taking the <filename>-dev</filename> packages as an | ||
3110 | example, the <filename>bar-dev</filename> package might | ||
3111 | provide headers and shared library symlinks needed by | ||
3112 | <filename>foo-dev</filename>, which shows the need | ||
3113 | for a dependency between the packages.</para> | ||
3114 | |||
3115 | <para>The dependencies added by | ||
3116 | <filename>depchains</filename> are in the form of | ||
3117 | <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>. | ||
3118 | <note> | ||
3119 | By default, <filename>foo-dev</filename> also has an | ||
3120 | <filename>RDEPENDS</filename>-style dependency on | ||
3121 | <filename>foo</filename>, because the default value of | ||
3122 | <filename>RDEPENDS_${PN}-dev</filename> (set in | ||
3123 | <filename>bitbake.conf</filename>) includes | ||
3124 | "${PN}". | ||
3125 | </note></para> | ||
3126 | |||
3127 | <para>To ensure that the dependency chain is never broken, | ||
3128 | <filename>-dev</filename> and <filename>-dbg</filename> | ||
3129 | packages are always generated by default, even if the | ||
3130 | packages turn out to be empty. | ||
3131 | See the | ||
3132 | <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink> | ||
3133 | variable for more information. | ||
3134 | </para></listitem> | ||
3135 | </itemizedlist> | ||
3136 | </para> | ||
3137 | |||
3138 | <para> | ||
3139 | The <filename>do_package</filename> task depends on the | ||
3140 | <filename>do_packagedata</filename> task of each recipe in | ||
3141 | <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> | ||
3142 | through use of a | ||
3143 | <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename> | ||
3144 | declaration, which guarantees that the required | ||
3145 | shared-library/module-to-package mapping information will be available | ||
3146 | when needed as long as <filename>DEPENDS</filename> has been | ||
3147 | correctly set. | ||
3148 | </para> | ||
3149 | </section> | ||
3150 | |||
3151 | <section id='fakeroot-and-pseudo'> | ||
3152 | <title>Fakeroot and Pseudo</title> | ||
3153 | |||
3154 | <para> | ||
3155 | Some tasks are easier to implement when allowed to perform certain | ||
3156 | operations that are normally reserved for the root user (e.g. | ||
3157 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>, | ||
3158 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>, | ||
3159 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>, | ||
3160 | and | ||
3161 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>). | ||
3162 | For example, the <filename>do_install</filename> task benefits | ||
3163 | from being able to set the UID and GID of installed files to | ||
3164 | arbitrary values. | ||
3165 | </para> | ||
3166 | |||
3167 | <para> | ||
3168 | One approach to allowing tasks to perform root-only operations | ||
3169 | would be to require | ||
3170 | <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> | ||
3171 | to run as root. | ||
3172 | However, this method is cumbersome and has security issues. | ||
3173 | The approach that is actually used is to run tasks that benefit | ||
3174 | from root privileges in a "fake" root environment. | ||
3175 | Within this environment, the task and its child processes believe | ||
3176 | that they are running as the root user, and see an internally | ||
3177 | consistent view of the filesystem. | ||
3178 | As long as generating the final output (e.g. a package or an image) | ||
3179 | does not require root privileges, the fact that some earlier | ||
3180 | steps ran in a fake root environment does not cause problems. | ||
3181 | </para> | ||
3182 | |||
3183 | <para> | ||
3184 | The capability to run tasks in a fake root environment is known as | ||
3185 | "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>", | ||
3186 | which is derived from the BitBake keyword/variable | ||
3187 | flag that requests a fake root environment for a task. | ||
3188 | </para> | ||
3189 | |||
3190 | <para> | ||
3191 | In the | ||
3192 | <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>, | ||
3193 | the program that implements fakeroot is known as | ||
3194 | <ulink url='https://www.yoctoproject.org/software-item/pseudo/'>Pseudo</ulink>. | ||
3195 | Pseudo overrides system calls by using the environment variable | ||
3196 | <filename>LD_PRELOAD</filename>, which results in the illusion | ||
3197 | of running as root. | ||
3198 | To keep track of "fake" file ownership and permissions resulting | ||
3199 | from operations that require root permissions, Pseudo uses | ||
3200 | an SQLite 3 database. | ||
3201 | This database is stored in | ||
3202 | <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename> | ||
3203 | for individual recipes. | ||
3204 | Storing the database in a file as opposed to in memory | ||
3205 | gives persistence between tasks and builds, which is not | ||
3206 | accomplished using fakeroot. | ||
3207 | <note><title>Caution</title> | ||
3208 | If you add your own task that manipulates the same files or | ||
3209 | directories as a fakeroot task, then that task also needs to | ||
3210 | run under fakeroot. | ||
3211 | Otherwise, the task cannot run root-only operations, and | ||
3212 | cannot see the fake file ownership and permissions set by the | ||
3213 | other task. | ||
3214 | You need to also add a dependency on | ||
3215 | <filename>virtual/fakeroot-native:do_populate_sysroot</filename>, | ||
3216 | giving the following: | ||
3217 | <literallayout class='monospaced'> | ||
3218 | fakeroot do_mytask () { | ||
3219 | ... | ||
3220 | } | ||
3221 | do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" | ||
3222 | </literallayout> | ||
3223 | </note> | ||
3224 | For more information, see the | ||
3225 | <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink> | ||
3226 | variables in the BitBake User Manual. | ||
3227 | You can also reference the | ||
3228 | "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>" | ||
3229 | article for background information on Fakeroot and Pseudo. | ||
3230 | </para> | ||
3231 | </section> | ||
3232 | </chapter> | ||
3233 | <!-- | ||
3234 | vim: expandtab tw=80 ts=4 | ||
3235 | --> | ||