diff options
Diffstat (limited to 'documentation/overview-manual/overview-manual-concepts.rst')
-rw-r--r-- | documentation/overview-manual/overview-manual-concepts.rst | 2185 |
1 files changed, 2185 insertions, 0 deletions
diff --git a/documentation/overview-manual/overview-manual-concepts.rst b/documentation/overview-manual/overview-manual-concepts.rst new file mode 100644 index 0000000000..6ce5f80af3 --- /dev/null +++ b/documentation/overview-manual/overview-manual-concepts.rst | |||
@@ -0,0 +1,2185 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ********************** | ||
4 | Yocto Project Concepts | ||
5 | ********************** | ||
6 | |||
7 | This chapter provides explanations for Yocto Project concepts that go | ||
8 | beyond the surface of "how-to" information and reference (or look-up) | ||
9 | material. Concepts such as components, the :term:`OpenEmbedded Build System` | ||
10 | workflow, | ||
11 | cross-development toolchains, shared state cache, and so forth are | ||
12 | explained. | ||
13 | |||
14 | Yocto Project Components | ||
15 | ======================== | ||
16 | |||
17 | The :term:`BitBake` task executor | ||
18 | together with various types of configuration files form the | ||
19 | :term:`OpenEmbedded-Core (OE-Core)`. This section | ||
20 | overviews these components by describing their use and how they | ||
21 | interact. | ||
22 | |||
23 | BitBake handles the parsing and execution of the data files. The data | ||
24 | itself is of various types: | ||
25 | |||
26 | - *Recipes:* Provides details about particular pieces of software. | ||
27 | |||
28 | - *Class Data:* Abstracts common build information (e.g. how to build a | ||
29 | Linux kernel). | ||
30 | |||
31 | - *Configuration Data:* Defines machine-specific settings, policy | ||
32 | decisions, and so forth. Configuration data acts as the glue to bind | ||
33 | everything together. | ||
34 | |||
35 | BitBake knows how to combine multiple data sources together and refers | ||
36 | to each data source as a layer. For information on layers, see the | ||
37 | ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" | ||
38 | section of the Yocto Project Development Tasks Manual. | ||
39 | |||
40 | Following are some brief details on these core components. For | ||
41 | additional information on how these components interact during a build, | ||
42 | see the | ||
43 | ":ref:`overview-manual/overview-manual-concepts:openembedded build system concepts`" | ||
44 | section. | ||
45 | |||
46 | .. _usingpoky-components-bitbake: | ||
47 | |||
48 | BitBake | ||
49 | ------- | ||
50 | |||
51 | BitBake is the tool at the heart of the :term:`OpenEmbedded Build System` | ||
52 | and is responsible | ||
53 | for parsing the :term:`Metadata`, generating | ||
54 | a list of tasks from it, and then executing those tasks. | ||
55 | |||
56 | This section briefly introduces BitBake. If you want more information on | ||
57 | BitBake, see the :doc:`BitBake User Manual <bitbake:index>`. | ||
58 | |||
59 | To see a list of the options BitBake supports, use either of the | ||
60 | following commands: | ||
61 | :: | ||
62 | |||
63 | $ bitbake -h | ||
64 | $ bitbake --help | ||
65 | |||
66 | The most common usage for BitBake is ``bitbake recipename``, where | ||
67 | ``recipename`` is the name of the recipe you want to build (referred | ||
68 | to as the "target"). The target often equates to the first part of a | ||
69 | recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``). | ||
70 | So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might | ||
71 | type the following: | ||
72 | :: | ||
73 | |||
74 | $ bitbake matchbox-desktop | ||
75 | |||
76 | Several different | ||
77 | versions of ``matchbox-desktop`` might exist. BitBake chooses the one | ||
78 | selected by the distribution configuration. You can get more details | ||
79 | about how BitBake chooses between different target versions and | ||
80 | providers in the | ||
81 | ":ref:`Preferences <bitbake:bb-bitbake-preferences>`" section | ||
82 | of the BitBake User Manual. | ||
83 | |||
84 | BitBake also tries to execute any dependent tasks first. So for example, | ||
85 | before building ``matchbox-desktop``, BitBake would build a cross | ||
86 | compiler and ``glibc`` if they had not already been built. | ||
87 | |||
88 | A useful BitBake option to consider is the ``-k`` or ``--continue`` | ||
89 | option. This option instructs BitBake to try and continue processing the | ||
90 | job as long as possible even after encountering an error. When an error | ||
91 | occurs, the target that failed and those that depend on it cannot be | ||
92 | remade. However, when you use this option other dependencies can still | ||
93 | be processed. | ||
94 | |||
95 | .. _overview-components-recipes: | ||
96 | |||
97 | Recipes | ||
98 | ------- | ||
99 | |||
100 | Files that have the ``.bb`` suffix are "recipes" files. In general, a | ||
101 | recipe contains information about a single piece of software. This | ||
102 | information includes the location from which to download the unaltered | ||
103 | source, any source patches to be applied to that source (if needed), | ||
104 | which special configuration options to apply, how to compile the source | ||
105 | files, and how to package the compiled output. | ||
106 | |||
107 | The term "package" is sometimes used to refer to recipes. However, since | ||
108 | the word "package" is used for the packaged output from the OpenEmbedded | ||
109 | build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids | ||
110 | using the term "package" when referring to recipes. | ||
111 | |||
112 | .. _overview-components-classes: | ||
113 | |||
114 | Classes | ||
115 | ------- | ||
116 | |||
117 | Class files (``.bbclass``) contain information that is useful to share | ||
118 | between recipes files. An example is the | ||
119 | :ref:`autotools <ref-classes-autotools>` class, | ||
120 | which contains common settings for any application that Autotools uses. | ||
121 | The ":ref:`ref-manual/ref-classes:Classes`" chapter in the | ||
122 | Yocto Project Reference Manual provides details about classes and how to | ||
123 | use them. | ||
124 | |||
125 | .. _overview-components-configurations: | ||
126 | |||
127 | Configurations | ||
128 | -------------- | ||
129 | |||
130 | The configuration files (``.conf``) define various configuration | ||
131 | variables that govern the OpenEmbedded build process. These files fall | ||
132 | into several areas that define machine configuration options, | ||
133 | distribution configuration options, compiler tuning options, general | ||
134 | common configuration options, and user configuration options in | ||
135 | ``conf/local.conf``, which is found in the :term:`Build Directory`. | ||
136 | |||
137 | |||
138 | .. _overview-layers: | ||
139 | |||
140 | Layers | ||
141 | ====== | ||
142 | |||
143 | Layers are repositories that contain related metadata (i.e. sets of | ||
144 | instructions) that tell the OpenEmbedded build system how to build a | ||
145 | target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__ | ||
146 | facilitates collaboration, sharing, customization, and reuse within the | ||
147 | Yocto Project development environment. Layers logically separate | ||
148 | information for your project. For example, you can use a layer to hold | ||
149 | all the configurations for a particular piece of hardware. Isolating | ||
150 | hardware-specific configurations allows you to share other metadata by | ||
151 | using a different layer where that metadata might be common across | ||
152 | several pieces of hardware. | ||
153 | |||
154 | Many layers exist that work in the Yocto Project development | ||
155 | environment. The `Yocto Project Curated Layer | ||
156 | Index <https://www.yoctoproject.org/software-overview/layers/>`__ | ||
157 | and `OpenEmbedded Layer | ||
158 | Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__ | ||
159 | both contain layers from which you can use or leverage. | ||
160 | |||
161 | By convention, layers in the Yocto Project follow a specific form. | ||
162 | Conforming to a known structure allows BitBake to make assumptions | ||
163 | during builds on where to find types of metadata. You can find | ||
164 | procedures and learn about tools (i.e. ``bitbake-layers``) for creating | ||
165 | layers suitable for the Yocto Project in the | ||
166 | ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" | ||
167 | section of the Yocto Project Development Tasks Manual. | ||
168 | |||
169 | .. _openembedded-build-system-build-concepts: | ||
170 | |||
171 | OpenEmbedded Build System Concepts | ||
172 | ================================== | ||
173 | |||
174 | This section takes a more detailed look inside the build process used by | ||
175 | the :term:`OpenEmbedded Build System`, | ||
176 | which is the build | ||
177 | system specific to the Yocto Project. At the heart of the build system | ||
178 | is BitBake, the task executor. | ||
179 | |||
180 | The following diagram represents the high-level workflow of a build. The | ||
181 | remainder of this section expands on the fundamental input, output, | ||
182 | process, and metadata logical blocks that make up the workflow. | ||
183 | |||
184 | .. image:: figures/YP-flow-diagram.png | ||
185 | :align: center | ||
186 | |||
187 | In general, the build's workflow consists of several functional areas: | ||
188 | |||
189 | - *User Configuration:* metadata you can use to control the build | ||
190 | process. | ||
191 | |||
192 | - *Metadata Layers:* Various layers that provide software, machine, and | ||
193 | distro metadata. | ||
194 | |||
195 | - *Source Files:* Upstream releases, local projects, and SCMs. | ||
196 | |||
197 | - *Build System:* Processes under the control of | ||
198 | :term:`BitBake`. This block expands | ||
199 | on how BitBake fetches source, applies patches, completes | ||
200 | compilation, analyzes output for package generation, creates and | ||
201 | tests packages, generates images, and generates cross-development | ||
202 | tools. | ||
203 | |||
204 | - *Package Feeds:* Directories containing output packages (RPM, DEB or | ||
205 | IPK), which are subsequently used in the construction of an image or | ||
206 | Software Development Kit (SDK), produced by the build system. These | ||
207 | feeds can also be copied and shared using a web server or other means | ||
208 | to facilitate extending or updating existing images on devices at | ||
209 | runtime if runtime package management is enabled. | ||
210 | |||
211 | - *Images:* Images produced by the workflow. | ||
212 | |||
213 | - *Application Development SDK:* Cross-development tools that are | ||
214 | produced along with an image or separately with BitBake. | ||
215 | |||
216 | User Configuration | ||
217 | ------------------ | ||
218 | |||
219 | User configuration helps define the build. Through user configuration, | ||
220 | you can tell BitBake the target architecture for which you are building | ||
221 | the image, where to store downloaded source, and other build properties. | ||
222 | |||
223 | The following figure shows an expanded representation of the "User | ||
224 | Configuration" box of the `general workflow | ||
225 | figure <#general-workflow-figure>`__: | ||
226 | |||
227 | .. image:: figures/user-configuration.png | ||
228 | :align: center | ||
229 | |||
230 | BitBake needs some basic configuration files in order to complete a | ||
231 | build. These files are ``*.conf`` files. The minimally necessary ones | ||
232 | reside as example files in the ``build/conf`` directory of the | ||
233 | :term:`Source Directory`. For simplicity, | ||
234 | this section refers to the Source Directory as the "Poky Directory." | ||
235 | |||
236 | When you clone the :term:`Poky` Git repository | ||
237 | or you download and unpack a Yocto Project release, you can set up the | ||
238 | Source Directory to be named anything you want. For this discussion, the | ||
239 | cloned repository uses the default name ``poky``. | ||
240 | |||
241 | .. note:: | ||
242 | |||
243 | The Poky repository is primarily an aggregation of existing | ||
244 | repositories. It is not a canonical upstream source. | ||
245 | |||
246 | The ``meta-poky`` layer inside Poky contains a ``conf`` directory that | ||
247 | has example configuration files. These example files are used as a basis | ||
248 | for creating actual configuration files when you source | ||
249 | :ref:`structure-core-script`, which is the | ||
250 | build environment script. | ||
251 | |||
252 | Sourcing the build environment script creates a | ||
253 | :term:`Build Directory` if one does not | ||
254 | already exist. BitBake uses the Build Directory for all its work during | ||
255 | builds. The Build Directory has a ``conf`` directory that contains | ||
256 | default versions of your ``local.conf`` and ``bblayers.conf`` | ||
257 | configuration files. These default configuration files are created only | ||
258 | if versions do not already exist in the Build Directory at the time you | ||
259 | source the build environment setup script. | ||
260 | |||
261 | Because the Poky repository is fundamentally an aggregation of existing | ||
262 | repositories, some users might be familiar with running the | ||
263 | :ref:`structure-core-script` script in the context of separate | ||
264 | :term:`OpenEmbedded-Core (OE-Core)` and BitBake | ||
265 | repositories rather than a single Poky repository. This discussion | ||
266 | assumes the script is executed from within a cloned or unpacked version | ||
267 | of Poky. | ||
268 | |||
269 | Depending on where the script is sourced, different sub-scripts are | ||
270 | called to set up the Build Directory (Yocto or OpenEmbedded). | ||
271 | Specifically, the script ``scripts/oe-setup-builddir`` inside the poky | ||
272 | directory sets up the Build Directory and seeds the directory (if | ||
273 | necessary) with configuration files appropriate for the Yocto Project | ||
274 | development environment. | ||
275 | |||
276 | .. note:: | ||
277 | |||
278 | The | ||
279 | scripts/oe-setup-builddir | ||
280 | script uses the | ||
281 | ``$TEMPLATECONF`` | ||
282 | variable to determine which sample configuration files to locate. | ||
283 | |||
284 | The ``local.conf`` file provides many basic variables that define a | ||
285 | build environment. Here is a list of a few. To see the default | ||
286 | configurations in a ``local.conf`` file created by the build environment | ||
287 | script, see the | ||
288 | :yocto_git:`local.conf.sample </cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample>` | ||
289 | in the ``meta-poky`` layer: | ||
290 | |||
291 | - *Target Machine Selection:* Controlled by the | ||
292 | :term:`MACHINE` variable. | ||
293 | |||
294 | - *Download Directory:* Controlled by the | ||
295 | :term:`DL_DIR` variable. | ||
296 | |||
297 | - *Shared State Directory:* Controlled by the | ||
298 | :term:`SSTATE_DIR` variable. | ||
299 | |||
300 | - *Build Output:* Controlled by the | ||
301 | :term:`TMPDIR` variable. | ||
302 | |||
303 | - *Distribution Policy:* Controlled by the | ||
304 | :term:`DISTRO` variable. | ||
305 | |||
306 | - *Packaging Format:* Controlled by the | ||
307 | :term:`PACKAGE_CLASSES` | ||
308 | variable. | ||
309 | |||
310 | - *SDK Target Architecture:* Controlled by the | ||
311 | :term:`SDKMACHINE` variable. | ||
312 | |||
313 | - *Extra Image Packages:* Controlled by the | ||
314 | :term:`EXTRA_IMAGE_FEATURES` | ||
315 | variable. | ||
316 | |||
317 | .. note:: | ||
318 | |||
319 | Configurations set in the | ||
320 | conf/local.conf | ||
321 | file can also be set in the | ||
322 | conf/site.conf | ||
323 | and | ||
324 | conf/auto.conf | ||
325 | configuration files. | ||
326 | |||
327 | The ``bblayers.conf`` file tells BitBake what layers you want considered | ||
328 | during the build. By default, the layers listed in this file include | ||
329 | layers minimally needed by the build system. However, you must manually | ||
330 | add any custom layers you have created. You can find more information on | ||
331 | working with the ``bblayers.conf`` file in the | ||
332 | ":ref:`dev-manual/dev-manual-common-tasks:enabling your layer`" | ||
333 | section in the Yocto Project Development Tasks Manual. | ||
334 | |||
335 | The files ``site.conf`` and ``auto.conf`` are not created by the | ||
336 | environment initialization script. If you want the ``site.conf`` file, | ||
337 | you need to create that yourself. The ``auto.conf`` file is typically | ||
338 | created by an autobuilder: | ||
339 | |||
340 | - *site.conf:* You can use the ``conf/site.conf`` configuration | ||
341 | file to configure multiple build directories. For example, suppose | ||
342 | you had several build environments and they shared some common | ||
343 | features. You can set these default build properties here. A good | ||
344 | example is perhaps the packaging format to use through the | ||
345 | :term:`PACKAGE_CLASSES` | ||
346 | variable. | ||
347 | |||
348 | One useful scenario for using the ``conf/site.conf`` file is to | ||
349 | extend your :term:`BBPATH` variable | ||
350 | to include the path to a ``conf/site.conf``. Then, when BitBake looks | ||
351 | for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file | ||
352 | and applies your common configurations found in the file. To override | ||
353 | configurations in a particular build directory, alter the similar | ||
354 | configurations within that build directory's ``conf/local.conf`` | ||
355 | file. | ||
356 | |||
357 | - *auto.conf:* The file is usually created and written to by an | ||
358 | autobuilder. The settings put into the file are typically the same as | ||
359 | you would find in the ``conf/local.conf`` or the ``conf/site.conf`` | ||
360 | files. | ||
361 | |||
362 | You can edit all configuration files to further define any particular | ||
363 | build environment. This process is represented by the "User | ||
364 | Configuration Edits" box in the figure. | ||
365 | |||
366 | When you launch your build with the ``bitbake target`` command, BitBake | ||
367 | sorts out the configurations to ultimately define your build | ||
368 | environment. It is important to understand that the | ||
369 | :term:`OpenEmbedded Build System` reads the | ||
370 | configuration files in a specific order: ``site.conf``, ``auto.conf``, | ||
371 | and ``local.conf``. And, the build system applies the normal assignment | ||
372 | statement rules as described in the | ||
373 | ":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter | ||
374 | of the BitBake User Manual. Because the files are parsed in a specific | ||
375 | order, variable assignments for the same variable could be affected. For | ||
376 | example, if the ``auto.conf`` file and the ``local.conf`` set variable1 | ||
377 | to different values, because the build system parses ``local.conf`` | ||
378 | after ``auto.conf``, variable1 is assigned the value from the | ||
379 | ``local.conf`` file. | ||
380 | |||
381 | Metadata, Machine Configuration, and Policy Configuration | ||
382 | --------------------------------------------------------- | ||
383 | |||
384 | The previous section described the user configurations that define | ||
385 | BitBake's global behavior. This section takes a closer look at the | ||
386 | layers the build system uses to further control the build. These layers | ||
387 | provide Metadata for the software, machine, and policies. | ||
388 | |||
389 | In general, three types of layer input exists. You can see them below | ||
390 | the "User Configuration" box in the `general workflow | ||
391 | figure <#general-workflow-figure>`__: | ||
392 | |||
393 | - *Metadata (.bb + Patches):* Software layers containing | ||
394 | user-supplied recipe files, patches, and append files. A good example | ||
395 | of a software layer might be the | ||
396 | `meta-qt5 layer <https://github.com/meta-qt5/meta-qt5>`__ from | ||
397 | the `OpenEmbedded Layer | ||
398 | Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__. | ||
399 | This layer is for version 5.0 of the popular | ||
400 | `Qt <https://wiki.qt.io/About_Qt>`__ cross-platform application | ||
401 | development framework for desktop, embedded and mobile. | ||
402 | |||
403 | - *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e. | ||
404 | "BSP Layer" in the following figure) providing machine-specific | ||
405 | configurations. This type of information is specific to a particular | ||
406 | target architecture. A good example of a BSP layer from the `Poky | ||
407 | Reference Distribution <#gs-reference-distribution-poky>`__ is the | ||
408 | :yocto_git:`meta-yocto-bsp </cgit/cgit.cgi/poky/tree/meta-yocto-bsp>` | ||
409 | layer. | ||
410 | |||
411 | - *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in | ||
412 | the following figure) providing top-level or general policies for the | ||
413 | images or SDKs being built for a particular distribution. For | ||
414 | example, in the Poky Reference Distribution the distro layer is the | ||
415 | :yocto_git:`meta-poky </cgit/cgit.cgi/poky/tree/meta-poky>` | ||
416 | layer. Within the distro layer is a ``conf/distro`` directory that | ||
417 | contains distro configuration files (e.g. | ||
418 | :yocto_git:`poky.conf </cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf>` | ||
419 | that contain many policy configurations for the Poky distribution. | ||
420 | |||
421 | The following figure shows an expanded representation of these three | ||
422 | layers from the `general workflow figure <#general-workflow-figure>`__: | ||
423 | |||
424 | .. image:: figures/layer-input.png | ||
425 | :align: center | ||
426 | |||
427 | In general, all layers have a similar structure. They all contain a | ||
428 | licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed, | ||
429 | a ``README`` file as good practice and especially if the layer is to be | ||
430 | distributed, a configuration directory, and recipe directories. You can | ||
431 | learn about the general structure for layers used with the Yocto Project | ||
432 | in the | ||
433 | ":ref:`dev-manual/dev-manual-common-tasks:creating your own layer`" | ||
434 | section in the | ||
435 | Yocto Project Development Tasks Manual. For a general discussion on | ||
436 | layers and the many layers from which you can draw, see the | ||
437 | "`Layers <#overview-layers>`__" and "`The Yocto Project Layer | ||
438 | Model <#the-yocto-project-layer-model>`__" sections both earlier in this | ||
439 | manual. | ||
440 | |||
441 | If you explored the previous links, you discovered some areas where many | ||
442 | layers that work with the Yocto Project exist. The `Source | ||
443 | Repositories <http://git.yoctoproject.org/>`__ also shows layers | ||
444 | categorized under "Yocto Metadata Layers." | ||
445 | |||
446 | .. note:: | ||
447 | |||
448 | Layers exist in the Yocto Project Source Repositories that cannot be | ||
449 | found in the OpenEmbedded Layer Index. These layers are either | ||
450 | deprecated or experimental in nature. | ||
451 | |||
452 | BitBake uses the ``conf/bblayers.conf`` file, which is part of the user | ||
453 | configuration, to find what layers it should be using as part of the | ||
454 | build. | ||
455 | |||
456 | Distro Layer | ||
457 | ~~~~~~~~~~~~ | ||
458 | |||
459 | The distribution layer provides policy configurations for your | ||
460 | distribution. Best practices dictate that you isolate these types of | ||
461 | configurations into their own layer. Settings you provide in | ||
462 | ``conf/distro/distro.conf`` override similar settings that BitBake finds | ||
463 | in your ``conf/local.conf`` file in the Build Directory. | ||
464 | |||
465 | The following list provides some explanation and references for what you | ||
466 | typically find in the distribution layer: | ||
467 | |||
468 | - *classes:* Class files (``.bbclass``) hold common functionality that | ||
469 | can be shared among recipes in the distribution. When your recipes | ||
470 | inherit a class, they take on the settings and functions for that | ||
471 | class. You can read more about class files in the | ||
472 | ":ref:`ref-manual/ref-classes:Classes`" chapter of the Yocto | ||
473 | Reference Manual. | ||
474 | |||
475 | - *conf:* This area holds configuration files for the layer | ||
476 | (``conf/layer.conf``), the distribution | ||
477 | (``conf/distro/distro.conf``), and any distribution-wide include | ||
478 | files. | ||
479 | |||
480 | - *recipes-*:* Recipes and append files that affect common | ||
481 | functionality across the distribution. This area could include | ||
482 | recipes and append files to add distribution-specific configuration, | ||
483 | initialization scripts, custom image recipes, and so forth. Examples | ||
484 | of ``recipes-*`` directories are ``recipes-core`` and | ||
485 | ``recipes-extra``. Hierarchy and contents within a ``recipes-*`` | ||
486 | directory can vary. Generally, these directories contain recipe files | ||
487 | (``*.bb``), recipe append files (``*.bbappend``), directories that | ||
488 | are distro-specific for configuration files, and so forth. | ||
489 | |||
490 | BSP Layer | ||
491 | ~~~~~~~~~ | ||
492 | |||
493 | The BSP Layer provides machine configurations that target specific | ||
494 | hardware. Everything in this layer is specific to the machine for which | ||
495 | you are building the image or the SDK. A common structure or form is | ||
496 | defined for BSP layers. You can learn more about this structure in the | ||
497 | :doc:`../bsp-guide/bsp-guide`. | ||
498 | |||
499 | .. note:: | ||
500 | |||
501 | In order for a BSP layer to be considered compliant with the Yocto | ||
502 | Project, it must meet some structural requirements. | ||
503 | |||
504 | The BSP Layer's configuration directory contains configuration files for | ||
505 | the machine (``conf/machine/machine.conf``) and, of course, the layer | ||
506 | (``conf/layer.conf``). | ||
507 | |||
508 | The remainder of the layer is dedicated to specific recipes by function: | ||
509 | ``recipes-bsp``, ``recipes-core``, ``recipes-graphics``, | ||
510 | ``recipes-kernel``, and so forth. Metadata can exist for multiple | ||
511 | formfactors, graphics support systems, and so forth. | ||
512 | |||
513 | .. note:: | ||
514 | |||
515 | While the figure shows several | ||
516 | recipes-\* | ||
517 | directories, not all these directories appear in all BSP layers. | ||
518 | |||
519 | Software Layer | ||
520 | ~~~~~~~~~~~~~~ | ||
521 | |||
522 | The software layer provides the Metadata for additional software | ||
523 | packages used during the build. This layer does not include Metadata | ||
524 | that is specific to the distribution or the machine, which are found in | ||
525 | their respective layers. | ||
526 | |||
527 | This layer contains any recipes, append files, and patches, that your | ||
528 | project needs. | ||
529 | |||
530 | .. _sources-dev-environment: | ||
531 | |||
532 | Sources | ||
533 | ------- | ||
534 | |||
535 | In order for the OpenEmbedded build system to create an image or any | ||
536 | target, it must be able to access source files. The `general workflow | ||
537 | figure <#general-workflow-figure>`__ represents source files using the | ||
538 | "Upstream Project Releases", "Local Projects", and "SCMs (optional)" | ||
539 | boxes. The figure represents mirrors, which also play a role in locating | ||
540 | source files, with the "Source Materials" box. | ||
541 | |||
542 | The method by which source files are ultimately organized is a function | ||
543 | of the project. For example, for released software, projects tend to use | ||
544 | tarballs or other archived files that can capture the state of a release | ||
545 | guaranteeing that it is statically represented. On the other hand, for a | ||
546 | project that is more dynamic or experimental in nature, a project might | ||
547 | keep source files in a repository controlled by a Source Control Manager | ||
548 | (SCM) such as Git. Pulling source from a repository allows you to | ||
549 | control the point in the repository (the revision) from which you want | ||
550 | to build software. Finally, a combination of the two might exist, which | ||
551 | would give the consumer a choice when deciding where to get source | ||
552 | files. | ||
553 | |||
554 | BitBake uses the :term:`SRC_URI` | ||
555 | variable to point to source files regardless of their location. Each | ||
556 | recipe must have a ``SRC_URI`` variable that points to the source. | ||
557 | |||
558 | Another area that plays a significant role in where source files come | ||
559 | from is pointed to by the | ||
560 | :term:`DL_DIR` variable. This area is | ||
561 | a cache that can hold previously downloaded source. You can also | ||
562 | instruct the OpenEmbedded build system to create tarballs from Git | ||
563 | repositories, which is not the default behavior, and store them in the | ||
564 | ``DL_DIR`` by using the | ||
565 | :term:`BB_GENERATE_MIRROR_TARBALLS` | ||
566 | variable. | ||
567 | |||
568 | Judicious use of a ``DL_DIR`` directory can save the build system a trip | ||
569 | across the Internet when looking for files. A good method for using a | ||
570 | download directory is to have ``DL_DIR`` point to an area outside of | ||
571 | your Build Directory. Doing so allows you to safely delete the Build | ||
572 | Directory if needed without fear of removing any downloaded source file. | ||
573 | |||
574 | The remainder of this section provides a deeper look into the source | ||
575 | files and the mirrors. Here is a more detailed look at the source file | ||
576 | area of the `general workflow figure <#general-workflow-figure>`__: | ||
577 | |||
578 | .. image:: figures/source-input.png | ||
579 | :align: center | ||
580 | |||
581 | Upstream Project Releases | ||
582 | ~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
583 | |||
584 | Upstream project releases exist anywhere in the form of an archived file | ||
585 | (e.g. tarball or zip file). These files correspond to individual | ||
586 | recipes. For example, the figure uses specific releases each for | ||
587 | BusyBox, Qt, and Dbus. An archive file can be for any released product | ||
588 | that can be built using a recipe. | ||
589 | |||
590 | Local Projects | ||
591 | ~~~~~~~~~~~~~~ | ||
592 | |||
593 | Local projects are custom bits of software the user provides. These bits | ||
594 | reside somewhere local to a project - perhaps a directory into which the | ||
595 | user checks in items (e.g. a local directory containing a development | ||
596 | source tree used by the group). | ||
597 | |||
598 | The canonical method through which to include a local project is to use | ||
599 | the :ref:`externalsrc <ref-classes-externalsrc>` | ||
600 | class to include that local project. You use either the ``local.conf`` | ||
601 | or a recipe's append file to override or set the recipe to point to the | ||
602 | local directory on your disk to pull in the whole source tree. | ||
603 | |||
604 | .. _scms: | ||
605 | |||
606 | Source Control Managers (Optional) | ||
607 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
608 | |||
609 | Another place from which the build system can get source files is with | ||
610 | :ref:`fetchers <bitbake:bb-fetchers>` employing various Source | ||
611 | Control Managers (SCMs) such as Git or Subversion. In such cases, a | ||
612 | repository is cloned or checked out. The | ||
613 | :ref:`ref-tasks-fetch` task inside | ||
614 | BitBake uses the :term:`SRC_URI` | ||
615 | variable and the argument's prefix to determine the correct fetcher | ||
616 | module. | ||
617 | |||
618 | .. note:: | ||
619 | |||
620 | For information on how to have the OpenEmbedded build system generate | ||
621 | tarballs for Git repositories and place them in the | ||
622 | DL_DIR | ||
623 | directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS` | ||
624 | variable in the Yocto Project Reference Manual. | ||
625 | |||
626 | When fetching a repository, BitBake uses the | ||
627 | :term:`SRCREV` variable to determine | ||
628 | the specific revision from which to build. | ||
629 | |||
630 | Source Mirror(s) | ||
631 | ~~~~~~~~~~~~~~~~ | ||
632 | |||
633 | Two kinds of mirrors exist: pre-mirrors and regular mirrors. The | ||
634 | :term:`PREMIRRORS` and | ||
635 | :term:`MIRRORS` variables point to | ||
636 | these, respectively. BitBake checks pre-mirrors before looking upstream | ||
637 | for any source files. Pre-mirrors are appropriate when you have a shared | ||
638 | directory that is not a directory defined by the | ||
639 | :term:`DL_DIR` variable. A Pre-mirror | ||
640 | typically points to a shared directory that is local to your | ||
641 | organization. | ||
642 | |||
643 | Regular mirrors can be any site across the Internet that is used as an | ||
644 | alternative location for source code should the primary site not be | ||
645 | functioning for some reason or another. | ||
646 | |||
647 | .. _package-feeds-dev-environment: | ||
648 | |||
649 | Package Feeds | ||
650 | ------------- | ||
651 | |||
652 | When the OpenEmbedded build system generates an image or an SDK, it gets | ||
653 | the packages from a package feed area located in the | ||
654 | :term:`Build Directory`. The `general | ||
655 | workflow figure <#general-workflow-figure>`__ shows this package feeds | ||
656 | area in the upper-right corner. | ||
657 | |||
658 | This section looks a little closer into the package feeds area used by | ||
659 | the build system. Here is a more detailed look at the area: | ||
660 | |||
661 | .. image:: figures/package-feeds.png | ||
662 | :align: center | ||
663 | |||
664 | Package feeds are an intermediary step in the build process. The | ||
665 | OpenEmbedded build system provides classes to generate different package | ||
666 | types, and you specify which classes to enable through the | ||
667 | :term:`PACKAGE_CLASSES` | ||
668 | variable. Before placing the packages into package feeds, the build | ||
669 | process validates them with generated output quality assurance checks | ||
670 | through the :ref:`insane <ref-classes-insane>` | ||
671 | class. | ||
672 | |||
673 | The package feed area resides in the Build Directory. The directory the | ||
674 | build system uses to temporarily store packages is determined by a | ||
675 | combination of variables and the particular package manager in use. See | ||
676 | the "Package Feeds" box in the illustration and note the information to | ||
677 | the right of that area. In particular, the following defines where | ||
678 | package files are kept: | ||
679 | |||
680 | - :term:`DEPLOY_DIR`: Defined as | ||
681 | ``tmp/deploy`` in the Build Directory. | ||
682 | |||
683 | - ``DEPLOY_DIR_*``: Depending on the package manager used, the package | ||
684 | type sub-folder. Given RPM, IPK, or DEB packaging and tarball | ||
685 | creation, the | ||
686 | :term:`DEPLOY_DIR_RPM`, | ||
687 | :term:`DEPLOY_DIR_IPK`, | ||
688 | :term:`DEPLOY_DIR_DEB`, or | ||
689 | :term:`DEPLOY_DIR_TAR`, | ||
690 | variables are used, respectively. | ||
691 | |||
692 | - :term:`PACKAGE_ARCH`: Defines | ||
693 | architecture-specific sub-folders. For example, packages could exist | ||
694 | for the i586 or qemux86 architectures. | ||
695 | |||
696 | BitBake uses the | ||
697 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | ||
698 | tasks to generate packages and place them into the package holding area | ||
699 | (e.g. ``do_package_write_ipk`` for IPK packages). See the | ||
700 | ":ref:`ref-tasks-package_write_deb`", | ||
701 | ":ref:`ref-tasks-package_write_ipk`", | ||
702 | ":ref:`ref-tasks-package_write_rpm`", | ||
703 | and | ||
704 | ":ref:`ref-tasks-package_write_tar`" | ||
705 | sections in the Yocto Project Reference Manual for additional | ||
706 | information. As an example, consider a scenario where an IPK packaging | ||
707 | manager is being used and package architecture support for both i586 and | ||
708 | qemux86 exist. Packages for the i586 architecture are placed in | ||
709 | ``build/tmp/deploy/ipk/i586``, while packages for the qemux86 | ||
710 | architecture are placed in ``build/tmp/deploy/ipk/qemux86``. | ||
711 | |||
712 | .. _bitbake-dev-environment: | ||
713 | |||
714 | BitBake Tool | ||
715 | ------------ | ||
716 | |||
717 | The OpenEmbedded build system uses | ||
718 | :term:`BitBake` to produce images and | ||
719 | Software Development Kits (SDKs). You can see from the `general workflow | ||
720 | figure <#general-workflow-figure>`__, the BitBake area consists of | ||
721 | several functional areas. This section takes a closer look at each of | ||
722 | those areas. | ||
723 | |||
724 | .. note:: | ||
725 | |||
726 | Separate documentation exists for the BitBake tool. See the | ||
727 | BitBake User Manual | ||
728 | for reference material on BitBake. | ||
729 | |||
730 | .. _source-fetching-dev-environment: | ||
731 | |||
732 | Source Fetching | ||
733 | ~~~~~~~~~~~~~~~ | ||
734 | |||
735 | The first stages of building a recipe are to fetch and unpack the source | ||
736 | code: | ||
737 | |||
738 | .. image:: figures/source-fetching.png | ||
739 | :align: center | ||
740 | |||
741 | The :ref:`ref-tasks-fetch` and | ||
742 | :ref:`ref-tasks-unpack` tasks fetch | ||
743 | the source files and unpack them into the | ||
744 | :term:`Build Directory`. | ||
745 | |||
746 | .. note:: | ||
747 | |||
748 | For every local file (e.g. | ||
749 | file:// | ||
750 | ) that is part of a recipe's | ||
751 | SRC_URI | ||
752 | statement, the OpenEmbedded build system takes a checksum of the file | ||
753 | for the recipe and inserts the checksum into the signature for the | ||
754 | do_fetch | ||
755 | task. If any local file has been modified, the | ||
756 | do_fetch | ||
757 | task and all tasks that depend on it are re-executed. | ||
758 | |||
759 | By default, everything is accomplished in the Build Directory, which has | ||
760 | a defined structure. For additional general information on the Build | ||
761 | Directory, see the ":ref:`structure-core-build`" section in | ||
762 | the Yocto Project Reference Manual. | ||
763 | |||
764 | Each recipe has an area in the Build Directory where the unpacked source | ||
765 | code resides. The :term:`S` variable points | ||
766 | to this area for a recipe's unpacked source code. The name of that | ||
767 | directory for any given recipe is defined from several different | ||
768 | variables. The preceding figure and the following list describe the | ||
769 | Build Directory's hierarchy: | ||
770 | |||
771 | - :term:`TMPDIR`: The base directory | ||
772 | where the OpenEmbedded build system performs all its work during the | ||
773 | build. The default base directory is the ``tmp`` directory. | ||
774 | |||
775 | - :term:`PACKAGE_ARCH`: The | ||
776 | architecture of the built package or packages. Depending on the | ||
777 | eventual destination of the package or packages (i.e. machine | ||
778 | architecture, :term:`Build Host`, SDK, or | ||
779 | specific machine), ``PACKAGE_ARCH`` varies. See the variable's | ||
780 | description for details. | ||
781 | |||
782 | - :term:`TARGET_OS`: The operating | ||
783 | system of the target device. A typical value would be "linux" (e.g. | ||
784 | "qemux86-poky-linux"). | ||
785 | |||
786 | - :term:`PN`: The name of the recipe used | ||
787 | to build the package. This variable can have multiple meanings. | ||
788 | However, when used in the context of input files, ``PN`` represents | ||
789 | the name of the recipe. | ||
790 | |||
791 | - :term:`WORKDIR`: The location | ||
792 | where the OpenEmbedded build system builds a recipe (i.e. does the | ||
793 | work to create the package). | ||
794 | |||
795 | - :term:`PV`: The version of the | ||
796 | recipe used to build the package. | ||
797 | |||
798 | - :term:`PR`: The revision of the | ||
799 | recipe used to build the package. | ||
800 | |||
801 | - :term:`S`: Contains the unpacked source | ||
802 | files for a given recipe. | ||
803 | |||
804 | - :term:`BPN`: The name of the recipe | ||
805 | used to build the package. The ``BPN`` variable is a version of | ||
806 | the ``PN`` variable but with common prefixes and suffixes removed. | ||
807 | |||
808 | - :term:`PV`: The version of the | ||
809 | recipe used to build the package. | ||
810 | |||
811 | .. note:: | ||
812 | |||
813 | In the previous figure, notice that two sample hierarchies exist: one | ||
814 | based on package architecture (i.e. | ||
815 | PACKAGE_ARCH | ||
816 | ) and one based on a machine (i.e. | ||
817 | MACHINE | ||
818 | ). The underlying structures are identical. The differentiator being | ||
819 | what the OpenEmbedded build system is using as a build target (e.g. | ||
820 | general architecture, a build host, an SDK, or a specific machine). | ||
821 | |||
822 | .. _patching-dev-environment: | ||
823 | |||
824 | Patching | ||
825 | ~~~~~~~~ | ||
826 | |||
827 | Once source code is fetched and unpacked, BitBake locates patch files | ||
828 | and applies them to the source files: | ||
829 | |||
830 | .. image:: figures/patching.png | ||
831 | :align: center | ||
832 | |||
833 | The :ref:`ref-tasks-patch` task uses a | ||
834 | recipe's :term:`SRC_URI` statements | ||
835 | and the :term:`FILESPATH` variable | ||
836 | to locate applicable patch files. | ||
837 | |||
838 | Default processing for patch files assumes the files have either | ||
839 | ``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters | ||
840 | to change the way the build system recognizes patch files. See the | ||
841 | :ref:`ref-tasks-patch` task for more | ||
842 | information. | ||
843 | |||
844 | BitBake finds and applies multiple patches for a single recipe in the | ||
845 | order in which it locates the patches. The ``FILESPATH`` variable | ||
846 | defines the default set of directories that the build system uses to | ||
847 | search for patch files. Once found, patches are applied to the recipe's | ||
848 | source files, which are located in the | ||
849 | :term:`S` directory. | ||
850 | |||
851 | For more information on how the source directories are created, see the | ||
852 | "`Source Fetching <#source-fetching-dev-environment>`__" section. For | ||
853 | more information on how to create patches and how the build system | ||
854 | processes patches, see the | ||
855 | ":ref:`dev-manual/dev-manual-common-tasks:patching code`" | ||
856 | section in the | ||
857 | Yocto Project Development Tasks Manual. You can also see the | ||
858 | ":ref:`sdk-manual/sdk-extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" | ||
859 | section in the Yocto Project Application Development and the Extensible | ||
860 | Software Development Kit (SDK) manual and the | ||
861 | ":ref:`kernel-dev/kernel-dev-common:using traditional kernel development to patch the kernel`" | ||
862 | section in the Yocto Project Linux Kernel Development Manual. | ||
863 | |||
864 | .. _configuration-compilation-and-staging-dev-environment: | ||
865 | |||
866 | Configuration, Compilation, and Staging | ||
867 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
868 | |||
869 | After source code is patched, BitBake executes tasks that configure and | ||
870 | compile the source code. Once compilation occurs, the files are copied | ||
871 | to a holding area (staged) in preparation for packaging: | ||
872 | |||
873 | .. image:: figures/configuration-compile-autoreconf.png | ||
874 | :align: center | ||
875 | |||
876 | This step in the build process consists of the following tasks: | ||
877 | |||
878 | - :ref:`ref-tasks-prepare_recipe_sysroot`: | ||
879 | This task sets up the two sysroots in | ||
880 | ``${``\ :term:`WORKDIR`\ ``}`` | ||
881 | (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that | ||
882 | during the packaging phase the sysroots can contain the contents of | ||
883 | the | ||
884 | :ref:`ref-tasks-populate_sysroot` | ||
885 | tasks of the recipes on which the recipe containing the tasks | ||
886 | depends. A sysroot exists for both the target and for the native | ||
887 | binaries, which run on the host system. | ||
888 | |||
889 | - *do_configure*: This task configures the source by enabling and | ||
890 | disabling any build-time and configuration options for the software | ||
891 | being built. Configurations can come from the recipe itself as well | ||
892 | as from an inherited class. Additionally, the software itself might | ||
893 | configure itself depending on the target for which it is being built. | ||
894 | |||
895 | The configurations handled by the | ||
896 | :ref:`ref-tasks-configure` task | ||
897 | are specific to configurations for the source code being built by the | ||
898 | recipe. | ||
899 | |||
900 | If you are using the | ||
901 | :ref:`autotools <ref-classes-autotools>` class, | ||
902 | you can add additional configuration options by using the | ||
903 | :term:`EXTRA_OECONF` or | ||
904 | :term:`PACKAGECONFIG_CONFARGS` | ||
905 | variables. For information on how this variable works within that | ||
906 | class, see the | ||
907 | :ref:`autotools <ref-classes-autotools>` class | ||
908 | :yocto_git:`here </cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass>`. | ||
909 | |||
910 | - *do_compile*: Once a configuration task has been satisfied, | ||
911 | BitBake compiles the source using the | ||
912 | :ref:`ref-tasks-compile` task. | ||
913 | Compilation occurs in the directory pointed to by the | ||
914 | :term:`B` variable. Realize that the | ||
915 | ``B`` directory is, by default, the same as the | ||
916 | :term:`S` directory. | ||
917 | |||
918 | - *do_install*: After compilation completes, BitBake executes the | ||
919 | :ref:`ref-tasks-install` task. | ||
920 | This task copies files from the ``B`` directory and places them in a | ||
921 | holding area pointed to by the :term:`D` | ||
922 | variable. Packaging occurs later using files from this holding | ||
923 | directory. | ||
924 | |||
925 | .. _package-splitting-dev-environment: | ||
926 | |||
927 | Package Splitting | ||
928 | ~~~~~~~~~~~~~~~~~ | ||
929 | |||
930 | After source code is configured, compiled, and staged, the build system | ||
931 | analyzes the results and splits the output into packages: | ||
932 | |||
933 | .. image:: figures/analysis-for-package-splitting.png | ||
934 | :align: center | ||
935 | |||
936 | The :ref:`ref-tasks-package` and | ||
937 | :ref:`ref-tasks-packagedata` | ||
938 | tasks combine to analyze the files found in the | ||
939 | :term:`D` directory and split them into | ||
940 | subsets based on available packages and files. Analysis involves the | ||
941 | following as well as other items: splitting out debugging symbols, | ||
942 | looking at shared library dependencies between packages, and looking at | ||
943 | package relationships. | ||
944 | |||
945 | The ``do_packagedata`` task creates package metadata based on the | ||
946 | analysis such that the build system can generate the final packages. The | ||
947 | :ref:`ref-tasks-populate_sysroot` | ||
948 | task stages (copies) a subset of the files installed by the | ||
949 | :ref:`ref-tasks-install` task into | ||
950 | the appropriate sysroot. Working, staged, and intermediate results of | ||
951 | the analysis and package splitting process use several areas: | ||
952 | |||
953 | - :term:`PKGD`: The destination | ||
954 | directory (i.e. ``package``) for packages before they are split into | ||
955 | individual packages. | ||
956 | |||
957 | - :term:`PKGDESTWORK`: A | ||
958 | temporary work area (i.e. ``pkgdata``) used by the ``do_package`` | ||
959 | task to save package metadata. | ||
960 | |||
961 | - :term:`PKGDEST`: The parent | ||
962 | directory (i.e. ``packages-split``) for packages after they have been | ||
963 | split. | ||
964 | |||
965 | - :term:`PKGDATA_DIR`: A shared, | ||
966 | global-state directory that holds packaging metadata generated during | ||
967 | the packaging process. The packaging process copies metadata from | ||
968 | ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally | ||
969 | available. | ||
970 | |||
971 | - :term:`STAGING_DIR_HOST`: | ||
972 | The path for the sysroot for the system on which a component is built | ||
973 | to run (i.e. ``recipe-sysroot``). | ||
974 | |||
975 | - :term:`STAGING_DIR_NATIVE`: | ||
976 | The path for the sysroot used when building components for the build | ||
977 | host (i.e. ``recipe-sysroot-native``). | ||
978 | |||
979 | - :term:`STAGING_DIR_TARGET`: | ||
980 | The path for the sysroot used when a component that is built to | ||
981 | execute on a system and it generates code for yet another machine | ||
982 | (e.g. cross-canadian recipes). | ||
983 | |||
984 | The :term:`FILES` variable defines the | ||
985 | files that go into each package in | ||
986 | :term:`PACKAGES`. If you want | ||
987 | details on how this is accomplished, you can look at | ||
988 | :yocto_git:`package.bbclass </cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass>`. | ||
989 | |||
990 | Depending on the type of packages being created (RPM, DEB, or IPK), the | ||
991 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | ||
992 | task creates the actual packages and places them in the Package Feed | ||
993 | area, which is ``${TMPDIR}/deploy``. You can see the "`Package | ||
994 | Feeds <#package-feeds-dev-environment>`__" section for more detail on | ||
995 | that part of the build process. | ||
996 | |||
997 | .. note:: | ||
998 | |||
999 | Support for creating feeds directly from the | ||
1000 | deploy/\* | ||
1001 | directories does not exist. Creating such feeds usually requires some | ||
1002 | kind of feed maintenance mechanism that would upload the new packages | ||
1003 | into an official package feed (e.g. the Ångström distribution). This | ||
1004 | functionality is highly distribution-specific and thus is not | ||
1005 | provided out of the box. | ||
1006 | |||
1007 | .. _image-generation-dev-environment: | ||
1008 | |||
1009 | Image Generation | ||
1010 | ~~~~~~~~~~~~~~~~ | ||
1011 | |||
1012 | Once packages are split and stored in the Package Feeds area, the build | ||
1013 | system uses BitBake to generate the root filesystem image: | ||
1014 | |||
1015 | .. image:: figures/image-generation.png | ||
1016 | :align: center | ||
1017 | |||
1018 | The image generation process consists of several stages and depends on | ||
1019 | several tasks and variables. The | ||
1020 | :ref:`ref-tasks-rootfs` task creates | ||
1021 | the root filesystem (file and directory structure) for an image. This | ||
1022 | task uses several key variables to help create the list of packages to | ||
1023 | actually install: | ||
1024 | |||
1025 | - :term:`IMAGE_INSTALL`: Lists | ||
1026 | out the base set of packages from which to install from the Package | ||
1027 | Feeds area. | ||
1028 | |||
1029 | - :term:`PACKAGE_EXCLUDE`: | ||
1030 | Specifies packages that should not be installed into the image. | ||
1031 | |||
1032 | - :term:`IMAGE_FEATURES`: | ||
1033 | Specifies features to include in the image. Most of these features | ||
1034 | map to additional packages for installation. | ||
1035 | |||
1036 | - :term:`PACKAGE_CLASSES`: | ||
1037 | Specifies the package backend (e.g. RPM, DEB, or IPK) to use and | ||
1038 | consequently helps determine where to locate packages within the | ||
1039 | Package Feeds area. | ||
1040 | |||
1041 | - :term:`IMAGE_LINGUAS`: | ||
1042 | Determines the language(s) for which additional language support | ||
1043 | packages are installed. | ||
1044 | |||
1045 | - :term:`PACKAGE_INSTALL`: | ||
1046 | The final list of packages passed to the package manager for | ||
1047 | installation into the image. | ||
1048 | |||
1049 | With :term:`IMAGE_ROOTFS` | ||
1050 | pointing to the location of the filesystem under construction and the | ||
1051 | ``PACKAGE_INSTALL`` variable providing the final list of packages to | ||
1052 | install, the root file system is created. | ||
1053 | |||
1054 | Package installation is under control of the package manager (e.g. | ||
1055 | dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package | ||
1056 | management is enabled for the target. At the end of the process, if | ||
1057 | package management is not enabled for the target, the package manager's | ||
1058 | data files are deleted from the root filesystem. As part of the final | ||
1059 | stage of package installation, post installation scripts that are part | ||
1060 | of the packages are run. Any scripts that fail to run on the build host | ||
1061 | are run on the target when the target system is first booted. If you are | ||
1062 | using a | ||
1063 | :ref:`read-only root filesystem <dev-manual/dev-manual-common-tasks:creating a read-only root filesystem>`, | ||
1064 | all the post installation scripts must succeed on the build host during | ||
1065 | the package installation phase since the root filesystem on the target | ||
1066 | is read-only. | ||
1067 | |||
1068 | The final stages of the ``do_rootfs`` task handle post processing. Post | ||
1069 | processing includes creation of a manifest file and optimizations. | ||
1070 | |||
1071 | The manifest file (``.manifest``) resides in the same directory as the | ||
1072 | root filesystem image. This file lists out, line-by-line, the installed | ||
1073 | packages. The manifest file is useful for the | ||
1074 | :ref:`testimage <ref-classes-testimage*>` class, | ||
1075 | for example, to determine whether or not to run specific tests. See the | ||
1076 | :term:`IMAGE_MANIFEST` | ||
1077 | variable for additional information. | ||
1078 | |||
1079 | Optimizing processes that are run across the image include ``mklibs``, | ||
1080 | ``prelink``, and any other post-processing commands as defined by the | ||
1081 | :term:`ROOTFS_POSTPROCESS_COMMAND` | ||
1082 | variable. The ``mklibs`` process optimizes the size of the libraries, | ||
1083 | while the ``prelink`` process optimizes the dynamic linking of shared | ||
1084 | libraries to reduce start up time of executables. | ||
1085 | |||
1086 | After the root filesystem is built, processing begins on the image | ||
1087 | through the :ref:`ref-tasks-image` | ||
1088 | task. The build system runs any pre-processing commands as defined by | ||
1089 | the | ||
1090 | :term:`IMAGE_PREPROCESS_COMMAND` | ||
1091 | variable. This variable specifies a list of functions to call before the | ||
1092 | build system creates the final image output files. | ||
1093 | |||
1094 | The build system dynamically creates ``do_image_*`` tasks as needed, | ||
1095 | based on the image types specified in the | ||
1096 | :term:`IMAGE_FSTYPES` variable. | ||
1097 | The process turns everything into an image file or a set of image files | ||
1098 | and can compress the root filesystem image to reduce the overall size of | ||
1099 | the image. The formats used for the root filesystem depend on the | ||
1100 | ``IMAGE_FSTYPES`` variable. Compression depends on whether the formats | ||
1101 | support compression. | ||
1102 | |||
1103 | As an example, a dynamically created task when creating a particular | ||
1104 | image type would take the following form: | ||
1105 | :: | ||
1106 | |||
1107 | do_image_type | ||
1108 | |||
1109 | So, if the type | ||
1110 | as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically | ||
1111 | generated task would be as follows: | ||
1112 | :: | ||
1113 | |||
1114 | do_image_ext4 | ||
1115 | |||
1116 | The final task involved in image creation is the | ||
1117 | :ref:`do_image_complete <ref-tasks-image-complete>` | ||
1118 | task. This task completes the image by applying any image post | ||
1119 | processing as defined through the | ||
1120 | :term:`IMAGE_POSTPROCESS_COMMAND` | ||
1121 | variable. The variable specifies a list of functions to call once the | ||
1122 | build system has created the final image output files. | ||
1123 | |||
1124 | .. note:: | ||
1125 | |||
1126 | The entire image generation process is run under | ||
1127 | Pseudo. Running under Pseudo ensures that the files in the root filesystem | ||
1128 | have correct ownership. | ||
1129 | |||
1130 | .. _sdk-generation-dev-environment: | ||
1131 | |||
1132 | SDK Generation | ||
1133 | ~~~~~~~~~~~~~~ | ||
1134 | |||
1135 | The OpenEmbedded build system uses BitBake to generate the Software | ||
1136 | Development Kit (SDK) installer scripts for both the standard SDK and | ||
1137 | the extensible SDK (eSDK): | ||
1138 | |||
1139 | .. image:: figures/sdk-generation.png | ||
1140 | :align: center | ||
1141 | |||
1142 | .. note:: | ||
1143 | |||
1144 | For more information on the cross-development toolchain generation, | ||
1145 | see the ":ref:`overview-manual/overview-manual-concepts:cross-development toolchain generation`" | ||
1146 | section. For information on advantages gained when building a | ||
1147 | cross-development toolchain using the do_populate_sdk task, see the | ||
1148 | ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" section in | ||
1149 | the Yocto Project Application Development and the Extensible Software | ||
1150 | Development Kit (eSDK) manual. | ||
1151 | |||
1152 | Like image generation, the SDK script process consists of several stages | ||
1153 | and depends on many variables. The | ||
1154 | :ref:`ref-tasks-populate_sdk` | ||
1155 | and | ||
1156 | :ref:`ref-tasks-populate_sdk_ext` | ||
1157 | tasks use these key variables to help create the list of packages to | ||
1158 | actually install. For information on the variables listed in the figure, | ||
1159 | see the "`Application Development SDK <#sdk-dev-environment>`__" | ||
1160 | section. | ||
1161 | |||
1162 | The ``do_populate_sdk`` task helps create the standard SDK and handles | ||
1163 | two parts: a target part and a host part. The target part is the part | ||
1164 | built for the target hardware and includes libraries and headers. The | ||
1165 | host part is the part of the SDK that runs on the | ||
1166 | :term:`SDKMACHINE`. | ||
1167 | |||
1168 | The ``do_populate_sdk_ext`` task helps create the extensible SDK and | ||
1169 | handles host and target parts differently than its counter part does for | ||
1170 | the standard SDK. For the extensible SDK, the task encapsulates the | ||
1171 | build system, which includes everything needed (host and target) for the | ||
1172 | SDK. | ||
1173 | |||
1174 | Regardless of the type of SDK being constructed, the tasks perform some | ||
1175 | cleanup after which a cross-development environment setup script and any | ||
1176 | needed configuration files are created. The final output is the | ||
1177 | Cross-development toolchain installation script (``.sh`` file), which | ||
1178 | includes the environment setup script. | ||
1179 | |||
1180 | Stamp Files and the Rerunning of Tasks | ||
1181 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
1182 | |||
1183 | For each task that completes successfully, BitBake writes a stamp file | ||
1184 | into the :term:`STAMPS_DIR` | ||
1185 | directory. The beginning of the stamp file's filename is determined by | ||
1186 | the :term:`STAMP` variable, and the end | ||
1187 | of the name consists of the task's name and current `input | ||
1188 | checksum <#overview-checksums>`__. | ||
1189 | |||
1190 | .. note:: | ||
1191 | |||
1192 | This naming scheme assumes that | ||
1193 | BB_SIGNATURE_HANDLER | ||
1194 | is "OEBasicHash", which is almost always the case in current | ||
1195 | OpenEmbedded. | ||
1196 | |||
1197 | To determine if a task needs to be rerun, BitBake checks if a stamp file | ||
1198 | with a matching input checksum exists for the task. If such a stamp file | ||
1199 | exists, the task's output is assumed to exist and still be valid. If the | ||
1200 | file does not exist, the task is rerun. | ||
1201 | |||
1202 | .. note:: | ||
1203 | |||
1204 | The stamp mechanism is more general than the shared state (sstate) | ||
1205 | cache mechanism described in the "`Setscene Tasks and Shared | ||
1206 | State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids | ||
1207 | rerunning any task that has a valid stamp file, not just tasks that | ||
1208 | can be accelerated through the sstate cache. | ||
1209 | |||
1210 | However, you should realize that stamp files only serve as a marker | ||
1211 | that some work has been done and that these files do not record task | ||
1212 | output. The actual task output would usually be somewhere in | ||
1213 | :term:`TMPDIR` (e.g. in some | ||
1214 | recipe's :term:`WORKDIR`.) What | ||
1215 | the sstate cache mechanism adds is a way to cache task output that | ||
1216 | can then be shared between build machines. | ||
1217 | |||
1218 | Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing | ||
1219 | ``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will | ||
1220 | properly be rerun to repopulate ``TMPDIR``. | ||
1221 | |||
1222 | If you want some task to always be considered "out of date", you can | ||
1223 | mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>` | ||
1224 | varflag. If some other task depends on such a task, then that task will | ||
1225 | also always be considered out of date, which might not be what you want. | ||
1226 | |||
1227 | For details on how to view information about a task's signature, see the | ||
1228 | ":ref:`dev-manual/dev-manual-common-tasks:viewing task variable dependencies`" | ||
1229 | section in the Yocto Project Development Tasks Manual. | ||
1230 | |||
1231 | Setscene Tasks and Shared State | ||
1232 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
1233 | |||
1234 | The description of tasks so far assumes that BitBake needs to build | ||
1235 | everything and no available prebuilt objects exist. BitBake does support | ||
1236 | skipping tasks if prebuilt objects are available. These objects are | ||
1237 | usually made available in the form of a shared state (sstate) cache. | ||
1238 | |||
1239 | .. note:: | ||
1240 | |||
1241 | For information on variables affecting sstate, see the | ||
1242 | :term:`SSTATE_DIR` | ||
1243 | and | ||
1244 | :term:`SSTATE_MIRRORS` | ||
1245 | variables. | ||
1246 | |||
1247 | The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a | ||
1248 | version of the task where instead of building something, BitBake can | ||
1249 | skip to the end result and simply place a set of files into specific | ||
1250 | locations as needed. In some cases, it makes sense to have a setscene | ||
1251 | task variant (e.g. generating package files in the | ||
1252 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | ||
1253 | task). In other cases, it does not make sense (e.g. a | ||
1254 | :ref:`ref-tasks-patch` task or a | ||
1255 | :ref:`ref-tasks-unpack` task) since | ||
1256 | the work involved would be equal to or greater than the underlying task. | ||
1257 | |||
1258 | In the build system, the common tasks that have setscene variants are | ||
1259 | :ref:`ref-tasks-package`, | ||
1260 | ``do_package_write_*``, | ||
1261 | :ref:`ref-tasks-deploy`, | ||
1262 | :ref:`ref-tasks-packagedata`, and | ||
1263 | :ref:`ref-tasks-populate_sysroot`. | ||
1264 | Notice that these tasks represent most of the tasks whose output is an | ||
1265 | end result. | ||
1266 | |||
1267 | The build system has knowledge of the relationship between these tasks | ||
1268 | and other preceding tasks. For example, if BitBake runs | ||
1269 | ``do_populate_sysroot_setscene`` for something, it does not make sense | ||
1270 | to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``, | ||
1271 | ``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if | ||
1272 | ``do_package`` needs to be run, BitBake needs to run those other tasks. | ||
1273 | |||
1274 | It becomes more complicated if everything can come from an sstate cache | ||
1275 | because some objects are simply not required at all. For example, you do | ||
1276 | not need a compiler or native tools, such as quilt, if nothing exists to | ||
1277 | compile or patch. If the ``do_package_write_*`` packages are available | ||
1278 | from sstate, BitBake does not need the ``do_package`` task data. | ||
1279 | |||
1280 | To handle all these complexities, BitBake runs in two phases. The first | ||
1281 | is the "setscene" stage. During this stage, BitBake first checks the | ||
1282 | sstate cache for any targets it is planning to build. BitBake does a | ||
1283 | fast check to see if the object exists rather than a complete download. | ||
1284 | If nothing exists, the second phase, which is the setscene stage, | ||
1285 | completes and the main build proceeds. | ||
1286 | |||
1287 | If objects are found in the sstate cache, the build system works | ||
1288 | backwards from the end targets specified by the user. For example, if an | ||
1289 | image is being built, the build system first looks for the packages | ||
1290 | needed for that image and the tools needed to construct an image. If | ||
1291 | those are available, the compiler is not needed. Thus, the compiler is | ||
1292 | not even downloaded. If something was found to be unavailable, or the | ||
1293 | download or setscene task fails, the build system then tries to install | ||
1294 | dependencies, such as the compiler, from the cache. | ||
1295 | |||
1296 | The availability of objects in the sstate cache is handled by the | ||
1297 | function specified by the | ||
1298 | :term:`bitbake:BB_HASHCHECK_FUNCTION` | ||
1299 | variable and returns a list of available objects. The function specified | ||
1300 | by the | ||
1301 | :term:`bitbake:BB_SETSCENE_DEPVALID` | ||
1302 | variable is the function that determines whether a given dependency | ||
1303 | needs to be followed, and whether for any given relationship the | ||
1304 | function needs to be passed. The function returns a True or False value. | ||
1305 | |||
1306 | .. _images-dev-environment: | ||
1307 | |||
1308 | Images | ||
1309 | ------ | ||
1310 | |||
1311 | The images produced by the build system are compressed forms of the root | ||
1312 | filesystem and are ready to boot on a target device. You can see from | ||
1313 | the `general workflow figure <#general-workflow-figure>`__ that BitBake | ||
1314 | output, in part, consists of images. This section takes a closer look at | ||
1315 | this output: | ||
1316 | |||
1317 | .. image:: figures/images.png | ||
1318 | :align: center | ||
1319 | |||
1320 | .. note:: | ||
1321 | |||
1322 | For a list of example images that the Yocto Project provides, see the | ||
1323 | ":doc:`../ref-manual/ref-images`" chapter in the Yocto Project Reference | ||
1324 | Manual. | ||
1325 | |||
1326 | The build process writes images out to the :term:`Build Directory` | ||
1327 | inside the | ||
1328 | ``tmp/deploy/images/machine/`` folder as shown in the figure. This | ||
1329 | folder contains any files expected to be loaded on the target device. | ||
1330 | The :term:`DEPLOY_DIR` variable | ||
1331 | points to the ``deploy`` directory, while the | ||
1332 | :term:`DEPLOY_DIR_IMAGE` | ||
1333 | variable points to the appropriate directory containing images for the | ||
1334 | current configuration. | ||
1335 | |||
1336 | - kernel-image: A kernel binary file. The | ||
1337 | :term:`KERNEL_IMAGETYPE` | ||
1338 | variable determines the naming scheme for the kernel image file. | ||
1339 | Depending on this variable, the file could begin with a variety of | ||
1340 | naming strings. The ``deploy/images/``\ machine directory can contain | ||
1341 | multiple image files for the machine. | ||
1342 | |||
1343 | - root-filesystem-image: Root filesystems for the target device (e.g. | ||
1344 | ``*.ext3`` or ``*.bz2`` files). The | ||
1345 | :term:`IMAGE_FSTYPES` | ||
1346 | variable determines the root filesystem image type. The | ||
1347 | ``deploy/images/``\ machine directory can contain multiple root | ||
1348 | filesystems for the machine. | ||
1349 | |||
1350 | - kernel-modules: Tarballs that contain all the modules built for the | ||
1351 | kernel. Kernel module tarballs exist for legacy purposes and can be | ||
1352 | suppressed by setting the | ||
1353 | :term:`MODULE_TARBALL_DEPLOY` | ||
1354 | variable to "0". The ``deploy/images/``\ machine directory can | ||
1355 | contain multiple kernel module tarballs for the machine. | ||
1356 | |||
1357 | - bootloaders: If applicable to the target machine, bootloaders | ||
1358 | supporting the image. The ``deploy/images/``\ machine directory can | ||
1359 | contain multiple bootloaders for the machine. | ||
1360 | |||
1361 | - symlinks: The ``deploy/images/``\ machine folder contains a symbolic | ||
1362 | link that points to the most recently built file for each machine. | ||
1363 | These links might be useful for external scripts that need to obtain | ||
1364 | the latest version of each file. | ||
1365 | |||
1366 | .. _sdk-dev-environment: | ||
1367 | |||
1368 | Application Development SDK | ||
1369 | --------------------------- | ||
1370 | |||
1371 | In the `general workflow figure <#general-workflow-figure>`__, the | ||
1372 | output labeled "Application Development SDK" represents an SDK. The SDK | ||
1373 | generation process differs depending on whether you build an extensible | ||
1374 | SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK | ||
1375 | (e.g. ``bitbake -c populate_sdk`` imagename). This section takes a | ||
1376 | closer look at this output: | ||
1377 | |||
1378 | .. image:: figures/sdk.png | ||
1379 | :align: center | ||
1380 | |||
1381 | The specific form of this output is a set of files that includes a | ||
1382 | self-extracting SDK installer (``*.sh``), host and target manifest | ||
1383 | files, and files used for SDK testing. When the SDK installer file is | ||
1384 | run, it installs the SDK. The SDK consists of a cross-development | ||
1385 | toolchain, a set of libraries and headers, and an SDK environment setup | ||
1386 | script. Running this installer essentially sets up your | ||
1387 | cross-development environment. You can think of the cross-toolchain as | ||
1388 | the "host" part because it runs on the SDK machine. You can think of the | ||
1389 | libraries and headers as the "target" part because they are built for | ||
1390 | the target hardware. The environment setup script is added so that you | ||
1391 | can initialize the environment before using the tools. | ||
1392 | |||
1393 | .. note:: | ||
1394 | |||
1395 | - The Yocto Project supports several methods by which you can set up | ||
1396 | this cross-development environment. These methods include | ||
1397 | downloading pre-built SDK installers or building and installing | ||
1398 | your own SDK installer. | ||
1399 | |||
1400 | - For background information on cross-development toolchains in the | ||
1401 | Yocto Project development environment, see the "`Cross-Development | ||
1402 | Toolchain Generation <#cross-development-toolchain-generation>`__" | ||
1403 | section. | ||
1404 | |||
1405 | - For information on setting up a cross-development environment, see | ||
1406 | the :doc:`../sdk-manual/sdk-manual` manual. | ||
1407 | |||
1408 | All the output files for an SDK are written to the ``deploy/sdk`` folder | ||
1409 | inside the :term:`Build Directory` as | ||
1410 | shown in the previous figure. Depending on the type of SDK, several | ||
1411 | variables exist that help configure these files. The following list | ||
1412 | shows the variables associated with an extensible SDK: | ||
1413 | |||
1414 | - :term:`DEPLOY_DIR`: Points to | ||
1415 | the ``deploy`` directory. | ||
1416 | |||
1417 | - :term:`SDK_EXT_TYPE`: | ||
1418 | Controls whether or not shared state artifacts are copied into the | ||
1419 | extensible SDK. By default, all required shared state artifacts are | ||
1420 | copied into the SDK. | ||
1421 | |||
1422 | - :term:`SDK_INCLUDE_PKGDATA`: | ||
1423 | Specifies whether or not packagedata is included in the extensible | ||
1424 | SDK for all recipes in the "world" target. | ||
1425 | |||
1426 | - :term:`SDK_INCLUDE_TOOLCHAIN`: | ||
1427 | Specifies whether or not the toolchain is included when building the | ||
1428 | extensible SDK. | ||
1429 | |||
1430 | - :term:`SDK_LOCAL_CONF_WHITELIST`: | ||
1431 | A list of variables allowed through from the build system | ||
1432 | configuration into the extensible SDK configuration. | ||
1433 | |||
1434 | - :term:`SDK_LOCAL_CONF_BLACKLIST`: | ||
1435 | A list of variables not allowed through from the build system | ||
1436 | configuration into the extensible SDK configuration. | ||
1437 | |||
1438 | - :term:`SDK_INHERIT_BLACKLIST`: | ||
1439 | A list of classes to remove from the | ||
1440 | :term:`INHERIT` value globally | ||
1441 | within the extensible SDK configuration. | ||
1442 | |||
1443 | This next list, shows the variables associated with a standard SDK: | ||
1444 | |||
1445 | - :term:`DEPLOY_DIR`: Points to | ||
1446 | the ``deploy`` directory. | ||
1447 | |||
1448 | - :term:`SDKMACHINE`: Specifies | ||
1449 | the architecture of the machine on which the cross-development tools | ||
1450 | are run to create packages for the target hardware. | ||
1451 | |||
1452 | - :term:`SDKIMAGE_FEATURES`: | ||
1453 | Lists the features to include in the "target" part of the SDK. | ||
1454 | |||
1455 | - :term:`TOOLCHAIN_HOST_TASK`: | ||
1456 | Lists packages that make up the host part of the SDK (i.e. the part | ||
1457 | that runs on the ``SDKMACHINE``). When you use | ||
1458 | ``bitbake -c populate_sdk imagename`` to create the SDK, a set of | ||
1459 | default packages apply. This variable allows you to add more | ||
1460 | packages. | ||
1461 | |||
1462 | - :term:`TOOLCHAIN_TARGET_TASK`: | ||
1463 | Lists packages that make up the target part of the SDK (i.e. the part | ||
1464 | built for the target hardware). | ||
1465 | |||
1466 | - :term:`SDKPATH`: Defines the | ||
1467 | default SDK installation path offered by the installation script. | ||
1468 | |||
1469 | - :term:`SDK_HOST_MANIFEST`: | ||
1470 | Lists all the installed packages that make up the host part of the | ||
1471 | SDK. This variable also plays a minor role for extensible SDK | ||
1472 | development as well. However, it is mainly used for the standard SDK. | ||
1473 | |||
1474 | - :term:`SDK_TARGET_MANIFEST`: | ||
1475 | Lists all the installed packages that make up the target part of the | ||
1476 | SDK. This variable also plays a minor role for extensible SDK | ||
1477 | development as well. However, it is mainly used for the standard SDK. | ||
1478 | |||
1479 | Cross-Development Toolchain Generation | ||
1480 | ====================================== | ||
1481 | |||
1482 | The Yocto Project does most of the work for you when it comes to | ||
1483 | creating :ref:`sdk-manual/sdk-intro:the cross-development toolchain`. This | ||
1484 | section provides some technical background on how cross-development | ||
1485 | toolchains are created and used. For more information on toolchains, you | ||
1486 | can also see the :doc:`../sdk-manual/sdk-manual` manual. | ||
1487 | |||
1488 | In the Yocto Project development environment, cross-development | ||
1489 | toolchains are used to build images and applications that run on the | ||
1490 | target hardware. With just a few commands, the OpenEmbedded build system | ||
1491 | creates these necessary toolchains for you. | ||
1492 | |||
1493 | The following figure shows a high-level build environment regarding | ||
1494 | toolchain construction and use. | ||
1495 | |||
1496 | .. image:: figures/cross-development-toolchains.png | ||
1497 | :align: center | ||
1498 | |||
1499 | Most of the work occurs on the Build Host. This is the machine used to | ||
1500 | build images and generally work within the the Yocto Project | ||
1501 | environment. When you run | ||
1502 | :term:`BitBake` to create an image, the | ||
1503 | OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a | ||
1504 | cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what | ||
1505 | BitBake uses to compile source files when creating the target image. You | ||
1506 | can think of ``gcc-cross`` simply as an automatically generated | ||
1507 | cross-compiler that is used internally within BitBake only. | ||
1508 | |||
1509 | .. note:: | ||
1510 | |||
1511 | The extensible SDK does not use | ||
1512 | gcc-cross-canadian | ||
1513 | since this SDK ships a copy of the OpenEmbedded build system and the | ||
1514 | sysroot within it contains | ||
1515 | gcc-cross | ||
1516 | . | ||
1517 | |||
1518 | The chain of events that occurs when ``gcc-cross`` is bootstrapped is as | ||
1519 | follows: | ||
1520 | :: | ||
1521 | |||
1522 | gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime | ||
1523 | |||
1524 | - ``gcc``: The build host's GNU Compiler Collection (GCC). | ||
1525 | |||
1526 | - ``binutils-cross``: The bare minimum binary utilities needed in order | ||
1527 | to run the ``gcc-cross-initial`` phase of the bootstrap operation. | ||
1528 | |||
1529 | - ``gcc-cross-initial``: An early stage of the bootstrap process for | ||
1530 | creating the cross-compiler. This stage builds enough of the | ||
1531 | ``gcc-cross``, the C library, and other pieces needed to finish | ||
1532 | building the final cross-compiler in later stages. This tool is a | ||
1533 | "native" package (i.e. it is designed to run on the build host). | ||
1534 | |||
1535 | - ``linux-libc-headers``: Headers needed for the cross-compiler. | ||
1536 | |||
1537 | - ``glibc-initial``: An initial version of the Embedded GNU C Library | ||
1538 | (GLIBC) needed to bootstrap ``glibc``. | ||
1539 | |||
1540 | - ``glibc``: The GNU C Library. | ||
1541 | |||
1542 | - ``gcc-cross``: The final stage of the bootstrap process for the | ||
1543 | cross-compiler. This stage results in the actual cross-compiler that | ||
1544 | BitBake uses when it builds an image for a targeted device. | ||
1545 | |||
1546 | .. note:: | ||
1547 | |||
1548 | If you are replacing this cross compiler toolchain with a custom | ||
1549 | version, you must replace | ||
1550 | gcc-cross | ||
1551 | . | ||
1552 | |||
1553 | This tool is also a "native" package (i.e. it is designed to run on | ||
1554 | the build host). | ||
1555 | |||
1556 | - ``gcc-runtime``: Runtime libraries resulting from the toolchain | ||
1557 | bootstrapping process. This tool produces a binary that consists of | ||
1558 | the runtime libraries need for the targeted device. | ||
1559 | |||
1560 | You can use the OpenEmbedded build system to build an installer for the | ||
1561 | relocatable SDK used to develop applications. When you run the | ||
1562 | installer, it installs the toolchain, which contains the development | ||
1563 | tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and | ||
1564 | other ``nativesdk-*`` tools), which are tools native to the SDK (i.e. | ||
1565 | native to :term:`SDK_ARCH`), you | ||
1566 | need to cross-compile and test your software. The figure shows the | ||
1567 | commands you use to easily build out this toolchain. This | ||
1568 | cross-development toolchain is built to execute on the | ||
1569 | :term:`SDKMACHINE`, which might or | ||
1570 | might not be the same machine as the Build Host. | ||
1571 | |||
1572 | .. note:: | ||
1573 | |||
1574 | If your target architecture is supported by the Yocto Project, you | ||
1575 | can take advantage of pre-built images that ship with the Yocto | ||
1576 | Project and already contain cross-development toolchain installers. | ||
1577 | |||
1578 | Here is the bootstrap process for the relocatable toolchain: | ||
1579 | :: | ||
1580 | |||
1581 | gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian | ||
1582 | |||
1583 | - ``gcc``: The build host's GNU Compiler Collection (GCC). | ||
1584 | |||
1585 | - ``binutils-crosssdk``: The bare minimum binary utilities needed in | ||
1586 | order to run the ``gcc-crosssdk-initial`` phase of the bootstrap | ||
1587 | operation. | ||
1588 | |||
1589 | - ``gcc-crosssdk-initial``: An early stage of the bootstrap process for | ||
1590 | creating the cross-compiler. This stage builds enough of the | ||
1591 | ``gcc-crosssdk`` and supporting pieces so that the final stage of the | ||
1592 | bootstrap process can produce the finished cross-compiler. This tool | ||
1593 | is a "native" binary that runs on the build host. | ||
1594 | |||
1595 | - ``linux-libc-headers``: Headers needed for the cross-compiler. | ||
1596 | |||
1597 | - ``glibc-initial``: An initial version of the Embedded GLIBC needed to | ||
1598 | bootstrap ``nativesdk-glibc``. | ||
1599 | |||
1600 | - ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the | ||
1601 | ``gcc-crosssdk``. | ||
1602 | |||
1603 | - ``gcc-crosssdk``: The final stage of the bootstrap process for the | ||
1604 | relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory | ||
1605 | compiler and never leaves the build host. Its purpose is to help in | ||
1606 | the bootstrap process to create the eventual ``gcc-cross-canadian`` | ||
1607 | compiler, which is relocatable. This tool is also a "native" package | ||
1608 | (i.e. it is designed to run on the build host). | ||
1609 | |||
1610 | - ``gcc-cross-canadian``: The final relocatable cross-compiler. When | ||
1611 | run on the :term:`SDKMACHINE`, | ||
1612 | this tool produces executable code that runs on the target device. | ||
1613 | Only one cross-canadian compiler is produced per architecture since | ||
1614 | they can be targeted at different processor optimizations using | ||
1615 | configurations passed to the compiler through the compile commands. | ||
1616 | This circumvents the need for multiple compilers and thus reduces the | ||
1617 | size of the toolchains. | ||
1618 | |||
1619 | .. note:: | ||
1620 | |||
1621 | For information on advantages gained when building a | ||
1622 | cross-development toolchain installer, see the | ||
1623 | ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" appendix | ||
1624 | in the Yocto Project Application Development and the | ||
1625 | Extensible Software Development Kit (eSDK) manual. | ||
1626 | |||
1627 | Shared State Cache | ||
1628 | ================== | ||
1629 | |||
1630 | By design, the OpenEmbedded build system builds everything from scratch | ||
1631 | unless :term:`BitBake` can determine | ||
1632 | that parts do not need to be rebuilt. Fundamentally, building from | ||
1633 | scratch is attractive as it means all parts are built fresh and no | ||
1634 | possibility of stale data exists that can cause problems. When | ||
1635 | developers hit problems, they typically default back to building from | ||
1636 | scratch so they have a know state from the start. | ||
1637 | |||
1638 | Building an image from scratch is both an advantage and a disadvantage | ||
1639 | to the process. As mentioned in the previous paragraph, building from | ||
1640 | scratch ensures that everything is current and starts from a known | ||
1641 | state. However, building from scratch also takes much longer as it | ||
1642 | generally means rebuilding things that do not necessarily need to be | ||
1643 | rebuilt. | ||
1644 | |||
1645 | The Yocto Project implements shared state code that supports incremental | ||
1646 | builds. The implementation of the shared state code answers the | ||
1647 | following questions that were fundamental roadblocks within the | ||
1648 | OpenEmbedded incremental build support system: | ||
1649 | |||
1650 | - What pieces of the system have changed and what pieces have not | ||
1651 | changed? | ||
1652 | |||
1653 | - How are changed pieces of software removed and replaced? | ||
1654 | |||
1655 | - How are pre-built components that do not need to be rebuilt from | ||
1656 | scratch used when they are available? | ||
1657 | |||
1658 | For the first question, the build system detects changes in the "inputs" | ||
1659 | to a given task by creating a checksum (or signature) of the task's | ||
1660 | inputs. If the checksum changes, the system assumes the inputs have | ||
1661 | changed and the task needs to be rerun. For the second question, the | ||
1662 | shared state (sstate) code tracks which tasks add which output to the | ||
1663 | build process. This means the output from a given task can be removed, | ||
1664 | upgraded or otherwise manipulated. The third question is partly | ||
1665 | addressed by the solution for the second question assuming the build | ||
1666 | system can fetch the sstate objects from remote locations and install | ||
1667 | them if they are deemed to be valid. | ||
1668 | |||
1669 | .. note:: | ||
1670 | |||
1671 | - The build system does not maintain | ||
1672 | :term:`PR` information as part of | ||
1673 | the shared state packages. Consequently, considerations exist that | ||
1674 | affect maintaining shared state feeds. For information on how the | ||
1675 | build system works with packages and can track incrementing ``PR`` | ||
1676 | information, see the ":ref:`dev-manual/dev-manual-common-tasks:automatically incrementing a package version number`" | ||
1677 | section in the Yocto Project Development Tasks Manual. | ||
1678 | |||
1679 | - The code in the build system that supports incremental builds is | ||
1680 | not simple code. For techniques that help you work around issues | ||
1681 | related to shared state code, see the | ||
1682 | ":ref:`dev-manual/dev-manual-common-tasks:viewing metadata used to create the input signature of a shared state task`" | ||
1683 | and | ||
1684 | ":ref:`dev-manual/dev-manual-common-tasks:invalidating shared state to force a task to run`" | ||
1685 | sections both in the Yocto Project Development Tasks Manual. | ||
1686 | |||
1687 | The rest of this section goes into detail about the overall incremental | ||
1688 | build architecture, the checksums (signatures), and shared state. | ||
1689 | |||
1690 | .. _concepts-overall-architecture: | ||
1691 | |||
1692 | Overall Architecture | ||
1693 | -------------------- | ||
1694 | |||
1695 | When determining what parts of the system need to be built, BitBake | ||
1696 | works on a per-task basis rather than a per-recipe basis. You might | ||
1697 | wonder why using a per-task basis is preferred over a per-recipe basis. | ||
1698 | To help explain, consider having the IPK packaging backend enabled and | ||
1699 | then switching to DEB. In this case, the | ||
1700 | :ref:`ref-tasks-install` and | ||
1701 | :ref:`ref-tasks-package` task outputs | ||
1702 | are still valid. However, with a per-recipe approach, the build would | ||
1703 | not include the ``.deb`` files. Consequently, you would have to | ||
1704 | invalidate the whole build and rerun it. Rerunning everything is not the | ||
1705 | best solution. Also, in this case, the core must be "taught" much about | ||
1706 | specific tasks. This methodology does not scale well and does not allow | ||
1707 | users to easily add new tasks in layers or as external recipes without | ||
1708 | touching the packaged-staging core. | ||
1709 | |||
1710 | .. _overview-checksums: | ||
1711 | |||
1712 | Checksums (Signatures) | ||
1713 | ---------------------- | ||
1714 | |||
1715 | The shared state code uses a checksum, which is a unique signature of a | ||
1716 | task's inputs, to determine if a task needs to be run again. Because it | ||
1717 | is a change in a task's inputs that triggers a rerun, the process needs | ||
1718 | to detect all the inputs to a given task. For shell tasks, this turns | ||
1719 | out to be fairly easy because the build process generates a "run" shell | ||
1720 | script for each task and it is possible to create a checksum that gives | ||
1721 | you a good idea of when the task's data changes. | ||
1722 | |||
1723 | To complicate the problem, there are things that should not be included | ||
1724 | in the checksum. First, there is the actual specific build path of a | ||
1725 | given task - the :term:`WORKDIR`. It | ||
1726 | does not matter if the work directory changes because it should not | ||
1727 | affect the output for target packages. Also, the build process has the | ||
1728 | objective of making native or cross packages relocatable. | ||
1729 | |||
1730 | .. note:: | ||
1731 | |||
1732 | Both native and cross packages run on the | ||
1733 | build host. However, cross packages generate output for the target | ||
1734 | architecture. | ||
1735 | |||
1736 | The checksum therefore needs to exclude ``WORKDIR``. The simplistic | ||
1737 | approach for excluding the work directory is to set ``WORKDIR`` to some | ||
1738 | fixed value and create the checksum for the "run" script. | ||
1739 | |||
1740 | Another problem results from the "run" scripts containing functions that | ||
1741 | might or might not get called. The incremental build solution contains | ||
1742 | code that figures out dependencies between shell functions. This code is | ||
1743 | used to prune the "run" scripts down to the minimum set, thereby | ||
1744 | alleviating this problem and making the "run" scripts much more readable | ||
1745 | as a bonus. | ||
1746 | |||
1747 | So far, solutions for shell scripts exist. What about Python tasks? The | ||
1748 | same approach applies even though these tasks are more difficult. The | ||
1749 | process needs to figure out what variables a Python function accesses | ||
1750 | and what functions it calls. Again, the incremental build solution | ||
1751 | contains code that first figures out the variable and function | ||
1752 | dependencies, and then creates a checksum for the data used as the input | ||
1753 | to the task. | ||
1754 | |||
1755 | Like the ``WORKDIR`` case, situations exist where dependencies should be | ||
1756 | ignored. For these situations, you can instruct the build process to | ||
1757 | ignore a dependency by using a line like the following: | ||
1758 | :: | ||
1759 | |||
1760 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
1761 | |||
1762 | This example ensures that the :term:`PACKAGE_ARCHS` variable | ||
1763 | does not depend on the value of :term:`MACHINE`, even if it does | ||
1764 | reference it. | ||
1765 | |||
1766 | Equally, there are cases where you need to add dependencies BitBake is | ||
1767 | not able to find. You can accomplish this by using a line like the | ||
1768 | following: | ||
1769 | :: | ||
1770 | |||
1771 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
1772 | |||
1773 | This example explicitly | ||
1774 | adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. | ||
1775 | |||
1776 | As an example, consider a case with in-line Python where BitBake is not | ||
1777 | able to figure out dependencies. When running in debug mode (i.e. using | ||
1778 | ``-DDD``), BitBake produces output when it discovers something for which | ||
1779 | it cannot figure out dependencies. The Yocto Project team has currently | ||
1780 | not managed to cover those dependencies in detail and is aware of the | ||
1781 | need to fix this situation. | ||
1782 | |||
1783 | Thus far, this section has limited discussion to the direct inputs into | ||
1784 | a task. Information based on direct inputs is referred to as the | ||
1785 | "basehash" in the code. However, the question of a task's indirect | ||
1786 | inputs still exits - items already built and present in the | ||
1787 | :term:`Build Directory`. The checksum (or | ||
1788 | signature) for a particular task needs to add the hashes of all the | ||
1789 | tasks on which the particular task depends. Choosing which dependencies | ||
1790 | to add is a policy decision. However, the effect is to generate a master | ||
1791 | checksum that combines the basehash and the hashes of the task's | ||
1792 | dependencies. | ||
1793 | |||
1794 | At the code level, a variety of ways exist by which both the basehash | ||
1795 | and the dependent task hashes can be influenced. Within the BitBake | ||
1796 | configuration file, you can give BitBake some extra information to help | ||
1797 | it construct the basehash. The following statement effectively results | ||
1798 | in a list of global variable dependency excludes (i.e. variables never | ||
1799 | included in any checksum): | ||
1800 | :: | ||
1801 | |||
1802 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\ | ||
1803 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\ | ||
1804 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ | ||
1805 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ | ||
1806 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
1807 | |||
1808 | The | ||
1809 | previous example excludes | ||
1810 | :term:`WORKDIR` since that variable | ||
1811 | is actually constructed as a path within | ||
1812 | :term:`TMPDIR`, which is on the | ||
1813 | whitelist. | ||
1814 | |||
1815 | The rules for deciding which hashes of dependent tasks to include | ||
1816 | through dependency chains are more complex and are generally | ||
1817 | accomplished with a Python function. The code in | ||
1818 | ``meta/lib/oe/sstatesig.py`` shows two examples of this and also | ||
1819 | illustrates how you can insert your own policy into the system if so | ||
1820 | desired. This file defines the two basic signature generators | ||
1821 | :term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and | ||
1822 | "OEBasicHash". By default, a dummy "noop" signature handler is enabled | ||
1823 | in BitBake. This means that behavior is unchanged from previous | ||
1824 | versions. OE-Core uses the "OEBasicHash" signature handler by default | ||
1825 | through this setting in the ``bitbake.conf`` file: | ||
1826 | :: | ||
1827 | |||
1828 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
1829 | |||
1830 | The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same | ||
1831 | as the "OEBasic" version but adds the task hash to the `stamp | ||
1832 | files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any | ||
1833 | metadata change that changes the task hash, automatically causing the | ||
1834 | task to be run again. This removes the need to bump | ||
1835 | :term:`PR` values, and changes to metadata | ||
1836 | automatically ripple across the build. | ||
1837 | |||
1838 | It is also worth noting that the end result of these signature | ||
1839 | generators is to make some dependency and hash information available to | ||
1840 | the build. This information includes: | ||
1841 | |||
1842 | - ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the | ||
1843 | recipe. | ||
1844 | |||
1845 | - ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each | ||
1846 | dependent task. | ||
1847 | |||
1848 | - ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for | ||
1849 | each task. | ||
1850 | |||
1851 | - ``BB_TASKHASH``: The hash of the currently running task. | ||
1852 | |||
1853 | Shared State | ||
1854 | ------------ | ||
1855 | |||
1856 | Checksums and dependencies, as discussed in the previous section, solve | ||
1857 | half the problem of supporting a shared state. The other half of the | ||
1858 | problem is being able to use checksum information during the build and | ||
1859 | being able to reuse or rebuild specific components. | ||
1860 | |||
1861 | The :ref:`sstate <ref-classes-sstate>` class is a | ||
1862 | relatively generic implementation of how to "capture" a snapshot of a | ||
1863 | given task. The idea is that the build process does not care about the | ||
1864 | source of a task's output. Output could be freshly built or it could be | ||
1865 | downloaded and unpacked from somewhere. In other words, the build | ||
1866 | process does not need to worry about its origin. | ||
1867 | |||
1868 | Two types of output exist. One type is just about creating a directory | ||
1869 | in :term:`WORKDIR`. A good example is | ||
1870 | the output of either | ||
1871 | :ref:`ref-tasks-install` or | ||
1872 | :ref:`ref-tasks-package`. The other | ||
1873 | type of output occurs when a set of data is merged into a shared | ||
1874 | directory tree such as the sysroot. | ||
1875 | |||
1876 | The Yocto Project team has tried to keep the details of the | ||
1877 | implementation hidden in ``sstate`` class. From a user's perspective, | ||
1878 | adding shared state wrapping to a task is as simple as this | ||
1879 | :ref:`ref-tasks-deploy` example taken | ||
1880 | from the :ref:`deploy <ref-classes-deploy>` class: | ||
1881 | :: | ||
1882 | |||
1883 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" | ||
1884 | SSTATETASKS += "do_deploy" | ||
1885 | do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" | ||
1886 | do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" | ||
1887 | |||
1888 | python do_deploy_setscene () { | ||
1889 | sstate_setscene(d) | ||
1890 | } | ||
1891 | addtask do_deploy_setscene | ||
1892 | do_deploy[dirs] = "${DEPLOYDIR} ${B}" | ||
1893 | do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" | ||
1894 | |||
1895 | The following list explains the previous example: | ||
1896 | |||
1897 | - Adding "do_deploy" to ``SSTATETASKS`` adds some required | ||
1898 | sstate-related processing, which is implemented in the | ||
1899 | :ref:`sstate <ref-classes-sstate>` class, to | ||
1900 | before and after the | ||
1901 | :ref:`ref-tasks-deploy` task. | ||
1902 | |||
1903 | - The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that | ||
1904 | ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally | ||
1905 | (i.e. when not using the sstate cache). This output becomes the input | ||
1906 | to the shared state cache. | ||
1907 | |||
1908 | - The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line | ||
1909 | causes the contents of the shared state cache to be copied to | ||
1910 | ``${DEPLOY_DIR_IMAGE}``. | ||
1911 | |||
1912 | .. note:: | ||
1913 | |||
1914 | If ``do_deploy`` is not already in the shared state cache or if its input | ||
1915 | checksum (signature) has changed from when the output was cached, the task | ||
1916 | runs to populate the shared state cache, after which the contents of the | ||
1917 | shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If | ||
1918 | ``do_deploy`` is in the shared state cache and its signature indicates | ||
1919 | that the cached output is still valid (i.e. if no relevant task inputs | ||
1920 | have changed), then the contents of the shared state cache copies | ||
1921 | directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task | ||
1922 | instead, skipping the ``do_deploy`` task. | ||
1923 | |||
1924 | - The following task definition is glue logic needed to make the | ||
1925 | previous settings effective: | ||
1926 | :: | ||
1927 | |||
1928 | python do_deploy_setscene () { | ||
1929 | sstate_setscene(d) | ||
1930 | } | ||
1931 | addtask do_deploy_setscene | ||
1932 | |||
1933 | ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task | ||
1934 | through the shared state cache if possible. If the task was | ||
1935 | accelerated, ``sstate_setscene()`` returns True. Otherwise, it | ||
1936 | returns False, and the normal ``do_deploy`` task runs. For more | ||
1937 | information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`" | ||
1938 | section in the BitBake User Manual. | ||
1939 | |||
1940 | - The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates | ||
1941 | ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and | ||
1942 | also sets the current working directory of ``do_deploy`` to ``${B}``. | ||
1943 | For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" | ||
1944 | section in the BitBake | ||
1945 | User Manual. | ||
1946 | |||
1947 | .. note:: | ||
1948 | |||
1949 | In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be | ||
1950 | the same, you can use ``sstate-plaindirs``. For example, to preserve the | ||
1951 | ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package`` | ||
1952 | task, use the following: | ||
1953 | :: | ||
1954 | |||
1955 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | ||
1956 | |||
1957 | |||
1958 | - The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends | ||
1959 | extra metadata to the `stamp | ||
1960 | file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the | ||
1961 | metadata makes the task specific to a machine's architecture. See | ||
1962 | ":ref:`bitbake:ref-bitbake-tasklist`" | ||
1963 | section in the BitBake User Manual for more information on the | ||
1964 | ``stamp-extra-info`` flag. | ||
1965 | |||
1966 | - ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with | ||
1967 | multiple directories. For example, the following declares | ||
1968 | ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories, | ||
1969 | which populates the shared state cache, and ``PKGDATA_DIR`` and | ||
1970 | ``SHLIBSDIR`` as the corresponding shared state output directories: | ||
1971 | :: | ||
1972 | |||
1973 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" | ||
1974 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" | ||
1975 | |||
1976 | - These methods also include the ability to take a lockfile when | ||
1977 | manipulating shared state directory structures, for cases where file | ||
1978 | additions or removals are sensitive: | ||
1979 | :: | ||
1980 | |||
1981 | do_package[sstate-lockfile] = "${PACKAGELOCK}" | ||
1982 | |||
1983 | Behind the scenes, the shared state code works by looking in | ||
1984 | :term:`SSTATE_DIR` and | ||
1985 | :term:`SSTATE_MIRRORS` for | ||
1986 | shared state files. Here is an example: | ||
1987 | :: | ||
1988 | |||
1989 | SSTATE_MIRRORS ?= "\ | ||
1990 | file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ | ||
1991 | file://.\* file:///some/local/dir/sstate/PATH" | ||
1992 | |||
1993 | .. note:: | ||
1994 | |||
1995 | The shared state directory (``SSTATE_DIR``) is organized into two-character | ||
1996 | subdirectories, where the subdirectory names are based on the first two | ||
1997 | characters of the hash. | ||
1998 | If the shared state directory structure for a mirror has the same structure | ||
1999 | as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build | ||
2000 | system to map to the appropriate subdirectory. | ||
2001 | |||
2002 | The shared state package validity can be detected just by looking at the | ||
2003 | filename since the filename contains the task checksum (or signature) as | ||
2004 | described earlier in this section. If a valid shared state package is | ||
2005 | found, the build process downloads it and uses it to accelerate the | ||
2006 | task. | ||
2007 | |||
2008 | The build processes use the ``*_setscene`` tasks for the task | ||
2009 | acceleration phase. BitBake goes through this phase before the main | ||
2010 | execution code and tries to accelerate any tasks for which it can find | ||
2011 | shared state packages. If a shared state package for a task is | ||
2012 | available, the shared state package is used. This means the task and any | ||
2013 | tasks on which it is dependent are not executed. | ||
2014 | |||
2015 | As a real world example, the aim is when building an IPK-based image, | ||
2016 | only the | ||
2017 | :ref:`ref-tasks-package_write_ipk` | ||
2018 | tasks would have their shared state packages fetched and extracted. | ||
2019 | Since the sysroot is not used, it would never get extracted. This is | ||
2020 | another reason why a task-based approach is preferred over a | ||
2021 | recipe-based approach, which would have to install the output from every | ||
2022 | task. | ||
2023 | |||
2024 | Automatically Added Runtime Dependencies | ||
2025 | ======================================== | ||
2026 | |||
2027 | The OpenEmbedded build system automatically adds common types of runtime | ||
2028 | dependencies between packages, which means that you do not need to | ||
2029 | explicitly declare the packages using | ||
2030 | :term:`RDEPENDS`. Three automatic | ||
2031 | mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that | ||
2032 | handle shared libraries, package configuration (pkg-config) modules, and | ||
2033 | ``-dev`` and ``-dbg`` packages, respectively. For other types of runtime | ||
2034 | dependencies, you must manually declare the dependencies. | ||
2035 | |||
2036 | - ``shlibdeps``: During the | ||
2037 | :ref:`ref-tasks-package` task of | ||
2038 | each recipe, all shared libraries installed by the recipe are | ||
2039 | located. For each shared library, the package that contains the | ||
2040 | shared library is registered as providing the shared library. More | ||
2041 | specifically, the package is registered as providing the | ||
2042 | `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The | ||
2043 | resulting shared-library-to-package mapping is saved globally in | ||
2044 | :term:`PKGDATA_DIR` by the | ||
2045 | :ref:`ref-tasks-packagedata` | ||
2046 | task. | ||
2047 | |||
2048 | Simultaneously, all executables and shared libraries installed by the | ||
2049 | recipe are inspected to see what shared libraries they link against. | ||
2050 | For each shared library dependency that is found, ``PKGDATA_DIR`` is | ||
2051 | queried to see if some package (likely from a different recipe) | ||
2052 | contains the shared library. If such a package is found, a runtime | ||
2053 | dependency is added from the package that depends on the shared | ||
2054 | library to the package that contains the library. | ||
2055 | |||
2056 | The automatically added runtime dependency also includes a version | ||
2057 | restriction. This version restriction specifies that at least the | ||
2058 | current version of the package that provides the shared library must | ||
2059 | be used, as if "package (>= version)" had been added to ``RDEPENDS``. | ||
2060 | This forces an upgrade of the package containing the shared library | ||
2061 | when installing the package that depends on the library, if needed. | ||
2062 | |||
2063 | If you want to avoid a package being registered as providing a | ||
2064 | particular shared library (e.g. because the library is for internal | ||
2065 | use only), then add the library to | ||
2066 | :term:`PRIVATE_LIBS` inside | ||
2067 | the package's recipe. | ||
2068 | |||
2069 | - ``pcdeps``: During the ``do_package`` task of each recipe, all | ||
2070 | pkg-config modules (``*.pc`` files) installed by the recipe are | ||
2071 | located. For each module, the package that contains the module is | ||
2072 | registered as providing the module. The resulting module-to-package | ||
2073 | mapping is saved globally in ``PKGDATA_DIR`` by the | ||
2074 | ``do_packagedata`` task. | ||
2075 | |||
2076 | Simultaneously, all pkg-config modules installed by the recipe are | ||
2077 | inspected to see what other pkg-config modules they depend on. A | ||
2078 | module is seen as depending on another module if it contains a | ||
2079 | "Requires:" line that specifies the other module. For each module | ||
2080 | dependency, ``PKGDATA_DIR`` is queried to see if some package | ||
2081 | contains the module. If such a package is found, a runtime dependency | ||
2082 | is added from the package that depends on the module to the package | ||
2083 | that contains the module. | ||
2084 | |||
2085 | .. note:: | ||
2086 | |||
2087 | The | ||
2088 | pcdeps | ||
2089 | mechanism most often infers dependencies between | ||
2090 | -dev | ||
2091 | packages. | ||
2092 | |||
2093 | - ``depchains``: If a package ``foo`` depends on a package ``bar``, | ||
2094 | then ``foo-dev`` and ``foo-dbg`` are also made to depend on | ||
2095 | ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev`` | ||
2096 | packages as an example, the ``bar-dev`` package might provide headers | ||
2097 | and shared library symlinks needed by ``foo-dev``, which shows the | ||
2098 | need for a dependency between the packages. | ||
2099 | |||
2100 | The dependencies added by ``depchains`` are in the form of | ||
2101 | :term:`RRECOMMENDS`. | ||
2102 | |||
2103 | .. note:: | ||
2104 | |||
2105 | By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on | ||
2106 | ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in | ||
2107 | bitbake.conf) includes "${PN}". | ||
2108 | |||
2109 | To ensure that the dependency chain is never broken, ``-dev`` and | ||
2110 | ``-dbg`` packages are always generated by default, even if the | ||
2111 | packages turn out to be empty. See the | ||
2112 | :term:`ALLOW_EMPTY` variable | ||
2113 | for more information. | ||
2114 | |||
2115 | The ``do_package`` task depends on the ``do_packagedata`` task of each | ||
2116 | recipe in :term:`DEPENDS` through use | ||
2117 | of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` | ||
2118 | declaration, which guarantees that the required | ||
2119 | shared-library/module-to-package mapping information will be available | ||
2120 | when needed as long as ``DEPENDS`` has been correctly set. | ||
2121 | |||
2122 | Fakeroot and Pseudo | ||
2123 | =================== | ||
2124 | |||
2125 | Some tasks are easier to implement when allowed to perform certain | ||
2126 | operations that are normally reserved for the root user (e.g. | ||
2127 | :ref:`ref-tasks-install`, | ||
2128 | :ref:`do_package_write* <ref-tasks-package_write_deb>`, | ||
2129 | :ref:`ref-tasks-rootfs`, and | ||
2130 | :ref:`do_image* <ref-tasks-image>`). For example, | ||
2131 | the ``do_install`` task benefits from being able to set the UID and GID | ||
2132 | of installed files to arbitrary values. | ||
2133 | |||
2134 | One approach to allowing tasks to perform root-only operations would be | ||
2135 | to require :term:`BitBake` to run as | ||
2136 | root. However, this method is cumbersome and has security issues. The | ||
2137 | approach that is actually used is to run tasks that benefit from root | ||
2138 | privileges in a "fake" root environment. Within this environment, the | ||
2139 | task and its child processes believe that they are running as the root | ||
2140 | user, and see an internally consistent view of the filesystem. As long | ||
2141 | as generating the final output (e.g. a package or an image) does not | ||
2142 | require root privileges, the fact that some earlier steps ran in a fake | ||
2143 | root environment does not cause problems. | ||
2144 | |||
2145 | The capability to run tasks in a fake root environment is known as | ||
2146 | "`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from | ||
2147 | the BitBake keyword/variable flag that requests a fake root environment | ||
2148 | for a task. | ||
2149 | |||
2150 | In the :term:`OpenEmbedded Build System`, | ||
2151 | the program that | ||
2152 | implements fakeroot is known as | ||
2153 | `Pseudo <https://www.yoctoproject.org/software-item/pseudo/>`__. Pseudo | ||
2154 | overrides system calls by using the environment variable ``LD_PRELOAD``, | ||
2155 | which results in the illusion of running as root. To keep track of | ||
2156 | "fake" file ownership and permissions resulting from operations that | ||
2157 | require root permissions, Pseudo uses an SQLite 3 database. This | ||
2158 | database is stored in | ||
2159 | ``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db`` | ||
2160 | for individual recipes. Storing the database in a file as opposed to in | ||
2161 | memory gives persistence between tasks and builds, which is not | ||
2162 | accomplished using fakeroot. | ||
2163 | |||
2164 | .. note:: | ||
2165 | |||
2166 | If you add your own task that manipulates the same files or | ||
2167 | directories as a fakeroot task, then that task also needs to run | ||
2168 | under fakeroot. Otherwise, the task cannot run root-only operations, | ||
2169 | and cannot see the fake file ownership and permissions set by the | ||
2170 | other task. You need to also add a dependency on | ||
2171 | virtual/fakeroot-native:do_populate_sysroot | ||
2172 | , giving the following: | ||
2173 | :: | ||
2174 | |||
2175 | fakeroot do_mytask () { | ||
2176 | ... | ||
2177 | } | ||
2178 | do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" | ||
2179 | |||
2180 | |||
2181 | For more information, see the | ||
2182 | :term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the | ||
2183 | BitBake User Manual. You can also reference the "`Why Not | ||
2184 | Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__" | ||
2185 | article for background information on Fakeroot and Pseudo. | ||