summaryrefslogtreecommitdiffstats
path: root/documentation/overview-manual/concepts.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/overview-manual/concepts.rst')
-rw-r--r--documentation/overview-manual/concepts.rst1029
1 files changed, 631 insertions, 398 deletions
diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst
index 257de44ec8..62f2327a7e 100644
--- a/documentation/overview-manual/concepts.rst
+++ b/documentation/overview-manual/concepts.rst
@@ -34,10 +34,10 @@ itself is of various types:
34 34
35BitBake knows how to combine multiple data sources together and refers 35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the 36to each data source as a layer. For information on layers, see the
37":ref:`dev-manual/common-tasks:understanding and creating layers`" 37":ref:`dev-manual/layers:understanding and creating layers`"
38section of the Yocto Project Development Tasks Manual. 38section of the Yocto Project Development Tasks Manual.
39 39
40Following are some brief details on these core components. For 40Here are some brief details on these core components. For
41additional information on how these components interact during a build, 41additional information on how these components interact during a build,
42see the 42see the
43":ref:`overview-manual/concepts:openembedded build system concepts`" 43":ref:`overview-manual/concepts:openembedded build system concepts`"
@@ -55,8 +55,7 @@ This section briefly introduces BitBake. If you want more information on
55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`. 55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`.
56 56
57To see a list of the options BitBake supports, use either of the 57To see a list of the options BitBake supports, use either of the
58following commands: 58following commands::
59::
60 59
61 $ bitbake -h 60 $ bitbake -h
62 $ bitbake --help 61 $ bitbake --help
@@ -66,17 +65,14 @@ The most common usage for BitBake is ``bitbake recipename``, where
66to as the "target"). The target often equates to the first part of a 65to as the "target"). The target often equates to the first part of a
67recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``). 66recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
68So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might 67So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
69type the following: 68type the following::
70::
71 69
72 $ bitbake matchbox-desktop 70 $ bitbake matchbox-desktop
73 71
74Several different 72Several different versions of ``matchbox-desktop`` might exist. BitBake chooses
75versions of ``matchbox-desktop`` might exist. BitBake chooses the one 73the one selected by the distribution configuration. You can get more details
76selected by the distribution configuration. You can get more details 74about how BitBake chooses between different target versions and providers in the
77about how BitBake chooses between different target versions and 75":ref:`bitbake-user-manual/bitbake-user-manual-execution:preferences`" section
78providers in the
79":ref:`Preferences <bitbake:bitbake-user-manual/bitbake-user-manual-execution:preferences>`" section
80of the BitBake User Manual. 76of the BitBake User Manual.
81 77
82BitBake also tries to execute any dependent tasks first. So for example, 78BitBake also tries to execute any dependent tasks first. So for example,
@@ -109,12 +105,11 @@ Classes
109------- 105-------
110 106
111Class files (``.bbclass``) contain information that is useful to share 107Class files (``.bbclass``) contain information that is useful to share
112between recipes files. An example is the 108between recipes files. An example is the :ref:`ref-classes-autotools` class,
113:ref:`autotools <ref-classes-autotools>` class, 109which contains common settings for any application that is built with
114which contains common settings for any application that Autotools uses. 110the :wikipedia:`GNU Autotools <GNU_Autotools>`.
115The ":ref:`ref-manual/classes:Classes`" chapter in the 111The ":ref:`ref-manual/classes:Classes`" chapter in the Yocto Project
116Yocto Project Reference Manual provides details about classes and how to 112Reference Manual provides details about classes and how to use them.
117use them.
118 113
119Configurations 114Configurations
120-------------- 115--------------
@@ -132,7 +127,7 @@ Layers
132 127
133Layers are repositories that contain related metadata (i.e. sets of 128Layers are repositories that contain related metadata (i.e. sets of
134instructions) that tell the OpenEmbedded build system how to build a 129instructions) that tell the OpenEmbedded build system how to build a
135target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__ 130target. :ref:`overview-manual/yp-intro:the yocto project layer model`
136facilitates collaboration, sharing, customization, and reuse within the 131facilitates collaboration, sharing, customization, and reuse within the
137Yocto Project development environment. Layers logically separate 132Yocto Project development environment. Layers logically separate
138information for your project. For example, you can use a layer to hold 133information for your project. For example, you can use a layer to hold
@@ -141,8 +136,8 @@ hardware-specific configurations allows you to share other metadata by
141using a different layer where that metadata might be common across 136using a different layer where that metadata might be common across
142several pieces of hardware. 137several pieces of hardware.
143 138
144Many layers exist that work in the Yocto Project development environment. The 139There are many layers working in the Yocto Project development environment. The
145:yocto_home:`Yocto Project Curated Layer Index </software-overview/layers/>` 140:yocto_home:`Yocto Project Compatible Layer Index </software-overview/layers/>`
146and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from 141and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from
147which you can use or leverage. 142which you can use or leverage.
148 143
@@ -151,7 +146,7 @@ Conforming to a known structure allows BitBake to make assumptions
151during builds on where to find types of metadata. You can find 146during builds on where to find types of metadata. You can find
152procedures and learn about tools (i.e. ``bitbake-layers``) for creating 147procedures and learn about tools (i.e. ``bitbake-layers``) for creating
153layers suitable for the Yocto Project in the 148layers suitable for the Yocto Project in the
154":ref:`dev-manual/common-tasks:understanding and creating layers`" 149":ref:`dev-manual/layers:understanding and creating layers`"
155section of the Yocto Project Development Tasks Manual. 150section of the Yocto Project Development Tasks Manual.
156 151
157OpenEmbedded Build System Concepts 152OpenEmbedded Build System Concepts
@@ -168,7 +163,7 @@ remainder of this section expands on the fundamental input, output,
168process, and metadata logical blocks that make up the workflow. 163process, and metadata logical blocks that make up the workflow.
169 164
170.. image:: figures/YP-flow-diagram.png 165.. image:: figures/YP-flow-diagram.png
171 :align: center 166 :width: 100%
172 167
173In general, the build's workflow consists of several functional areas: 168In general, the build's workflow consists of several functional areas:
174 169
@@ -207,11 +202,11 @@ you can tell BitBake the target architecture for which you are building
207the image, where to store downloaded source, and other build properties. 202the image, where to store downloaded source, and other build properties.
208 203
209The following figure shows an expanded representation of the "User 204The following figure shows an expanded representation of the "User
210Configuration" box of the `general workflow 205Configuration" box of the :ref:`general workflow
211figure <#general-workflow-figure>`__: 206figure <overview-manual/concepts:openembedded build system concepts>`:
212 207
213.. image:: figures/user-configuration.png 208.. image:: figures/user-configuration.png
214 :align: center 209 :width: 100%
215 210
216BitBake needs some basic configuration files in order to complete a 211BitBake needs some basic configuration files in order to complete a
217build. These files are ``*.conf`` files. The minimally necessary ones 212build. These files are ``*.conf`` files. The minimally necessary ones
@@ -235,13 +230,12 @@ for creating actual configuration files when you source
235:ref:`structure-core-script`, which is the 230:ref:`structure-core-script`, which is the
236build environment script. 231build environment script.
237 232
238Sourcing the build environment script creates a 233Sourcing the build environment script creates a :term:`Build Directory`
239:term:`Build Directory` if one does not 234if one does not already exist. BitBake uses the :term:`Build Directory`
240already exist. BitBake uses the Build Directory for all its work during 235for all its work during builds. The Build Directory has a ``conf`` directory
241builds. The Build Directory has a ``conf`` directory that contains 236that contains default versions of your ``local.conf`` and ``bblayers.conf``
242default versions of your ``local.conf`` and ``bblayers.conf``
243configuration files. These default configuration files are created only 237configuration files. These default configuration files are created only
244if versions do not already exist in the Build Directory at the time you 238if versions do not already exist in the :term:`Build Directory` at the time you
245source the build environment setup script. 239source the build environment setup script.
246 240
247Because the Poky repository is fundamentally an aggregation of existing 241Because the Poky repository is fundamentally an aggregation of existing
@@ -253,9 +247,9 @@ assumes the script is executed from within a cloned or unpacked version
253of Poky. 247of Poky.
254 248
255Depending on where the script is sourced, different sub-scripts are 249Depending on where the script is sourced, different sub-scripts are
256called to set up the Build Directory (Yocto or OpenEmbedded). 250called to set up the :term:`Build Directory` (Yocto or OpenEmbedded).
257Specifically, the script ``scripts/oe-setup-builddir`` inside the poky 251Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
258directory sets up the Build Directory and seeds the directory (if 252directory sets up the :term:`Build Directory` and seeds the directory (if
259necessary) with configuration files appropriate for the Yocto Project 253necessary) with configuration files appropriate for the Yocto Project
260development environment. 254development environment.
261 255
@@ -271,7 +265,7 @@ The ``local.conf`` file provides many basic variables that define a
271build environment. Here is a list of a few. To see the default 265build environment. Here is a list of a few. To see the default
272configurations in a ``local.conf`` file created by the build environment 266configurations in a ``local.conf`` file created by the build environment
273script, see the 267script, see the
274:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/local.conf.sample>` 268:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/templates/default/local.conf.sample>`
275in the ``meta-poky`` layer: 269in the ``meta-poky`` layer:
276 270
277- *Target Machine Selection:* Controlled by the 271- *Target Machine Selection:* Controlled by the
@@ -302,25 +296,20 @@ in the ``meta-poky`` layer:
302 296
303.. note:: 297.. note::
304 298
305 Configurations set in the 299 Configurations set in the ``conf/local.conf`` file can also be set
306 conf/local.conf 300 in the ``conf/site.conf`` and ``conf/auto.conf`` configuration files.
307 file can also be set in the
308 conf/site.conf
309 and
310 conf/auto.conf
311 configuration files.
312 301
313The ``bblayers.conf`` file tells BitBake what layers you want considered 302The ``bblayers.conf`` file tells BitBake what layers you want considered
314during the build. By default, the layers listed in this file include 303during the build. By default, the layers listed in this file include
315layers minimally needed by the build system. However, you must manually 304layers minimally needed by the build system. However, you must manually
316add any custom layers you have created. You can find more information on 305add any custom layers you have created. You can find more information on
317working with the ``bblayers.conf`` file in the 306working with the ``bblayers.conf`` file in the
318":ref:`dev-manual/common-tasks:enabling your layer`" 307":ref:`dev-manual/layers:enabling your layer`"
319section in the Yocto Project Development Tasks Manual. 308section in the Yocto Project Development Tasks Manual.
320 309
321The files ``site.conf`` and ``auto.conf`` are not created by the 310The files ``site.conf`` and ``auto.conf`` are not created by the
322environment initialization script. If you want the ``site.conf`` file, 311environment initialization script. If you want the ``site.conf`` file,
323you need to create that yourself. The ``auto.conf`` file is typically 312you need to create it yourself. The ``auto.conf`` file is typically
324created by an autobuilder: 313created by an autobuilder:
325 314
326- *site.conf:* You can use the ``conf/site.conf`` configuration 315- *site.conf:* You can use the ``conf/site.conf`` configuration
@@ -328,17 +317,7 @@ created by an autobuilder:
328 you had several build environments and they shared some common 317 you had several build environments and they shared some common
329 features. You can set these default build properties here. A good 318 features. You can set these default build properties here. A good
330 example is perhaps the packaging format to use through the 319 example is perhaps the packaging format to use through the
331 :term:`PACKAGE_CLASSES` 320 :term:`PACKAGE_CLASSES` variable.
332 variable.
333
334 One useful scenario for using the ``conf/site.conf`` file is to
335 extend your :term:`BBPATH` variable
336 to include the path to a ``conf/site.conf``. Then, when BitBake looks
337 for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file
338 and applies your common configurations found in the file. To override
339 configurations in a particular build directory, alter the similar
340 configurations within that build directory's ``conf/local.conf``
341 file.
342 321
343- *auto.conf:* The file is usually created and written to by an 322- *auto.conf:* The file is usually created and written to by an
344 autobuilder. The settings put into the file are typically the same as 323 autobuilder. The settings put into the file are typically the same as
@@ -372,9 +351,9 @@ BitBake's global behavior. This section takes a closer look at the
372layers the build system uses to further control the build. These layers 351layers the build system uses to further control the build. These layers
373provide Metadata for the software, machine, and policies. 352provide Metadata for the software, machine, and policies.
374 353
375In general, three types of layer input exists. You can see them below 354In general, there are three types of layer input. You can see them below
376the "User Configuration" box in the `general workflow 355the "User Configuration" box in the `general workflow
377figure <#general-workflow-figure>`__: 356figure <overview-manual/concepts:openembedded build system concepts>`:
378 357
379- *Metadata (.bb + Patches):* Software layers containing 358- *Metadata (.bb + Patches):* Software layers containing
380 user-supplied recipe files, patches, and append files. A good example 359 user-supplied recipe files, patches, and append files. A good example
@@ -387,8 +366,8 @@ figure <#general-workflow-figure>`__:
387- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e. 366- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
388 "BSP Layer" in the following figure) providing machine-specific 367 "BSP Layer" in the following figure) providing machine-specific
389 configurations. This type of information is specific to a particular 368 configurations. This type of information is specific to a particular
390 target architecture. A good example of a BSP layer from the `Poky 369 target architecture. A good example of a BSP layer from the
391 Reference Distribution <#gs-reference-distribution-poky>`__ is the 370 :ref:`overview-manual/yp-intro:reference distribution (poky)` is the
392 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>` 371 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>`
393 layer. 372 layer.
394 373
@@ -403,10 +382,12 @@ figure <#general-workflow-figure>`__:
403 that contain many policy configurations for the Poky distribution. 382 that contain many policy configurations for the Poky distribution.
404 383
405The following figure shows an expanded representation of these three 384The following figure shows an expanded representation of these three
406layers from the `general workflow figure <#general-workflow-figure>`__: 385layers from the :ref:`general workflow figure
386<overview-manual/concepts:openembedded build system concepts>`:
407 387
408.. image:: figures/layer-input.png 388.. image:: figures/layer-input.png
409 :align: center 389 :align: center
390 :width: 70%
410 391
411In general, all layers have a similar structure. They all contain a 392In general, all layers have a similar structure. They all contain a
412licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed, 393licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
@@ -414,13 +395,13 @@ a ``README`` file as good practice and especially if the layer is to be
414distributed, a configuration directory, and recipe directories. You can 395distributed, a configuration directory, and recipe directories. You can
415learn about the general structure for layers used with the Yocto Project 396learn about the general structure for layers used with the Yocto Project
416in the 397in the
417":ref:`dev-manual/common-tasks:creating your own layer`" 398":ref:`dev-manual/layers:creating your own layer`"
418section in the 399section in the
419Yocto Project Development Tasks Manual. For a general discussion on 400Yocto Project Development Tasks Manual. For a general discussion on
420layers and the many layers from which you can draw, see the 401layers and the many layers from which you can draw, see the
421"`Layers <#overview-layers>`__" and "`The Yocto Project Layer 402":ref:`overview-manual/concepts:layers`" and
422Model <#the-yocto-project-layer-model>`__" sections both earlier in this 403":ref:`overview-manual/yp-intro:the yocto project layer model`" sections both
423manual. 404earlier in this manual.
424 405
425If you explored the previous links, you discovered some areas where many 406If you explored the previous links, you discovered some areas where many
426layers that work with the Yocto Project exist. The :yocto_git:`Source 407layers that work with the Yocto Project exist. The :yocto_git:`Source
@@ -428,8 +409,8 @@ Repositories <>` also shows layers categorized under "Yocto Metadata Layers."
428 409
429.. note:: 410.. note::
430 411
431 Layers exist in the Yocto Project Source Repositories that cannot be 412 There are layers in the Yocto Project Source Repositories that cannot be
432 found in the OpenEmbedded Layer Index. These layers are either 413 found in the OpenEmbedded Layer Index. Such layers are either
433 deprecated or experimental in nature. 414 deprecated or experimental in nature.
434 415
435BitBake uses the ``conf/bblayers.conf`` file, which is part of the user 416BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
@@ -443,7 +424,7 @@ The distribution layer provides policy configurations for your
443distribution. Best practices dictate that you isolate these types of 424distribution. Best practices dictate that you isolate these types of
444configurations into their own layer. Settings you provide in 425configurations into their own layer. Settings you provide in
445``conf/distro/distro.conf`` override similar settings that BitBake finds 426``conf/distro/distro.conf`` override similar settings that BitBake finds
446in your ``conf/local.conf`` file in the Build Directory. 427in your ``conf/local.conf`` file in the :term:`Build Directory`.
447 428
448The following list provides some explanation and references for what you 429The following list provides some explanation and references for what you
449typically find in the distribution layer: 430typically find in the distribution layer:
@@ -490,7 +471,7 @@ the machine (``conf/machine/machine.conf``) and, of course, the layer
490 471
491The remainder of the layer is dedicated to specific recipes by function: 472The remainder of the layer is dedicated to specific recipes by function:
492``recipes-bsp``, ``recipes-core``, ``recipes-graphics``, 473``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
493``recipes-kernel``, and so forth. Metadata can exist for multiple 474``recipes-kernel``, and so forth. There can be metadata for multiple
494formfactors, graphics support systems, and so forth. 475formfactors, graphics support systems, and so forth.
495 476
496.. note:: 477.. note::
@@ -514,11 +495,12 @@ Sources
514------- 495-------
515 496
516In order for the OpenEmbedded build system to create an image or any 497In order for the OpenEmbedded build system to create an image or any
517target, it must be able to access source files. The `general workflow 498target, it must be able to access source files. The :ref:`general workflow
518figure <#general-workflow-figure>`__ represents source files using the 499figure <overview-manual/concepts:openembedded build system concepts>`
519"Upstream Project Releases", "Local Projects", and "SCMs (optional)" 500represents source files using the "Upstream Project Releases", "Local
520boxes. The figure represents mirrors, which also play a role in locating 501Projects", and "SCMs (optional)" boxes. The figure represents mirrors,
521source files, with the "Source Materials" box. 502which also play a role in locating source files, with the "Source
503Materials" box.
522 504
523The method by which source files are ultimately organized is a function 505The method by which source files are ultimately organized is a function
524of the project. For example, for released software, projects tend to use 506of the project. For example, for released software, projects tend to use
@@ -528,13 +510,11 @@ project that is more dynamic or experimental in nature, a project might
528keep source files in a repository controlled by a Source Control Manager 510keep source files in a repository controlled by a Source Control Manager
529(SCM) such as Git. Pulling source from a repository allows you to 511(SCM) such as Git. Pulling source from a repository allows you to
530control the point in the repository (the revision) from which you want 512control the point in the repository (the revision) from which you want
531to build software. Finally, a combination of the two might exist, which 513to build software. A combination of the two is also possible.
532would give the consumer a choice when deciding where to get source
533files.
534 514
535BitBake uses the :term:`SRC_URI` 515BitBake uses the :term:`SRC_URI`
536variable to point to source files regardless of their location. Each 516variable to point to source files regardless of their location. Each
537recipe must have a ``SRC_URI`` variable that points to the source. 517recipe must have a :term:`SRC_URI` variable that points to the source.
538 518
539Another area that plays a significant role in where source files come 519Another area that plays a significant role in where source files come
540from is pointed to by the 520from is pointed to by the
@@ -542,22 +522,24 @@ from is pointed to by the
542a cache that can hold previously downloaded source. You can also 522a cache that can hold previously downloaded source. You can also
543instruct the OpenEmbedded build system to create tarballs from Git 523instruct the OpenEmbedded build system to create tarballs from Git
544repositories, which is not the default behavior, and store them in the 524repositories, which is not the default behavior, and store them in the
545``DL_DIR`` by using the 525:term:`DL_DIR` by using the
546:term:`BB_GENERATE_MIRROR_TARBALLS` 526:term:`BB_GENERATE_MIRROR_TARBALLS`
547variable. 527variable.
548 528
549Judicious use of a ``DL_DIR`` directory can save the build system a trip 529Judicious use of a :term:`DL_DIR` directory can save the build system a trip
550across the Internet when looking for files. A good method for using a 530across the Internet when looking for files. A good method for using a download
551download directory is to have ``DL_DIR`` point to an area outside of 531directory is to have :term:`DL_DIR` point to an area outside of your
552your Build Directory. Doing so allows you to safely delete the Build 532:term:`Build Directory`. Doing so allows you to safely delete the
553Directory if needed without fear of removing any downloaded source file. 533:term:`Build Directory` if needed without fear of removing any downloaded
534source file.
554 535
555The remainder of this section provides a deeper look into the source 536The remainder of this section provides a deeper look into the source
556files and the mirrors. Here is a more detailed look at the source file 537files and the mirrors. Here is a more detailed look at the source file
557area of the `general workflow figure <#general-workflow-figure>`__: 538area of the :ref:`general workflow figure <overview-manual/concepts:openembedded build system concepts>`:
558 539
559.. image:: figures/source-input.png 540.. image:: figures/source-input.png
560 :align: center 541 :align: center
542 :width: 70%
561 543
562Upstream Project Releases 544Upstream Project Releases
563~~~~~~~~~~~~~~~~~~~~~~~~~ 545~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -572,33 +554,30 @@ Local Projects
572~~~~~~~~~~~~~~ 554~~~~~~~~~~~~~~
573 555
574Local projects are custom bits of software the user provides. These bits 556Local projects are custom bits of software the user provides. These bits
575reside somewhere local to a project - perhaps a directory into which the 557reside somewhere local to a project --- perhaps a directory into which the
576user checks in items (e.g. a local directory containing a development 558user checks in items (e.g. a local directory containing a development
577source tree used by the group). 559source tree used by the group).
578 560
579The canonical method through which to include a local project is to use 561The canonical method through which to include a local project is to use the
580the :ref:`externalsrc <ref-classes-externalsrc>` 562:ref:`ref-classes-externalsrc` class to include that local project. You use
581class to include that local project. You use either the ``local.conf`` 563either the ``local.conf`` or a recipe's append file to override or set the
582or a recipe's append file to override or set the recipe to point to the 564recipe to point to the local directory on your disk to pull in the whole
583local directory on your disk to pull in the whole source tree. 565source tree.
584 566
585Source Control Managers (Optional) 567Source Control Managers (Optional)
586~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 568~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
587 569
588Another place from which the build system can get source files is with 570Another place from which the build system can get source files is with
589:ref:`fetchers <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` employing various Source 571:ref:`bitbake-user-manual/bitbake-user-manual-fetching:fetchers` employing
590Control Managers (SCMs) such as Git or Subversion. In such cases, a 572various Source Control Managers (SCMs) such as Git or Subversion. In such
591repository is cloned or checked out. The 573cases, a repository is cloned or checked out. The :ref:`ref-tasks-fetch` task
592:ref:`ref-tasks-fetch` task inside 574inside BitBake uses the :term:`SRC_URI` variable and the argument's prefix to
593BitBake uses the :term:`SRC_URI` 575determine the correct fetcher module.
594variable and the argument's prefix to determine the correct fetcher
595module.
596 576
597.. note:: 577.. note::
598 578
599 For information on how to have the OpenEmbedded build system generate 579 For information on how to have the OpenEmbedded build system generate
600 tarballs for Git repositories and place them in the 580 tarballs for Git repositories and place them in the :term:`DL_DIR`
601 DL_DIR
602 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS` 581 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS`
603 variable in the Yocto Project Reference Manual. 582 variable in the Yocto Project Reference Manual.
604 583
@@ -609,7 +588,7 @@ the specific revision from which to build.
609Source Mirror(s) 588Source Mirror(s)
610~~~~~~~~~~~~~~~~ 589~~~~~~~~~~~~~~~~
611 590
612Two kinds of mirrors exist: pre-mirrors and regular mirrors. The 591There are two kinds of mirrors: pre-mirrors and regular mirrors. The
613:term:`PREMIRRORS` and 592:term:`PREMIRRORS` and
614:term:`MIRRORS` variables point to 593:term:`MIRRORS` variables point to
615these, respectively. BitBake checks pre-mirrors before looking upstream 594these, respectively. BitBake checks pre-mirrors before looking upstream
@@ -628,15 +607,15 @@ Package Feeds
628 607
629When the OpenEmbedded build system generates an image or an SDK, it gets 608When the OpenEmbedded build system generates an image or an SDK, it gets
630the packages from a package feed area located in the 609the packages from a package feed area located in the
631:term:`Build Directory`. The `general 610:term:`Build Directory`. The :ref:`general workflow figure
632workflow figure <#general-workflow-figure>`__ shows this package feeds 611<overview-manual/concepts:openembedded build system concepts>`
633area in the upper-right corner. 612shows this package feeds area in the upper-right corner.
634 613
635This section looks a little closer into the package feeds area used by 614This section looks a little closer into the package feeds area used by
636the build system. Here is a more detailed look at the area: 615the build system. Here is a more detailed look at the area:
637 616
638.. image:: figures/package-feeds.png 617.. image:: figures/package-feeds.png
639 :align: center 618 :width: 100%
640 619
641Package feeds are an intermediary step in the build process. The 620Package feeds are an intermediary step in the build process. The
642OpenEmbedded build system provides classes to generate different package 621OpenEmbedded build system provides classes to generate different package
@@ -644,31 +623,28 @@ types, and you specify which classes to enable through the
644:term:`PACKAGE_CLASSES` 623:term:`PACKAGE_CLASSES`
645variable. Before placing the packages into package feeds, the build 624variable. Before placing the packages into package feeds, the build
646process validates them with generated output quality assurance checks 625process validates them with generated output quality assurance checks
647through the :ref:`insane <ref-classes-insane>` 626through the :ref:`ref-classes-insane` class.
648class.
649 627
650The package feed area resides in the Build Directory. The directory the 628The package feed area resides in the :term:`Build Directory`. The directory the
651build system uses to temporarily store packages is determined by a 629build system uses to temporarily store packages is determined by a
652combination of variables and the particular package manager in use. See 630combination of variables and the particular package manager in use. See
653the "Package Feeds" box in the illustration and note the information to 631the "Package Feeds" box in the illustration and note the information to
654the right of that area. In particular, the following defines where 632the right of that area. In particular, the following defines where
655package files are kept: 633package files are kept:
656 634
657- :term:`DEPLOY_DIR`: Defined as 635- :term:`DEPLOY_DIR`: Defined as ``tmp/deploy`` in the :term:`Build Directory`.
658 ``tmp/deploy`` in the Build Directory.
659 636
660- ``DEPLOY_DIR_*``: Depending on the package manager used, the package 637- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
661 type sub-folder. Given RPM, IPK, or DEB packaging and tarball 638 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
662 creation, the 639 creation, the
663 :term:`DEPLOY_DIR_RPM`, 640 :term:`DEPLOY_DIR_RPM`,
664 :term:`DEPLOY_DIR_IPK`, 641 :term:`DEPLOY_DIR_IPK`, or
665 :term:`DEPLOY_DIR_DEB`, or 642 :term:`DEPLOY_DIR_DEB`
666 :term:`DEPLOY_DIR_TAR`,
667 variables are used, respectively. 643 variables are used, respectively.
668 644
669- :term:`PACKAGE_ARCH`: Defines 645- :term:`PACKAGE_ARCH`: Defines
670 architecture-specific sub-folders. For example, packages could exist 646 architecture-specific sub-folders. For example, packages could be
671 for the i586 or qemux86 architectures. 647 available for the i586 or qemux86 architectures.
672 648
673BitBake uses the 649BitBake uses the
674:ref:`do_package_write_* <ref-tasks-package_write_deb>` 650:ref:`do_package_write_* <ref-tasks-package_write_deb>`
@@ -676,13 +652,12 @@ tasks to generate packages and place them into the package holding area
676(e.g. ``do_package_write_ipk`` for IPK packages). See the 652(e.g. ``do_package_write_ipk`` for IPK packages). See the
677":ref:`ref-tasks-package_write_deb`", 653":ref:`ref-tasks-package_write_deb`",
678":ref:`ref-tasks-package_write_ipk`", 654":ref:`ref-tasks-package_write_ipk`",
679":ref:`ref-tasks-package_write_rpm`",
680and 655and
681":ref:`ref-tasks-package_write_tar`" 656":ref:`ref-tasks-package_write_rpm`"
682sections in the Yocto Project Reference Manual for additional 657sections in the Yocto Project Reference Manual for additional
683information. As an example, consider a scenario where an IPK packaging 658information. As an example, consider a scenario where an IPK packaging
684manager is being used and package architecture support for both i586 and 659manager is being used and there is package architecture support for both
685qemux86 exist. Packages for the i586 architecture are placed in 660i586 and qemux86. Packages for the i586 architecture are placed in
686``build/tmp/deploy/ipk/i586``, while packages for the qemux86 661``build/tmp/deploy/ipk/i586``, while packages for the qemux86
687architecture are placed in ``build/tmp/deploy/ipk/qemux86``. 662architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
688 663
@@ -691,15 +666,15 @@ BitBake Tool
691 666
692The OpenEmbedded build system uses 667The OpenEmbedded build system uses
693:term:`BitBake` to produce images and 668:term:`BitBake` to produce images and
694Software Development Kits (SDKs). You can see from the `general workflow 669Software Development Kits (SDKs). You can see from the :ref:`general workflow
695figure <#general-workflow-figure>`__, the BitBake area consists of 670figure <overview-manual/concepts:openembedded build system concepts>`,
696several functional areas. This section takes a closer look at each of 671the BitBake area consists of several functional areas. This section takes a
697those areas. 672closer look at each of those areas.
698 673
699.. note:: 674.. note::
700 675
701 Separate documentation exists for the BitBake tool. See the 676 Documentation for the BitBake tool is available separately. See the
702 BitBake User Manual 677 :doc:`BitBake User Manual <bitbake:index>`
703 for reference material on BitBake. 678 for reference material on BitBake.
704 679
705Source Fetching 680Source Fetching
@@ -709,37 +684,30 @@ The first stages of building a recipe are to fetch and unpack the source
709code: 684code:
710 685
711.. image:: figures/source-fetching.png 686.. image:: figures/source-fetching.png
712 :align: center 687 :width: 100%
713 688
714The :ref:`ref-tasks-fetch` and 689The :ref:`ref-tasks-fetch` and :ref:`ref-tasks-unpack` tasks fetch
715:ref:`ref-tasks-unpack` tasks fetch 690the source files and unpack them into the :term:`Build Directory`.
716the source files and unpack them into the
717:term:`Build Directory`.
718 691
719.. note:: 692.. note::
720 693
721 For every local file (e.g. 694 For every local file (e.g. ``file://``) that is part of a recipe's
722 file:// 695 :term:`SRC_URI` statement, the OpenEmbedded build system takes a
723 ) that is part of a recipe's 696 checksum of the file for the recipe and inserts the checksum into
724 SRC_URI 697 the signature for the :ref:`ref-tasks-fetch` task. If any local
725 statement, the OpenEmbedded build system takes a checksum of the file 698 file has been modified, the :ref:`ref-tasks-fetch` task and all
726 for the recipe and inserts the checksum into the signature for the 699 tasks that depend on it are re-executed.
727 do_fetch 700
728 task. If any local file has been modified, the 701By default, everything is accomplished in the :term:`Build Directory`, which has
729 do_fetch 702a defined structure. For additional general information on the
730 task and all tasks that depend on it are re-executed. 703:term:`Build Directory`, see the ":ref:`structure-core-build`" section in
731
732By default, everything is accomplished in the Build Directory, which has
733a defined structure. For additional general information on the Build
734Directory, see the ":ref:`structure-core-build`" section in
735the Yocto Project Reference Manual. 704the Yocto Project Reference Manual.
736 705
737Each recipe has an area in the Build Directory where the unpacked source 706Each recipe has an area in the :term:`Build Directory` where the unpacked
738code resides. The :term:`S` variable points 707source code resides. The :term:`S` variable points to this area for a recipe's
739to this area for a recipe's unpacked source code. The name of that 708unpacked source code. The name of that directory for any given recipe is
740directory for any given recipe is defined from several different 709defined from several different variables. The preceding figure and the
741variables. The preceding figure and the following list describe the 710following list describe the :term:`Build Directory`'s hierarchy:
742Build Directory's hierarchy:
743 711
744- :term:`TMPDIR`: The base directory 712- :term:`TMPDIR`: The base directory
745 where the OpenEmbedded build system performs all its work during the 713 where the OpenEmbedded build system performs all its work during the
@@ -749,7 +717,7 @@ Build Directory's hierarchy:
749 architecture of the built package or packages. Depending on the 717 architecture of the built package or packages. Depending on the
750 eventual destination of the package or packages (i.e. machine 718 eventual destination of the package or packages (i.e. machine
751 architecture, :term:`Build Host`, SDK, or 719 architecture, :term:`Build Host`, SDK, or
752 specific machine), ``PACKAGE_ARCH`` varies. See the variable's 720 specific machine), :term:`PACKAGE_ARCH` varies. See the variable's
753 description for details. 721 description for details.
754 722
755- :term:`TARGET_OS`: The operating 723- :term:`TARGET_OS`: The operating
@@ -758,7 +726,7 @@ Build Directory's hierarchy:
758 726
759- :term:`PN`: The name of the recipe used 727- :term:`PN`: The name of the recipe used
760 to build the package. This variable can have multiple meanings. 728 to build the package. This variable can have multiple meanings.
761 However, when used in the context of input files, ``PN`` represents 729 However, when used in the context of input files, :term:`PN` represents
762 the name of the recipe. 730 the name of the recipe.
763 731
764- :term:`WORKDIR`: The location 732- :term:`WORKDIR`: The location
@@ -775,20 +743,18 @@ Build Directory's hierarchy:
775 files for a given recipe. 743 files for a given recipe.
776 744
777 - :term:`BPN`: The name of the recipe 745 - :term:`BPN`: The name of the recipe
778 used to build the package. The ``BPN`` variable is a version of 746 used to build the package. The :term:`BPN` variable is a version of
779 the ``PN`` variable but with common prefixes and suffixes removed. 747 the :term:`PN` variable but with common prefixes and suffixes removed.
780 748
781 - :term:`PV`: The version of the 749 - :term:`PV`: The version of the
782 recipe used to build the package. 750 recipe used to build the package.
783 751
784.. note:: 752.. note::
785 753
786 In the previous figure, notice that two sample hierarchies exist: one 754 In the previous figure, notice that there are two sample hierarchies:
787 based on package architecture (i.e. 755 one based on package architecture (i.e. :term:`PACKAGE_ARCH`)
788 PACKAGE_ARCH 756 and one based on a machine (i.e. :term:`MACHINE`).
789 ) and one based on a machine (i.e. 757 The underlying structures are identical. The differentiator being
790 MACHINE
791 ). The underlying structures are identical. The differentiator being
792 what the OpenEmbedded build system is using as a build target (e.g. 758 what the OpenEmbedded build system is using as a build target (e.g.
793 general architecture, a build host, an SDK, or a specific machine). 759 general architecture, a build host, an SDK, or a specific machine).
794 760
@@ -799,7 +765,7 @@ Once source code is fetched and unpacked, BitBake locates patch files
799and applies them to the source files: 765and applies them to the source files:
800 766
801.. image:: figures/patching.png 767.. image:: figures/patching.png
802 :align: center 768 :width: 100%
803 769
804The :ref:`ref-tasks-patch` task uses a 770The :ref:`ref-tasks-patch` task uses a
805recipe's :term:`SRC_URI` statements 771recipe's :term:`SRC_URI` statements
@@ -807,23 +773,23 @@ and the :term:`FILESPATH` variable
807to locate applicable patch files. 773to locate applicable patch files.
808 774
809Default processing for patch files assumes the files have either 775Default processing for patch files assumes the files have either
810``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters 776``*.patch`` or ``*.diff`` file types. You can use :term:`SRC_URI` parameters
811to change the way the build system recognizes patch files. See the 777to change the way the build system recognizes patch files. See the
812:ref:`ref-tasks-patch` task for more 778:ref:`ref-tasks-patch` task for more
813information. 779information.
814 780
815BitBake finds and applies multiple patches for a single recipe in the 781BitBake finds and applies multiple patches for a single recipe in the
816order in which it locates the patches. The ``FILESPATH`` variable 782order in which it locates the patches. The :term:`FILESPATH` variable
817defines the default set of directories that the build system uses to 783defines the default set of directories that the build system uses to
818search for patch files. Once found, patches are applied to the recipe's 784search for patch files. Once found, patches are applied to the recipe's
819source files, which are located in the 785source files, which are located in the
820:term:`S` directory. 786:term:`S` directory.
821 787
822For more information on how the source directories are created, see the 788For more information on how the source directories are created, see the
823"`Source Fetching <#source-fetching-dev-environment>`__" section. For 789":ref:`overview-manual/concepts:source fetching`" section. For
824more information on how to create patches and how the build system 790more information on how to create patches and how the build system
825processes patches, see the 791processes patches, see the
826":ref:`dev-manual/common-tasks:patching code`" 792":ref:`dev-manual/new-recipe:patching code`"
827section in the 793section in the
828Yocto Project Development Tasks Manual. You can also see the 794Yocto Project Development Tasks Manual. You can also see the
829":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" 795":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
@@ -840,7 +806,7 @@ compile the source code. Once compilation occurs, the files are copied
840to a holding area (staged) in preparation for packaging: 806to a holding area (staged) in preparation for packaging:
841 807
842.. image:: figures/configuration-compile-autoreconf.png 808.. image:: figures/configuration-compile-autoreconf.png
843 :align: center 809 :width: 100%
844 810
845This step in the build process consists of the following tasks: 811This step in the build process consists of the following tasks:
846 812
@@ -866,27 +832,25 @@ This step in the build process consists of the following tasks:
866 are specific to configurations for the source code being built by the 832 are specific to configurations for the source code being built by the
867 recipe. 833 recipe.
868 834
869 If you are using the 835 If you are using the :ref:`ref-classes-autotools` class,
870 :ref:`autotools <ref-classes-autotools>` class,
871 you can add additional configuration options by using the 836 you can add additional configuration options by using the
872 :term:`EXTRA_OECONF` or 837 :term:`EXTRA_OECONF` or
873 :term:`PACKAGECONFIG_CONFARGS` 838 :term:`PACKAGECONFIG_CONFARGS`
874 variables. For information on how this variable works within that 839 variables. For information on how this variable works within that
875 class, see the 840 class, see the :ref:`ref-classes-autotools` class
876 :ref:`autotools <ref-classes-autotools>` class 841 :yocto_git:`here </poky/tree/meta/classes-recipe/autotools.bbclass>`.
877 :yocto_git:`here </poky/tree/meta/classes/autotools.bbclass>`.
878 842
879- *do_compile*: Once a configuration task has been satisfied, 843- *do_compile*: Once a configuration task has been satisfied,
880 BitBake compiles the source using the 844 BitBake compiles the source using the
881 :ref:`ref-tasks-compile` task. 845 :ref:`ref-tasks-compile` task.
882 Compilation occurs in the directory pointed to by the 846 Compilation occurs in the directory pointed to by the
883 :term:`B` variable. Realize that the 847 :term:`B` variable. Realize that the
884 ``B`` directory is, by default, the same as the 848 :term:`B` directory is, by default, the same as the
885 :term:`S` directory. 849 :term:`S` directory.
886 850
887- *do_install*: After compilation completes, BitBake executes the 851- *do_install*: After compilation completes, BitBake executes the
888 :ref:`ref-tasks-install` task. 852 :ref:`ref-tasks-install` task.
889 This task copies files from the ``B`` directory and places them in a 853 This task copies files from the :term:`B` directory and places them in a
890 holding area pointed to by the :term:`D` 854 holding area pointed to by the :term:`D`
891 variable. Packaging occurs later using files from this holding 855 variable. Packaging occurs later using files from this holding
892 directory. 856 directory.
@@ -898,7 +862,7 @@ After source code is configured, compiled, and staged, the build system
898analyzes the results and splits the output into packages: 862analyzes the results and splits the output into packages:
899 863
900.. image:: figures/analysis-for-package-splitting.png 864.. image:: figures/analysis-for-package-splitting.png
901 :align: center 865 :width: 100%
902 866
903The :ref:`ref-tasks-package` and 867The :ref:`ref-tasks-package` and
904:ref:`ref-tasks-packagedata` 868:ref:`ref-tasks-packagedata`
@@ -909,7 +873,7 @@ following as well as other items: splitting out debugging symbols,
909looking at shared library dependencies between packages, and looking at 873looking at shared library dependencies between packages, and looking at
910package relationships. 874package relationships.
911 875
912The ``do_packagedata`` task creates package metadata based on the 876The :ref:`ref-tasks-packagedata` task creates package metadata based on the
913analysis such that the build system can generate the final packages. The 877analysis such that the build system can generate the final packages. The
914:ref:`ref-tasks-populate_sysroot` 878:ref:`ref-tasks-populate_sysroot`
915task stages (copies) a subset of the files installed by the 879task stages (copies) a subset of the files installed by the
@@ -922,7 +886,7 @@ the analysis and package splitting process use several areas:
922 individual packages. 886 individual packages.
923 887
924- :term:`PKGDESTWORK`: A 888- :term:`PKGDESTWORK`: A
925 temporary work area (i.e. ``pkgdata``) used by the ``do_package`` 889 temporary work area (i.e. ``pkgdata``) used by the :ref:`ref-tasks-package`
926 task to save package metadata. 890 task to save package metadata.
927 891
928- :term:`PKGDEST`: The parent 892- :term:`PKGDEST`: The parent
@@ -932,7 +896,7 @@ the analysis and package splitting process use several areas:
932- :term:`PKGDATA_DIR`: A shared, 896- :term:`PKGDATA_DIR`: A shared,
933 global-state directory that holds packaging metadata generated during 897 global-state directory that holds packaging metadata generated during
934 the packaging process. The packaging process copies metadata from 898 the packaging process. The packaging process copies metadata from
935 ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally 899 :term:`PKGDESTWORK` to the :term:`PKGDATA_DIR` area where it becomes globally
936 available. 900 available.
937 901
938- :term:`STAGING_DIR_HOST`: 902- :term:`STAGING_DIR_HOST`:
@@ -946,25 +910,24 @@ the analysis and package splitting process use several areas:
946- :term:`STAGING_DIR_TARGET`: 910- :term:`STAGING_DIR_TARGET`:
947 The path for the sysroot used when a component that is built to 911 The path for the sysroot used when a component that is built to
948 execute on a system and it generates code for yet another machine 912 execute on a system and it generates code for yet another machine
949 (e.g. cross-canadian recipes). 913 (e.g. :ref:`ref-classes-cross-canadian` recipes).
950 914
951The :term:`FILES` variable defines the 915The :term:`FILES` variable defines the
952files that go into each package in 916files that go into each package in
953:term:`PACKAGES`. If you want 917:term:`PACKAGES`. If you want
954details on how this is accomplished, you can look at 918details on how this is accomplished, you can look at
955:yocto_git:`package.bbclass </poky/tree/meta/classes/package.bbclass>`. 919:yocto_git:`package.bbclass </poky/tree/meta/classes-global/package.bbclass>`.
956 920
957Depending on the type of packages being created (RPM, DEB, or IPK), the 921Depending on the type of packages being created (RPM, DEB, or IPK), the
958:ref:`do_package_write_* <ref-tasks-package_write_deb>` 922:ref:`do_package_write_* <ref-tasks-package_write_deb>`
959task creates the actual packages and places them in the Package Feed 923task creates the actual packages and places them in the Package Feed
960area, which is ``${TMPDIR}/deploy``. You can see the "`Package 924area, which is ``${TMPDIR}/deploy``. You can see the
961Feeds <#package-feeds-dev-environment>`__" section for more detail on 925":ref:`overview-manual/concepts:package feeds`" section for more detail on
962that part of the build process. 926that part of the build process.
963 927
964.. note:: 928.. note::
965 929
966 Support for creating feeds directly from the 930 Support for creating feeds directly from the ``deploy/*``
967 deploy/\*
968 directories does not exist. Creating such feeds usually requires some 931 directories does not exist. Creating such feeds usually requires some
969 kind of feed maintenance mechanism that would upload the new packages 932 kind of feed maintenance mechanism that would upload the new packages
970 into an official package feed (e.g. the Ångström distribution). This 933 into an official package feed (e.g. the Ångström distribution). This
@@ -978,7 +941,7 @@ Once packages are split and stored in the Package Feeds area, the build
978system uses BitBake to generate the root filesystem image: 941system uses BitBake to generate the root filesystem image:
979 942
980.. image:: figures/image-generation.png 943.. image:: figures/image-generation.png
981 :align: center 944 :width: 100%
982 945
983The image generation process consists of several stages and depends on 946The image generation process consists of several stages and depends on
984several tasks and variables. The 947several tasks and variables. The
@@ -1013,7 +976,7 @@ actually install:
1013 976
1014With :term:`IMAGE_ROOTFS` 977With :term:`IMAGE_ROOTFS`
1015pointing to the location of the filesystem under construction and the 978pointing to the location of the filesystem under construction and the
1016``PACKAGE_INSTALL`` variable providing the final list of packages to 979:term:`PACKAGE_INSTALL` variable providing the final list of packages to
1017install, the root file system is created. 980install, the root file system is created.
1018 981
1019Package installation is under control of the package manager (e.g. 982Package installation is under control of the package manager (e.g.
@@ -1024,29 +987,25 @@ data files are deleted from the root filesystem. As part of the final
1024stage of package installation, post installation scripts that are part 987stage of package installation, post installation scripts that are part
1025of the packages are run. Any scripts that fail to run on the build host 988of the packages are run. Any scripts that fail to run on the build host
1026are run on the target when the target system is first booted. If you are 989are run on the target when the target system is first booted. If you are
1027using a 990using a
1028:ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`, 991:ref:`read-only root filesystem <dev-manual/read-only-rootfs:creating a read-only root filesystem>`,
1029all the post installation scripts must succeed on the build host during 992all the post installation scripts must succeed on the build host during
1030the package installation phase since the root filesystem on the target 993the package installation phase since the root filesystem on the target
1031is read-only. 994is read-only.
1032 995
1033The final stages of the ``do_rootfs`` task handle post processing. Post 996The final stages of the :ref:`ref-tasks-rootfs` task handle post processing. Post
1034processing includes creation of a manifest file and optimizations. 997processing includes creation of a manifest file and optimizations.
1035 998
1036The manifest file (``.manifest``) resides in the same directory as the 999The manifest file (``.manifest``) resides in the same directory as the root
1037root filesystem image. This file lists out, line-by-line, the installed 1000filesystem image. This file lists out, line-by-line, the installed packages.
1038packages. The manifest file is useful for the 1001The manifest file is useful for the :ref:`ref-classes-testimage` class,
1039:ref:`testimage <ref-classes-testimage*>` class,
1040for example, to determine whether or not to run specific tests. See the 1002for example, to determine whether or not to run specific tests. See the
1041:term:`IMAGE_MANIFEST` 1003:term:`IMAGE_MANIFEST` variable for additional information.
1042variable for additional information.
1043 1004
1044Optimizing processes that are run across the image include ``mklibs``, 1005Optimizing processes that are run across the image include ``mklibs``
1045``prelink``, and any other post-processing commands as defined by the 1006and any other post-processing commands as defined by the
1046:term:`ROOTFS_POSTPROCESS_COMMAND` 1007:term:`ROOTFS_POSTPROCESS_COMMAND`
1047variable. The ``mklibs`` process optimizes the size of the libraries, 1008variable. The ``mklibs`` process optimizes the size of the libraries.
1048while the ``prelink`` process optimizes the dynamic linking of shared
1049libraries to reduce start up time of executables.
1050 1009
1051After the root filesystem is built, processing begins on the image 1010After the root filesystem is built, processing begins on the image
1052through the :ref:`ref-tasks-image` 1011through the :ref:`ref-tasks-image`
@@ -1056,25 +1015,23 @@ the
1056variable. This variable specifies a list of functions to call before the 1015variable. This variable specifies a list of functions to call before the
1057build system creates the final image output files. 1016build system creates the final image output files.
1058 1017
1059The build system dynamically creates ``do_image_*`` tasks as needed, 1018The build system dynamically creates :ref:`do_image_* <ref-tasks-image>` tasks as needed,
1060based on the image types specified in the 1019based on the image types specified in the
1061:term:`IMAGE_FSTYPES` variable. 1020:term:`IMAGE_FSTYPES` variable.
1062The process turns everything into an image file or a set of image files 1021The process turns everything into an image file or a set of image files
1063and can compress the root filesystem image to reduce the overall size of 1022and can compress the root filesystem image to reduce the overall size of
1064the image. The formats used for the root filesystem depend on the 1023the image. The formats used for the root filesystem depend on the
1065``IMAGE_FSTYPES`` variable. Compression depends on whether the formats 1024:term:`IMAGE_FSTYPES` variable. Compression depends on whether the formats
1066support compression. 1025support compression.
1067 1026
1068As an example, a dynamically created task when creating a particular 1027As an example, a dynamically created task when creating a particular
1069image type would take the following form: 1028image type would take the following form::
1070::
1071 1029
1072 do_image_type 1030 do_image_type
1073 1031
1074So, if the type 1032So, if the type
1075as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically 1033as specified by the :term:`IMAGE_FSTYPES` were ``ext4``, the dynamically
1076generated task would be as follows: 1034generated task would be as follows::
1077::
1078 1035
1079 do_image_ext4 1036 do_image_ext4
1080 1037
@@ -1100,14 +1057,14 @@ Development Kit (SDK) installer scripts for both the standard SDK and
1100the extensible SDK (eSDK): 1057the extensible SDK (eSDK):
1101 1058
1102.. image:: figures/sdk-generation.png 1059.. image:: figures/sdk-generation.png
1103 :align: center 1060 :width: 100%
1104 1061
1105.. note:: 1062.. note::
1106 1063
1107 For more information on the cross-development toolchain generation, 1064 For more information on the cross-development toolchain generation,
1108 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`" 1065 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`"
1109 section. For information on advantages gained when building a 1066 section. For information on advantages gained when building a
1110 cross-development toolchain using the do_populate_sdk task, see the 1067 cross-development toolchain using the :ref:`ref-tasks-populate_sdk` task, see the
1111 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in 1068 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in
1112 the Yocto Project Application Development and the Extensible Software 1069 the Yocto Project Application Development and the Extensible Software
1113 Development Kit (eSDK) manual. 1070 Development Kit (eSDK) manual.
@@ -1119,16 +1076,16 @@ and
1119:ref:`ref-tasks-populate_sdk_ext` 1076:ref:`ref-tasks-populate_sdk_ext`
1120tasks use these key variables to help create the list of packages to 1077tasks use these key variables to help create the list of packages to
1121actually install. For information on the variables listed in the figure, 1078actually install. For information on the variables listed in the figure,
1122see the "`Application Development SDK <#sdk-dev-environment>`__" 1079see the ":ref:`overview-manual/concepts:application development sdk`"
1123section. 1080section.
1124 1081
1125The ``do_populate_sdk`` task helps create the standard SDK and handles 1082The :ref:`ref-tasks-populate_sdk` task helps create the standard SDK and handles
1126two parts: a target part and a host part. The target part is the part 1083two parts: a target part and a host part. The target part is the part
1127built for the target hardware and includes libraries and headers. The 1084built for the target hardware and includes libraries and headers. The
1128host part is the part of the SDK that runs on the 1085host part is the part of the SDK that runs on the
1129:term:`SDKMACHINE`. 1086:term:`SDKMACHINE`.
1130 1087
1131The ``do_populate_sdk_ext`` task helps create the extensible SDK and 1088The :ref:`ref-tasks-populate_sdk_ext` task helps create the extensible SDK and
1132handles host and target parts differently than its counter part does for 1089handles host and target parts differently than its counter part does for
1133the standard SDK. For the extensible SDK, the task encapsulates the 1090the standard SDK. For the extensible SDK, the task encapsulates the
1134build system, which includes everything needed (host and target) for the 1091build system, which includes everything needed (host and target) for the
@@ -1147,28 +1104,27 @@ For each task that completes successfully, BitBake writes a stamp file
1147into the :term:`STAMPS_DIR` 1104into the :term:`STAMPS_DIR`
1148directory. The beginning of the stamp file's filename is determined by 1105directory. The beginning of the stamp file's filename is determined by
1149the :term:`STAMP` variable, and the end 1106the :term:`STAMP` variable, and the end
1150of the name consists of the task's name and current `input 1107of the name consists of the task's name and current :ref:`input
1151checksum <#overview-checksums>`__. 1108checksum <overview-manual/concepts:checksums (signatures)>`.
1152 1109
1153.. note:: 1110.. note::
1154 1111
1155 This naming scheme assumes that 1112 This naming scheme assumes that :term:`BB_SIGNATURE_HANDLER`
1156 BB_SIGNATURE_HANDLER
1157 is "OEBasicHash", which is almost always the case in current 1113 is "OEBasicHash", which is almost always the case in current
1158 OpenEmbedded. 1114 OpenEmbedded.
1159 1115
1160To determine if a task needs to be rerun, BitBake checks if a stamp file 1116To determine if a task needs to be rerun, BitBake checks if a stamp file
1161with a matching input checksum exists for the task. If such a stamp file 1117with a matching input checksum exists for the task. In this case,
1162exists, the task's output is assumed to exist and still be valid. If the 1118the task's output is assumed to exist and still be valid. Otherwise,
1163file does not exist, the task is rerun. 1119the task is rerun.
1164 1120
1165.. note:: 1121.. note::
1166 1122
1167 The stamp mechanism is more general than the shared state (sstate) 1123 The stamp mechanism is more general than the shared state (sstate)
1168 cache mechanism described in the "`Setscene Tasks and Shared 1124 cache mechanism described in the
1169 State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids 1125 ":ref:`overview-manual/concepts:setscene tasks and shared state`" section.
1170 rerunning any task that has a valid stamp file, not just tasks that 1126 BitBake avoids rerunning any task that has a valid stamp file, not just
1171 can be accelerated through the sstate cache. 1127 tasks that can be accelerated through the sstate cache.
1172 1128
1173 However, you should realize that stamp files only serve as a marker 1129 However, you should realize that stamp files only serve as a marker
1174 that some work has been done and that these files do not record task 1130 that some work has been done and that these files do not record task
@@ -1178,17 +1134,17 @@ file does not exist, the task is rerun.
1178 the sstate cache mechanism adds is a way to cache task output that 1134 the sstate cache mechanism adds is a way to cache task output that
1179 can then be shared between build machines. 1135 can then be shared between build machines.
1180 1136
1181Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing 1137Since :term:`STAMPS_DIR` is usually a subdirectory of :term:`TMPDIR`, removing
1182``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will 1138:term:`TMPDIR` will also remove :term:`STAMPS_DIR`, which means tasks will
1183properly be rerun to repopulate ``TMPDIR``. 1139properly be rerun to repopulate :term:`TMPDIR`.
1184 1140
1185If you want some task to always be considered "out of date", you can 1141If you want some task to always be considered "out of date", you can
1186mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>` 1142mark it with the :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`
1187varflag. If some other task depends on such a task, then that task will 1143varflag. If some other task depends on such a task, then that task will
1188also always be considered out of date, which might not be what you want. 1144also always be considered out of date, which might not be what you want.
1189 1145
1190For details on how to view information about a task's signature, see the 1146For details on how to view information about a task's signature, see the
1191":ref:`dev-manual/common-tasks:viewing task variable dependencies`" 1147":ref:`dev-manual/debugging:viewing task variable dependencies`"
1192section in the Yocto Project Development Tasks Manual. 1148section in the Yocto Project Development Tasks Manual.
1193 1149
1194Setscene Tasks and Shared State 1150Setscene Tasks and Shared State
@@ -1207,7 +1163,7 @@ usually made available in the form of a shared state (sstate) cache.
1207 :term:`SSTATE_MIRRORS` 1163 :term:`SSTATE_MIRRORS`
1208 variables. 1164 variables.
1209 1165
1210The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a 1166The idea of a setscene task (i.e ``do_taskname_setscene``) is a
1211version of the task where instead of building something, BitBake can 1167version of the task where instead of building something, BitBake can
1212skip to the end result and simply place a set of files into specific 1168skip to the end result and simply place a set of files into specific
1213locations as needed. In some cases, it makes sense to have a setscene 1169locations as needed. In some cases, it makes sense to have a setscene
@@ -1220,7 +1176,7 @@ the work involved would be equal to or greater than the underlying task.
1220 1176
1221In the build system, the common tasks that have setscene variants are 1177In the build system, the common tasks that have setscene variants are
1222:ref:`ref-tasks-package`, 1178:ref:`ref-tasks-package`,
1223``do_package_write_*``, 1179:ref:`do_package_write_* <ref-tasks-package_write_deb>`,
1224:ref:`ref-tasks-deploy`, 1180:ref:`ref-tasks-deploy`,
1225:ref:`ref-tasks-packagedata`, and 1181:ref:`ref-tasks-packagedata`, and
1226:ref:`ref-tasks-populate_sysroot`. 1182:ref:`ref-tasks-populate_sysroot`.
@@ -1230,20 +1186,20 @@ end result.
1230The build system has knowledge of the relationship between these tasks 1186The build system has knowledge of the relationship between these tasks
1231and other preceding tasks. For example, if BitBake runs 1187and other preceding tasks. For example, if BitBake runs
1232``do_populate_sysroot_setscene`` for something, it does not make sense 1188``do_populate_sysroot_setscene`` for something, it does not make sense
1233to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``, 1189to run any of the :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
1234``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if 1190:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, and :ref:`ref-tasks-install` tasks. However, if
1235``do_package`` needs to be run, BitBake needs to run those other tasks. 1191:ref:`ref-tasks-package` needs to be run, BitBake needs to run those other tasks.
1236 1192
1237It becomes more complicated if everything can come from an sstate cache 1193It becomes more complicated if everything can come from an sstate cache
1238because some objects are simply not required at all. For example, you do 1194because some objects are simply not required at all. For example, you do
1239not need a compiler or native tools, such as quilt, if nothing exists to 1195not need a compiler or native tools, such as quilt, if there isn't anything
1240compile or patch. If the ``do_package_write_*`` packages are available 1196to compile or patch. If the :ref:`do_package_write_* <ref-tasks-package_write_deb>` packages are available
1241from sstate, BitBake does not need the ``do_package`` task data. 1197from sstate, BitBake does not need the :ref:`ref-tasks-package` task data.
1242 1198
1243To handle all these complexities, BitBake runs in two phases. The first 1199To handle all these complexities, BitBake runs in two phases. The first
1244is the "setscene" stage. During this stage, BitBake first checks the 1200is the "setscene" stage. During this stage, BitBake first checks the
1245sstate cache for any targets it is planning to build. BitBake does a 1201sstate cache for any targets it is planning to build. BitBake does a
1246fast check to see if the object exists rather than a complete download. 1202fast check to see if the object exists rather than doing a complete download.
1247If nothing exists, the second phase, which is the setscene stage, 1203If nothing exists, the second phase, which is the setscene stage,
1248completes and the main build proceeds. 1204completes and the main build proceeds.
1249 1205
@@ -1257,11 +1213,9 @@ download or setscene task fails, the build system then tries to install
1257dependencies, such as the compiler, from the cache. 1213dependencies, such as the compiler, from the cache.
1258 1214
1259The availability of objects in the sstate cache is handled by the 1215The availability of objects in the sstate cache is handled by the
1260function specified by the 1216function specified by the :term:`BB_HASHCHECK_FUNCTION`
1261:term:`bitbake:BB_HASHCHECK_FUNCTION`
1262variable and returns a list of available objects. The function specified 1217variable and returns a list of available objects. The function specified
1263by the 1218by the :term:`BB_SETSCENE_DEPVALID`
1264:term:`bitbake:BB_SETSCENE_DEPVALID`
1265variable is the function that determines whether a given dependency 1219variable is the function that determines whether a given dependency
1266needs to be followed, and whether for any given relationship the 1220needs to be followed, and whether for any given relationship the
1267function needs to be passed. The function returns a True or False value. 1221function needs to be passed. The function returns a True or False value.
@@ -1271,12 +1225,14 @@ Images
1271 1225
1272The images produced by the build system are compressed forms of the root 1226The images produced by the build system are compressed forms of the root
1273filesystem and are ready to boot on a target device. You can see from 1227filesystem and are ready to boot on a target device. You can see from
1274the `general workflow figure <#general-workflow-figure>`__ that BitBake 1228the :ref:`general workflow figure
1229<overview-manual/concepts:openembedded build system concepts>` that BitBake
1275output, in part, consists of images. This section takes a closer look at 1230output, in part, consists of images. This section takes a closer look at
1276this output: 1231this output:
1277 1232
1278.. image:: figures/images.png 1233.. image:: figures/images.png
1279 :align: center 1234 :align: center
1235 :width: 75%
1280 1236
1281.. note:: 1237.. note::
1282 1238
@@ -1284,15 +1240,12 @@ this output:
1284 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference 1240 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference
1285 Manual. 1241 Manual.
1286 1242
1287The build process writes images out to the :term:`Build Directory` 1243The build process writes images out to the :term:`Build Directory` inside
1288inside the 1244the ``tmp/deploy/images/machine/`` folder as shown in the figure. This
1289``tmp/deploy/images/machine/`` folder as shown in the figure. This
1290folder contains any files expected to be loaded on the target device. 1245folder contains any files expected to be loaded on the target device.
1291The :term:`DEPLOY_DIR` variable 1246The :term:`DEPLOY_DIR` variable points to the ``deploy`` directory, while the
1292points to the ``deploy`` directory, while the 1247:term:`DEPLOY_DIR_IMAGE` variable points to the appropriate directory
1293:term:`DEPLOY_DIR_IMAGE` 1248containing images for the current configuration.
1294variable points to the appropriate directory containing images for the
1295current configuration.
1296 1249
1297- kernel-image: A kernel binary file. The 1250- kernel-image: A kernel binary file. The
1298 :term:`KERNEL_IMAGETYPE` 1251 :term:`KERNEL_IMAGETYPE`
@@ -1327,7 +1280,8 @@ current configuration.
1327Application Development SDK 1280Application Development SDK
1328--------------------------- 1281---------------------------
1329 1282
1330In the `general workflow figure <#general-workflow-figure>`__, the 1283In the :ref:`general workflow figure
1284<overview-manual/concepts:openembedded build system concepts>`, the
1331output labeled "Application Development SDK" represents an SDK. The SDK 1285output labeled "Application Development SDK" represents an SDK. The SDK
1332generation process differs depending on whether you build an extensible 1286generation process differs depending on whether you build an extensible
1333SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK 1287SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
@@ -1335,7 +1289,7 @@ SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1335closer look at this output: 1289closer look at this output:
1336 1290
1337.. image:: figures/sdk.png 1291.. image:: figures/sdk.png
1338 :align: center 1292 :width: 100%
1339 1293
1340The specific form of this output is a set of files that includes a 1294The specific form of this output is a set of files that includes a
1341self-extracting SDK installer (``*.sh``), host and target manifest 1295self-extracting SDK installer (``*.sh``), host and target manifest
@@ -1357,18 +1311,17 @@ can initialize the environment before using the tools.
1357 your own SDK installer. 1311 your own SDK installer.
1358 1312
1359 - For background information on cross-development toolchains in the 1313 - For background information on cross-development toolchains in the
1360 Yocto Project development environment, see the "`Cross-Development 1314 Yocto Project development environment, see the
1361 Toolchain Generation <#cross-development-toolchain-generation>`__" 1315 ":ref:`overview-manual/concepts:cross-development toolchain generation`"
1362 section. 1316 section.
1363 1317
1364 - For information on setting up a cross-development environment, see 1318 - For information on setting up a cross-development environment, see
1365 the :doc:`/sdk-manual/index` manual. 1319 the :doc:`/sdk-manual/index` manual.
1366 1320
1367All the output files for an SDK are written to the ``deploy/sdk`` folder 1321All the output files for an SDK are written to the ``deploy/sdk`` folder
1368inside the :term:`Build Directory` as 1322inside the :term:`Build Directory` as shown in the previous figure. Depending
1369shown in the previous figure. Depending on the type of SDK, several 1323on the type of SDK, there are several variables to configure these files.
1370variables exist that help configure these files. The following list 1324The variables associated with an extensible SDK are:
1371shows the variables associated with an extensible SDK:
1372 1325
1373- :term:`DEPLOY_DIR`: Points to 1326- :term:`DEPLOY_DIR`: Points to
1374 the ``deploy`` directory. 1327 the ``deploy`` directory.
@@ -1386,15 +1339,15 @@ shows the variables associated with an extensible SDK:
1386 Specifies whether or not the toolchain is included when building the 1339 Specifies whether or not the toolchain is included when building the
1387 extensible SDK. 1340 extensible SDK.
1388 1341
1389- :term:`SDK_LOCAL_CONF_WHITELIST`: 1342- :term:`ESDK_LOCALCONF_ALLOW`:
1390 A list of variables allowed through from the build system 1343 A list of variables allowed through from the build system
1391 configuration into the extensible SDK configuration. 1344 configuration into the extensible SDK configuration.
1392 1345
1393- :term:`SDK_LOCAL_CONF_BLACKLIST`: 1346- :term:`ESDK_LOCALCONF_REMOVE`:
1394 A list of variables not allowed through from the build system 1347 A list of variables not allowed through from the build system
1395 configuration into the extensible SDK configuration. 1348 configuration into the extensible SDK configuration.
1396 1349
1397- :term:`SDK_INHERIT_BLACKLIST`: 1350- :term:`ESDK_CLASS_INHERIT_DISABLE`:
1398 A list of classes to remove from the 1351 A list of classes to remove from the
1399 :term:`INHERIT` value globally 1352 :term:`INHERIT` value globally
1400 within the extensible SDK configuration. 1353 within the extensible SDK configuration.
@@ -1413,7 +1366,7 @@ This next list, shows the variables associated with a standard SDK:
1413 1366
1414- :term:`TOOLCHAIN_HOST_TASK`: 1367- :term:`TOOLCHAIN_HOST_TASK`:
1415 Lists packages that make up the host part of the SDK (i.e. the part 1368 Lists packages that make up the host part of the SDK (i.e. the part
1416 that runs on the ``SDKMACHINE``). When you use 1369 that runs on the :term:`SDKMACHINE`). When you use
1417 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of 1370 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1418 default packages apply. This variable allows you to add more 1371 default packages apply. This variable allows you to add more
1419 packages. 1372 packages.
@@ -1422,7 +1375,7 @@ This next list, shows the variables associated with a standard SDK:
1422 Lists packages that make up the target part of the SDK (i.e. the part 1375 Lists packages that make up the target part of the SDK (i.e. the part
1423 built for the target hardware). 1376 built for the target hardware).
1424 1377
1425- :term:`SDKPATH`: Defines the 1378- :term:`SDKPATHINSTALL`: Defines the
1426 default SDK installation path offered by the installation script. 1379 default SDK installation path offered by the installation script.
1427 1380
1428- :term:`SDK_HOST_MANIFEST`: 1381- :term:`SDK_HOST_MANIFEST`:
@@ -1453,10 +1406,10 @@ The following figure shows a high-level build environment regarding
1453toolchain construction and use. 1406toolchain construction and use.
1454 1407
1455.. image:: figures/cross-development-toolchains.png 1408.. image:: figures/cross-development-toolchains.png
1456 :align: center 1409 :width: 100%
1457 1410
1458Most of the work occurs on the Build Host. This is the machine used to 1411Most of the work occurs on the Build Host. This is the machine used to
1459build images and generally work within the the Yocto Project 1412build images and generally work within the Yocto Project
1460environment. When you run 1413environment. When you run
1461:term:`BitBake` to create an image, the 1414:term:`BitBake` to create an image, the
1462OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a 1415OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
@@ -1467,15 +1420,11 @@ cross-compiler that is used internally within BitBake only.
1467 1420
1468.. note:: 1421.. note::
1469 1422
1470 The extensible SDK does not use 1423 The extensible SDK does not use ``gcc-cross-canadian``
1471 gcc-cross-canadian
1472 since this SDK ships a copy of the OpenEmbedded build system and the 1424 since this SDK ships a copy of the OpenEmbedded build system and the
1473 sysroot within it contains 1425 sysroot within it contains ``gcc-cross``.
1474 gcc-cross
1475 .
1476 1426
1477The chain of events that occurs when the standard toolchain is bootstrapped: 1427The chain of events that occurs when the standard toolchain is bootstrapped::
1478::
1479 1428
1480 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime 1429 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime
1481 1430
@@ -1511,12 +1460,11 @@ relocatable SDK used to develop applications. When you run the
1511installer, it installs the toolchain, which contains the development 1460installer, it installs the toolchain, which contains the development
1512tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and 1461tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1513other ``nativesdk-*`` tools), which are tools native to the SDK (i.e. 1462other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1514native to :term:`SDK_ARCH`), you 1463native to :term:`SDK_ARCH`), you need to cross-compile and test your
1515need to cross-compile and test your software. The figure shows the 1464software. The figure shows the commands you use to easily build out
1516commands you use to easily build out this toolchain. This 1465this toolchain. This cross-development toolchain is built to execute on the
1517cross-development toolchain is built to execute on the 1466:term:`SDKMACHINE`, which might or might not be the same machine as
1518:term:`SDKMACHINE`, which might or 1467the Build Host.
1519might not be the same machine as the Build Host.
1520 1468
1521.. note:: 1469.. note::
1522 1470
@@ -1524,8 +1472,7 @@ might not be the same machine as the Build Host.
1524 can take advantage of pre-built images that ship with the Yocto 1472 can take advantage of pre-built images that ship with the Yocto
1525 Project and already contain cross-development toolchain installers. 1473 Project and already contain cross-development toolchain installers.
1526 1474
1527Here is the bootstrap process for the relocatable toolchain: 1475Here is the bootstrap process for the relocatable toolchain::
1528::
1529 1476
1530 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian 1477 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1531 1478
@@ -1579,10 +1526,10 @@ Shared State Cache
1579By design, the OpenEmbedded build system builds everything from scratch 1526By design, the OpenEmbedded build system builds everything from scratch
1580unless :term:`BitBake` can determine 1527unless :term:`BitBake` can determine
1581that parts do not need to be rebuilt. Fundamentally, building from 1528that parts do not need to be rebuilt. Fundamentally, building from
1582scratch is attractive as it means all parts are built fresh and no 1529scratch is attractive as it means all parts are built fresh and there is
1583possibility of stale data exists that can cause problems. When 1530no possibility of stale data that can cause problems. When
1584developers hit problems, they typically default back to building from 1531developers hit problems, they typically default back to building from
1585scratch so they have a know state from the start. 1532scratch so they have a known state from the start.
1586 1533
1587Building an image from scratch is both an advantage and a disadvantage 1534Building an image from scratch is both an advantage and a disadvantage
1588to the process. As mentioned in the previous paragraph, building from 1535to the process. As mentioned in the previous paragraph, building from
@@ -1619,18 +1566,18 @@ them if they are deemed to be valid.
1619 1566
1620 - The build system does not maintain 1567 - The build system does not maintain
1621 :term:`PR` information as part of 1568 :term:`PR` information as part of
1622 the shared state packages. Consequently, considerations exist that 1569 the shared state packages. Consequently, there are considerations that
1623 affect maintaining shared state feeds. For information on how the 1570 affect maintaining shared state feeds. For information on how the
1624 build system works with packages and can track incrementing ``PR`` 1571 build system works with packages and can track incrementing :term:`PR`
1625 information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`" 1572 information, see the ":ref:`dev-manual/packages:automatically incrementing a package version number`"
1626 section in the Yocto Project Development Tasks Manual. 1573 section in the Yocto Project Development Tasks Manual.
1627 1574
1628 - The code in the build system that supports incremental builds is 1575 - The code in the build system that supports incremental builds is
1629 not simple code. For techniques that help you work around issues 1576 complex. For techniques that help you work around issues
1630 related to shared state code, see the 1577 related to shared state code, see the
1631 ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`" 1578 ":ref:`dev-manual/debugging:viewing metadata used to create the input signature of a shared state task`"
1632 and 1579 and
1633 ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`" 1580 ":ref:`dev-manual/debugging:invalidating shared state to force a task to run`"
1634 sections both in the Yocto Project Development Tasks Manual. 1581 sections both in the Yocto Project Development Tasks Manual.
1635 1582
1636The rest of this section goes into detail about the overall incremental 1583The rest of this section goes into detail about the overall incremental
@@ -1667,7 +1614,7 @@ you a good idea of when the task's data changes.
1667 1614
1668To complicate the problem, there are things that should not be included 1615To complicate the problem, there are things that should not be included
1669in the checksum. First, there is the actual specific build path of a 1616in the checksum. First, there is the actual specific build path of a
1670given task - the :term:`WORKDIR`. It 1617given task --- the :term:`WORKDIR`. It
1671does not matter if the work directory changes because it should not 1618does not matter if the work directory changes because it should not
1672affect the output for target packages. Also, the build process has the 1619affect the output for target packages. Also, the build process has the
1673objective of making native or cross packages relocatable. 1620objective of making native or cross packages relocatable.
@@ -1678,8 +1625,8 @@ objective of making native or cross packages relocatable.
1678 build host. However, cross packages generate output for the target 1625 build host. However, cross packages generate output for the target
1679 architecture. 1626 architecture.
1680 1627
1681The checksum therefore needs to exclude ``WORKDIR``. The simplistic 1628The checksum therefore needs to exclude :term:`WORKDIR`. The simplistic
1682approach for excluding the work directory is to set ``WORKDIR`` to some 1629approach for excluding the work directory is to set :term:`WORKDIR` to some
1683fixed value and create the checksum for the "run" script. 1630fixed value and create the checksum for the "run" script.
1684 1631
1685Another problem results from the "run" scripts containing functions that 1632Another problem results from the "run" scripts containing functions that
@@ -1689,7 +1636,7 @@ used to prune the "run" scripts down to the minimum set, thereby
1689alleviating this problem and making the "run" scripts much more readable 1636alleviating this problem and making the "run" scripts much more readable
1690as a bonus. 1637as a bonus.
1691 1638
1692So far, solutions for shell scripts exist. What about Python tasks? The 1639So far, there are solutions for shell scripts. What about Python tasks? The
1693same approach applies even though these tasks are more difficult. The 1640same approach applies even though these tasks are more difficult. The
1694process needs to figure out what variables a Python function accesses 1641process needs to figure out what variables a Python function accesses
1695and what functions it calls. Again, the incremental build solution 1642and what functions it calls. Again, the incremental build solution
@@ -1697,10 +1644,9 @@ contains code that first figures out the variable and function
1697dependencies, and then creates a checksum for the data used as the input 1644dependencies, and then creates a checksum for the data used as the input
1698to the task. 1645to the task.
1699 1646
1700Like the ``WORKDIR`` case, situations exist where dependencies should be 1647Like the :term:`WORKDIR` case, there can be situations where dependencies should be
1701ignored. For these situations, you can instruct the build process to 1648ignored. For these situations, you can instruct the build process to
1702ignore a dependency by using a line like the following: 1649ignore a dependency by using a line like the following::
1703::
1704 1650
1705 PACKAGE_ARCHS[vardepsexclude] = "MACHINE" 1651 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1706 1652
@@ -1710,13 +1656,12 @@ reference it.
1710 1656
1711Equally, there are cases where you need to add dependencies BitBake is 1657Equally, there are cases where you need to add dependencies BitBake is
1712not able to find. You can accomplish this by using a line like the 1658not able to find. You can accomplish this by using a line like the
1713following: 1659following::
1714::
1715 1660
1716 PACKAGE_ARCHS[vardeps] = "MACHINE" 1661 PACKAGE_ARCHS[vardeps] = "MACHINE"
1717 1662
1718This example explicitly 1663This example explicitly
1719adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. 1664adds the :term:`MACHINE` variable as a dependency for :term:`PACKAGE_ARCHS`.
1720 1665
1721As an example, consider a case with in-line Python where BitBake is not 1666As an example, consider a case with in-line Python where BitBake is not
1722able to figure out dependencies. When running in debug mode (i.e. using 1667able to figure out dependencies. When running in debug mode (i.e. using
@@ -1728,34 +1673,29 @@ need to fix this situation.
1728Thus far, this section has limited discussion to the direct inputs into 1673Thus far, this section has limited discussion to the direct inputs into
1729a task. Information based on direct inputs is referred to as the 1674a task. Information based on direct inputs is referred to as the
1730"basehash" in the code. However, the question of a task's indirect 1675"basehash" in the code. However, the question of a task's indirect
1731inputs still exits - items already built and present in the 1676inputs still exits --- items already built and present in the
1732:term:`Build Directory`. The checksum (or 1677:term:`Build Directory`. The checksum (or
1733signature) for a particular task needs to add the hashes of all the 1678signature) for a particular task needs to add the hashes of all the
1734tasks on which the particular task depends. Choosing which dependencies 1679tasks on which the particular task depends. Choosing which dependencies
1735to add is a policy decision. However, the effect is to generate a master 1680to add is a policy decision. However, the effect is to generate a
1736checksum that combines the basehash and the hashes of the task's 1681checksum that combines the basehash and the hashes of the task's
1737dependencies. 1682dependencies.
1738 1683
1739At the code level, a variety of ways exist by which both the basehash 1684At the code level, there are multiple ways by which both the basehash
1740and the dependent task hashes can be influenced. Within the BitBake 1685and the dependent task hashes can be influenced. Within the BitBake
1741configuration file, you can give BitBake some extra information to help 1686configuration file, you can give BitBake some extra information to help
1742it construct the basehash. The following statement effectively results 1687it construct the basehash. The following statement effectively results
1743in a list of global variable dependency excludes (i.e. variables never 1688in a list of global variable dependency excludes (i.e. variables never
1744included in any checksum): 1689included in any checksum)::
1745::
1746 1690
1747 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\ 1691 BB_BASEHASH_IGNORE_VARS ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\
1748 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\ 1692 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\
1749 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ 1693 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\
1750 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ 1694 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\
1751 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" 1695 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1752 1696
1753The 1697The previous example does not include :term:`WORKDIR` since that variable is
1754previous example excludes 1698actually constructed as a path within :term:`TMPDIR`, which is included above.
1755:term:`WORKDIR` since that variable
1756is actually constructed as a path within
1757:term:`TMPDIR`, which is on the
1758whitelist.
1759 1699
1760The rules for deciding which hashes of dependent tasks to include 1700The rules for deciding which hashes of dependent tasks to include
1761through dependency chains are more complex and are generally 1701through dependency chains are more complex and are generally
@@ -1767,16 +1707,15 @@ desired. This file defines the two basic signature generators
1767"OEBasicHash". By default, a dummy "noop" signature handler is enabled 1707"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1768in BitBake. This means that behavior is unchanged from previous 1708in BitBake. This means that behavior is unchanged from previous
1769versions. OE-Core uses the "OEBasicHash" signature handler by default 1709versions. OE-Core uses the "OEBasicHash" signature handler by default
1770through this setting in the ``bitbake.conf`` file: 1710through this setting in the ``bitbake.conf`` file::
1771::
1772 1711
1773 BB_SIGNATURE_HANDLER ?= "OEBasicHash" 1712 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1774 1713
1775The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same 1714The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same
1776as the "OEBasic" version but adds the task hash to the `stamp 1715as the "OEBasic" version but adds the task hash to the :ref:`stamp
1777files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any 1716files <overview-manual/concepts:stamp files and the rerunning of tasks>`. This
1778metadata change that changes the task hash, automatically causing the 1717results in any metadata change that changes the task hash, automatically causing
1779task to be run again. This removes the need to bump 1718the task to be run again. This removes the need to bump
1780:term:`PR` values, and changes to metadata 1719:term:`PR` values, and changes to metadata
1781automatically ripple across the build. 1720automatically ripple across the build.
1782 1721
@@ -1784,16 +1723,13 @@ It is also worth noting that the end result of these signature
1784generators is to make some dependency and hash information available to 1723generators is to make some dependency and hash information available to
1785the build. This information includes: 1724the build. This information includes:
1786 1725
1787- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the 1726- ``BB_BASEHASH:task-``\ taskname: The base hashes for each task in the
1788 recipe. 1727 recipe.
1789 1728
1790- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each 1729- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1791 dependent task. 1730 dependent task.
1792 1731
1793- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for 1732- :term:`BB_TASKHASH`: The hash of the currently running task.
1794 each task.
1795
1796- ``BB_TASKHASH``: The hash of the currently running task.
1797 1733
1798Shared State 1734Shared State
1799------------ 1735------------
@@ -1803,12 +1739,11 @@ half the problem of supporting a shared state. The other half of the
1803problem is being able to use checksum information during the build and 1739problem is being able to use checksum information during the build and
1804being able to reuse or rebuild specific components. 1740being able to reuse or rebuild specific components.
1805 1741
1806The :ref:`sstate <ref-classes-sstate>` class is a 1742The :ref:`ref-classes-sstate` class is a relatively generic implementation of
1807relatively generic implementation of how to "capture" a snapshot of a 1743how to "capture" a snapshot of a given task. The idea is that the build process
1808given task. The idea is that the build process does not care about the 1744does not care about the source of a task's output. Output could be freshly
1809source of a task's output. Output could be freshly built or it could be 1745built or it could be downloaded and unpacked from somewhere. In other words,
1810downloaded and unpacked from somewhere. In other words, the build 1746the build process does not need to worry about its origin.
1811process does not need to worry about its origin.
1812 1747
1813Two types of output exist. One type is just about creating a directory 1748Two types of output exist. One type is just about creating a directory
1814in :term:`WORKDIR`. A good example is 1749in :term:`WORKDIR`. A good example is
@@ -1819,11 +1754,9 @@ type of output occurs when a set of data is merged into a shared
1819directory tree such as the sysroot. 1754directory tree such as the sysroot.
1820 1755
1821The Yocto Project team has tried to keep the details of the 1756The Yocto Project team has tried to keep the details of the
1822implementation hidden in ``sstate`` class. From a user's perspective, 1757implementation hidden in the :ref:`ref-classes-sstate` class. From a user's perspective,
1823adding shared state wrapping to a task is as simple as this 1758adding shared state wrapping to a task is as simple as this
1824:ref:`ref-tasks-deploy` example taken 1759:ref:`ref-tasks-deploy` example taken from the :ref:`ref-classes-deploy` class::
1825from the :ref:`deploy <ref-classes-deploy>` class:
1826::
1827 1760
1828 DEPLOYDIR = "${WORKDIR}/deploy-${PN}" 1761 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
1829 SSTATETASKS += "do_deploy" 1762 SSTATETASKS += "do_deploy"
@@ -1839,14 +1772,12 @@ from the :ref:`deploy <ref-classes-deploy>` class:
1839 1772
1840The following list explains the previous example: 1773The following list explains the previous example:
1841 1774
1842- Adding "do_deploy" to ``SSTATETASKS`` adds some required 1775- Adding ``do_deploy`` to ``SSTATETASKS`` adds some required sstate-related
1843 sstate-related processing, which is implemented in the 1776 processing, which is implemented in the :ref:`ref-classes-sstate` class, to
1844 :ref:`sstate <ref-classes-sstate>` class, to 1777 before and after the :ref:`ref-tasks-deploy` task.
1845 before and after the
1846 :ref:`ref-tasks-deploy` task.
1847 1778
1848- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that 1779- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1849 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally 1780 :ref:`ref-tasks-deploy` places its output in ``${DEPLOYDIR}`` when run normally
1850 (i.e. when not using the sstate cache). This output becomes the input 1781 (i.e. when not using the sstate cache). This output becomes the input
1851 to the shared state cache. 1782 to the shared state cache.
1852 1783
@@ -1856,92 +1787,85 @@ The following list explains the previous example:
1856 1787
1857 .. note:: 1788 .. note::
1858 1789
1859 If ``do_deploy`` is not already in the shared state cache or if its input 1790 If :ref:`ref-tasks-deploy` is not already in the shared state cache or if its input
1860 checksum (signature) has changed from when the output was cached, the task 1791 checksum (signature) has changed from when the output was cached, the task
1861 runs to populate the shared state cache, after which the contents of the 1792 runs to populate the shared state cache, after which the contents of the
1862 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If 1793 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If
1863 ``do_deploy`` is in the shared state cache and its signature indicates 1794 :ref:`ref-tasks-deploy` is in the shared state cache and its signature indicates
1864 that the cached output is still valid (i.e. if no relevant task inputs 1795 that the cached output is still valid (i.e. if no relevant task inputs
1865 have changed), then the contents of the shared state cache copies 1796 have changed), then the contents of the shared state cache copies
1866 directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task 1797 directly to ${:term:`DEPLOY_DIR_IMAGE`} by the ``do_deploy_setscene`` task
1867 instead, skipping the ``do_deploy`` task. 1798 instead, skipping the :ref:`ref-tasks-deploy` task.
1868 1799
1869- The following task definition is glue logic needed to make the 1800- The following task definition is glue logic needed to make the
1870 previous settings effective: 1801 previous settings effective::
1871 ::
1872 1802
1873 python do_deploy_setscene () { 1803 python do_deploy_setscene () {
1874 sstate_setscene(d) 1804 sstate_setscene(d)
1875 } 1805 }
1876 addtask do_deploy_setscene 1806 addtask do_deploy_setscene
1877 1807
1878 ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task 1808 ``sstate_setscene()`` takes the flags above as input and accelerates the
1879 through the shared state cache if possible. If the task was 1809 :ref:`ref-tasks-deploy` task through the shared state cache if possible. If
1880 accelerated, ``sstate_setscene()`` returns True. Otherwise, it 1810 the task was accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1881 returns False, and the normal ``do_deploy`` task runs. For more 1811 returns False, and the normal :ref:`ref-tasks-deploy` task runs. For more
1882 information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`" 1812 information, see the ":ref:`bitbake-user-manual/bitbake-user-manual-execution:setscene`"
1883 section in the BitBake User Manual. 1813 section in the BitBake User Manual.
1884 1814
1885- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates 1815- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates ``${DEPLOYDIR}``
1886 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and 1816 and ``${B}`` before the :ref:`ref-tasks-deploy` task runs, and also sets the
1887 also sets the current working directory of ``do_deploy`` to ``${B}``. 1817 current working directory of :ref:`ref-tasks-deploy` to ``${B}``. For more
1888 For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" 1818 information, see the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
1889 section in the BitBake 1819 section in the BitBake User Manual.
1890 User Manual.
1891 1820
1892 .. note:: 1821 .. note::
1893 1822
1894 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be 1823 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be
1895 the same, you can use ``sstate-plaindirs``. For example, to preserve the 1824 the same, you can use ``sstate-plaindirs``. For example, to preserve the
1896 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package`` 1825 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the :ref:`ref-tasks-package`
1897 task, use the following: 1826 task, use the following::
1898 ::
1899 1827
1900 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" 1828 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1901 1829
1902 1830
1903- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends 1831- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends extra
1904 extra metadata to the `stamp 1832 metadata to the :ref:`stamp file <overview-manual/concepts:stamp files and the rerunning of tasks>`.
1905 file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the 1833 In this case, the metadata makes the task specific to a machine's architecture.
1906 metadata makes the task specific to a machine's architecture. See 1834 See the ":ref:`bitbake-user-manual/bitbake-user-manual-execution:the task list`"
1907 ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:the task list`"
1908 section in the BitBake User Manual for more information on the 1835 section in the BitBake User Manual for more information on the
1909 ``stamp-extra-info`` flag. 1836 ``stamp-extra-info`` flag.
1910 1837
1911- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with 1838- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1912 multiple directories. For example, the following declares 1839 multiple directories. For example, the following declares
1913 ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories, 1840 :term:`PKGDESTWORK` and ``SHLIBWORK`` as shared state input directories,
1914 which populates the shared state cache, and ``PKGDATA_DIR`` and 1841 which populates the shared state cache, and :term:`PKGDATA_DIR` and
1915 ``SHLIBSDIR`` as the corresponding shared state output directories: 1842 ``SHLIBSDIR`` as the corresponding shared state output directories::
1916 ::
1917 1843
1918 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" 1844 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1919 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" 1845 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1920 1846
1921- These methods also include the ability to take a lockfile when 1847- These methods also include the ability to take a lockfile when
1922 manipulating shared state directory structures, for cases where file 1848 manipulating shared state directory structures, for cases where file
1923 additions or removals are sensitive: 1849 additions or removals are sensitive::
1924 ::
1925 1850
1926 do_package[sstate-lockfile] = "${PACKAGELOCK}" 1851 do_package[sstate-lockfile] = "${PACKAGELOCK}"
1927 1852
1928Behind the scenes, the shared state code works by looking in 1853Behind the scenes, the shared state code works by looking in
1929:term:`SSTATE_DIR` and 1854:term:`SSTATE_DIR` and
1930:term:`SSTATE_MIRRORS` for 1855:term:`SSTATE_MIRRORS` for
1931shared state files. Here is an example: 1856shared state files. Here is an example::
1932::
1933 1857
1934 SSTATE_MIRRORS ?= "\ 1858 SSTATE_MIRRORS ?= "\
1935 file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ 1859 file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \
1936 file://.\* file:///some/local/dir/sstate/PATH" 1860 file://.* file:///some/local/dir/sstate/PATH"
1937 1861
1938.. note:: 1862.. note::
1939 1863
1940 The shared state directory (``SSTATE_DIR``) is organized into two-character 1864 The shared state directory (:term:`SSTATE_DIR`) is organized into two-character
1941 subdirectories, where the subdirectory names are based on the first two 1865 subdirectories, where the subdirectory names are based on the first two
1942 characters of the hash. 1866 characters of the hash.
1943 If the shared state directory structure for a mirror has the same structure 1867 If the shared state directory structure for a mirror has the same structure
1944 as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build 1868 as :term:`SSTATE_DIR`, you must specify "PATH" as part of the URI to enable the build
1945 system to map to the appropriate subdirectory. 1869 system to map to the appropriate subdirectory.
1946 1870
1947The shared state package validity can be detected just by looking at the 1871The shared state package validity can be detected just by looking at the
@@ -1966,14 +1890,155 @@ another reason why a task-based approach is preferred over a
1966recipe-based approach, which would have to install the output from every 1890recipe-based approach, which would have to install the output from every
1967task. 1891task.
1968 1892
1893Hash Equivalence
1894----------------
1895
1896The above section explained how BitBake skips the execution of tasks
1897whose output can already be found in the Shared State cache.
1898
1899During a build, it may often be the case that the output / result of a task might
1900be unchanged despite changes in the task's input values. An example might be
1901whitespace changes in some input C code. In project terms, this is what we define
1902as "equivalence".
1903
1904To keep track of such equivalence, BitBake has to manage three hashes
1905for each task:
1906
1907- The *task hash* explained earlier: computed from the recipe metadata,
1908 the task code and the task hash values from its dependencies.
1909 When changes are made, these task hashes are therefore modified,
1910 causing the task to re-execute. The task hashes of tasks depending on this
1911 task are therefore modified too, causing the whole dependency
1912 chain to re-execute.
1913
1914- The *output hash*, a new hash computed from the output of Shared State tasks,
1915 tasks that save their resulting output to a Shared State tarball.
1916 The mapping between the task hash and its output hash is reported
1917 to a new *Hash Equivalence* server. This mapping is stored in a database
1918 by the server for future reference.
1919
1920- The *unihash*, a new hash, initially set to the task hash for the task.
1921 This is used to track the *unicity* of task output, and we will explain
1922 how its value is maintained.
1923
1924When Hash Equivalence is enabled, BitBake computes the task hash
1925for each task by using the unihash of its dependencies, instead
1926of their task hash.
1927
1928Now, imagine that a Shared State task is modified because of a change in
1929its code or metadata, or because of a change in its dependencies.
1930Since this modifies its task hash, this task will need re-executing.
1931Its output hash will therefore be computed again.
1932
1933Then, the new mapping between the new task hash and its output hash
1934will be reported to the Hash Equivalence server. The server will
1935let BitBake know whether this output hash is the same as a previously
1936reported output hash, for a different task hash.
1937
1938If the output hash is already known, BitBake will update the task's
1939unihash to match the original task hash that generated that output.
1940Thanks to this, the depending tasks will keep a previously recorded
1941task hash, and BitBake will be able to retrieve their output from
1942the Shared State cache, instead of re-executing them. Similarly, the
1943output of further downstream tasks can also be retrieved from Shared
1944State.
1945
1946If the output hash is unknown, a new entry will be created on the Hash
1947Equivalence server, matching the task hash to that output.
1948The depending tasks, still having a new task hash because of the
1949change, will need to re-execute as expected. The change propagates
1950to the depending tasks.
1951
1952To summarize, when Hash Equivalence is enabled, a change in one of the
1953tasks in BitBake's run queue doesn't have to propagate to all the
1954downstream tasks that depend on the output of this task, causing a
1955full rebuild of such tasks, and so on with the next depending tasks.
1956Instead, when the output of this task remains identical to previously
1957recorded output, BitBake can safely retrieve all the downstream
1958task output from the Shared State cache.
1959
1960.. note::
1961
1962 Having :doc:`/test-manual/reproducible-builds` is a key ingredient for
1963 the stability of the task's output hash. Therefore, the effectiveness
1964 of Hash Equivalence strongly depends on it.
1965
1966 Recipes that are not reproducible may have undesired behavior if hash
1967 equivalence is enabled, since the non-reproducible diverging output maybe be
1968 remapped to an older sstate object in the cache by the server. If a recipe
1969 is non-reproducible in trivial ways, such as different timestamps, this is
1970 likely not a problem. However recipes that have more dramatic changes (such
1971 as completely different file names) will likely outright fail since the
1972 downstream sstate objects are not actually equivalent to what was just
1973 built.
1974
1975This applies to multiple scenarios:
1976
1977- A "trivial" change to a recipe that doesn't impact its generated output,
1978 such as whitespace changes, modifications to unused code paths or
1979 in the ordering of variables.
1980
1981- Shared library updates, for example to fix a security vulnerability.
1982 For sure, the programs using such a library should be rebuilt, but
1983 their new binaries should remain identical. The corresponding tasks should
1984 have a different output hash because of the change in the hash of their
1985 library dependency, but thanks to their output being identical, Hash
1986 Equivalence will stop the propagation down the dependency chain.
1987
1988- Native tool updates. Though the depending tasks should be rebuilt,
1989 it's likely that they will generate the same output and be marked
1990 as equivalent.
1991
1992This mechanism is enabled by default in Poky, and is controlled by three
1993variables:
1994
1995- :term:`bitbake:BB_HASHSERVE`, specifying a local or remote Hash
1996 Equivalence server to use.
1997
1998- :term:`BB_HASHSERVE_UPSTREAM`, when ``BB_HASHSERVE = "auto"``,
1999 allowing to connect the local server to an upstream one.
2000
2001- :term:`bitbake:BB_SIGNATURE_HANDLER`, which must be set to ``OEEquivHash``.
2002
2003Therefore, the default configuration in Poky corresponds to the
2004below settings::
2005
2006 BB_HASHSERVE = "auto"
2007 BB_SIGNATURE_HANDLER = "OEEquivHash"
2008
2009Rather than starting a local server, another possibility is to rely
2010on a Hash Equivalence server on a network, by setting::
2011
2012 BB_HASHSERVE = "<HOSTNAME>:<PORT>"
2013
2014.. note::
2015
2016 The shared Hash Equivalence server needs to be maintained together with the
2017 Shared State cache. Otherwise, the server could report Shared State hashes
2018 that only exist on specific clients.
2019
2020 We therefore recommend that one Hash Equivalence server be set up to
2021 correspond with a given Shared State cache, and to start this server
2022 in *read-only mode*, so that it doesn't store equivalences for
2023 Shared State caches that are local to clients.
2024
2025 See the :term:`BB_HASHSERVE` reference for details about starting
2026 a Hash Equivalence server.
2027
2028See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__
2029of Joshua Watt's `Hash Equivalence and Reproducible Builds
2030<https://elinux.org/images/3/37/Hash_Equivalence_and_Reproducible_Builds.pdf>`__
2031presentation at ELC 2020 for a very synthetic introduction to the
2032Hash Equivalence implementation in the Yocto Project.
2033
1969Automatically Added Runtime Dependencies 2034Automatically Added Runtime Dependencies
1970======================================== 2035========================================
1971 2036
1972The OpenEmbedded build system automatically adds common types of runtime 2037The OpenEmbedded build system automatically adds common types of runtime
1973dependencies between packages, which means that you do not need to 2038dependencies between packages, which means that you do not need to
1974explicitly declare the packages using 2039explicitly declare the packages using
1975:term:`RDEPENDS`. Three automatic 2040:term:`RDEPENDS`. There are three automatic
1976mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that 2041mechanisms (``shlibdeps``, ``pcdeps``, and ``depchains``) that
1977handle shared libraries, package configuration (pkg-config) modules, and 2042handle shared libraries, package configuration (pkg-config) modules, and
1978``-dev`` and ``-dbg`` packages, respectively. For other types of runtime 2043``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
1979dependencies, you must manually declare the dependencies. 2044dependencies, you must manually declare the dependencies.
@@ -1984,7 +2049,7 @@ dependencies, you must manually declare the dependencies.
1984 located. For each shared library, the package that contains the 2049 located. For each shared library, the package that contains the
1985 shared library is registered as providing the shared library. More 2050 shared library is registered as providing the shared library. More
1986 specifically, the package is registered as providing the 2051 specifically, the package is registered as providing the
1987 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The 2052 :wikipedia:`soname <Soname>` of the library. The
1988 resulting shared-library-to-package mapping is saved globally in 2053 resulting shared-library-to-package mapping is saved globally in
1989 :term:`PKGDATA_DIR` by the 2054 :term:`PKGDATA_DIR` by the
1990 :ref:`ref-tasks-packagedata` 2055 :ref:`ref-tasks-packagedata`
@@ -1992,7 +2057,7 @@ dependencies, you must manually declare the dependencies.
1992 2057
1993 Simultaneously, all executables and shared libraries installed by the 2058 Simultaneously, all executables and shared libraries installed by the
1994 recipe are inspected to see what shared libraries they link against. 2059 recipe are inspected to see what shared libraries they link against.
1995 For each shared library dependency that is found, ``PKGDATA_DIR`` is 2060 For each shared library dependency that is found, :term:`PKGDATA_DIR` is
1996 queried to see if some package (likely from a different recipe) 2061 queried to see if some package (likely from a different recipe)
1997 contains the shared library. If such a package is found, a runtime 2062 contains the shared library. If such a package is found, a runtime
1998 dependency is added from the package that depends on the shared 2063 dependency is added from the package that depends on the shared
@@ -2001,7 +2066,7 @@ dependencies, you must manually declare the dependencies.
2001 The automatically added runtime dependency also includes a version 2066 The automatically added runtime dependency also includes a version
2002 restriction. This version restriction specifies that at least the 2067 restriction. This version restriction specifies that at least the
2003 current version of the package that provides the shared library must 2068 current version of the package that provides the shared library must
2004 be used, as if "package (>= version)" had been added to ``RDEPENDS``. 2069 be used, as if "package (>= version)" had been added to :term:`RDEPENDS`.
2005 This forces an upgrade of the package containing the shared library 2070 This forces an upgrade of the package containing the shared library
2006 when installing the package that depends on the library, if needed. 2071 when installing the package that depends on the library, if needed.
2007 2072
@@ -2011,18 +2076,18 @@ dependencies, you must manually declare the dependencies.
2011 :term:`PRIVATE_LIBS` inside 2076 :term:`PRIVATE_LIBS` inside
2012 the package's recipe. 2077 the package's recipe.
2013 2078
2014- ``pcdeps``: During the ``do_package`` task of each recipe, all 2079- ``pcdeps``: During the :ref:`ref-tasks-package` task of each recipe, all
2015 pkg-config modules (``*.pc`` files) installed by the recipe are 2080 pkg-config modules (``*.pc`` files) installed by the recipe are
2016 located. For each module, the package that contains the module is 2081 located. For each module, the package that contains the module is
2017 registered as providing the module. The resulting module-to-package 2082 registered as providing the module. The resulting module-to-package
2018 mapping is saved globally in ``PKGDATA_DIR`` by the 2083 mapping is saved globally in :term:`PKGDATA_DIR` by the
2019 ``do_packagedata`` task. 2084 :ref:`ref-tasks-packagedata` task.
2020 2085
2021 Simultaneously, all pkg-config modules installed by the recipe are 2086 Simultaneously, all pkg-config modules installed by the recipe are
2022 inspected to see what other pkg-config modules they depend on. A 2087 inspected to see what other pkg-config modules they depend on. A
2023 module is seen as depending on another module if it contains a 2088 module is seen as depending on another module if it contains a
2024 "Requires:" line that specifies the other module. For each module 2089 "Requires:" line that specifies the other module. For each module
2025 dependency, ``PKGDATA_DIR`` is queried to see if some package 2090 dependency, :term:`PKGDATA_DIR` is queried to see if some package
2026 contains the module. If such a package is found, a runtime dependency 2091 contains the module. If such a package is found, a runtime dependency
2027 is added from the package that depends on the module to the package 2092 is added from the package that depends on the module to the package
2028 that contains the module. 2093 that contains the module.
@@ -2047,9 +2112,9 @@ dependencies, you must manually declare the dependencies.
2047 2112
2048 .. note:: 2113 .. note::
2049 2114
2050 By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on 2115 By default, ``foo-dev`` also has an :term:`RDEPENDS`-style dependency on
2051 ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in 2116 ``foo``, because the default value of ``RDEPENDS:${PN}-dev`` (set in
2052 bitbake.conf) includes "${PN}". 2117 ``bitbake.conf``) includes "${PN}".
2053 2118
2054 To ensure that the dependency chain is never broken, ``-dev`` and 2119 To ensure that the dependency chain is never broken, ``-dev`` and
2055 ``-dbg`` packages are always generated by default, even if the 2120 ``-dbg`` packages are always generated by default, even if the
@@ -2057,12 +2122,12 @@ dependencies, you must manually declare the dependencies.
2057 :term:`ALLOW_EMPTY` variable 2122 :term:`ALLOW_EMPTY` variable
2058 for more information. 2123 for more information.
2059 2124
2060The ``do_package`` task depends on the ``do_packagedata`` task of each 2125The :ref:`ref-tasks-package` task depends on the :ref:`ref-tasks-packagedata`
2061recipe in :term:`DEPENDS` through use 2126task of each recipe in :term:`DEPENDS` through use of a
2062of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` 2127``[``\ :ref:`deptask <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2063declaration, which guarantees that the required 2128declaration, which guarantees that the required shared-library /
2064shared-library/module-to-package mapping information will be available 2129module-to-package mapping information will be available when needed as long as
2065when needed as long as ``DEPENDS`` has been correctly set. 2130:term:`DEPENDS` has been correctly set.
2066 2131
2067Fakeroot and Pseudo 2132Fakeroot and Pseudo
2068=================== 2133===================
@@ -2072,8 +2137,8 @@ operations that are normally reserved for the root user (e.g.
2072:ref:`ref-tasks-install`, 2137:ref:`ref-tasks-install`,
2073:ref:`do_package_write* <ref-tasks-package_write_deb>`, 2138:ref:`do_package_write* <ref-tasks-package_write_deb>`,
2074:ref:`ref-tasks-rootfs`, and 2139:ref:`ref-tasks-rootfs`, and
2075:ref:`do_image* <ref-tasks-image>`). For example, 2140:ref:`do_image_* <ref-tasks-image>`). For example,
2076the ``do_install`` task benefits from being able to set the UID and GID 2141the :ref:`ref-tasks-install` task benefits from being able to set the UID and GID
2077of installed files to arbitrary values. 2142of installed files to arbitrary values.
2078 2143
2079One approach to allowing tasks to perform root-only operations would be 2144One approach to allowing tasks to perform root-only operations would be
@@ -2111,9 +2176,7 @@ accomplished using fakeroot.
2111 under fakeroot. Otherwise, the task cannot run root-only operations, 2176 under fakeroot. Otherwise, the task cannot run root-only operations,
2112 and cannot see the fake file ownership and permissions set by the 2177 and cannot see the fake file ownership and permissions set by the
2113 other task. You need to also add a dependency on 2178 other task. You need to also add a dependency on
2114 virtual/fakeroot-native:do_populate_sysroot 2179 ``virtual/fakeroot-native:do_populate_sysroot``, giving the following::
2115 , giving the following:
2116 ::
2117 2180
2118 fakeroot do_mytask () { 2181 fakeroot do_mytask () {
2119 ... 2182 ...
@@ -2126,3 +2189,173 @@ For more information, see the
2126BitBake User Manual. You can also reference the "`Why Not 2189BitBake User Manual. You can also reference the "`Why Not
2127Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__" 2190Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2128article for background information on Fakeroot and Pseudo. 2191article for background information on Fakeroot and Pseudo.
2192
2193BitBake Tasks Map
2194=================
2195
2196To understand how BitBake operates in the build directory and environment
2197we can consider the following recipes and diagram, to have full picture
2198about the tasks that BitBake runs to generate the final package file
2199for the recipe.
2200
2201We will have two recipes as an example:
2202
2203- ``libhello``: A recipe that provides a shared library
2204- ``sayhello``: A recipe that uses ``libhello`` library to do its job
2205
2206.. note::
2207
2208 ``sayhello`` depends on ``libhello`` at compile time as it needs the shared
2209 library to do the dynamic linking process. It also depends on it at runtime
2210 as the shared library loader needs to find the library.
2211 For more details about dependencies check :ref:`ref-varlocality-recipe-dependencies`.
2212
2213``libhello`` sources are as follows:
2214
2215- ``LICENSE``: This is the license associated with this library
2216- ``Makefile``: The file used by ``make`` to build the library
2217- ``hellolib.c``: The implementation of the library
2218- ``hellolib.h``: The C header of the library
2219
2220``sayhello`` sources are as follows:
2221
2222- ``LICENSE``: This is the license associated with this project
2223- ``Makefile``: The file used by ``make`` to build the project
2224- ``sayhello.c``: The source file of the project
2225
2226Before presenting the contents of each file, here are the steps
2227that we need to follow to accomplish what we want in the first place,
2228which is integrating ``sayhello`` in our root file system:
2229
2230#. Create a Git repository for each project with the corresponding files
2231
2232#. Create a recipe for each project
2233
2234#. Make sure that ``sayhello`` recipe :term:`DEPENDS` on ``libhello``
2235
2236#. Make sure that ``sayhello`` recipe :term:`RDEPENDS` on ``libhello``
2237
2238#. Add ``sayhello`` to :term:`IMAGE_INSTALL` to integrate it into
2239 the root file system
2240
2241The contents of ``libhello/Makefile`` are::
2242
2243 LIB=libhello.so
2244
2245 all: $(LIB)
2246
2247 $(LIB): hellolib.o
2248 $(CC) $< -Wl,-soname,$(LIB).1 -fPIC $(LDFLAGS) -shared -o $(LIB).1.0
2249
2250 %.o: %.c
2251 $(CC) -c $<
2252
2253 clean:
2254 rm -rf *.o *.so*
2255
2256.. note::
2257
2258 When creating shared libraries, it is strongly recommended to follow the Linux
2259 conventions and guidelines (see `this article
2260 <https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
2261 for some background).
2262
2263.. note::
2264
2265 When creating ``Makefile`` files, it is strongly recommended to use ``CC``, ``LDFLAGS``
2266 and ``CFLAGS`` as BitBake will set them as environment variables according
2267 to your build configuration.
2268
2269The contents of ``libhello/hellolib.h`` are::
2270
2271 #ifndef HELLOLIB_H
2272 #define HELLOLIB_H
2273
2274 void Hello();
2275
2276 #endif
2277
2278The contents of ``libhello/hellolib.c`` are::
2279
2280 #include <stdio.h>
2281
2282 void Hello(){
2283 puts("Hello from a Yocto demo \n");
2284 }
2285
2286The contents of ``sayhello/Makefile`` are::
2287
2288 EXEC=sayhello
2289 LDFLAGS += -lhello
2290
2291 all: $(EXEC)
2292
2293 $(EXEC): sayhello.c
2294 $(CC) $< $(LDFLAGS) $(CFLAGS) -o $(EXEC)
2295
2296 clean:
2297 rm -rf $(EXEC) *.o
2298
2299The contents of ``sayhello/sayhello.c`` are::
2300
2301 #include <hellolib.h>
2302
2303 int main(){
2304 Hello();
2305 return 0;
2306 }
2307
2308The contents of ``libhello_0.1.bb`` are::
2309
2310 SUMMARY = "Hello demo library"
2311 DESCRIPTION = "Hello shared library used in Yocto demo"
2312
2313 # NOTE: Set the License according to the LICENSE file of your project
2314 # and then add LIC_FILES_CHKSUM accordingly
2315 LICENSE = "CLOSED"
2316
2317 # Assuming the branch is main
2318 # Change <username> accordingly
2319 SRC_URI = "git://github.com/<username>/libhello;branch=main;protocol=https"
2320
2321 S = "${WORKDIR}/git"
2322
2323 do_install(){
2324 install -d ${D}${includedir}
2325 install -d ${D}${libdir}
2326
2327 install hellolib.h ${D}${includedir}
2328 oe_soinstall ${PN}.so.${PV} ${D}${libdir}
2329 }
2330
2331The contents of ``sayhello_0.1.bb`` are::
2332
2333 SUMMARY = "SayHello demo"
2334 DESCRIPTION = "SayHello project used in Yocto demo"
2335
2336 # NOTE: Set the License according to the LICENSE file of your project
2337 # and then add LIC_FILES_CHKSUM accordingly
2338 LICENSE = "CLOSED"
2339
2340 # Assuming the branch is main
2341 # Change <username> accordingly
2342 SRC_URI = "git://github.com/<username>/sayhello;branch=main;protocol=https"
2343
2344 DEPENDS += "libhello"
2345 RDEPENDS:${PN} += "libhello"
2346
2347 S = "${WORKDIR}/git"
2348
2349 do_install(){
2350 install -d ${D}/usr/bin
2351 install -m 0700 sayhello ${D}/usr/bin
2352 }
2353
2354After placing the recipes in a custom layer we can run ``bitbake sayhello``
2355to build the recipe.
2356
2357The following diagram shows the sequences of tasks that BitBake
2358executes to accomplish that.
2359
2360.. image:: svg/bitbake_tasks_map.*
2361 :width: 100%