diff options
Diffstat (limited to 'documentation/overview-manual/concepts.rst')
-rw-r--r-- | documentation/overview-manual/concepts.rst | 1029 |
1 files changed, 631 insertions, 398 deletions
diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst index 257de44ec8..62f2327a7e 100644 --- a/documentation/overview-manual/concepts.rst +++ b/documentation/overview-manual/concepts.rst | |||
@@ -34,10 +34,10 @@ itself is of various types: | |||
34 | 34 | ||
35 | BitBake knows how to combine multiple data sources together and refers | 35 | BitBake knows how to combine multiple data sources together and refers |
36 | to each data source as a layer. For information on layers, see the | 36 | to each data source as a layer. For information on layers, see the |
37 | ":ref:`dev-manual/common-tasks:understanding and creating layers`" | 37 | ":ref:`dev-manual/layers:understanding and creating layers`" |
38 | section of the Yocto Project Development Tasks Manual. | 38 | section of the Yocto Project Development Tasks Manual. |
39 | 39 | ||
40 | Following are some brief details on these core components. For | 40 | Here are some brief details on these core components. For |
41 | additional information on how these components interact during a build, | 41 | additional information on how these components interact during a build, |
42 | see the | 42 | see the |
43 | ":ref:`overview-manual/concepts:openembedded build system concepts`" | 43 | ":ref:`overview-manual/concepts:openembedded build system concepts`" |
@@ -55,8 +55,7 @@ This section briefly introduces BitBake. If you want more information on | |||
55 | BitBake, see the :doc:`BitBake User Manual <bitbake:index>`. | 55 | BitBake, see the :doc:`BitBake User Manual <bitbake:index>`. |
56 | 56 | ||
57 | To see a list of the options BitBake supports, use either of the | 57 | To see a list of the options BitBake supports, use either of the |
58 | following commands: | 58 | following commands:: |
59 | :: | ||
60 | 59 | ||
61 | $ bitbake -h | 60 | $ bitbake -h |
62 | $ bitbake --help | 61 | $ bitbake --help |
@@ -66,17 +65,14 @@ The most common usage for BitBake is ``bitbake recipename``, where | |||
66 | to as the "target"). The target often equates to the first part of a | 65 | to as the "target"). The target often equates to the first part of a |
67 | recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``). | 66 | recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``). |
68 | So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might | 67 | So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might |
69 | type the following: | 68 | type the following:: |
70 | :: | ||
71 | 69 | ||
72 | $ bitbake matchbox-desktop | 70 | $ bitbake matchbox-desktop |
73 | 71 | ||
74 | Several different | 72 | Several different versions of ``matchbox-desktop`` might exist. BitBake chooses |
75 | versions of ``matchbox-desktop`` might exist. BitBake chooses the one | 73 | the one selected by the distribution configuration. You can get more details |
76 | selected by the distribution configuration. You can get more details | 74 | about how BitBake chooses between different target versions and providers in the |
77 | about how BitBake chooses between different target versions and | 75 | ":ref:`bitbake-user-manual/bitbake-user-manual-execution:preferences`" section |
78 | providers in the | ||
79 | ":ref:`Preferences <bitbake:bitbake-user-manual/bitbake-user-manual-execution:preferences>`" section | ||
80 | of the BitBake User Manual. | 76 | of the BitBake User Manual. |
81 | 77 | ||
82 | BitBake also tries to execute any dependent tasks first. So for example, | 78 | BitBake also tries to execute any dependent tasks first. So for example, |
@@ -109,12 +105,11 @@ Classes | |||
109 | ------- | 105 | ------- |
110 | 106 | ||
111 | Class files (``.bbclass``) contain information that is useful to share | 107 | Class files (``.bbclass``) contain information that is useful to share |
112 | between recipes files. An example is the | 108 | between recipes files. An example is the :ref:`ref-classes-autotools` class, |
113 | :ref:`autotools <ref-classes-autotools>` class, | 109 | which contains common settings for any application that is built with |
114 | which contains common settings for any application that Autotools uses. | 110 | the :wikipedia:`GNU Autotools <GNU_Autotools>`. |
115 | The ":ref:`ref-manual/classes:Classes`" chapter in the | 111 | The ":ref:`ref-manual/classes:Classes`" chapter in the Yocto Project |
116 | Yocto Project Reference Manual provides details about classes and how to | 112 | Reference Manual provides details about classes and how to use them. |
117 | use them. | ||
118 | 113 | ||
119 | Configurations | 114 | Configurations |
120 | -------------- | 115 | -------------- |
@@ -132,7 +127,7 @@ Layers | |||
132 | 127 | ||
133 | Layers are repositories that contain related metadata (i.e. sets of | 128 | Layers are repositories that contain related metadata (i.e. sets of |
134 | instructions) that tell the OpenEmbedded build system how to build a | 129 | instructions) that tell the OpenEmbedded build system how to build a |
135 | target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__ | 130 | target. :ref:`overview-manual/yp-intro:the yocto project layer model` |
136 | facilitates collaboration, sharing, customization, and reuse within the | 131 | facilitates collaboration, sharing, customization, and reuse within the |
137 | Yocto Project development environment. Layers logically separate | 132 | Yocto Project development environment. Layers logically separate |
138 | information for your project. For example, you can use a layer to hold | 133 | information for your project. For example, you can use a layer to hold |
@@ -141,8 +136,8 @@ hardware-specific configurations allows you to share other metadata by | |||
141 | using a different layer where that metadata might be common across | 136 | using a different layer where that metadata might be common across |
142 | several pieces of hardware. | 137 | several pieces of hardware. |
143 | 138 | ||
144 | Many layers exist that work in the Yocto Project development environment. The | 139 | There are many layers working in the Yocto Project development environment. The |
145 | :yocto_home:`Yocto Project Curated Layer Index </software-overview/layers/>` | 140 | :yocto_home:`Yocto Project Compatible Layer Index </software-overview/layers/>` |
146 | and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from | 141 | and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from |
147 | which you can use or leverage. | 142 | which you can use or leverage. |
148 | 143 | ||
@@ -151,7 +146,7 @@ Conforming to a known structure allows BitBake to make assumptions | |||
151 | during builds on where to find types of metadata. You can find | 146 | during builds on where to find types of metadata. You can find |
152 | procedures and learn about tools (i.e. ``bitbake-layers``) for creating | 147 | procedures and learn about tools (i.e. ``bitbake-layers``) for creating |
153 | layers suitable for the Yocto Project in the | 148 | layers suitable for the Yocto Project in the |
154 | ":ref:`dev-manual/common-tasks:understanding and creating layers`" | 149 | ":ref:`dev-manual/layers:understanding and creating layers`" |
155 | section of the Yocto Project Development Tasks Manual. | 150 | section of the Yocto Project Development Tasks Manual. |
156 | 151 | ||
157 | OpenEmbedded Build System Concepts | 152 | OpenEmbedded Build System Concepts |
@@ -168,7 +163,7 @@ remainder of this section expands on the fundamental input, output, | |||
168 | process, and metadata logical blocks that make up the workflow. | 163 | process, and metadata logical blocks that make up the workflow. |
169 | 164 | ||
170 | .. image:: figures/YP-flow-diagram.png | 165 | .. image:: figures/YP-flow-diagram.png |
171 | :align: center | 166 | :width: 100% |
172 | 167 | ||
173 | In general, the build's workflow consists of several functional areas: | 168 | In general, the build's workflow consists of several functional areas: |
174 | 169 | ||
@@ -207,11 +202,11 @@ you can tell BitBake the target architecture for which you are building | |||
207 | the image, where to store downloaded source, and other build properties. | 202 | the image, where to store downloaded source, and other build properties. |
208 | 203 | ||
209 | The following figure shows an expanded representation of the "User | 204 | The following figure shows an expanded representation of the "User |
210 | Configuration" box of the `general workflow | 205 | Configuration" box of the :ref:`general workflow |
211 | figure <#general-workflow-figure>`__: | 206 | figure <overview-manual/concepts:openembedded build system concepts>`: |
212 | 207 | ||
213 | .. image:: figures/user-configuration.png | 208 | .. image:: figures/user-configuration.png |
214 | :align: center | 209 | :width: 100% |
215 | 210 | ||
216 | BitBake needs some basic configuration files in order to complete a | 211 | BitBake needs some basic configuration files in order to complete a |
217 | build. These files are ``*.conf`` files. The minimally necessary ones | 212 | build. These files are ``*.conf`` files. The minimally necessary ones |
@@ -235,13 +230,12 @@ for creating actual configuration files when you source | |||
235 | :ref:`structure-core-script`, which is the | 230 | :ref:`structure-core-script`, which is the |
236 | build environment script. | 231 | build environment script. |
237 | 232 | ||
238 | Sourcing the build environment script creates a | 233 | Sourcing the build environment script creates a :term:`Build Directory` |
239 | :term:`Build Directory` if one does not | 234 | if one does not already exist. BitBake uses the :term:`Build Directory` |
240 | already exist. BitBake uses the Build Directory for all its work during | 235 | for all its work during builds. The Build Directory has a ``conf`` directory |
241 | builds. The Build Directory has a ``conf`` directory that contains | 236 | that contains default versions of your ``local.conf`` and ``bblayers.conf`` |
242 | default versions of your ``local.conf`` and ``bblayers.conf`` | ||
243 | configuration files. These default configuration files are created only | 237 | configuration files. These default configuration files are created only |
244 | if versions do not already exist in the Build Directory at the time you | 238 | if versions do not already exist in the :term:`Build Directory` at the time you |
245 | source the build environment setup script. | 239 | source the build environment setup script. |
246 | 240 | ||
247 | Because the Poky repository is fundamentally an aggregation of existing | 241 | Because the Poky repository is fundamentally an aggregation of existing |
@@ -253,9 +247,9 @@ assumes the script is executed from within a cloned or unpacked version | |||
253 | of Poky. | 247 | of Poky. |
254 | 248 | ||
255 | Depending on where the script is sourced, different sub-scripts are | 249 | Depending on where the script is sourced, different sub-scripts are |
256 | called to set up the Build Directory (Yocto or OpenEmbedded). | 250 | called to set up the :term:`Build Directory` (Yocto or OpenEmbedded). |
257 | Specifically, the script ``scripts/oe-setup-builddir`` inside the poky | 251 | Specifically, the script ``scripts/oe-setup-builddir`` inside the poky |
258 | directory sets up the Build Directory and seeds the directory (if | 252 | directory sets up the :term:`Build Directory` and seeds the directory (if |
259 | necessary) with configuration files appropriate for the Yocto Project | 253 | necessary) with configuration files appropriate for the Yocto Project |
260 | development environment. | 254 | development environment. |
261 | 255 | ||
@@ -271,7 +265,7 @@ The ``local.conf`` file provides many basic variables that define a | |||
271 | build environment. Here is a list of a few. To see the default | 265 | build environment. Here is a list of a few. To see the default |
272 | configurations in a ``local.conf`` file created by the build environment | 266 | configurations in a ``local.conf`` file created by the build environment |
273 | script, see the | 267 | script, see the |
274 | :yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/local.conf.sample>` | 268 | :yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/templates/default/local.conf.sample>` |
275 | in the ``meta-poky`` layer: | 269 | in the ``meta-poky`` layer: |
276 | 270 | ||
277 | - *Target Machine Selection:* Controlled by the | 271 | - *Target Machine Selection:* Controlled by the |
@@ -302,25 +296,20 @@ in the ``meta-poky`` layer: | |||
302 | 296 | ||
303 | .. note:: | 297 | .. note:: |
304 | 298 | ||
305 | Configurations set in the | 299 | Configurations set in the ``conf/local.conf`` file can also be set |
306 | conf/local.conf | 300 | in the ``conf/site.conf`` and ``conf/auto.conf`` configuration files. |
307 | file can also be set in the | ||
308 | conf/site.conf | ||
309 | and | ||
310 | conf/auto.conf | ||
311 | configuration files. | ||
312 | 301 | ||
313 | The ``bblayers.conf`` file tells BitBake what layers you want considered | 302 | The ``bblayers.conf`` file tells BitBake what layers you want considered |
314 | during the build. By default, the layers listed in this file include | 303 | during the build. By default, the layers listed in this file include |
315 | layers minimally needed by the build system. However, you must manually | 304 | layers minimally needed by the build system. However, you must manually |
316 | add any custom layers you have created. You can find more information on | 305 | add any custom layers you have created. You can find more information on |
317 | working with the ``bblayers.conf`` file in the | 306 | working with the ``bblayers.conf`` file in the |
318 | ":ref:`dev-manual/common-tasks:enabling your layer`" | 307 | ":ref:`dev-manual/layers:enabling your layer`" |
319 | section in the Yocto Project Development Tasks Manual. | 308 | section in the Yocto Project Development Tasks Manual. |
320 | 309 | ||
321 | The files ``site.conf`` and ``auto.conf`` are not created by the | 310 | The files ``site.conf`` and ``auto.conf`` are not created by the |
322 | environment initialization script. If you want the ``site.conf`` file, | 311 | environment initialization script. If you want the ``site.conf`` file, |
323 | you need to create that yourself. The ``auto.conf`` file is typically | 312 | you need to create it yourself. The ``auto.conf`` file is typically |
324 | created by an autobuilder: | 313 | created by an autobuilder: |
325 | 314 | ||
326 | - *site.conf:* You can use the ``conf/site.conf`` configuration | 315 | - *site.conf:* You can use the ``conf/site.conf`` configuration |
@@ -328,17 +317,7 @@ created by an autobuilder: | |||
328 | you had several build environments and they shared some common | 317 | you had several build environments and they shared some common |
329 | features. You can set these default build properties here. A good | 318 | features. You can set these default build properties here. A good |
330 | example is perhaps the packaging format to use through the | 319 | example is perhaps the packaging format to use through the |
331 | :term:`PACKAGE_CLASSES` | 320 | :term:`PACKAGE_CLASSES` variable. |
332 | variable. | ||
333 | |||
334 | One useful scenario for using the ``conf/site.conf`` file is to | ||
335 | extend your :term:`BBPATH` variable | ||
336 | to include the path to a ``conf/site.conf``. Then, when BitBake looks | ||
337 | for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file | ||
338 | and applies your common configurations found in the file. To override | ||
339 | configurations in a particular build directory, alter the similar | ||
340 | configurations within that build directory's ``conf/local.conf`` | ||
341 | file. | ||
342 | 321 | ||
343 | - *auto.conf:* The file is usually created and written to by an | 322 | - *auto.conf:* The file is usually created and written to by an |
344 | autobuilder. The settings put into the file are typically the same as | 323 | autobuilder. The settings put into the file are typically the same as |
@@ -372,9 +351,9 @@ BitBake's global behavior. This section takes a closer look at the | |||
372 | layers the build system uses to further control the build. These layers | 351 | layers the build system uses to further control the build. These layers |
373 | provide Metadata for the software, machine, and policies. | 352 | provide Metadata for the software, machine, and policies. |
374 | 353 | ||
375 | In general, three types of layer input exists. You can see them below | 354 | In general, there are three types of layer input. You can see them below |
376 | the "User Configuration" box in the `general workflow | 355 | the "User Configuration" box in the `general workflow |
377 | figure <#general-workflow-figure>`__: | 356 | figure <overview-manual/concepts:openembedded build system concepts>`: |
378 | 357 | ||
379 | - *Metadata (.bb + Patches):* Software layers containing | 358 | - *Metadata (.bb + Patches):* Software layers containing |
380 | user-supplied recipe files, patches, and append files. A good example | 359 | user-supplied recipe files, patches, and append files. A good example |
@@ -387,8 +366,8 @@ figure <#general-workflow-figure>`__: | |||
387 | - *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e. | 366 | - *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e. |
388 | "BSP Layer" in the following figure) providing machine-specific | 367 | "BSP Layer" in the following figure) providing machine-specific |
389 | configurations. This type of information is specific to a particular | 368 | configurations. This type of information is specific to a particular |
390 | target architecture. A good example of a BSP layer from the `Poky | 369 | target architecture. A good example of a BSP layer from the |
391 | Reference Distribution <#gs-reference-distribution-poky>`__ is the | 370 | :ref:`overview-manual/yp-intro:reference distribution (poky)` is the |
392 | :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>` | 371 | :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>` |
393 | layer. | 372 | layer. |
394 | 373 | ||
@@ -403,10 +382,12 @@ figure <#general-workflow-figure>`__: | |||
403 | that contain many policy configurations for the Poky distribution. | 382 | that contain many policy configurations for the Poky distribution. |
404 | 383 | ||
405 | The following figure shows an expanded representation of these three | 384 | The following figure shows an expanded representation of these three |
406 | layers from the `general workflow figure <#general-workflow-figure>`__: | 385 | layers from the :ref:`general workflow figure |
386 | <overview-manual/concepts:openembedded build system concepts>`: | ||
407 | 387 | ||
408 | .. image:: figures/layer-input.png | 388 | .. image:: figures/layer-input.png |
409 | :align: center | 389 | :align: center |
390 | :width: 70% | ||
410 | 391 | ||
411 | In general, all layers have a similar structure. They all contain a | 392 | In general, all layers have a similar structure. They all contain a |
412 | licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed, | 393 | licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed, |
@@ -414,13 +395,13 @@ a ``README`` file as good practice and especially if the layer is to be | |||
414 | distributed, a configuration directory, and recipe directories. You can | 395 | distributed, a configuration directory, and recipe directories. You can |
415 | learn about the general structure for layers used with the Yocto Project | 396 | learn about the general structure for layers used with the Yocto Project |
416 | in the | 397 | in the |
417 | ":ref:`dev-manual/common-tasks:creating your own layer`" | 398 | ":ref:`dev-manual/layers:creating your own layer`" |
418 | section in the | 399 | section in the |
419 | Yocto Project Development Tasks Manual. For a general discussion on | 400 | Yocto Project Development Tasks Manual. For a general discussion on |
420 | layers and the many layers from which you can draw, see the | 401 | layers and the many layers from which you can draw, see the |
421 | "`Layers <#overview-layers>`__" and "`The Yocto Project Layer | 402 | ":ref:`overview-manual/concepts:layers`" and |
422 | Model <#the-yocto-project-layer-model>`__" sections both earlier in this | 403 | ":ref:`overview-manual/yp-intro:the yocto project layer model`" sections both |
423 | manual. | 404 | earlier in this manual. |
424 | 405 | ||
425 | If you explored the previous links, you discovered some areas where many | 406 | If you explored the previous links, you discovered some areas where many |
426 | layers that work with the Yocto Project exist. The :yocto_git:`Source | 407 | layers that work with the Yocto Project exist. The :yocto_git:`Source |
@@ -428,8 +409,8 @@ Repositories <>` also shows layers categorized under "Yocto Metadata Layers." | |||
428 | 409 | ||
429 | .. note:: | 410 | .. note:: |
430 | 411 | ||
431 | Layers exist in the Yocto Project Source Repositories that cannot be | 412 | There are layers in the Yocto Project Source Repositories that cannot be |
432 | found in the OpenEmbedded Layer Index. These layers are either | 413 | found in the OpenEmbedded Layer Index. Such layers are either |
433 | deprecated or experimental in nature. | 414 | deprecated or experimental in nature. |
434 | 415 | ||
435 | BitBake uses the ``conf/bblayers.conf`` file, which is part of the user | 416 | BitBake uses the ``conf/bblayers.conf`` file, which is part of the user |
@@ -443,7 +424,7 @@ The distribution layer provides policy configurations for your | |||
443 | distribution. Best practices dictate that you isolate these types of | 424 | distribution. Best practices dictate that you isolate these types of |
444 | configurations into their own layer. Settings you provide in | 425 | configurations into their own layer. Settings you provide in |
445 | ``conf/distro/distro.conf`` override similar settings that BitBake finds | 426 | ``conf/distro/distro.conf`` override similar settings that BitBake finds |
446 | in your ``conf/local.conf`` file in the Build Directory. | 427 | in your ``conf/local.conf`` file in the :term:`Build Directory`. |
447 | 428 | ||
448 | The following list provides some explanation and references for what you | 429 | The following list provides some explanation and references for what you |
449 | typically find in the distribution layer: | 430 | typically find in the distribution layer: |
@@ -490,7 +471,7 @@ the machine (``conf/machine/machine.conf``) and, of course, the layer | |||
490 | 471 | ||
491 | The remainder of the layer is dedicated to specific recipes by function: | 472 | The remainder of the layer is dedicated to specific recipes by function: |
492 | ``recipes-bsp``, ``recipes-core``, ``recipes-graphics``, | 473 | ``recipes-bsp``, ``recipes-core``, ``recipes-graphics``, |
493 | ``recipes-kernel``, and so forth. Metadata can exist for multiple | 474 | ``recipes-kernel``, and so forth. There can be metadata for multiple |
494 | formfactors, graphics support systems, and so forth. | 475 | formfactors, graphics support systems, and so forth. |
495 | 476 | ||
496 | .. note:: | 477 | .. note:: |
@@ -514,11 +495,12 @@ Sources | |||
514 | ------- | 495 | ------- |
515 | 496 | ||
516 | In order for the OpenEmbedded build system to create an image or any | 497 | In order for the OpenEmbedded build system to create an image or any |
517 | target, it must be able to access source files. The `general workflow | 498 | target, it must be able to access source files. The :ref:`general workflow |
518 | figure <#general-workflow-figure>`__ represents source files using the | 499 | figure <overview-manual/concepts:openembedded build system concepts>` |
519 | "Upstream Project Releases", "Local Projects", and "SCMs (optional)" | 500 | represents source files using the "Upstream Project Releases", "Local |
520 | boxes. The figure represents mirrors, which also play a role in locating | 501 | Projects", and "SCMs (optional)" boxes. The figure represents mirrors, |
521 | source files, with the "Source Materials" box. | 502 | which also play a role in locating source files, with the "Source |
503 | Materials" box. | ||
522 | 504 | ||
523 | The method by which source files are ultimately organized is a function | 505 | The method by which source files are ultimately organized is a function |
524 | of the project. For example, for released software, projects tend to use | 506 | of the project. For example, for released software, projects tend to use |
@@ -528,13 +510,11 @@ project that is more dynamic or experimental in nature, a project might | |||
528 | keep source files in a repository controlled by a Source Control Manager | 510 | keep source files in a repository controlled by a Source Control Manager |
529 | (SCM) such as Git. Pulling source from a repository allows you to | 511 | (SCM) such as Git. Pulling source from a repository allows you to |
530 | control the point in the repository (the revision) from which you want | 512 | control the point in the repository (the revision) from which you want |
531 | to build software. Finally, a combination of the two might exist, which | 513 | to build software. A combination of the two is also possible. |
532 | would give the consumer a choice when deciding where to get source | ||
533 | files. | ||
534 | 514 | ||
535 | BitBake uses the :term:`SRC_URI` | 515 | BitBake uses the :term:`SRC_URI` |
536 | variable to point to source files regardless of their location. Each | 516 | variable to point to source files regardless of their location. Each |
537 | recipe must have a ``SRC_URI`` variable that points to the source. | 517 | recipe must have a :term:`SRC_URI` variable that points to the source. |
538 | 518 | ||
539 | Another area that plays a significant role in where source files come | 519 | Another area that plays a significant role in where source files come |
540 | from is pointed to by the | 520 | from is pointed to by the |
@@ -542,22 +522,24 @@ from is pointed to by the | |||
542 | a cache that can hold previously downloaded source. You can also | 522 | a cache that can hold previously downloaded source. You can also |
543 | instruct the OpenEmbedded build system to create tarballs from Git | 523 | instruct the OpenEmbedded build system to create tarballs from Git |
544 | repositories, which is not the default behavior, and store them in the | 524 | repositories, which is not the default behavior, and store them in the |
545 | ``DL_DIR`` by using the | 525 | :term:`DL_DIR` by using the |
546 | :term:`BB_GENERATE_MIRROR_TARBALLS` | 526 | :term:`BB_GENERATE_MIRROR_TARBALLS` |
547 | variable. | 527 | variable. |
548 | 528 | ||
549 | Judicious use of a ``DL_DIR`` directory can save the build system a trip | 529 | Judicious use of a :term:`DL_DIR` directory can save the build system a trip |
550 | across the Internet when looking for files. A good method for using a | 530 | across the Internet when looking for files. A good method for using a download |
551 | download directory is to have ``DL_DIR`` point to an area outside of | 531 | directory is to have :term:`DL_DIR` point to an area outside of your |
552 | your Build Directory. Doing so allows you to safely delete the Build | 532 | :term:`Build Directory`. Doing so allows you to safely delete the |
553 | Directory if needed without fear of removing any downloaded source file. | 533 | :term:`Build Directory` if needed without fear of removing any downloaded |
534 | source file. | ||
554 | 535 | ||
555 | The remainder of this section provides a deeper look into the source | 536 | The remainder of this section provides a deeper look into the source |
556 | files and the mirrors. Here is a more detailed look at the source file | 537 | files and the mirrors. Here is a more detailed look at the source file |
557 | area of the `general workflow figure <#general-workflow-figure>`__: | 538 | area of the :ref:`general workflow figure <overview-manual/concepts:openembedded build system concepts>`: |
558 | 539 | ||
559 | .. image:: figures/source-input.png | 540 | .. image:: figures/source-input.png |
560 | :align: center | 541 | :align: center |
542 | :width: 70% | ||
561 | 543 | ||
562 | Upstream Project Releases | 544 | Upstream Project Releases |
563 | ~~~~~~~~~~~~~~~~~~~~~~~~~ | 545 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
@@ -572,33 +554,30 @@ Local Projects | |||
572 | ~~~~~~~~~~~~~~ | 554 | ~~~~~~~~~~~~~~ |
573 | 555 | ||
574 | Local projects are custom bits of software the user provides. These bits | 556 | Local projects are custom bits of software the user provides. These bits |
575 | reside somewhere local to a project - perhaps a directory into which the | 557 | reside somewhere local to a project --- perhaps a directory into which the |
576 | user checks in items (e.g. a local directory containing a development | 558 | user checks in items (e.g. a local directory containing a development |
577 | source tree used by the group). | 559 | source tree used by the group). |
578 | 560 | ||
579 | The canonical method through which to include a local project is to use | 561 | The canonical method through which to include a local project is to use the |
580 | the :ref:`externalsrc <ref-classes-externalsrc>` | 562 | :ref:`ref-classes-externalsrc` class to include that local project. You use |
581 | class to include that local project. You use either the ``local.conf`` | 563 | either the ``local.conf`` or a recipe's append file to override or set the |
582 | or a recipe's append file to override or set the recipe to point to the | 564 | recipe to point to the local directory on your disk to pull in the whole |
583 | local directory on your disk to pull in the whole source tree. | 565 | source tree. |
584 | 566 | ||
585 | Source Control Managers (Optional) | 567 | Source Control Managers (Optional) |
586 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | 568 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
587 | 569 | ||
588 | Another place from which the build system can get source files is with | 570 | Another place from which the build system can get source files is with |
589 | :ref:`fetchers <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` employing various Source | 571 | :ref:`bitbake-user-manual/bitbake-user-manual-fetching:fetchers` employing |
590 | Control Managers (SCMs) such as Git or Subversion. In such cases, a | 572 | various Source Control Managers (SCMs) such as Git or Subversion. In such |
591 | repository is cloned or checked out. The | 573 | cases, a repository is cloned or checked out. The :ref:`ref-tasks-fetch` task |
592 | :ref:`ref-tasks-fetch` task inside | 574 | inside BitBake uses the :term:`SRC_URI` variable and the argument's prefix to |
593 | BitBake uses the :term:`SRC_URI` | 575 | determine the correct fetcher module. |
594 | variable and the argument's prefix to determine the correct fetcher | ||
595 | module. | ||
596 | 576 | ||
597 | .. note:: | 577 | .. note:: |
598 | 578 | ||
599 | For information on how to have the OpenEmbedded build system generate | 579 | For information on how to have the OpenEmbedded build system generate |
600 | tarballs for Git repositories and place them in the | 580 | tarballs for Git repositories and place them in the :term:`DL_DIR` |
601 | DL_DIR | ||
602 | directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS` | 581 | directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS` |
603 | variable in the Yocto Project Reference Manual. | 582 | variable in the Yocto Project Reference Manual. |
604 | 583 | ||
@@ -609,7 +588,7 @@ the specific revision from which to build. | |||
609 | Source Mirror(s) | 588 | Source Mirror(s) |
610 | ~~~~~~~~~~~~~~~~ | 589 | ~~~~~~~~~~~~~~~~ |
611 | 590 | ||
612 | Two kinds of mirrors exist: pre-mirrors and regular mirrors. The | 591 | There are two kinds of mirrors: pre-mirrors and regular mirrors. The |
613 | :term:`PREMIRRORS` and | 592 | :term:`PREMIRRORS` and |
614 | :term:`MIRRORS` variables point to | 593 | :term:`MIRRORS` variables point to |
615 | these, respectively. BitBake checks pre-mirrors before looking upstream | 594 | these, respectively. BitBake checks pre-mirrors before looking upstream |
@@ -628,15 +607,15 @@ Package Feeds | |||
628 | 607 | ||
629 | When the OpenEmbedded build system generates an image or an SDK, it gets | 608 | When the OpenEmbedded build system generates an image or an SDK, it gets |
630 | the packages from a package feed area located in the | 609 | the packages from a package feed area located in the |
631 | :term:`Build Directory`. The `general | 610 | :term:`Build Directory`. The :ref:`general workflow figure |
632 | workflow figure <#general-workflow-figure>`__ shows this package feeds | 611 | <overview-manual/concepts:openembedded build system concepts>` |
633 | area in the upper-right corner. | 612 | shows this package feeds area in the upper-right corner. |
634 | 613 | ||
635 | This section looks a little closer into the package feeds area used by | 614 | This section looks a little closer into the package feeds area used by |
636 | the build system. Here is a more detailed look at the area: | 615 | the build system. Here is a more detailed look at the area: |
637 | 616 | ||
638 | .. image:: figures/package-feeds.png | 617 | .. image:: figures/package-feeds.png |
639 | :align: center | 618 | :width: 100% |
640 | 619 | ||
641 | Package feeds are an intermediary step in the build process. The | 620 | Package feeds are an intermediary step in the build process. The |
642 | OpenEmbedded build system provides classes to generate different package | 621 | OpenEmbedded build system provides classes to generate different package |
@@ -644,31 +623,28 @@ types, and you specify which classes to enable through the | |||
644 | :term:`PACKAGE_CLASSES` | 623 | :term:`PACKAGE_CLASSES` |
645 | variable. Before placing the packages into package feeds, the build | 624 | variable. Before placing the packages into package feeds, the build |
646 | process validates them with generated output quality assurance checks | 625 | process validates them with generated output quality assurance checks |
647 | through the :ref:`insane <ref-classes-insane>` | 626 | through the :ref:`ref-classes-insane` class. |
648 | class. | ||
649 | 627 | ||
650 | The package feed area resides in the Build Directory. The directory the | 628 | The package feed area resides in the :term:`Build Directory`. The directory the |
651 | build system uses to temporarily store packages is determined by a | 629 | build system uses to temporarily store packages is determined by a |
652 | combination of variables and the particular package manager in use. See | 630 | combination of variables and the particular package manager in use. See |
653 | the "Package Feeds" box in the illustration and note the information to | 631 | the "Package Feeds" box in the illustration and note the information to |
654 | the right of that area. In particular, the following defines where | 632 | the right of that area. In particular, the following defines where |
655 | package files are kept: | 633 | package files are kept: |
656 | 634 | ||
657 | - :term:`DEPLOY_DIR`: Defined as | 635 | - :term:`DEPLOY_DIR`: Defined as ``tmp/deploy`` in the :term:`Build Directory`. |
658 | ``tmp/deploy`` in the Build Directory. | ||
659 | 636 | ||
660 | - ``DEPLOY_DIR_*``: Depending on the package manager used, the package | 637 | - ``DEPLOY_DIR_*``: Depending on the package manager used, the package |
661 | type sub-folder. Given RPM, IPK, or DEB packaging and tarball | 638 | type sub-folder. Given RPM, IPK, or DEB packaging and tarball |
662 | creation, the | 639 | creation, the |
663 | :term:`DEPLOY_DIR_RPM`, | 640 | :term:`DEPLOY_DIR_RPM`, |
664 | :term:`DEPLOY_DIR_IPK`, | 641 | :term:`DEPLOY_DIR_IPK`, or |
665 | :term:`DEPLOY_DIR_DEB`, or | 642 | :term:`DEPLOY_DIR_DEB` |
666 | :term:`DEPLOY_DIR_TAR`, | ||
667 | variables are used, respectively. | 643 | variables are used, respectively. |
668 | 644 | ||
669 | - :term:`PACKAGE_ARCH`: Defines | 645 | - :term:`PACKAGE_ARCH`: Defines |
670 | architecture-specific sub-folders. For example, packages could exist | 646 | architecture-specific sub-folders. For example, packages could be |
671 | for the i586 or qemux86 architectures. | 647 | available for the i586 or qemux86 architectures. |
672 | 648 | ||
673 | BitBake uses the | 649 | BitBake uses the |
674 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | 650 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` |
@@ -676,13 +652,12 @@ tasks to generate packages and place them into the package holding area | |||
676 | (e.g. ``do_package_write_ipk`` for IPK packages). See the | 652 | (e.g. ``do_package_write_ipk`` for IPK packages). See the |
677 | ":ref:`ref-tasks-package_write_deb`", | 653 | ":ref:`ref-tasks-package_write_deb`", |
678 | ":ref:`ref-tasks-package_write_ipk`", | 654 | ":ref:`ref-tasks-package_write_ipk`", |
679 | ":ref:`ref-tasks-package_write_rpm`", | ||
680 | and | 655 | and |
681 | ":ref:`ref-tasks-package_write_tar`" | 656 | ":ref:`ref-tasks-package_write_rpm`" |
682 | sections in the Yocto Project Reference Manual for additional | 657 | sections in the Yocto Project Reference Manual for additional |
683 | information. As an example, consider a scenario where an IPK packaging | 658 | information. As an example, consider a scenario where an IPK packaging |
684 | manager is being used and package architecture support for both i586 and | 659 | manager is being used and there is package architecture support for both |
685 | qemux86 exist. Packages for the i586 architecture are placed in | 660 | i586 and qemux86. Packages for the i586 architecture are placed in |
686 | ``build/tmp/deploy/ipk/i586``, while packages for the qemux86 | 661 | ``build/tmp/deploy/ipk/i586``, while packages for the qemux86 |
687 | architecture are placed in ``build/tmp/deploy/ipk/qemux86``. | 662 | architecture are placed in ``build/tmp/deploy/ipk/qemux86``. |
688 | 663 | ||
@@ -691,15 +666,15 @@ BitBake Tool | |||
691 | 666 | ||
692 | The OpenEmbedded build system uses | 667 | The OpenEmbedded build system uses |
693 | :term:`BitBake` to produce images and | 668 | :term:`BitBake` to produce images and |
694 | Software Development Kits (SDKs). You can see from the `general workflow | 669 | Software Development Kits (SDKs). You can see from the :ref:`general workflow |
695 | figure <#general-workflow-figure>`__, the BitBake area consists of | 670 | figure <overview-manual/concepts:openembedded build system concepts>`, |
696 | several functional areas. This section takes a closer look at each of | 671 | the BitBake area consists of several functional areas. This section takes a |
697 | those areas. | 672 | closer look at each of those areas. |
698 | 673 | ||
699 | .. note:: | 674 | .. note:: |
700 | 675 | ||
701 | Separate documentation exists for the BitBake tool. See the | 676 | Documentation for the BitBake tool is available separately. See the |
702 | BitBake User Manual | 677 | :doc:`BitBake User Manual <bitbake:index>` |
703 | for reference material on BitBake. | 678 | for reference material on BitBake. |
704 | 679 | ||
705 | Source Fetching | 680 | Source Fetching |
@@ -709,37 +684,30 @@ The first stages of building a recipe are to fetch and unpack the source | |||
709 | code: | 684 | code: |
710 | 685 | ||
711 | .. image:: figures/source-fetching.png | 686 | .. image:: figures/source-fetching.png |
712 | :align: center | 687 | :width: 100% |
713 | 688 | ||
714 | The :ref:`ref-tasks-fetch` and | 689 | The :ref:`ref-tasks-fetch` and :ref:`ref-tasks-unpack` tasks fetch |
715 | :ref:`ref-tasks-unpack` tasks fetch | 690 | the source files and unpack them into the :term:`Build Directory`. |
716 | the source files and unpack them into the | ||
717 | :term:`Build Directory`. | ||
718 | 691 | ||
719 | .. note:: | 692 | .. note:: |
720 | 693 | ||
721 | For every local file (e.g. | 694 | For every local file (e.g. ``file://``) that is part of a recipe's |
722 | file:// | 695 | :term:`SRC_URI` statement, the OpenEmbedded build system takes a |
723 | ) that is part of a recipe's | 696 | checksum of the file for the recipe and inserts the checksum into |
724 | SRC_URI | 697 | the signature for the :ref:`ref-tasks-fetch` task. If any local |
725 | statement, the OpenEmbedded build system takes a checksum of the file | 698 | file has been modified, the :ref:`ref-tasks-fetch` task and all |
726 | for the recipe and inserts the checksum into the signature for the | 699 | tasks that depend on it are re-executed. |
727 | do_fetch | 700 | |
728 | task. If any local file has been modified, the | 701 | By default, everything is accomplished in the :term:`Build Directory`, which has |
729 | do_fetch | 702 | a defined structure. For additional general information on the |
730 | task and all tasks that depend on it are re-executed. | 703 | :term:`Build Directory`, see the ":ref:`structure-core-build`" section in |
731 | |||
732 | By default, everything is accomplished in the Build Directory, which has | ||
733 | a defined structure. For additional general information on the Build | ||
734 | Directory, see the ":ref:`structure-core-build`" section in | ||
735 | the Yocto Project Reference Manual. | 704 | the Yocto Project Reference Manual. |
736 | 705 | ||
737 | Each recipe has an area in the Build Directory where the unpacked source | 706 | Each recipe has an area in the :term:`Build Directory` where the unpacked |
738 | code resides. The :term:`S` variable points | 707 | source code resides. The :term:`S` variable points to this area for a recipe's |
739 | to this area for a recipe's unpacked source code. The name of that | 708 | unpacked source code. The name of that directory for any given recipe is |
740 | directory for any given recipe is defined from several different | 709 | defined from several different variables. The preceding figure and the |
741 | variables. The preceding figure and the following list describe the | 710 | following list describe the :term:`Build Directory`'s hierarchy: |
742 | Build Directory's hierarchy: | ||
743 | 711 | ||
744 | - :term:`TMPDIR`: The base directory | 712 | - :term:`TMPDIR`: The base directory |
745 | where the OpenEmbedded build system performs all its work during the | 713 | where the OpenEmbedded build system performs all its work during the |
@@ -749,7 +717,7 @@ Build Directory's hierarchy: | |||
749 | architecture of the built package or packages. Depending on the | 717 | architecture of the built package or packages. Depending on the |
750 | eventual destination of the package or packages (i.e. machine | 718 | eventual destination of the package or packages (i.e. machine |
751 | architecture, :term:`Build Host`, SDK, or | 719 | architecture, :term:`Build Host`, SDK, or |
752 | specific machine), ``PACKAGE_ARCH`` varies. See the variable's | 720 | specific machine), :term:`PACKAGE_ARCH` varies. See the variable's |
753 | description for details. | 721 | description for details. |
754 | 722 | ||
755 | - :term:`TARGET_OS`: The operating | 723 | - :term:`TARGET_OS`: The operating |
@@ -758,7 +726,7 @@ Build Directory's hierarchy: | |||
758 | 726 | ||
759 | - :term:`PN`: The name of the recipe used | 727 | - :term:`PN`: The name of the recipe used |
760 | to build the package. This variable can have multiple meanings. | 728 | to build the package. This variable can have multiple meanings. |
761 | However, when used in the context of input files, ``PN`` represents | 729 | However, when used in the context of input files, :term:`PN` represents |
762 | the name of the recipe. | 730 | the name of the recipe. |
763 | 731 | ||
764 | - :term:`WORKDIR`: The location | 732 | - :term:`WORKDIR`: The location |
@@ -775,20 +743,18 @@ Build Directory's hierarchy: | |||
775 | files for a given recipe. | 743 | files for a given recipe. |
776 | 744 | ||
777 | - :term:`BPN`: The name of the recipe | 745 | - :term:`BPN`: The name of the recipe |
778 | used to build the package. The ``BPN`` variable is a version of | 746 | used to build the package. The :term:`BPN` variable is a version of |
779 | the ``PN`` variable but with common prefixes and suffixes removed. | 747 | the :term:`PN` variable but with common prefixes and suffixes removed. |
780 | 748 | ||
781 | - :term:`PV`: The version of the | 749 | - :term:`PV`: The version of the |
782 | recipe used to build the package. | 750 | recipe used to build the package. |
783 | 751 | ||
784 | .. note:: | 752 | .. note:: |
785 | 753 | ||
786 | In the previous figure, notice that two sample hierarchies exist: one | 754 | In the previous figure, notice that there are two sample hierarchies: |
787 | based on package architecture (i.e. | 755 | one based on package architecture (i.e. :term:`PACKAGE_ARCH`) |
788 | PACKAGE_ARCH | 756 | and one based on a machine (i.e. :term:`MACHINE`). |
789 | ) and one based on a machine (i.e. | 757 | The underlying structures are identical. The differentiator being |
790 | MACHINE | ||
791 | ). The underlying structures are identical. The differentiator being | ||
792 | what the OpenEmbedded build system is using as a build target (e.g. | 758 | what the OpenEmbedded build system is using as a build target (e.g. |
793 | general architecture, a build host, an SDK, or a specific machine). | 759 | general architecture, a build host, an SDK, or a specific machine). |
794 | 760 | ||
@@ -799,7 +765,7 @@ Once source code is fetched and unpacked, BitBake locates patch files | |||
799 | and applies them to the source files: | 765 | and applies them to the source files: |
800 | 766 | ||
801 | .. image:: figures/patching.png | 767 | .. image:: figures/patching.png |
802 | :align: center | 768 | :width: 100% |
803 | 769 | ||
804 | The :ref:`ref-tasks-patch` task uses a | 770 | The :ref:`ref-tasks-patch` task uses a |
805 | recipe's :term:`SRC_URI` statements | 771 | recipe's :term:`SRC_URI` statements |
@@ -807,23 +773,23 @@ and the :term:`FILESPATH` variable | |||
807 | to locate applicable patch files. | 773 | to locate applicable patch files. |
808 | 774 | ||
809 | Default processing for patch files assumes the files have either | 775 | Default processing for patch files assumes the files have either |
810 | ``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters | 776 | ``*.patch`` or ``*.diff`` file types. You can use :term:`SRC_URI` parameters |
811 | to change the way the build system recognizes patch files. See the | 777 | to change the way the build system recognizes patch files. See the |
812 | :ref:`ref-tasks-patch` task for more | 778 | :ref:`ref-tasks-patch` task for more |
813 | information. | 779 | information. |
814 | 780 | ||
815 | BitBake finds and applies multiple patches for a single recipe in the | 781 | BitBake finds and applies multiple patches for a single recipe in the |
816 | order in which it locates the patches. The ``FILESPATH`` variable | 782 | order in which it locates the patches. The :term:`FILESPATH` variable |
817 | defines the default set of directories that the build system uses to | 783 | defines the default set of directories that the build system uses to |
818 | search for patch files. Once found, patches are applied to the recipe's | 784 | search for patch files. Once found, patches are applied to the recipe's |
819 | source files, which are located in the | 785 | source files, which are located in the |
820 | :term:`S` directory. | 786 | :term:`S` directory. |
821 | 787 | ||
822 | For more information on how the source directories are created, see the | 788 | For more information on how the source directories are created, see the |
823 | "`Source Fetching <#source-fetching-dev-environment>`__" section. For | 789 | ":ref:`overview-manual/concepts:source fetching`" section. For |
824 | more information on how to create patches and how the build system | 790 | more information on how to create patches and how the build system |
825 | processes patches, see the | 791 | processes patches, see the |
826 | ":ref:`dev-manual/common-tasks:patching code`" | 792 | ":ref:`dev-manual/new-recipe:patching code`" |
827 | section in the | 793 | section in the |
828 | Yocto Project Development Tasks Manual. You can also see the | 794 | Yocto Project Development Tasks Manual. You can also see the |
829 | ":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" | 795 | ":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" |
@@ -840,7 +806,7 @@ compile the source code. Once compilation occurs, the files are copied | |||
840 | to a holding area (staged) in preparation for packaging: | 806 | to a holding area (staged) in preparation for packaging: |
841 | 807 | ||
842 | .. image:: figures/configuration-compile-autoreconf.png | 808 | .. image:: figures/configuration-compile-autoreconf.png |
843 | :align: center | 809 | :width: 100% |
844 | 810 | ||
845 | This step in the build process consists of the following tasks: | 811 | This step in the build process consists of the following tasks: |
846 | 812 | ||
@@ -866,27 +832,25 @@ This step in the build process consists of the following tasks: | |||
866 | are specific to configurations for the source code being built by the | 832 | are specific to configurations for the source code being built by the |
867 | recipe. | 833 | recipe. |
868 | 834 | ||
869 | If you are using the | 835 | If you are using the :ref:`ref-classes-autotools` class, |
870 | :ref:`autotools <ref-classes-autotools>` class, | ||
871 | you can add additional configuration options by using the | 836 | you can add additional configuration options by using the |
872 | :term:`EXTRA_OECONF` or | 837 | :term:`EXTRA_OECONF` or |
873 | :term:`PACKAGECONFIG_CONFARGS` | 838 | :term:`PACKAGECONFIG_CONFARGS` |
874 | variables. For information on how this variable works within that | 839 | variables. For information on how this variable works within that |
875 | class, see the | 840 | class, see the :ref:`ref-classes-autotools` class |
876 | :ref:`autotools <ref-classes-autotools>` class | 841 | :yocto_git:`here </poky/tree/meta/classes-recipe/autotools.bbclass>`. |
877 | :yocto_git:`here </poky/tree/meta/classes/autotools.bbclass>`. | ||
878 | 842 | ||
879 | - *do_compile*: Once a configuration task has been satisfied, | 843 | - *do_compile*: Once a configuration task has been satisfied, |
880 | BitBake compiles the source using the | 844 | BitBake compiles the source using the |
881 | :ref:`ref-tasks-compile` task. | 845 | :ref:`ref-tasks-compile` task. |
882 | Compilation occurs in the directory pointed to by the | 846 | Compilation occurs in the directory pointed to by the |
883 | :term:`B` variable. Realize that the | 847 | :term:`B` variable. Realize that the |
884 | ``B`` directory is, by default, the same as the | 848 | :term:`B` directory is, by default, the same as the |
885 | :term:`S` directory. | 849 | :term:`S` directory. |
886 | 850 | ||
887 | - *do_install*: After compilation completes, BitBake executes the | 851 | - *do_install*: After compilation completes, BitBake executes the |
888 | :ref:`ref-tasks-install` task. | 852 | :ref:`ref-tasks-install` task. |
889 | This task copies files from the ``B`` directory and places them in a | 853 | This task copies files from the :term:`B` directory and places them in a |
890 | holding area pointed to by the :term:`D` | 854 | holding area pointed to by the :term:`D` |
891 | variable. Packaging occurs later using files from this holding | 855 | variable. Packaging occurs later using files from this holding |
892 | directory. | 856 | directory. |
@@ -898,7 +862,7 @@ After source code is configured, compiled, and staged, the build system | |||
898 | analyzes the results and splits the output into packages: | 862 | analyzes the results and splits the output into packages: |
899 | 863 | ||
900 | .. image:: figures/analysis-for-package-splitting.png | 864 | .. image:: figures/analysis-for-package-splitting.png |
901 | :align: center | 865 | :width: 100% |
902 | 866 | ||
903 | The :ref:`ref-tasks-package` and | 867 | The :ref:`ref-tasks-package` and |
904 | :ref:`ref-tasks-packagedata` | 868 | :ref:`ref-tasks-packagedata` |
@@ -909,7 +873,7 @@ following as well as other items: splitting out debugging symbols, | |||
909 | looking at shared library dependencies between packages, and looking at | 873 | looking at shared library dependencies between packages, and looking at |
910 | package relationships. | 874 | package relationships. |
911 | 875 | ||
912 | The ``do_packagedata`` task creates package metadata based on the | 876 | The :ref:`ref-tasks-packagedata` task creates package metadata based on the |
913 | analysis such that the build system can generate the final packages. The | 877 | analysis such that the build system can generate the final packages. The |
914 | :ref:`ref-tasks-populate_sysroot` | 878 | :ref:`ref-tasks-populate_sysroot` |
915 | task stages (copies) a subset of the files installed by the | 879 | task stages (copies) a subset of the files installed by the |
@@ -922,7 +886,7 @@ the analysis and package splitting process use several areas: | |||
922 | individual packages. | 886 | individual packages. |
923 | 887 | ||
924 | - :term:`PKGDESTWORK`: A | 888 | - :term:`PKGDESTWORK`: A |
925 | temporary work area (i.e. ``pkgdata``) used by the ``do_package`` | 889 | temporary work area (i.e. ``pkgdata``) used by the :ref:`ref-tasks-package` |
926 | task to save package metadata. | 890 | task to save package metadata. |
927 | 891 | ||
928 | - :term:`PKGDEST`: The parent | 892 | - :term:`PKGDEST`: The parent |
@@ -932,7 +896,7 @@ the analysis and package splitting process use several areas: | |||
932 | - :term:`PKGDATA_DIR`: A shared, | 896 | - :term:`PKGDATA_DIR`: A shared, |
933 | global-state directory that holds packaging metadata generated during | 897 | global-state directory that holds packaging metadata generated during |
934 | the packaging process. The packaging process copies metadata from | 898 | the packaging process. The packaging process copies metadata from |
935 | ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally | 899 | :term:`PKGDESTWORK` to the :term:`PKGDATA_DIR` area where it becomes globally |
936 | available. | 900 | available. |
937 | 901 | ||
938 | - :term:`STAGING_DIR_HOST`: | 902 | - :term:`STAGING_DIR_HOST`: |
@@ -946,25 +910,24 @@ the analysis and package splitting process use several areas: | |||
946 | - :term:`STAGING_DIR_TARGET`: | 910 | - :term:`STAGING_DIR_TARGET`: |
947 | The path for the sysroot used when a component that is built to | 911 | The path for the sysroot used when a component that is built to |
948 | execute on a system and it generates code for yet another machine | 912 | execute on a system and it generates code for yet another machine |
949 | (e.g. cross-canadian recipes). | 913 | (e.g. :ref:`ref-classes-cross-canadian` recipes). |
950 | 914 | ||
951 | The :term:`FILES` variable defines the | 915 | The :term:`FILES` variable defines the |
952 | files that go into each package in | 916 | files that go into each package in |
953 | :term:`PACKAGES`. If you want | 917 | :term:`PACKAGES`. If you want |
954 | details on how this is accomplished, you can look at | 918 | details on how this is accomplished, you can look at |
955 | :yocto_git:`package.bbclass </poky/tree/meta/classes/package.bbclass>`. | 919 | :yocto_git:`package.bbclass </poky/tree/meta/classes-global/package.bbclass>`. |
956 | 920 | ||
957 | Depending on the type of packages being created (RPM, DEB, or IPK), the | 921 | Depending on the type of packages being created (RPM, DEB, or IPK), the |
958 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | 922 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` |
959 | task creates the actual packages and places them in the Package Feed | 923 | task creates the actual packages and places them in the Package Feed |
960 | area, which is ``${TMPDIR}/deploy``. You can see the "`Package | 924 | area, which is ``${TMPDIR}/deploy``. You can see the |
961 | Feeds <#package-feeds-dev-environment>`__" section for more detail on | 925 | ":ref:`overview-manual/concepts:package feeds`" section for more detail on |
962 | that part of the build process. | 926 | that part of the build process. |
963 | 927 | ||
964 | .. note:: | 928 | .. note:: |
965 | 929 | ||
966 | Support for creating feeds directly from the | 930 | Support for creating feeds directly from the ``deploy/*`` |
967 | deploy/\* | ||
968 | directories does not exist. Creating such feeds usually requires some | 931 | directories does not exist. Creating such feeds usually requires some |
969 | kind of feed maintenance mechanism that would upload the new packages | 932 | kind of feed maintenance mechanism that would upload the new packages |
970 | into an official package feed (e.g. the Ångström distribution). This | 933 | into an official package feed (e.g. the Ångström distribution). This |
@@ -978,7 +941,7 @@ Once packages are split and stored in the Package Feeds area, the build | |||
978 | system uses BitBake to generate the root filesystem image: | 941 | system uses BitBake to generate the root filesystem image: |
979 | 942 | ||
980 | .. image:: figures/image-generation.png | 943 | .. image:: figures/image-generation.png |
981 | :align: center | 944 | :width: 100% |
982 | 945 | ||
983 | The image generation process consists of several stages and depends on | 946 | The image generation process consists of several stages and depends on |
984 | several tasks and variables. The | 947 | several tasks and variables. The |
@@ -1013,7 +976,7 @@ actually install: | |||
1013 | 976 | ||
1014 | With :term:`IMAGE_ROOTFS` | 977 | With :term:`IMAGE_ROOTFS` |
1015 | pointing to the location of the filesystem under construction and the | 978 | pointing to the location of the filesystem under construction and the |
1016 | ``PACKAGE_INSTALL`` variable providing the final list of packages to | 979 | :term:`PACKAGE_INSTALL` variable providing the final list of packages to |
1017 | install, the root file system is created. | 980 | install, the root file system is created. |
1018 | 981 | ||
1019 | Package installation is under control of the package manager (e.g. | 982 | Package installation is under control of the package manager (e.g. |
@@ -1024,29 +987,25 @@ data files are deleted from the root filesystem. As part of the final | |||
1024 | stage of package installation, post installation scripts that are part | 987 | stage of package installation, post installation scripts that are part |
1025 | of the packages are run. Any scripts that fail to run on the build host | 988 | of the packages are run. Any scripts that fail to run on the build host |
1026 | are run on the target when the target system is first booted. If you are | 989 | are run on the target when the target system is first booted. If you are |
1027 | using a | 990 | using a |
1028 | :ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`, | 991 | :ref:`read-only root filesystem <dev-manual/read-only-rootfs:creating a read-only root filesystem>`, |
1029 | all the post installation scripts must succeed on the build host during | 992 | all the post installation scripts must succeed on the build host during |
1030 | the package installation phase since the root filesystem on the target | 993 | the package installation phase since the root filesystem on the target |
1031 | is read-only. | 994 | is read-only. |
1032 | 995 | ||
1033 | The final stages of the ``do_rootfs`` task handle post processing. Post | 996 | The final stages of the :ref:`ref-tasks-rootfs` task handle post processing. Post |
1034 | processing includes creation of a manifest file and optimizations. | 997 | processing includes creation of a manifest file and optimizations. |
1035 | 998 | ||
1036 | The manifest file (``.manifest``) resides in the same directory as the | 999 | The manifest file (``.manifest``) resides in the same directory as the root |
1037 | root filesystem image. This file lists out, line-by-line, the installed | 1000 | filesystem image. This file lists out, line-by-line, the installed packages. |
1038 | packages. The manifest file is useful for the | 1001 | The manifest file is useful for the :ref:`ref-classes-testimage` class, |
1039 | :ref:`testimage <ref-classes-testimage*>` class, | ||
1040 | for example, to determine whether or not to run specific tests. See the | 1002 | for example, to determine whether or not to run specific tests. See the |
1041 | :term:`IMAGE_MANIFEST` | 1003 | :term:`IMAGE_MANIFEST` variable for additional information. |
1042 | variable for additional information. | ||
1043 | 1004 | ||
1044 | Optimizing processes that are run across the image include ``mklibs``, | 1005 | Optimizing processes that are run across the image include ``mklibs`` |
1045 | ``prelink``, and any other post-processing commands as defined by the | 1006 | and any other post-processing commands as defined by the |
1046 | :term:`ROOTFS_POSTPROCESS_COMMAND` | 1007 | :term:`ROOTFS_POSTPROCESS_COMMAND` |
1047 | variable. The ``mklibs`` process optimizes the size of the libraries, | 1008 | variable. The ``mklibs`` process optimizes the size of the libraries. |
1048 | while the ``prelink`` process optimizes the dynamic linking of shared | ||
1049 | libraries to reduce start up time of executables. | ||
1050 | 1009 | ||
1051 | After the root filesystem is built, processing begins on the image | 1010 | After the root filesystem is built, processing begins on the image |
1052 | through the :ref:`ref-tasks-image` | 1011 | through the :ref:`ref-tasks-image` |
@@ -1056,25 +1015,23 @@ the | |||
1056 | variable. This variable specifies a list of functions to call before the | 1015 | variable. This variable specifies a list of functions to call before the |
1057 | build system creates the final image output files. | 1016 | build system creates the final image output files. |
1058 | 1017 | ||
1059 | The build system dynamically creates ``do_image_*`` tasks as needed, | 1018 | The build system dynamically creates :ref:`do_image_* <ref-tasks-image>` tasks as needed, |
1060 | based on the image types specified in the | 1019 | based on the image types specified in the |
1061 | :term:`IMAGE_FSTYPES` variable. | 1020 | :term:`IMAGE_FSTYPES` variable. |
1062 | The process turns everything into an image file or a set of image files | 1021 | The process turns everything into an image file or a set of image files |
1063 | and can compress the root filesystem image to reduce the overall size of | 1022 | and can compress the root filesystem image to reduce the overall size of |
1064 | the image. The formats used for the root filesystem depend on the | 1023 | the image. The formats used for the root filesystem depend on the |
1065 | ``IMAGE_FSTYPES`` variable. Compression depends on whether the formats | 1024 | :term:`IMAGE_FSTYPES` variable. Compression depends on whether the formats |
1066 | support compression. | 1025 | support compression. |
1067 | 1026 | ||
1068 | As an example, a dynamically created task when creating a particular | 1027 | As an example, a dynamically created task when creating a particular |
1069 | image type would take the following form: | 1028 | image type would take the following form:: |
1070 | :: | ||
1071 | 1029 | ||
1072 | do_image_type | 1030 | do_image_type |
1073 | 1031 | ||
1074 | So, if the type | 1032 | So, if the type |
1075 | as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically | 1033 | as specified by the :term:`IMAGE_FSTYPES` were ``ext4``, the dynamically |
1076 | generated task would be as follows: | 1034 | generated task would be as follows:: |
1077 | :: | ||
1078 | 1035 | ||
1079 | do_image_ext4 | 1036 | do_image_ext4 |
1080 | 1037 | ||
@@ -1100,14 +1057,14 @@ Development Kit (SDK) installer scripts for both the standard SDK and | |||
1100 | the extensible SDK (eSDK): | 1057 | the extensible SDK (eSDK): |
1101 | 1058 | ||
1102 | .. image:: figures/sdk-generation.png | 1059 | .. image:: figures/sdk-generation.png |
1103 | :align: center | 1060 | :width: 100% |
1104 | 1061 | ||
1105 | .. note:: | 1062 | .. note:: |
1106 | 1063 | ||
1107 | For more information on the cross-development toolchain generation, | 1064 | For more information on the cross-development toolchain generation, |
1108 | see the ":ref:`overview-manual/concepts:cross-development toolchain generation`" | 1065 | see the ":ref:`overview-manual/concepts:cross-development toolchain generation`" |
1109 | section. For information on advantages gained when building a | 1066 | section. For information on advantages gained when building a |
1110 | cross-development toolchain using the do_populate_sdk task, see the | 1067 | cross-development toolchain using the :ref:`ref-tasks-populate_sdk` task, see the |
1111 | ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in | 1068 | ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in |
1112 | the Yocto Project Application Development and the Extensible Software | 1069 | the Yocto Project Application Development and the Extensible Software |
1113 | Development Kit (eSDK) manual. | 1070 | Development Kit (eSDK) manual. |
@@ -1119,16 +1076,16 @@ and | |||
1119 | :ref:`ref-tasks-populate_sdk_ext` | 1076 | :ref:`ref-tasks-populate_sdk_ext` |
1120 | tasks use these key variables to help create the list of packages to | 1077 | tasks use these key variables to help create the list of packages to |
1121 | actually install. For information on the variables listed in the figure, | 1078 | actually install. For information on the variables listed in the figure, |
1122 | see the "`Application Development SDK <#sdk-dev-environment>`__" | 1079 | see the ":ref:`overview-manual/concepts:application development sdk`" |
1123 | section. | 1080 | section. |
1124 | 1081 | ||
1125 | The ``do_populate_sdk`` task helps create the standard SDK and handles | 1082 | The :ref:`ref-tasks-populate_sdk` task helps create the standard SDK and handles |
1126 | two parts: a target part and a host part. The target part is the part | 1083 | two parts: a target part and a host part. The target part is the part |
1127 | built for the target hardware and includes libraries and headers. The | 1084 | built for the target hardware and includes libraries and headers. The |
1128 | host part is the part of the SDK that runs on the | 1085 | host part is the part of the SDK that runs on the |
1129 | :term:`SDKMACHINE`. | 1086 | :term:`SDKMACHINE`. |
1130 | 1087 | ||
1131 | The ``do_populate_sdk_ext`` task helps create the extensible SDK and | 1088 | The :ref:`ref-tasks-populate_sdk_ext` task helps create the extensible SDK and |
1132 | handles host and target parts differently than its counter part does for | 1089 | handles host and target parts differently than its counter part does for |
1133 | the standard SDK. For the extensible SDK, the task encapsulates the | 1090 | the standard SDK. For the extensible SDK, the task encapsulates the |
1134 | build system, which includes everything needed (host and target) for the | 1091 | build system, which includes everything needed (host and target) for the |
@@ -1147,28 +1104,27 @@ For each task that completes successfully, BitBake writes a stamp file | |||
1147 | into the :term:`STAMPS_DIR` | 1104 | into the :term:`STAMPS_DIR` |
1148 | directory. The beginning of the stamp file's filename is determined by | 1105 | directory. The beginning of the stamp file's filename is determined by |
1149 | the :term:`STAMP` variable, and the end | 1106 | the :term:`STAMP` variable, and the end |
1150 | of the name consists of the task's name and current `input | 1107 | of the name consists of the task's name and current :ref:`input |
1151 | checksum <#overview-checksums>`__. | 1108 | checksum <overview-manual/concepts:checksums (signatures)>`. |
1152 | 1109 | ||
1153 | .. note:: | 1110 | .. note:: |
1154 | 1111 | ||
1155 | This naming scheme assumes that | 1112 | This naming scheme assumes that :term:`BB_SIGNATURE_HANDLER` |
1156 | BB_SIGNATURE_HANDLER | ||
1157 | is "OEBasicHash", which is almost always the case in current | 1113 | is "OEBasicHash", which is almost always the case in current |
1158 | OpenEmbedded. | 1114 | OpenEmbedded. |
1159 | 1115 | ||
1160 | To determine if a task needs to be rerun, BitBake checks if a stamp file | 1116 | To determine if a task needs to be rerun, BitBake checks if a stamp file |
1161 | with a matching input checksum exists for the task. If such a stamp file | 1117 | with a matching input checksum exists for the task. In this case, |
1162 | exists, the task's output is assumed to exist and still be valid. If the | 1118 | the task's output is assumed to exist and still be valid. Otherwise, |
1163 | file does not exist, the task is rerun. | 1119 | the task is rerun. |
1164 | 1120 | ||
1165 | .. note:: | 1121 | .. note:: |
1166 | 1122 | ||
1167 | The stamp mechanism is more general than the shared state (sstate) | 1123 | The stamp mechanism is more general than the shared state (sstate) |
1168 | cache mechanism described in the "`Setscene Tasks and Shared | 1124 | cache mechanism described in the |
1169 | State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids | 1125 | ":ref:`overview-manual/concepts:setscene tasks and shared state`" section. |
1170 | rerunning any task that has a valid stamp file, not just tasks that | 1126 | BitBake avoids rerunning any task that has a valid stamp file, not just |
1171 | can be accelerated through the sstate cache. | 1127 | tasks that can be accelerated through the sstate cache. |
1172 | 1128 | ||
1173 | However, you should realize that stamp files only serve as a marker | 1129 | However, you should realize that stamp files only serve as a marker |
1174 | that some work has been done and that these files do not record task | 1130 | that some work has been done and that these files do not record task |
@@ -1178,17 +1134,17 @@ file does not exist, the task is rerun. | |||
1178 | the sstate cache mechanism adds is a way to cache task output that | 1134 | the sstate cache mechanism adds is a way to cache task output that |
1179 | can then be shared between build machines. | 1135 | can then be shared between build machines. |
1180 | 1136 | ||
1181 | Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing | 1137 | Since :term:`STAMPS_DIR` is usually a subdirectory of :term:`TMPDIR`, removing |
1182 | ``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will | 1138 | :term:`TMPDIR` will also remove :term:`STAMPS_DIR`, which means tasks will |
1183 | properly be rerun to repopulate ``TMPDIR``. | 1139 | properly be rerun to repopulate :term:`TMPDIR`. |
1184 | 1140 | ||
1185 | If you want some task to always be considered "out of date", you can | 1141 | If you want some task to always be considered "out of date", you can |
1186 | mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>` | 1142 | mark it with the :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>` |
1187 | varflag. If some other task depends on such a task, then that task will | 1143 | varflag. If some other task depends on such a task, then that task will |
1188 | also always be considered out of date, which might not be what you want. | 1144 | also always be considered out of date, which might not be what you want. |
1189 | 1145 | ||
1190 | For details on how to view information about a task's signature, see the | 1146 | For details on how to view information about a task's signature, see the |
1191 | ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" | 1147 | ":ref:`dev-manual/debugging:viewing task variable dependencies`" |
1192 | section in the Yocto Project Development Tasks Manual. | 1148 | section in the Yocto Project Development Tasks Manual. |
1193 | 1149 | ||
1194 | Setscene Tasks and Shared State | 1150 | Setscene Tasks and Shared State |
@@ -1207,7 +1163,7 @@ usually made available in the form of a shared state (sstate) cache. | |||
1207 | :term:`SSTATE_MIRRORS` | 1163 | :term:`SSTATE_MIRRORS` |
1208 | variables. | 1164 | variables. |
1209 | 1165 | ||
1210 | The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a | 1166 | The idea of a setscene task (i.e ``do_taskname_setscene``) is a |
1211 | version of the task where instead of building something, BitBake can | 1167 | version of the task where instead of building something, BitBake can |
1212 | skip to the end result and simply place a set of files into specific | 1168 | skip to the end result and simply place a set of files into specific |
1213 | locations as needed. In some cases, it makes sense to have a setscene | 1169 | locations as needed. In some cases, it makes sense to have a setscene |
@@ -1220,7 +1176,7 @@ the work involved would be equal to or greater than the underlying task. | |||
1220 | 1176 | ||
1221 | In the build system, the common tasks that have setscene variants are | 1177 | In the build system, the common tasks that have setscene variants are |
1222 | :ref:`ref-tasks-package`, | 1178 | :ref:`ref-tasks-package`, |
1223 | ``do_package_write_*``, | 1179 | :ref:`do_package_write_* <ref-tasks-package_write_deb>`, |
1224 | :ref:`ref-tasks-deploy`, | 1180 | :ref:`ref-tasks-deploy`, |
1225 | :ref:`ref-tasks-packagedata`, and | 1181 | :ref:`ref-tasks-packagedata`, and |
1226 | :ref:`ref-tasks-populate_sysroot`. | 1182 | :ref:`ref-tasks-populate_sysroot`. |
@@ -1230,20 +1186,20 @@ end result. | |||
1230 | The build system has knowledge of the relationship between these tasks | 1186 | The build system has knowledge of the relationship between these tasks |
1231 | and other preceding tasks. For example, if BitBake runs | 1187 | and other preceding tasks. For example, if BitBake runs |
1232 | ``do_populate_sysroot_setscene`` for something, it does not make sense | 1188 | ``do_populate_sysroot_setscene`` for something, it does not make sense |
1233 | to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``, | 1189 | to run any of the :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`, |
1234 | ``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if | 1190 | :ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, and :ref:`ref-tasks-install` tasks. However, if |
1235 | ``do_package`` needs to be run, BitBake needs to run those other tasks. | 1191 | :ref:`ref-tasks-package` needs to be run, BitBake needs to run those other tasks. |
1236 | 1192 | ||
1237 | It becomes more complicated if everything can come from an sstate cache | 1193 | It becomes more complicated if everything can come from an sstate cache |
1238 | because some objects are simply not required at all. For example, you do | 1194 | because some objects are simply not required at all. For example, you do |
1239 | not need a compiler or native tools, such as quilt, if nothing exists to | 1195 | not need a compiler or native tools, such as quilt, if there isn't anything |
1240 | compile or patch. If the ``do_package_write_*`` packages are available | 1196 | to compile or patch. If the :ref:`do_package_write_* <ref-tasks-package_write_deb>` packages are available |
1241 | from sstate, BitBake does not need the ``do_package`` task data. | 1197 | from sstate, BitBake does not need the :ref:`ref-tasks-package` task data. |
1242 | 1198 | ||
1243 | To handle all these complexities, BitBake runs in two phases. The first | 1199 | To handle all these complexities, BitBake runs in two phases. The first |
1244 | is the "setscene" stage. During this stage, BitBake first checks the | 1200 | is the "setscene" stage. During this stage, BitBake first checks the |
1245 | sstate cache for any targets it is planning to build. BitBake does a | 1201 | sstate cache for any targets it is planning to build. BitBake does a |
1246 | fast check to see if the object exists rather than a complete download. | 1202 | fast check to see if the object exists rather than doing a complete download. |
1247 | If nothing exists, the second phase, which is the setscene stage, | 1203 | If nothing exists, the second phase, which is the setscene stage, |
1248 | completes and the main build proceeds. | 1204 | completes and the main build proceeds. |
1249 | 1205 | ||
@@ -1257,11 +1213,9 @@ download or setscene task fails, the build system then tries to install | |||
1257 | dependencies, such as the compiler, from the cache. | 1213 | dependencies, such as the compiler, from the cache. |
1258 | 1214 | ||
1259 | The availability of objects in the sstate cache is handled by the | 1215 | The availability of objects in the sstate cache is handled by the |
1260 | function specified by the | 1216 | function specified by the :term:`BB_HASHCHECK_FUNCTION` |
1261 | :term:`bitbake:BB_HASHCHECK_FUNCTION` | ||
1262 | variable and returns a list of available objects. The function specified | 1217 | variable and returns a list of available objects. The function specified |
1263 | by the | 1218 | by the :term:`BB_SETSCENE_DEPVALID` |
1264 | :term:`bitbake:BB_SETSCENE_DEPVALID` | ||
1265 | variable is the function that determines whether a given dependency | 1219 | variable is the function that determines whether a given dependency |
1266 | needs to be followed, and whether for any given relationship the | 1220 | needs to be followed, and whether for any given relationship the |
1267 | function needs to be passed. The function returns a True or False value. | 1221 | function needs to be passed. The function returns a True or False value. |
@@ -1271,12 +1225,14 @@ Images | |||
1271 | 1225 | ||
1272 | The images produced by the build system are compressed forms of the root | 1226 | The images produced by the build system are compressed forms of the root |
1273 | filesystem and are ready to boot on a target device. You can see from | 1227 | filesystem and are ready to boot on a target device. You can see from |
1274 | the `general workflow figure <#general-workflow-figure>`__ that BitBake | 1228 | the :ref:`general workflow figure |
1229 | <overview-manual/concepts:openembedded build system concepts>` that BitBake | ||
1275 | output, in part, consists of images. This section takes a closer look at | 1230 | output, in part, consists of images. This section takes a closer look at |
1276 | this output: | 1231 | this output: |
1277 | 1232 | ||
1278 | .. image:: figures/images.png | 1233 | .. image:: figures/images.png |
1279 | :align: center | 1234 | :align: center |
1235 | :width: 75% | ||
1280 | 1236 | ||
1281 | .. note:: | 1237 | .. note:: |
1282 | 1238 | ||
@@ -1284,15 +1240,12 @@ this output: | |||
1284 | ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference | 1240 | ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference |
1285 | Manual. | 1241 | Manual. |
1286 | 1242 | ||
1287 | The build process writes images out to the :term:`Build Directory` | 1243 | The build process writes images out to the :term:`Build Directory` inside |
1288 | inside the | 1244 | the ``tmp/deploy/images/machine/`` folder as shown in the figure. This |
1289 | ``tmp/deploy/images/machine/`` folder as shown in the figure. This | ||
1290 | folder contains any files expected to be loaded on the target device. | 1245 | folder contains any files expected to be loaded on the target device. |
1291 | The :term:`DEPLOY_DIR` variable | 1246 | The :term:`DEPLOY_DIR` variable points to the ``deploy`` directory, while the |
1292 | points to the ``deploy`` directory, while the | 1247 | :term:`DEPLOY_DIR_IMAGE` variable points to the appropriate directory |
1293 | :term:`DEPLOY_DIR_IMAGE` | 1248 | containing images for the current configuration. |
1294 | variable points to the appropriate directory containing images for the | ||
1295 | current configuration. | ||
1296 | 1249 | ||
1297 | - kernel-image: A kernel binary file. The | 1250 | - kernel-image: A kernel binary file. The |
1298 | :term:`KERNEL_IMAGETYPE` | 1251 | :term:`KERNEL_IMAGETYPE` |
@@ -1327,7 +1280,8 @@ current configuration. | |||
1327 | Application Development SDK | 1280 | Application Development SDK |
1328 | --------------------------- | 1281 | --------------------------- |
1329 | 1282 | ||
1330 | In the `general workflow figure <#general-workflow-figure>`__, the | 1283 | In the :ref:`general workflow figure |
1284 | <overview-manual/concepts:openembedded build system concepts>`, the | ||
1331 | output labeled "Application Development SDK" represents an SDK. The SDK | 1285 | output labeled "Application Development SDK" represents an SDK. The SDK |
1332 | generation process differs depending on whether you build an extensible | 1286 | generation process differs depending on whether you build an extensible |
1333 | SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK | 1287 | SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK |
@@ -1335,7 +1289,7 @@ SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK | |||
1335 | closer look at this output: | 1289 | closer look at this output: |
1336 | 1290 | ||
1337 | .. image:: figures/sdk.png | 1291 | .. image:: figures/sdk.png |
1338 | :align: center | 1292 | :width: 100% |
1339 | 1293 | ||
1340 | The specific form of this output is a set of files that includes a | 1294 | The specific form of this output is a set of files that includes a |
1341 | self-extracting SDK installer (``*.sh``), host and target manifest | 1295 | self-extracting SDK installer (``*.sh``), host and target manifest |
@@ -1357,18 +1311,17 @@ can initialize the environment before using the tools. | |||
1357 | your own SDK installer. | 1311 | your own SDK installer. |
1358 | 1312 | ||
1359 | - For background information on cross-development toolchains in the | 1313 | - For background information on cross-development toolchains in the |
1360 | Yocto Project development environment, see the "`Cross-Development | 1314 | Yocto Project development environment, see the |
1361 | Toolchain Generation <#cross-development-toolchain-generation>`__" | 1315 | ":ref:`overview-manual/concepts:cross-development toolchain generation`" |
1362 | section. | 1316 | section. |
1363 | 1317 | ||
1364 | - For information on setting up a cross-development environment, see | 1318 | - For information on setting up a cross-development environment, see |
1365 | the :doc:`/sdk-manual/index` manual. | 1319 | the :doc:`/sdk-manual/index` manual. |
1366 | 1320 | ||
1367 | All the output files for an SDK are written to the ``deploy/sdk`` folder | 1321 | All the output files for an SDK are written to the ``deploy/sdk`` folder |
1368 | inside the :term:`Build Directory` as | 1322 | inside the :term:`Build Directory` as shown in the previous figure. Depending |
1369 | shown in the previous figure. Depending on the type of SDK, several | 1323 | on the type of SDK, there are several variables to configure these files. |
1370 | variables exist that help configure these files. The following list | 1324 | The variables associated with an extensible SDK are: |
1371 | shows the variables associated with an extensible SDK: | ||
1372 | 1325 | ||
1373 | - :term:`DEPLOY_DIR`: Points to | 1326 | - :term:`DEPLOY_DIR`: Points to |
1374 | the ``deploy`` directory. | 1327 | the ``deploy`` directory. |
@@ -1386,15 +1339,15 @@ shows the variables associated with an extensible SDK: | |||
1386 | Specifies whether or not the toolchain is included when building the | 1339 | Specifies whether or not the toolchain is included when building the |
1387 | extensible SDK. | 1340 | extensible SDK. |
1388 | 1341 | ||
1389 | - :term:`SDK_LOCAL_CONF_WHITELIST`: | 1342 | - :term:`ESDK_LOCALCONF_ALLOW`: |
1390 | A list of variables allowed through from the build system | 1343 | A list of variables allowed through from the build system |
1391 | configuration into the extensible SDK configuration. | 1344 | configuration into the extensible SDK configuration. |
1392 | 1345 | ||
1393 | - :term:`SDK_LOCAL_CONF_BLACKLIST`: | 1346 | - :term:`ESDK_LOCALCONF_REMOVE`: |
1394 | A list of variables not allowed through from the build system | 1347 | A list of variables not allowed through from the build system |
1395 | configuration into the extensible SDK configuration. | 1348 | configuration into the extensible SDK configuration. |
1396 | 1349 | ||
1397 | - :term:`SDK_INHERIT_BLACKLIST`: | 1350 | - :term:`ESDK_CLASS_INHERIT_DISABLE`: |
1398 | A list of classes to remove from the | 1351 | A list of classes to remove from the |
1399 | :term:`INHERIT` value globally | 1352 | :term:`INHERIT` value globally |
1400 | within the extensible SDK configuration. | 1353 | within the extensible SDK configuration. |
@@ -1413,7 +1366,7 @@ This next list, shows the variables associated with a standard SDK: | |||
1413 | 1366 | ||
1414 | - :term:`TOOLCHAIN_HOST_TASK`: | 1367 | - :term:`TOOLCHAIN_HOST_TASK`: |
1415 | Lists packages that make up the host part of the SDK (i.e. the part | 1368 | Lists packages that make up the host part of the SDK (i.e. the part |
1416 | that runs on the ``SDKMACHINE``). When you use | 1369 | that runs on the :term:`SDKMACHINE`). When you use |
1417 | ``bitbake -c populate_sdk imagename`` to create the SDK, a set of | 1370 | ``bitbake -c populate_sdk imagename`` to create the SDK, a set of |
1418 | default packages apply. This variable allows you to add more | 1371 | default packages apply. This variable allows you to add more |
1419 | packages. | 1372 | packages. |
@@ -1422,7 +1375,7 @@ This next list, shows the variables associated with a standard SDK: | |||
1422 | Lists packages that make up the target part of the SDK (i.e. the part | 1375 | Lists packages that make up the target part of the SDK (i.e. the part |
1423 | built for the target hardware). | 1376 | built for the target hardware). |
1424 | 1377 | ||
1425 | - :term:`SDKPATH`: Defines the | 1378 | - :term:`SDKPATHINSTALL`: Defines the |
1426 | default SDK installation path offered by the installation script. | 1379 | default SDK installation path offered by the installation script. |
1427 | 1380 | ||
1428 | - :term:`SDK_HOST_MANIFEST`: | 1381 | - :term:`SDK_HOST_MANIFEST`: |
@@ -1453,10 +1406,10 @@ The following figure shows a high-level build environment regarding | |||
1453 | toolchain construction and use. | 1406 | toolchain construction and use. |
1454 | 1407 | ||
1455 | .. image:: figures/cross-development-toolchains.png | 1408 | .. image:: figures/cross-development-toolchains.png |
1456 | :align: center | 1409 | :width: 100% |
1457 | 1410 | ||
1458 | Most of the work occurs on the Build Host. This is the machine used to | 1411 | Most of the work occurs on the Build Host. This is the machine used to |
1459 | build images and generally work within the the Yocto Project | 1412 | build images and generally work within the Yocto Project |
1460 | environment. When you run | 1413 | environment. When you run |
1461 | :term:`BitBake` to create an image, the | 1414 | :term:`BitBake` to create an image, the |
1462 | OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a | 1415 | OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a |
@@ -1467,15 +1420,11 @@ cross-compiler that is used internally within BitBake only. | |||
1467 | 1420 | ||
1468 | .. note:: | 1421 | .. note:: |
1469 | 1422 | ||
1470 | The extensible SDK does not use | 1423 | The extensible SDK does not use ``gcc-cross-canadian`` |
1471 | gcc-cross-canadian | ||
1472 | since this SDK ships a copy of the OpenEmbedded build system and the | 1424 | since this SDK ships a copy of the OpenEmbedded build system and the |
1473 | sysroot within it contains | 1425 | sysroot within it contains ``gcc-cross``. |
1474 | gcc-cross | ||
1475 | . | ||
1476 | 1426 | ||
1477 | The chain of events that occurs when the standard toolchain is bootstrapped: | 1427 | The chain of events that occurs when the standard toolchain is bootstrapped:: |
1478 | :: | ||
1479 | 1428 | ||
1480 | binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime | 1429 | binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime |
1481 | 1430 | ||
@@ -1511,12 +1460,11 @@ relocatable SDK used to develop applications. When you run the | |||
1511 | installer, it installs the toolchain, which contains the development | 1460 | installer, it installs the toolchain, which contains the development |
1512 | tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and | 1461 | tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and |
1513 | other ``nativesdk-*`` tools), which are tools native to the SDK (i.e. | 1462 | other ``nativesdk-*`` tools), which are tools native to the SDK (i.e. |
1514 | native to :term:`SDK_ARCH`), you | 1463 | native to :term:`SDK_ARCH`), you need to cross-compile and test your |
1515 | need to cross-compile and test your software. The figure shows the | 1464 | software. The figure shows the commands you use to easily build out |
1516 | commands you use to easily build out this toolchain. This | 1465 | this toolchain. This cross-development toolchain is built to execute on the |
1517 | cross-development toolchain is built to execute on the | 1466 | :term:`SDKMACHINE`, which might or might not be the same machine as |
1518 | :term:`SDKMACHINE`, which might or | 1467 | the Build Host. |
1519 | might not be the same machine as the Build Host. | ||
1520 | 1468 | ||
1521 | .. note:: | 1469 | .. note:: |
1522 | 1470 | ||
@@ -1524,8 +1472,7 @@ might not be the same machine as the Build Host. | |||
1524 | can take advantage of pre-built images that ship with the Yocto | 1472 | can take advantage of pre-built images that ship with the Yocto |
1525 | Project and already contain cross-development toolchain installers. | 1473 | Project and already contain cross-development toolchain installers. |
1526 | 1474 | ||
1527 | Here is the bootstrap process for the relocatable toolchain: | 1475 | Here is the bootstrap process for the relocatable toolchain:: |
1528 | :: | ||
1529 | 1476 | ||
1530 | gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian | 1477 | gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian |
1531 | 1478 | ||
@@ -1579,10 +1526,10 @@ Shared State Cache | |||
1579 | By design, the OpenEmbedded build system builds everything from scratch | 1526 | By design, the OpenEmbedded build system builds everything from scratch |
1580 | unless :term:`BitBake` can determine | 1527 | unless :term:`BitBake` can determine |
1581 | that parts do not need to be rebuilt. Fundamentally, building from | 1528 | that parts do not need to be rebuilt. Fundamentally, building from |
1582 | scratch is attractive as it means all parts are built fresh and no | 1529 | scratch is attractive as it means all parts are built fresh and there is |
1583 | possibility of stale data exists that can cause problems. When | 1530 | no possibility of stale data that can cause problems. When |
1584 | developers hit problems, they typically default back to building from | 1531 | developers hit problems, they typically default back to building from |
1585 | scratch so they have a know state from the start. | 1532 | scratch so they have a known state from the start. |
1586 | 1533 | ||
1587 | Building an image from scratch is both an advantage and a disadvantage | 1534 | Building an image from scratch is both an advantage and a disadvantage |
1588 | to the process. As mentioned in the previous paragraph, building from | 1535 | to the process. As mentioned in the previous paragraph, building from |
@@ -1619,18 +1566,18 @@ them if they are deemed to be valid. | |||
1619 | 1566 | ||
1620 | - The build system does not maintain | 1567 | - The build system does not maintain |
1621 | :term:`PR` information as part of | 1568 | :term:`PR` information as part of |
1622 | the shared state packages. Consequently, considerations exist that | 1569 | the shared state packages. Consequently, there are considerations that |
1623 | affect maintaining shared state feeds. For information on how the | 1570 | affect maintaining shared state feeds. For information on how the |
1624 | build system works with packages and can track incrementing ``PR`` | 1571 | build system works with packages and can track incrementing :term:`PR` |
1625 | information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`" | 1572 | information, see the ":ref:`dev-manual/packages:automatically incrementing a package version number`" |
1626 | section in the Yocto Project Development Tasks Manual. | 1573 | section in the Yocto Project Development Tasks Manual. |
1627 | 1574 | ||
1628 | - The code in the build system that supports incremental builds is | 1575 | - The code in the build system that supports incremental builds is |
1629 | not simple code. For techniques that help you work around issues | 1576 | complex. For techniques that help you work around issues |
1630 | related to shared state code, see the | 1577 | related to shared state code, see the |
1631 | ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`" | 1578 | ":ref:`dev-manual/debugging:viewing metadata used to create the input signature of a shared state task`" |
1632 | and | 1579 | and |
1633 | ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`" | 1580 | ":ref:`dev-manual/debugging:invalidating shared state to force a task to run`" |
1634 | sections both in the Yocto Project Development Tasks Manual. | 1581 | sections both in the Yocto Project Development Tasks Manual. |
1635 | 1582 | ||
1636 | The rest of this section goes into detail about the overall incremental | 1583 | The rest of this section goes into detail about the overall incremental |
@@ -1667,7 +1614,7 @@ you a good idea of when the task's data changes. | |||
1667 | 1614 | ||
1668 | To complicate the problem, there are things that should not be included | 1615 | To complicate the problem, there are things that should not be included |
1669 | in the checksum. First, there is the actual specific build path of a | 1616 | in the checksum. First, there is the actual specific build path of a |
1670 | given task - the :term:`WORKDIR`. It | 1617 | given task --- the :term:`WORKDIR`. It |
1671 | does not matter if the work directory changes because it should not | 1618 | does not matter if the work directory changes because it should not |
1672 | affect the output for target packages. Also, the build process has the | 1619 | affect the output for target packages. Also, the build process has the |
1673 | objective of making native or cross packages relocatable. | 1620 | objective of making native or cross packages relocatable. |
@@ -1678,8 +1625,8 @@ objective of making native or cross packages relocatable. | |||
1678 | build host. However, cross packages generate output for the target | 1625 | build host. However, cross packages generate output for the target |
1679 | architecture. | 1626 | architecture. |
1680 | 1627 | ||
1681 | The checksum therefore needs to exclude ``WORKDIR``. The simplistic | 1628 | The checksum therefore needs to exclude :term:`WORKDIR`. The simplistic |
1682 | approach for excluding the work directory is to set ``WORKDIR`` to some | 1629 | approach for excluding the work directory is to set :term:`WORKDIR` to some |
1683 | fixed value and create the checksum for the "run" script. | 1630 | fixed value and create the checksum for the "run" script. |
1684 | 1631 | ||
1685 | Another problem results from the "run" scripts containing functions that | 1632 | Another problem results from the "run" scripts containing functions that |
@@ -1689,7 +1636,7 @@ used to prune the "run" scripts down to the minimum set, thereby | |||
1689 | alleviating this problem and making the "run" scripts much more readable | 1636 | alleviating this problem and making the "run" scripts much more readable |
1690 | as a bonus. | 1637 | as a bonus. |
1691 | 1638 | ||
1692 | So far, solutions for shell scripts exist. What about Python tasks? The | 1639 | So far, there are solutions for shell scripts. What about Python tasks? The |
1693 | same approach applies even though these tasks are more difficult. The | 1640 | same approach applies even though these tasks are more difficult. The |
1694 | process needs to figure out what variables a Python function accesses | 1641 | process needs to figure out what variables a Python function accesses |
1695 | and what functions it calls. Again, the incremental build solution | 1642 | and what functions it calls. Again, the incremental build solution |
@@ -1697,10 +1644,9 @@ contains code that first figures out the variable and function | |||
1697 | dependencies, and then creates a checksum for the data used as the input | 1644 | dependencies, and then creates a checksum for the data used as the input |
1698 | to the task. | 1645 | to the task. |
1699 | 1646 | ||
1700 | Like the ``WORKDIR`` case, situations exist where dependencies should be | 1647 | Like the :term:`WORKDIR` case, there can be situations where dependencies should be |
1701 | ignored. For these situations, you can instruct the build process to | 1648 | ignored. For these situations, you can instruct the build process to |
1702 | ignore a dependency by using a line like the following: | 1649 | ignore a dependency by using a line like the following:: |
1703 | :: | ||
1704 | 1650 | ||
1705 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | 1651 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" |
1706 | 1652 | ||
@@ -1710,13 +1656,12 @@ reference it. | |||
1710 | 1656 | ||
1711 | Equally, there are cases where you need to add dependencies BitBake is | 1657 | Equally, there are cases where you need to add dependencies BitBake is |
1712 | not able to find. You can accomplish this by using a line like the | 1658 | not able to find. You can accomplish this by using a line like the |
1713 | following: | 1659 | following:: |
1714 | :: | ||
1715 | 1660 | ||
1716 | PACKAGE_ARCHS[vardeps] = "MACHINE" | 1661 | PACKAGE_ARCHS[vardeps] = "MACHINE" |
1717 | 1662 | ||
1718 | This example explicitly | 1663 | This example explicitly |
1719 | adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. | 1664 | adds the :term:`MACHINE` variable as a dependency for :term:`PACKAGE_ARCHS`. |
1720 | 1665 | ||
1721 | As an example, consider a case with in-line Python where BitBake is not | 1666 | As an example, consider a case with in-line Python where BitBake is not |
1722 | able to figure out dependencies. When running in debug mode (i.e. using | 1667 | able to figure out dependencies. When running in debug mode (i.e. using |
@@ -1728,34 +1673,29 @@ need to fix this situation. | |||
1728 | Thus far, this section has limited discussion to the direct inputs into | 1673 | Thus far, this section has limited discussion to the direct inputs into |
1729 | a task. Information based on direct inputs is referred to as the | 1674 | a task. Information based on direct inputs is referred to as the |
1730 | "basehash" in the code. However, the question of a task's indirect | 1675 | "basehash" in the code. However, the question of a task's indirect |
1731 | inputs still exits - items already built and present in the | 1676 | inputs still exits --- items already built and present in the |
1732 | :term:`Build Directory`. The checksum (or | 1677 | :term:`Build Directory`. The checksum (or |
1733 | signature) for a particular task needs to add the hashes of all the | 1678 | signature) for a particular task needs to add the hashes of all the |
1734 | tasks on which the particular task depends. Choosing which dependencies | 1679 | tasks on which the particular task depends. Choosing which dependencies |
1735 | to add is a policy decision. However, the effect is to generate a master | 1680 | to add is a policy decision. However, the effect is to generate a |
1736 | checksum that combines the basehash and the hashes of the task's | 1681 | checksum that combines the basehash and the hashes of the task's |
1737 | dependencies. | 1682 | dependencies. |
1738 | 1683 | ||
1739 | At the code level, a variety of ways exist by which both the basehash | 1684 | At the code level, there are multiple ways by which both the basehash |
1740 | and the dependent task hashes can be influenced. Within the BitBake | 1685 | and the dependent task hashes can be influenced. Within the BitBake |
1741 | configuration file, you can give BitBake some extra information to help | 1686 | configuration file, you can give BitBake some extra information to help |
1742 | it construct the basehash. The following statement effectively results | 1687 | it construct the basehash. The following statement effectively results |
1743 | in a list of global variable dependency excludes (i.e. variables never | 1688 | in a list of global variable dependency excludes (i.e. variables never |
1744 | included in any checksum): | 1689 | included in any checksum):: |
1745 | :: | ||
1746 | 1690 | ||
1747 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\ | 1691 | BB_BASEHASH_IGNORE_VARS ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\ |
1748 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\ | 1692 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\ |
1749 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ | 1693 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ |
1750 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ | 1694 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ |
1751 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | 1695 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" |
1752 | 1696 | ||
1753 | The | 1697 | The previous example does not include :term:`WORKDIR` since that variable is |
1754 | previous example excludes | 1698 | actually constructed as a path within :term:`TMPDIR`, which is included above. |
1755 | :term:`WORKDIR` since that variable | ||
1756 | is actually constructed as a path within | ||
1757 | :term:`TMPDIR`, which is on the | ||
1758 | whitelist. | ||
1759 | 1699 | ||
1760 | The rules for deciding which hashes of dependent tasks to include | 1700 | The rules for deciding which hashes of dependent tasks to include |
1761 | through dependency chains are more complex and are generally | 1701 | through dependency chains are more complex and are generally |
@@ -1767,16 +1707,15 @@ desired. This file defines the two basic signature generators | |||
1767 | "OEBasicHash". By default, a dummy "noop" signature handler is enabled | 1707 | "OEBasicHash". By default, a dummy "noop" signature handler is enabled |
1768 | in BitBake. This means that behavior is unchanged from previous | 1708 | in BitBake. This means that behavior is unchanged from previous |
1769 | versions. OE-Core uses the "OEBasicHash" signature handler by default | 1709 | versions. OE-Core uses the "OEBasicHash" signature handler by default |
1770 | through this setting in the ``bitbake.conf`` file: | 1710 | through this setting in the ``bitbake.conf`` file:: |
1771 | :: | ||
1772 | 1711 | ||
1773 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | 1712 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" |
1774 | 1713 | ||
1775 | The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same | 1714 | The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same |
1776 | as the "OEBasic" version but adds the task hash to the `stamp | 1715 | as the "OEBasic" version but adds the task hash to the :ref:`stamp |
1777 | files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any | 1716 | files <overview-manual/concepts:stamp files and the rerunning of tasks>`. This |
1778 | metadata change that changes the task hash, automatically causing the | 1717 | results in any metadata change that changes the task hash, automatically causing |
1779 | task to be run again. This removes the need to bump | 1718 | the task to be run again. This removes the need to bump |
1780 | :term:`PR` values, and changes to metadata | 1719 | :term:`PR` values, and changes to metadata |
1781 | automatically ripple across the build. | 1720 | automatically ripple across the build. |
1782 | 1721 | ||
@@ -1784,16 +1723,13 @@ It is also worth noting that the end result of these signature | |||
1784 | generators is to make some dependency and hash information available to | 1723 | generators is to make some dependency and hash information available to |
1785 | the build. This information includes: | 1724 | the build. This information includes: |
1786 | 1725 | ||
1787 | - ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the | 1726 | - ``BB_BASEHASH:task-``\ taskname: The base hashes for each task in the |
1788 | recipe. | 1727 | recipe. |
1789 | 1728 | ||
1790 | - ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each | 1729 | - ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each |
1791 | dependent task. | 1730 | dependent task. |
1792 | 1731 | ||
1793 | - ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for | 1732 | - :term:`BB_TASKHASH`: The hash of the currently running task. |
1794 | each task. | ||
1795 | |||
1796 | - ``BB_TASKHASH``: The hash of the currently running task. | ||
1797 | 1733 | ||
1798 | Shared State | 1734 | Shared State |
1799 | ------------ | 1735 | ------------ |
@@ -1803,12 +1739,11 @@ half the problem of supporting a shared state. The other half of the | |||
1803 | problem is being able to use checksum information during the build and | 1739 | problem is being able to use checksum information during the build and |
1804 | being able to reuse or rebuild specific components. | 1740 | being able to reuse or rebuild specific components. |
1805 | 1741 | ||
1806 | The :ref:`sstate <ref-classes-sstate>` class is a | 1742 | The :ref:`ref-classes-sstate` class is a relatively generic implementation of |
1807 | relatively generic implementation of how to "capture" a snapshot of a | 1743 | how to "capture" a snapshot of a given task. The idea is that the build process |
1808 | given task. The idea is that the build process does not care about the | 1744 | does not care about the source of a task's output. Output could be freshly |
1809 | source of a task's output. Output could be freshly built or it could be | 1745 | built or it could be downloaded and unpacked from somewhere. In other words, |
1810 | downloaded and unpacked from somewhere. In other words, the build | 1746 | the build process does not need to worry about its origin. |
1811 | process does not need to worry about its origin. | ||
1812 | 1747 | ||
1813 | Two types of output exist. One type is just about creating a directory | 1748 | Two types of output exist. One type is just about creating a directory |
1814 | in :term:`WORKDIR`. A good example is | 1749 | in :term:`WORKDIR`. A good example is |
@@ -1819,11 +1754,9 @@ type of output occurs when a set of data is merged into a shared | |||
1819 | directory tree such as the sysroot. | 1754 | directory tree such as the sysroot. |
1820 | 1755 | ||
1821 | The Yocto Project team has tried to keep the details of the | 1756 | The Yocto Project team has tried to keep the details of the |
1822 | implementation hidden in ``sstate`` class. From a user's perspective, | 1757 | implementation hidden in the :ref:`ref-classes-sstate` class. From a user's perspective, |
1823 | adding shared state wrapping to a task is as simple as this | 1758 | adding shared state wrapping to a task is as simple as this |
1824 | :ref:`ref-tasks-deploy` example taken | 1759 | :ref:`ref-tasks-deploy` example taken from the :ref:`ref-classes-deploy` class:: |
1825 | from the :ref:`deploy <ref-classes-deploy>` class: | ||
1826 | :: | ||
1827 | 1760 | ||
1828 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" | 1761 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" |
1829 | SSTATETASKS += "do_deploy" | 1762 | SSTATETASKS += "do_deploy" |
@@ -1839,14 +1772,12 @@ from the :ref:`deploy <ref-classes-deploy>` class: | |||
1839 | 1772 | ||
1840 | The following list explains the previous example: | 1773 | The following list explains the previous example: |
1841 | 1774 | ||
1842 | - Adding "do_deploy" to ``SSTATETASKS`` adds some required | 1775 | - Adding ``do_deploy`` to ``SSTATETASKS`` adds some required sstate-related |
1843 | sstate-related processing, which is implemented in the | 1776 | processing, which is implemented in the :ref:`ref-classes-sstate` class, to |
1844 | :ref:`sstate <ref-classes-sstate>` class, to | 1777 | before and after the :ref:`ref-tasks-deploy` task. |
1845 | before and after the | ||
1846 | :ref:`ref-tasks-deploy` task. | ||
1847 | 1778 | ||
1848 | - The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that | 1779 | - The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that |
1849 | ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally | 1780 | :ref:`ref-tasks-deploy` places its output in ``${DEPLOYDIR}`` when run normally |
1850 | (i.e. when not using the sstate cache). This output becomes the input | 1781 | (i.e. when not using the sstate cache). This output becomes the input |
1851 | to the shared state cache. | 1782 | to the shared state cache. |
1852 | 1783 | ||
@@ -1856,92 +1787,85 @@ The following list explains the previous example: | |||
1856 | 1787 | ||
1857 | .. note:: | 1788 | .. note:: |
1858 | 1789 | ||
1859 | If ``do_deploy`` is not already in the shared state cache or if its input | 1790 | If :ref:`ref-tasks-deploy` is not already in the shared state cache or if its input |
1860 | checksum (signature) has changed from when the output was cached, the task | 1791 | checksum (signature) has changed from when the output was cached, the task |
1861 | runs to populate the shared state cache, after which the contents of the | 1792 | runs to populate the shared state cache, after which the contents of the |
1862 | shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If | 1793 | shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If |
1863 | ``do_deploy`` is in the shared state cache and its signature indicates | 1794 | :ref:`ref-tasks-deploy` is in the shared state cache and its signature indicates |
1864 | that the cached output is still valid (i.e. if no relevant task inputs | 1795 | that the cached output is still valid (i.e. if no relevant task inputs |
1865 | have changed), then the contents of the shared state cache copies | 1796 | have changed), then the contents of the shared state cache copies |
1866 | directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task | 1797 | directly to ${:term:`DEPLOY_DIR_IMAGE`} by the ``do_deploy_setscene`` task |
1867 | instead, skipping the ``do_deploy`` task. | 1798 | instead, skipping the :ref:`ref-tasks-deploy` task. |
1868 | 1799 | ||
1869 | - The following task definition is glue logic needed to make the | 1800 | - The following task definition is glue logic needed to make the |
1870 | previous settings effective: | 1801 | previous settings effective:: |
1871 | :: | ||
1872 | 1802 | ||
1873 | python do_deploy_setscene () { | 1803 | python do_deploy_setscene () { |
1874 | sstate_setscene(d) | 1804 | sstate_setscene(d) |
1875 | } | 1805 | } |
1876 | addtask do_deploy_setscene | 1806 | addtask do_deploy_setscene |
1877 | 1807 | ||
1878 | ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task | 1808 | ``sstate_setscene()`` takes the flags above as input and accelerates the |
1879 | through the shared state cache if possible. If the task was | 1809 | :ref:`ref-tasks-deploy` task through the shared state cache if possible. If |
1880 | accelerated, ``sstate_setscene()`` returns True. Otherwise, it | 1810 | the task was accelerated, ``sstate_setscene()`` returns True. Otherwise, it |
1881 | returns False, and the normal ``do_deploy`` task runs. For more | 1811 | returns False, and the normal :ref:`ref-tasks-deploy` task runs. For more |
1882 | information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`" | 1812 | information, see the ":ref:`bitbake-user-manual/bitbake-user-manual-execution:setscene`" |
1883 | section in the BitBake User Manual. | 1813 | section in the BitBake User Manual. |
1884 | 1814 | ||
1885 | - The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates | 1815 | - The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates ``${DEPLOYDIR}`` |
1886 | ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and | 1816 | and ``${B}`` before the :ref:`ref-tasks-deploy` task runs, and also sets the |
1887 | also sets the current working directory of ``do_deploy`` to ``${B}``. | 1817 | current working directory of :ref:`ref-tasks-deploy` to ``${B}``. For more |
1888 | For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" | 1818 | information, see the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" |
1889 | section in the BitBake | 1819 | section in the BitBake User Manual. |
1890 | User Manual. | ||
1891 | 1820 | ||
1892 | .. note:: | 1821 | .. note:: |
1893 | 1822 | ||
1894 | In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be | 1823 | In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be |
1895 | the same, you can use ``sstate-plaindirs``. For example, to preserve the | 1824 | the same, you can use ``sstate-plaindirs``. For example, to preserve the |
1896 | ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package`` | 1825 | ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the :ref:`ref-tasks-package` |
1897 | task, use the following: | 1826 | task, use the following:: |
1898 | :: | ||
1899 | 1827 | ||
1900 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | 1828 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" |
1901 | 1829 | ||
1902 | 1830 | ||
1903 | - The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends | 1831 | - The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends extra |
1904 | extra metadata to the `stamp | 1832 | metadata to the :ref:`stamp file <overview-manual/concepts:stamp files and the rerunning of tasks>`. |
1905 | file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the | 1833 | In this case, the metadata makes the task specific to a machine's architecture. |
1906 | metadata makes the task specific to a machine's architecture. See | 1834 | See the ":ref:`bitbake-user-manual/bitbake-user-manual-execution:the task list`" |
1907 | ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:the task list`" | ||
1908 | section in the BitBake User Manual for more information on the | 1835 | section in the BitBake User Manual for more information on the |
1909 | ``stamp-extra-info`` flag. | 1836 | ``stamp-extra-info`` flag. |
1910 | 1837 | ||
1911 | - ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with | 1838 | - ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with |
1912 | multiple directories. For example, the following declares | 1839 | multiple directories. For example, the following declares |
1913 | ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories, | 1840 | :term:`PKGDESTWORK` and ``SHLIBWORK`` as shared state input directories, |
1914 | which populates the shared state cache, and ``PKGDATA_DIR`` and | 1841 | which populates the shared state cache, and :term:`PKGDATA_DIR` and |
1915 | ``SHLIBSDIR`` as the corresponding shared state output directories: | 1842 | ``SHLIBSDIR`` as the corresponding shared state output directories:: |
1916 | :: | ||
1917 | 1843 | ||
1918 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" | 1844 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" |
1919 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" | 1845 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" |
1920 | 1846 | ||
1921 | - These methods also include the ability to take a lockfile when | 1847 | - These methods also include the ability to take a lockfile when |
1922 | manipulating shared state directory structures, for cases where file | 1848 | manipulating shared state directory structures, for cases where file |
1923 | additions or removals are sensitive: | 1849 | additions or removals are sensitive:: |
1924 | :: | ||
1925 | 1850 | ||
1926 | do_package[sstate-lockfile] = "${PACKAGELOCK}" | 1851 | do_package[sstate-lockfile] = "${PACKAGELOCK}" |
1927 | 1852 | ||
1928 | Behind the scenes, the shared state code works by looking in | 1853 | Behind the scenes, the shared state code works by looking in |
1929 | :term:`SSTATE_DIR` and | 1854 | :term:`SSTATE_DIR` and |
1930 | :term:`SSTATE_MIRRORS` for | 1855 | :term:`SSTATE_MIRRORS` for |
1931 | shared state files. Here is an example: | 1856 | shared state files. Here is an example:: |
1932 | :: | ||
1933 | 1857 | ||
1934 | SSTATE_MIRRORS ?= "\ | 1858 | SSTATE_MIRRORS ?= "\ |
1935 | file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ | 1859 | file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \ |
1936 | file://.\* file:///some/local/dir/sstate/PATH" | 1860 | file://.* file:///some/local/dir/sstate/PATH" |
1937 | 1861 | ||
1938 | .. note:: | 1862 | .. note:: |
1939 | 1863 | ||
1940 | The shared state directory (``SSTATE_DIR``) is organized into two-character | 1864 | The shared state directory (:term:`SSTATE_DIR`) is organized into two-character |
1941 | subdirectories, where the subdirectory names are based on the first two | 1865 | subdirectories, where the subdirectory names are based on the first two |
1942 | characters of the hash. | 1866 | characters of the hash. |
1943 | If the shared state directory structure for a mirror has the same structure | 1867 | If the shared state directory structure for a mirror has the same structure |
1944 | as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build | 1868 | as :term:`SSTATE_DIR`, you must specify "PATH" as part of the URI to enable the build |
1945 | system to map to the appropriate subdirectory. | 1869 | system to map to the appropriate subdirectory. |
1946 | 1870 | ||
1947 | The shared state package validity can be detected just by looking at the | 1871 | The shared state package validity can be detected just by looking at the |
@@ -1966,14 +1890,155 @@ another reason why a task-based approach is preferred over a | |||
1966 | recipe-based approach, which would have to install the output from every | 1890 | recipe-based approach, which would have to install the output from every |
1967 | task. | 1891 | task. |
1968 | 1892 | ||
1893 | Hash Equivalence | ||
1894 | ---------------- | ||
1895 | |||
1896 | The above section explained how BitBake skips the execution of tasks | ||
1897 | whose output can already be found in the Shared State cache. | ||
1898 | |||
1899 | During a build, it may often be the case that the output / result of a task might | ||
1900 | be unchanged despite changes in the task's input values. An example might be | ||
1901 | whitespace changes in some input C code. In project terms, this is what we define | ||
1902 | as "equivalence". | ||
1903 | |||
1904 | To keep track of such equivalence, BitBake has to manage three hashes | ||
1905 | for each task: | ||
1906 | |||
1907 | - The *task hash* explained earlier: computed from the recipe metadata, | ||
1908 | the task code and the task hash values from its dependencies. | ||
1909 | When changes are made, these task hashes are therefore modified, | ||
1910 | causing the task to re-execute. The task hashes of tasks depending on this | ||
1911 | task are therefore modified too, causing the whole dependency | ||
1912 | chain to re-execute. | ||
1913 | |||
1914 | - The *output hash*, a new hash computed from the output of Shared State tasks, | ||
1915 | tasks that save their resulting output to a Shared State tarball. | ||
1916 | The mapping between the task hash and its output hash is reported | ||
1917 | to a new *Hash Equivalence* server. This mapping is stored in a database | ||
1918 | by the server for future reference. | ||
1919 | |||
1920 | - The *unihash*, a new hash, initially set to the task hash for the task. | ||
1921 | This is used to track the *unicity* of task output, and we will explain | ||
1922 | how its value is maintained. | ||
1923 | |||
1924 | When Hash Equivalence is enabled, BitBake computes the task hash | ||
1925 | for each task by using the unihash of its dependencies, instead | ||
1926 | of their task hash. | ||
1927 | |||
1928 | Now, imagine that a Shared State task is modified because of a change in | ||
1929 | its code or metadata, or because of a change in its dependencies. | ||
1930 | Since this modifies its task hash, this task will need re-executing. | ||
1931 | Its output hash will therefore be computed again. | ||
1932 | |||
1933 | Then, the new mapping between the new task hash and its output hash | ||
1934 | will be reported to the Hash Equivalence server. The server will | ||
1935 | let BitBake know whether this output hash is the same as a previously | ||
1936 | reported output hash, for a different task hash. | ||
1937 | |||
1938 | If the output hash is already known, BitBake will update the task's | ||
1939 | unihash to match the original task hash that generated that output. | ||
1940 | Thanks to this, the depending tasks will keep a previously recorded | ||
1941 | task hash, and BitBake will be able to retrieve their output from | ||
1942 | the Shared State cache, instead of re-executing them. Similarly, the | ||
1943 | output of further downstream tasks can also be retrieved from Shared | ||
1944 | State. | ||
1945 | |||
1946 | If the output hash is unknown, a new entry will be created on the Hash | ||
1947 | Equivalence server, matching the task hash to that output. | ||
1948 | The depending tasks, still having a new task hash because of the | ||
1949 | change, will need to re-execute as expected. The change propagates | ||
1950 | to the depending tasks. | ||
1951 | |||
1952 | To summarize, when Hash Equivalence is enabled, a change in one of the | ||
1953 | tasks in BitBake's run queue doesn't have to propagate to all the | ||
1954 | downstream tasks that depend on the output of this task, causing a | ||
1955 | full rebuild of such tasks, and so on with the next depending tasks. | ||
1956 | Instead, when the output of this task remains identical to previously | ||
1957 | recorded output, BitBake can safely retrieve all the downstream | ||
1958 | task output from the Shared State cache. | ||
1959 | |||
1960 | .. note:: | ||
1961 | |||
1962 | Having :doc:`/test-manual/reproducible-builds` is a key ingredient for | ||
1963 | the stability of the task's output hash. Therefore, the effectiveness | ||
1964 | of Hash Equivalence strongly depends on it. | ||
1965 | |||
1966 | Recipes that are not reproducible may have undesired behavior if hash | ||
1967 | equivalence is enabled, since the non-reproducible diverging output maybe be | ||
1968 | remapped to an older sstate object in the cache by the server. If a recipe | ||
1969 | is non-reproducible in trivial ways, such as different timestamps, this is | ||
1970 | likely not a problem. However recipes that have more dramatic changes (such | ||
1971 | as completely different file names) will likely outright fail since the | ||
1972 | downstream sstate objects are not actually equivalent to what was just | ||
1973 | built. | ||
1974 | |||
1975 | This applies to multiple scenarios: | ||
1976 | |||
1977 | - A "trivial" change to a recipe that doesn't impact its generated output, | ||
1978 | such as whitespace changes, modifications to unused code paths or | ||
1979 | in the ordering of variables. | ||
1980 | |||
1981 | - Shared library updates, for example to fix a security vulnerability. | ||
1982 | For sure, the programs using such a library should be rebuilt, but | ||
1983 | their new binaries should remain identical. The corresponding tasks should | ||
1984 | have a different output hash because of the change in the hash of their | ||
1985 | library dependency, but thanks to their output being identical, Hash | ||
1986 | Equivalence will stop the propagation down the dependency chain. | ||
1987 | |||
1988 | - Native tool updates. Though the depending tasks should be rebuilt, | ||
1989 | it's likely that they will generate the same output and be marked | ||
1990 | as equivalent. | ||
1991 | |||
1992 | This mechanism is enabled by default in Poky, and is controlled by three | ||
1993 | variables: | ||
1994 | |||
1995 | - :term:`bitbake:BB_HASHSERVE`, specifying a local or remote Hash | ||
1996 | Equivalence server to use. | ||
1997 | |||
1998 | - :term:`BB_HASHSERVE_UPSTREAM`, when ``BB_HASHSERVE = "auto"``, | ||
1999 | allowing to connect the local server to an upstream one. | ||
2000 | |||
2001 | - :term:`bitbake:BB_SIGNATURE_HANDLER`, which must be set to ``OEEquivHash``. | ||
2002 | |||
2003 | Therefore, the default configuration in Poky corresponds to the | ||
2004 | below settings:: | ||
2005 | |||
2006 | BB_HASHSERVE = "auto" | ||
2007 | BB_SIGNATURE_HANDLER = "OEEquivHash" | ||
2008 | |||
2009 | Rather than starting a local server, another possibility is to rely | ||
2010 | on a Hash Equivalence server on a network, by setting:: | ||
2011 | |||
2012 | BB_HASHSERVE = "<HOSTNAME>:<PORT>" | ||
2013 | |||
2014 | .. note:: | ||
2015 | |||
2016 | The shared Hash Equivalence server needs to be maintained together with the | ||
2017 | Shared State cache. Otherwise, the server could report Shared State hashes | ||
2018 | that only exist on specific clients. | ||
2019 | |||
2020 | We therefore recommend that one Hash Equivalence server be set up to | ||
2021 | correspond with a given Shared State cache, and to start this server | ||
2022 | in *read-only mode*, so that it doesn't store equivalences for | ||
2023 | Shared State caches that are local to clients. | ||
2024 | |||
2025 | See the :term:`BB_HASHSERVE` reference for details about starting | ||
2026 | a Hash Equivalence server. | ||
2027 | |||
2028 | See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__ | ||
2029 | of Joshua Watt's `Hash Equivalence and Reproducible Builds | ||
2030 | <https://elinux.org/images/3/37/Hash_Equivalence_and_Reproducible_Builds.pdf>`__ | ||
2031 | presentation at ELC 2020 for a very synthetic introduction to the | ||
2032 | Hash Equivalence implementation in the Yocto Project. | ||
2033 | |||
1969 | Automatically Added Runtime Dependencies | 2034 | Automatically Added Runtime Dependencies |
1970 | ======================================== | 2035 | ======================================== |
1971 | 2036 | ||
1972 | The OpenEmbedded build system automatically adds common types of runtime | 2037 | The OpenEmbedded build system automatically adds common types of runtime |
1973 | dependencies between packages, which means that you do not need to | 2038 | dependencies between packages, which means that you do not need to |
1974 | explicitly declare the packages using | 2039 | explicitly declare the packages using |
1975 | :term:`RDEPENDS`. Three automatic | 2040 | :term:`RDEPENDS`. There are three automatic |
1976 | mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that | 2041 | mechanisms (``shlibdeps``, ``pcdeps``, and ``depchains``) that |
1977 | handle shared libraries, package configuration (pkg-config) modules, and | 2042 | handle shared libraries, package configuration (pkg-config) modules, and |
1978 | ``-dev`` and ``-dbg`` packages, respectively. For other types of runtime | 2043 | ``-dev`` and ``-dbg`` packages, respectively. For other types of runtime |
1979 | dependencies, you must manually declare the dependencies. | 2044 | dependencies, you must manually declare the dependencies. |
@@ -1984,7 +2049,7 @@ dependencies, you must manually declare the dependencies. | |||
1984 | located. For each shared library, the package that contains the | 2049 | located. For each shared library, the package that contains the |
1985 | shared library is registered as providing the shared library. More | 2050 | shared library is registered as providing the shared library. More |
1986 | specifically, the package is registered as providing the | 2051 | specifically, the package is registered as providing the |
1987 | `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The | 2052 | :wikipedia:`soname <Soname>` of the library. The |
1988 | resulting shared-library-to-package mapping is saved globally in | 2053 | resulting shared-library-to-package mapping is saved globally in |
1989 | :term:`PKGDATA_DIR` by the | 2054 | :term:`PKGDATA_DIR` by the |
1990 | :ref:`ref-tasks-packagedata` | 2055 | :ref:`ref-tasks-packagedata` |
@@ -1992,7 +2057,7 @@ dependencies, you must manually declare the dependencies. | |||
1992 | 2057 | ||
1993 | Simultaneously, all executables and shared libraries installed by the | 2058 | Simultaneously, all executables and shared libraries installed by the |
1994 | recipe are inspected to see what shared libraries they link against. | 2059 | recipe are inspected to see what shared libraries they link against. |
1995 | For each shared library dependency that is found, ``PKGDATA_DIR`` is | 2060 | For each shared library dependency that is found, :term:`PKGDATA_DIR` is |
1996 | queried to see if some package (likely from a different recipe) | 2061 | queried to see if some package (likely from a different recipe) |
1997 | contains the shared library. If such a package is found, a runtime | 2062 | contains the shared library. If such a package is found, a runtime |
1998 | dependency is added from the package that depends on the shared | 2063 | dependency is added from the package that depends on the shared |
@@ -2001,7 +2066,7 @@ dependencies, you must manually declare the dependencies. | |||
2001 | The automatically added runtime dependency also includes a version | 2066 | The automatically added runtime dependency also includes a version |
2002 | restriction. This version restriction specifies that at least the | 2067 | restriction. This version restriction specifies that at least the |
2003 | current version of the package that provides the shared library must | 2068 | current version of the package that provides the shared library must |
2004 | be used, as if "package (>= version)" had been added to ``RDEPENDS``. | 2069 | be used, as if "package (>= version)" had been added to :term:`RDEPENDS`. |
2005 | This forces an upgrade of the package containing the shared library | 2070 | This forces an upgrade of the package containing the shared library |
2006 | when installing the package that depends on the library, if needed. | 2071 | when installing the package that depends on the library, if needed. |
2007 | 2072 | ||
@@ -2011,18 +2076,18 @@ dependencies, you must manually declare the dependencies. | |||
2011 | :term:`PRIVATE_LIBS` inside | 2076 | :term:`PRIVATE_LIBS` inside |
2012 | the package's recipe. | 2077 | the package's recipe. |
2013 | 2078 | ||
2014 | - ``pcdeps``: During the ``do_package`` task of each recipe, all | 2079 | - ``pcdeps``: During the :ref:`ref-tasks-package` task of each recipe, all |
2015 | pkg-config modules (``*.pc`` files) installed by the recipe are | 2080 | pkg-config modules (``*.pc`` files) installed by the recipe are |
2016 | located. For each module, the package that contains the module is | 2081 | located. For each module, the package that contains the module is |
2017 | registered as providing the module. The resulting module-to-package | 2082 | registered as providing the module. The resulting module-to-package |
2018 | mapping is saved globally in ``PKGDATA_DIR`` by the | 2083 | mapping is saved globally in :term:`PKGDATA_DIR` by the |
2019 | ``do_packagedata`` task. | 2084 | :ref:`ref-tasks-packagedata` task. |
2020 | 2085 | ||
2021 | Simultaneously, all pkg-config modules installed by the recipe are | 2086 | Simultaneously, all pkg-config modules installed by the recipe are |
2022 | inspected to see what other pkg-config modules they depend on. A | 2087 | inspected to see what other pkg-config modules they depend on. A |
2023 | module is seen as depending on another module if it contains a | 2088 | module is seen as depending on another module if it contains a |
2024 | "Requires:" line that specifies the other module. For each module | 2089 | "Requires:" line that specifies the other module. For each module |
2025 | dependency, ``PKGDATA_DIR`` is queried to see if some package | 2090 | dependency, :term:`PKGDATA_DIR` is queried to see if some package |
2026 | contains the module. If such a package is found, a runtime dependency | 2091 | contains the module. If such a package is found, a runtime dependency |
2027 | is added from the package that depends on the module to the package | 2092 | is added from the package that depends on the module to the package |
2028 | that contains the module. | 2093 | that contains the module. |
@@ -2047,9 +2112,9 @@ dependencies, you must manually declare the dependencies. | |||
2047 | 2112 | ||
2048 | .. note:: | 2113 | .. note:: |
2049 | 2114 | ||
2050 | By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on | 2115 | By default, ``foo-dev`` also has an :term:`RDEPENDS`-style dependency on |
2051 | ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in | 2116 | ``foo``, because the default value of ``RDEPENDS:${PN}-dev`` (set in |
2052 | bitbake.conf) includes "${PN}". | 2117 | ``bitbake.conf``) includes "${PN}". |
2053 | 2118 | ||
2054 | To ensure that the dependency chain is never broken, ``-dev`` and | 2119 | To ensure that the dependency chain is never broken, ``-dev`` and |
2055 | ``-dbg`` packages are always generated by default, even if the | 2120 | ``-dbg`` packages are always generated by default, even if the |
@@ -2057,12 +2122,12 @@ dependencies, you must manually declare the dependencies. | |||
2057 | :term:`ALLOW_EMPTY` variable | 2122 | :term:`ALLOW_EMPTY` variable |
2058 | for more information. | 2123 | for more information. |
2059 | 2124 | ||
2060 | The ``do_package`` task depends on the ``do_packagedata`` task of each | 2125 | The :ref:`ref-tasks-package` task depends on the :ref:`ref-tasks-packagedata` |
2061 | recipe in :term:`DEPENDS` through use | 2126 | task of each recipe in :term:`DEPENDS` through use of a |
2062 | of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` | 2127 | ``[``\ :ref:`deptask <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` |
2063 | declaration, which guarantees that the required | 2128 | declaration, which guarantees that the required shared-library / |
2064 | shared-library/module-to-package mapping information will be available | 2129 | module-to-package mapping information will be available when needed as long as |
2065 | when needed as long as ``DEPENDS`` has been correctly set. | 2130 | :term:`DEPENDS` has been correctly set. |
2066 | 2131 | ||
2067 | Fakeroot and Pseudo | 2132 | Fakeroot and Pseudo |
2068 | =================== | 2133 | =================== |
@@ -2072,8 +2137,8 @@ operations that are normally reserved for the root user (e.g. | |||
2072 | :ref:`ref-tasks-install`, | 2137 | :ref:`ref-tasks-install`, |
2073 | :ref:`do_package_write* <ref-tasks-package_write_deb>`, | 2138 | :ref:`do_package_write* <ref-tasks-package_write_deb>`, |
2074 | :ref:`ref-tasks-rootfs`, and | 2139 | :ref:`ref-tasks-rootfs`, and |
2075 | :ref:`do_image* <ref-tasks-image>`). For example, | 2140 | :ref:`do_image_* <ref-tasks-image>`). For example, |
2076 | the ``do_install`` task benefits from being able to set the UID and GID | 2141 | the :ref:`ref-tasks-install` task benefits from being able to set the UID and GID |
2077 | of installed files to arbitrary values. | 2142 | of installed files to arbitrary values. |
2078 | 2143 | ||
2079 | One approach to allowing tasks to perform root-only operations would be | 2144 | One approach to allowing tasks to perform root-only operations would be |
@@ -2111,9 +2176,7 @@ accomplished using fakeroot. | |||
2111 | under fakeroot. Otherwise, the task cannot run root-only operations, | 2176 | under fakeroot. Otherwise, the task cannot run root-only operations, |
2112 | and cannot see the fake file ownership and permissions set by the | 2177 | and cannot see the fake file ownership and permissions set by the |
2113 | other task. You need to also add a dependency on | 2178 | other task. You need to also add a dependency on |
2114 | virtual/fakeroot-native:do_populate_sysroot | 2179 | ``virtual/fakeroot-native:do_populate_sysroot``, giving the following:: |
2115 | , giving the following: | ||
2116 | :: | ||
2117 | 2180 | ||
2118 | fakeroot do_mytask () { | 2181 | fakeroot do_mytask () { |
2119 | ... | 2182 | ... |
@@ -2126,3 +2189,173 @@ For more information, see the | |||
2126 | BitBake User Manual. You can also reference the "`Why Not | 2189 | BitBake User Manual. You can also reference the "`Why Not |
2127 | Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__" | 2190 | Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__" |
2128 | article for background information on Fakeroot and Pseudo. | 2191 | article for background information on Fakeroot and Pseudo. |
2192 | |||
2193 | BitBake Tasks Map | ||
2194 | ================= | ||
2195 | |||
2196 | To understand how BitBake operates in the build directory and environment | ||
2197 | we can consider the following recipes and diagram, to have full picture | ||
2198 | about the tasks that BitBake runs to generate the final package file | ||
2199 | for the recipe. | ||
2200 | |||
2201 | We will have two recipes as an example: | ||
2202 | |||
2203 | - ``libhello``: A recipe that provides a shared library | ||
2204 | - ``sayhello``: A recipe that uses ``libhello`` library to do its job | ||
2205 | |||
2206 | .. note:: | ||
2207 | |||
2208 | ``sayhello`` depends on ``libhello`` at compile time as it needs the shared | ||
2209 | library to do the dynamic linking process. It also depends on it at runtime | ||
2210 | as the shared library loader needs to find the library. | ||
2211 | For more details about dependencies check :ref:`ref-varlocality-recipe-dependencies`. | ||
2212 | |||
2213 | ``libhello`` sources are as follows: | ||
2214 | |||
2215 | - ``LICENSE``: This is the license associated with this library | ||
2216 | - ``Makefile``: The file used by ``make`` to build the library | ||
2217 | - ``hellolib.c``: The implementation of the library | ||
2218 | - ``hellolib.h``: The C header of the library | ||
2219 | |||
2220 | ``sayhello`` sources are as follows: | ||
2221 | |||
2222 | - ``LICENSE``: This is the license associated with this project | ||
2223 | - ``Makefile``: The file used by ``make`` to build the project | ||
2224 | - ``sayhello.c``: The source file of the project | ||
2225 | |||
2226 | Before presenting the contents of each file, here are the steps | ||
2227 | that we need to follow to accomplish what we want in the first place, | ||
2228 | which is integrating ``sayhello`` in our root file system: | ||
2229 | |||
2230 | #. Create a Git repository for each project with the corresponding files | ||
2231 | |||
2232 | #. Create a recipe for each project | ||
2233 | |||
2234 | #. Make sure that ``sayhello`` recipe :term:`DEPENDS` on ``libhello`` | ||
2235 | |||
2236 | #. Make sure that ``sayhello`` recipe :term:`RDEPENDS` on ``libhello`` | ||
2237 | |||
2238 | #. Add ``sayhello`` to :term:`IMAGE_INSTALL` to integrate it into | ||
2239 | the root file system | ||
2240 | |||
2241 | The contents of ``libhello/Makefile`` are:: | ||
2242 | |||
2243 | LIB=libhello.so | ||
2244 | |||
2245 | all: $(LIB) | ||
2246 | |||
2247 | $(LIB): hellolib.o | ||
2248 | $(CC) $< -Wl,-soname,$(LIB).1 -fPIC $(LDFLAGS) -shared -o $(LIB).1.0 | ||
2249 | |||
2250 | %.o: %.c | ||
2251 | $(CC) -c $< | ||
2252 | |||
2253 | clean: | ||
2254 | rm -rf *.o *.so* | ||
2255 | |||
2256 | .. note:: | ||
2257 | |||
2258 | When creating shared libraries, it is strongly recommended to follow the Linux | ||
2259 | conventions and guidelines (see `this article | ||
2260 | <https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__ | ||
2261 | for some background). | ||
2262 | |||
2263 | .. note:: | ||
2264 | |||
2265 | When creating ``Makefile`` files, it is strongly recommended to use ``CC``, ``LDFLAGS`` | ||
2266 | and ``CFLAGS`` as BitBake will set them as environment variables according | ||
2267 | to your build configuration. | ||
2268 | |||
2269 | The contents of ``libhello/hellolib.h`` are:: | ||
2270 | |||
2271 | #ifndef HELLOLIB_H | ||
2272 | #define HELLOLIB_H | ||
2273 | |||
2274 | void Hello(); | ||
2275 | |||
2276 | #endif | ||
2277 | |||
2278 | The contents of ``libhello/hellolib.c`` are:: | ||
2279 | |||
2280 | #include <stdio.h> | ||
2281 | |||
2282 | void Hello(){ | ||
2283 | puts("Hello from a Yocto demo \n"); | ||
2284 | } | ||
2285 | |||
2286 | The contents of ``sayhello/Makefile`` are:: | ||
2287 | |||
2288 | EXEC=sayhello | ||
2289 | LDFLAGS += -lhello | ||
2290 | |||
2291 | all: $(EXEC) | ||
2292 | |||
2293 | $(EXEC): sayhello.c | ||
2294 | $(CC) $< $(LDFLAGS) $(CFLAGS) -o $(EXEC) | ||
2295 | |||
2296 | clean: | ||
2297 | rm -rf $(EXEC) *.o | ||
2298 | |||
2299 | The contents of ``sayhello/sayhello.c`` are:: | ||
2300 | |||
2301 | #include <hellolib.h> | ||
2302 | |||
2303 | int main(){ | ||
2304 | Hello(); | ||
2305 | return 0; | ||
2306 | } | ||
2307 | |||
2308 | The contents of ``libhello_0.1.bb`` are:: | ||
2309 | |||
2310 | SUMMARY = "Hello demo library" | ||
2311 | DESCRIPTION = "Hello shared library used in Yocto demo" | ||
2312 | |||
2313 | # NOTE: Set the License according to the LICENSE file of your project | ||
2314 | # and then add LIC_FILES_CHKSUM accordingly | ||
2315 | LICENSE = "CLOSED" | ||
2316 | |||
2317 | # Assuming the branch is main | ||
2318 | # Change <username> accordingly | ||
2319 | SRC_URI = "git://github.com/<username>/libhello;branch=main;protocol=https" | ||
2320 | |||
2321 | S = "${WORKDIR}/git" | ||
2322 | |||
2323 | do_install(){ | ||
2324 | install -d ${D}${includedir} | ||
2325 | install -d ${D}${libdir} | ||
2326 | |||
2327 | install hellolib.h ${D}${includedir} | ||
2328 | oe_soinstall ${PN}.so.${PV} ${D}${libdir} | ||
2329 | } | ||
2330 | |||
2331 | The contents of ``sayhello_0.1.bb`` are:: | ||
2332 | |||
2333 | SUMMARY = "SayHello demo" | ||
2334 | DESCRIPTION = "SayHello project used in Yocto demo" | ||
2335 | |||
2336 | # NOTE: Set the License according to the LICENSE file of your project | ||
2337 | # and then add LIC_FILES_CHKSUM accordingly | ||
2338 | LICENSE = "CLOSED" | ||
2339 | |||
2340 | # Assuming the branch is main | ||
2341 | # Change <username> accordingly | ||
2342 | SRC_URI = "git://github.com/<username>/sayhello;branch=main;protocol=https" | ||
2343 | |||
2344 | DEPENDS += "libhello" | ||
2345 | RDEPENDS:${PN} += "libhello" | ||
2346 | |||
2347 | S = "${WORKDIR}/git" | ||
2348 | |||
2349 | do_install(){ | ||
2350 | install -d ${D}/usr/bin | ||
2351 | install -m 0700 sayhello ${D}/usr/bin | ||
2352 | } | ||
2353 | |||
2354 | After placing the recipes in a custom layer we can run ``bitbake sayhello`` | ||
2355 | to build the recipe. | ||
2356 | |||
2357 | The following diagram shows the sequences of tasks that BitBake | ||
2358 | executes to accomplish that. | ||
2359 | |||
2360 | .. image:: svg/bitbake_tasks_map.* | ||
2361 | :width: 100% | ||