diff options
Diffstat (limited to 'documentation/overview-manual')
13 files changed, 3941 insertions, 33 deletions
diff --git a/documentation/overview-manual/history.rst b/documentation/overview-manual/history.rst new file mode 100644 index 0000000000..0273d28b90 --- /dev/null +++ b/documentation/overview-manual/history.rst | |||
@@ -0,0 +1,28 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | *********************** | ||
4 | Manual Revision History | ||
5 | *********************** | ||
6 | |||
7 | .. list-table:: | ||
8 | :widths: 10 15 40 | ||
9 | :header-rows: 1 | ||
10 | |||
11 | * - Revision | ||
12 | - Date | ||
13 | - Note | ||
14 | * - 2.5 | ||
15 | - May 2018 | ||
16 | - The initial document released with the Yocto Project 2.5 Release | ||
17 | * - 2.6 | ||
18 | - November 2018 | ||
19 | - Released with the Yocto Project 2.6 Release. | ||
20 | * - 2.7 | ||
21 | - May 2019 | ||
22 | - Released with the Yocto Project 2.7 Release. | ||
23 | * - 3.0 | ||
24 | - October 2019 | ||
25 | - Released with the Yocto Project 3.0 Release. | ||
26 | * - 3.1 | ||
27 | - April 2020 | ||
28 | - Released with the Yocto Project 3.1 Release. | ||
diff --git a/documentation/overview-manual/overview-manual-concepts.rst b/documentation/overview-manual/overview-manual-concepts.rst new file mode 100644 index 0000000000..6ce5f80af3 --- /dev/null +++ b/documentation/overview-manual/overview-manual-concepts.rst | |||
@@ -0,0 +1,2185 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ********************** | ||
4 | Yocto Project Concepts | ||
5 | ********************** | ||
6 | |||
7 | This chapter provides explanations for Yocto Project concepts that go | ||
8 | beyond the surface of "how-to" information and reference (or look-up) | ||
9 | material. Concepts such as components, the :term:`OpenEmbedded Build System` | ||
10 | workflow, | ||
11 | cross-development toolchains, shared state cache, and so forth are | ||
12 | explained. | ||
13 | |||
14 | Yocto Project Components | ||
15 | ======================== | ||
16 | |||
17 | The :term:`BitBake` task executor | ||
18 | together with various types of configuration files form the | ||
19 | :term:`OpenEmbedded-Core (OE-Core)`. This section | ||
20 | overviews these components by describing their use and how they | ||
21 | interact. | ||
22 | |||
23 | BitBake handles the parsing and execution of the data files. The data | ||
24 | itself is of various types: | ||
25 | |||
26 | - *Recipes:* Provides details about particular pieces of software. | ||
27 | |||
28 | - *Class Data:* Abstracts common build information (e.g. how to build a | ||
29 | Linux kernel). | ||
30 | |||
31 | - *Configuration Data:* Defines machine-specific settings, policy | ||
32 | decisions, and so forth. Configuration data acts as the glue to bind | ||
33 | everything together. | ||
34 | |||
35 | BitBake knows how to combine multiple data sources together and refers | ||
36 | to each data source as a layer. For information on layers, see the | ||
37 | ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" | ||
38 | section of the Yocto Project Development Tasks Manual. | ||
39 | |||
40 | Following are some brief details on these core components. For | ||
41 | additional information on how these components interact during a build, | ||
42 | see the | ||
43 | ":ref:`overview-manual/overview-manual-concepts:openembedded build system concepts`" | ||
44 | section. | ||
45 | |||
46 | .. _usingpoky-components-bitbake: | ||
47 | |||
48 | BitBake | ||
49 | ------- | ||
50 | |||
51 | BitBake is the tool at the heart of the :term:`OpenEmbedded Build System` | ||
52 | and is responsible | ||
53 | for parsing the :term:`Metadata`, generating | ||
54 | a list of tasks from it, and then executing those tasks. | ||
55 | |||
56 | This section briefly introduces BitBake. If you want more information on | ||
57 | BitBake, see the :doc:`BitBake User Manual <bitbake:index>`. | ||
58 | |||
59 | To see a list of the options BitBake supports, use either of the | ||
60 | following commands: | ||
61 | :: | ||
62 | |||
63 | $ bitbake -h | ||
64 | $ bitbake --help | ||
65 | |||
66 | The most common usage for BitBake is ``bitbake recipename``, where | ||
67 | ``recipename`` is the name of the recipe you want to build (referred | ||
68 | to as the "target"). The target often equates to the first part of a | ||
69 | recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``). | ||
70 | So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might | ||
71 | type the following: | ||
72 | :: | ||
73 | |||
74 | $ bitbake matchbox-desktop | ||
75 | |||
76 | Several different | ||
77 | versions of ``matchbox-desktop`` might exist. BitBake chooses the one | ||
78 | selected by the distribution configuration. You can get more details | ||
79 | about how BitBake chooses between different target versions and | ||
80 | providers in the | ||
81 | ":ref:`Preferences <bitbake:bb-bitbake-preferences>`" section | ||
82 | of the BitBake User Manual. | ||
83 | |||
84 | BitBake also tries to execute any dependent tasks first. So for example, | ||
85 | before building ``matchbox-desktop``, BitBake would build a cross | ||
86 | compiler and ``glibc`` if they had not already been built. | ||
87 | |||
88 | A useful BitBake option to consider is the ``-k`` or ``--continue`` | ||
89 | option. This option instructs BitBake to try and continue processing the | ||
90 | job as long as possible even after encountering an error. When an error | ||
91 | occurs, the target that failed and those that depend on it cannot be | ||
92 | remade. However, when you use this option other dependencies can still | ||
93 | be processed. | ||
94 | |||
95 | .. _overview-components-recipes: | ||
96 | |||
97 | Recipes | ||
98 | ------- | ||
99 | |||
100 | Files that have the ``.bb`` suffix are "recipes" files. In general, a | ||
101 | recipe contains information about a single piece of software. This | ||
102 | information includes the location from which to download the unaltered | ||
103 | source, any source patches to be applied to that source (if needed), | ||
104 | which special configuration options to apply, how to compile the source | ||
105 | files, and how to package the compiled output. | ||
106 | |||
107 | The term "package" is sometimes used to refer to recipes. However, since | ||
108 | the word "package" is used for the packaged output from the OpenEmbedded | ||
109 | build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids | ||
110 | using the term "package" when referring to recipes. | ||
111 | |||
112 | .. _overview-components-classes: | ||
113 | |||
114 | Classes | ||
115 | ------- | ||
116 | |||
117 | Class files (``.bbclass``) contain information that is useful to share | ||
118 | between recipes files. An example is the | ||
119 | :ref:`autotools <ref-classes-autotools>` class, | ||
120 | which contains common settings for any application that Autotools uses. | ||
121 | The ":ref:`ref-manual/ref-classes:Classes`" chapter in the | ||
122 | Yocto Project Reference Manual provides details about classes and how to | ||
123 | use them. | ||
124 | |||
125 | .. _overview-components-configurations: | ||
126 | |||
127 | Configurations | ||
128 | -------------- | ||
129 | |||
130 | The configuration files (``.conf``) define various configuration | ||
131 | variables that govern the OpenEmbedded build process. These files fall | ||
132 | into several areas that define machine configuration options, | ||
133 | distribution configuration options, compiler tuning options, general | ||
134 | common configuration options, and user configuration options in | ||
135 | ``conf/local.conf``, which is found in the :term:`Build Directory`. | ||
136 | |||
137 | |||
138 | .. _overview-layers: | ||
139 | |||
140 | Layers | ||
141 | ====== | ||
142 | |||
143 | Layers are repositories that contain related metadata (i.e. sets of | ||
144 | instructions) that tell the OpenEmbedded build system how to build a | ||
145 | target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__ | ||
146 | facilitates collaboration, sharing, customization, and reuse within the | ||
147 | Yocto Project development environment. Layers logically separate | ||
148 | information for your project. For example, you can use a layer to hold | ||
149 | all the configurations for a particular piece of hardware. Isolating | ||
150 | hardware-specific configurations allows you to share other metadata by | ||
151 | using a different layer where that metadata might be common across | ||
152 | several pieces of hardware. | ||
153 | |||
154 | Many layers exist that work in the Yocto Project development | ||
155 | environment. The `Yocto Project Curated Layer | ||
156 | Index <https://www.yoctoproject.org/software-overview/layers/>`__ | ||
157 | and `OpenEmbedded Layer | ||
158 | Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__ | ||
159 | both contain layers from which you can use or leverage. | ||
160 | |||
161 | By convention, layers in the Yocto Project follow a specific form. | ||
162 | Conforming to a known structure allows BitBake to make assumptions | ||
163 | during builds on where to find types of metadata. You can find | ||
164 | procedures and learn about tools (i.e. ``bitbake-layers``) for creating | ||
165 | layers suitable for the Yocto Project in the | ||
166 | ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" | ||
167 | section of the Yocto Project Development Tasks Manual. | ||
168 | |||
169 | .. _openembedded-build-system-build-concepts: | ||
170 | |||
171 | OpenEmbedded Build System Concepts | ||
172 | ================================== | ||
173 | |||
174 | This section takes a more detailed look inside the build process used by | ||
175 | the :term:`OpenEmbedded Build System`, | ||
176 | which is the build | ||
177 | system specific to the Yocto Project. At the heart of the build system | ||
178 | is BitBake, the task executor. | ||
179 | |||
180 | The following diagram represents the high-level workflow of a build. The | ||
181 | remainder of this section expands on the fundamental input, output, | ||
182 | process, and metadata logical blocks that make up the workflow. | ||
183 | |||
184 | .. image:: figures/YP-flow-diagram.png | ||
185 | :align: center | ||
186 | |||
187 | In general, the build's workflow consists of several functional areas: | ||
188 | |||
189 | - *User Configuration:* metadata you can use to control the build | ||
190 | process. | ||
191 | |||
192 | - *Metadata Layers:* Various layers that provide software, machine, and | ||
193 | distro metadata. | ||
194 | |||
195 | - *Source Files:* Upstream releases, local projects, and SCMs. | ||
196 | |||
197 | - *Build System:* Processes under the control of | ||
198 | :term:`BitBake`. This block expands | ||
199 | on how BitBake fetches source, applies patches, completes | ||
200 | compilation, analyzes output for package generation, creates and | ||
201 | tests packages, generates images, and generates cross-development | ||
202 | tools. | ||
203 | |||
204 | - *Package Feeds:* Directories containing output packages (RPM, DEB or | ||
205 | IPK), which are subsequently used in the construction of an image or | ||
206 | Software Development Kit (SDK), produced by the build system. These | ||
207 | feeds can also be copied and shared using a web server or other means | ||
208 | to facilitate extending or updating existing images on devices at | ||
209 | runtime if runtime package management is enabled. | ||
210 | |||
211 | - *Images:* Images produced by the workflow. | ||
212 | |||
213 | - *Application Development SDK:* Cross-development tools that are | ||
214 | produced along with an image or separately with BitBake. | ||
215 | |||
216 | User Configuration | ||
217 | ------------------ | ||
218 | |||
219 | User configuration helps define the build. Through user configuration, | ||
220 | you can tell BitBake the target architecture for which you are building | ||
221 | the image, where to store downloaded source, and other build properties. | ||
222 | |||
223 | The following figure shows an expanded representation of the "User | ||
224 | Configuration" box of the `general workflow | ||
225 | figure <#general-workflow-figure>`__: | ||
226 | |||
227 | .. image:: figures/user-configuration.png | ||
228 | :align: center | ||
229 | |||
230 | BitBake needs some basic configuration files in order to complete a | ||
231 | build. These files are ``*.conf`` files. The minimally necessary ones | ||
232 | reside as example files in the ``build/conf`` directory of the | ||
233 | :term:`Source Directory`. For simplicity, | ||
234 | this section refers to the Source Directory as the "Poky Directory." | ||
235 | |||
236 | When you clone the :term:`Poky` Git repository | ||
237 | or you download and unpack a Yocto Project release, you can set up the | ||
238 | Source Directory to be named anything you want. For this discussion, the | ||
239 | cloned repository uses the default name ``poky``. | ||
240 | |||
241 | .. note:: | ||
242 | |||
243 | The Poky repository is primarily an aggregation of existing | ||
244 | repositories. It is not a canonical upstream source. | ||
245 | |||
246 | The ``meta-poky`` layer inside Poky contains a ``conf`` directory that | ||
247 | has example configuration files. These example files are used as a basis | ||
248 | for creating actual configuration files when you source | ||
249 | :ref:`structure-core-script`, which is the | ||
250 | build environment script. | ||
251 | |||
252 | Sourcing the build environment script creates a | ||
253 | :term:`Build Directory` if one does not | ||
254 | already exist. BitBake uses the Build Directory for all its work during | ||
255 | builds. The Build Directory has a ``conf`` directory that contains | ||
256 | default versions of your ``local.conf`` and ``bblayers.conf`` | ||
257 | configuration files. These default configuration files are created only | ||
258 | if versions do not already exist in the Build Directory at the time you | ||
259 | source the build environment setup script. | ||
260 | |||
261 | Because the Poky repository is fundamentally an aggregation of existing | ||
262 | repositories, some users might be familiar with running the | ||
263 | :ref:`structure-core-script` script in the context of separate | ||
264 | :term:`OpenEmbedded-Core (OE-Core)` and BitBake | ||
265 | repositories rather than a single Poky repository. This discussion | ||
266 | assumes the script is executed from within a cloned or unpacked version | ||
267 | of Poky. | ||
268 | |||
269 | Depending on where the script is sourced, different sub-scripts are | ||
270 | called to set up the Build Directory (Yocto or OpenEmbedded). | ||
271 | Specifically, the script ``scripts/oe-setup-builddir`` inside the poky | ||
272 | directory sets up the Build Directory and seeds the directory (if | ||
273 | necessary) with configuration files appropriate for the Yocto Project | ||
274 | development environment. | ||
275 | |||
276 | .. note:: | ||
277 | |||
278 | The | ||
279 | scripts/oe-setup-builddir | ||
280 | script uses the | ||
281 | ``$TEMPLATECONF`` | ||
282 | variable to determine which sample configuration files to locate. | ||
283 | |||
284 | The ``local.conf`` file provides many basic variables that define a | ||
285 | build environment. Here is a list of a few. To see the default | ||
286 | configurations in a ``local.conf`` file created by the build environment | ||
287 | script, see the | ||
288 | :yocto_git:`local.conf.sample </cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample>` | ||
289 | in the ``meta-poky`` layer: | ||
290 | |||
291 | - *Target Machine Selection:* Controlled by the | ||
292 | :term:`MACHINE` variable. | ||
293 | |||
294 | - *Download Directory:* Controlled by the | ||
295 | :term:`DL_DIR` variable. | ||
296 | |||
297 | - *Shared State Directory:* Controlled by the | ||
298 | :term:`SSTATE_DIR` variable. | ||
299 | |||
300 | - *Build Output:* Controlled by the | ||
301 | :term:`TMPDIR` variable. | ||
302 | |||
303 | - *Distribution Policy:* Controlled by the | ||
304 | :term:`DISTRO` variable. | ||
305 | |||
306 | - *Packaging Format:* Controlled by the | ||
307 | :term:`PACKAGE_CLASSES` | ||
308 | variable. | ||
309 | |||
310 | - *SDK Target Architecture:* Controlled by the | ||
311 | :term:`SDKMACHINE` variable. | ||
312 | |||
313 | - *Extra Image Packages:* Controlled by the | ||
314 | :term:`EXTRA_IMAGE_FEATURES` | ||
315 | variable. | ||
316 | |||
317 | .. note:: | ||
318 | |||
319 | Configurations set in the | ||
320 | conf/local.conf | ||
321 | file can also be set in the | ||
322 | conf/site.conf | ||
323 | and | ||
324 | conf/auto.conf | ||
325 | configuration files. | ||
326 | |||
327 | The ``bblayers.conf`` file tells BitBake what layers you want considered | ||
328 | during the build. By default, the layers listed in this file include | ||
329 | layers minimally needed by the build system. However, you must manually | ||
330 | add any custom layers you have created. You can find more information on | ||
331 | working with the ``bblayers.conf`` file in the | ||
332 | ":ref:`dev-manual/dev-manual-common-tasks:enabling your layer`" | ||
333 | section in the Yocto Project Development Tasks Manual. | ||
334 | |||
335 | The files ``site.conf`` and ``auto.conf`` are not created by the | ||
336 | environment initialization script. If you want the ``site.conf`` file, | ||
337 | you need to create that yourself. The ``auto.conf`` file is typically | ||
338 | created by an autobuilder: | ||
339 | |||
340 | - *site.conf:* You can use the ``conf/site.conf`` configuration | ||
341 | file to configure multiple build directories. For example, suppose | ||
342 | you had several build environments and they shared some common | ||
343 | features. You can set these default build properties here. A good | ||
344 | example is perhaps the packaging format to use through the | ||
345 | :term:`PACKAGE_CLASSES` | ||
346 | variable. | ||
347 | |||
348 | One useful scenario for using the ``conf/site.conf`` file is to | ||
349 | extend your :term:`BBPATH` variable | ||
350 | to include the path to a ``conf/site.conf``. Then, when BitBake looks | ||
351 | for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file | ||
352 | and applies your common configurations found in the file. To override | ||
353 | configurations in a particular build directory, alter the similar | ||
354 | configurations within that build directory's ``conf/local.conf`` | ||
355 | file. | ||
356 | |||
357 | - *auto.conf:* The file is usually created and written to by an | ||
358 | autobuilder. The settings put into the file are typically the same as | ||
359 | you would find in the ``conf/local.conf`` or the ``conf/site.conf`` | ||
360 | files. | ||
361 | |||
362 | You can edit all configuration files to further define any particular | ||
363 | build environment. This process is represented by the "User | ||
364 | Configuration Edits" box in the figure. | ||
365 | |||
366 | When you launch your build with the ``bitbake target`` command, BitBake | ||
367 | sorts out the configurations to ultimately define your build | ||
368 | environment. It is important to understand that the | ||
369 | :term:`OpenEmbedded Build System` reads the | ||
370 | configuration files in a specific order: ``site.conf``, ``auto.conf``, | ||
371 | and ``local.conf``. And, the build system applies the normal assignment | ||
372 | statement rules as described in the | ||
373 | ":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter | ||
374 | of the BitBake User Manual. Because the files are parsed in a specific | ||
375 | order, variable assignments for the same variable could be affected. For | ||
376 | example, if the ``auto.conf`` file and the ``local.conf`` set variable1 | ||
377 | to different values, because the build system parses ``local.conf`` | ||
378 | after ``auto.conf``, variable1 is assigned the value from the | ||
379 | ``local.conf`` file. | ||
380 | |||
381 | Metadata, Machine Configuration, and Policy Configuration | ||
382 | --------------------------------------------------------- | ||
383 | |||
384 | The previous section described the user configurations that define | ||
385 | BitBake's global behavior. This section takes a closer look at the | ||
386 | layers the build system uses to further control the build. These layers | ||
387 | provide Metadata for the software, machine, and policies. | ||
388 | |||
389 | In general, three types of layer input exists. You can see them below | ||
390 | the "User Configuration" box in the `general workflow | ||
391 | figure <#general-workflow-figure>`__: | ||
392 | |||
393 | - *Metadata (.bb + Patches):* Software layers containing | ||
394 | user-supplied recipe files, patches, and append files. A good example | ||
395 | of a software layer might be the | ||
396 | `meta-qt5 layer <https://github.com/meta-qt5/meta-qt5>`__ from | ||
397 | the `OpenEmbedded Layer | ||
398 | Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__. | ||
399 | This layer is for version 5.0 of the popular | ||
400 | `Qt <https://wiki.qt.io/About_Qt>`__ cross-platform application | ||
401 | development framework for desktop, embedded and mobile. | ||
402 | |||
403 | - *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e. | ||
404 | "BSP Layer" in the following figure) providing machine-specific | ||
405 | configurations. This type of information is specific to a particular | ||
406 | target architecture. A good example of a BSP layer from the `Poky | ||
407 | Reference Distribution <#gs-reference-distribution-poky>`__ is the | ||
408 | :yocto_git:`meta-yocto-bsp </cgit/cgit.cgi/poky/tree/meta-yocto-bsp>` | ||
409 | layer. | ||
410 | |||
411 | - *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in | ||
412 | the following figure) providing top-level or general policies for the | ||
413 | images or SDKs being built for a particular distribution. For | ||
414 | example, in the Poky Reference Distribution the distro layer is the | ||
415 | :yocto_git:`meta-poky </cgit/cgit.cgi/poky/tree/meta-poky>` | ||
416 | layer. Within the distro layer is a ``conf/distro`` directory that | ||
417 | contains distro configuration files (e.g. | ||
418 | :yocto_git:`poky.conf </cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf>` | ||
419 | that contain many policy configurations for the Poky distribution. | ||
420 | |||
421 | The following figure shows an expanded representation of these three | ||
422 | layers from the `general workflow figure <#general-workflow-figure>`__: | ||
423 | |||
424 | .. image:: figures/layer-input.png | ||
425 | :align: center | ||
426 | |||
427 | In general, all layers have a similar structure. They all contain a | ||
428 | licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed, | ||
429 | a ``README`` file as good practice and especially if the layer is to be | ||
430 | distributed, a configuration directory, and recipe directories. You can | ||
431 | learn about the general structure for layers used with the Yocto Project | ||
432 | in the | ||
433 | ":ref:`dev-manual/dev-manual-common-tasks:creating your own layer`" | ||
434 | section in the | ||
435 | Yocto Project Development Tasks Manual. For a general discussion on | ||
436 | layers and the many layers from which you can draw, see the | ||
437 | "`Layers <#overview-layers>`__" and "`The Yocto Project Layer | ||
438 | Model <#the-yocto-project-layer-model>`__" sections both earlier in this | ||
439 | manual. | ||
440 | |||
441 | If you explored the previous links, you discovered some areas where many | ||
442 | layers that work with the Yocto Project exist. The `Source | ||
443 | Repositories <http://git.yoctoproject.org/>`__ also shows layers | ||
444 | categorized under "Yocto Metadata Layers." | ||
445 | |||
446 | .. note:: | ||
447 | |||
448 | Layers exist in the Yocto Project Source Repositories that cannot be | ||
449 | found in the OpenEmbedded Layer Index. These layers are either | ||
450 | deprecated or experimental in nature. | ||
451 | |||
452 | BitBake uses the ``conf/bblayers.conf`` file, which is part of the user | ||
453 | configuration, to find what layers it should be using as part of the | ||
454 | build. | ||
455 | |||
456 | Distro Layer | ||
457 | ~~~~~~~~~~~~ | ||
458 | |||
459 | The distribution layer provides policy configurations for your | ||
460 | distribution. Best practices dictate that you isolate these types of | ||
461 | configurations into their own layer. Settings you provide in | ||
462 | ``conf/distro/distro.conf`` override similar settings that BitBake finds | ||
463 | in your ``conf/local.conf`` file in the Build Directory. | ||
464 | |||
465 | The following list provides some explanation and references for what you | ||
466 | typically find in the distribution layer: | ||
467 | |||
468 | - *classes:* Class files (``.bbclass``) hold common functionality that | ||
469 | can be shared among recipes in the distribution. When your recipes | ||
470 | inherit a class, they take on the settings and functions for that | ||
471 | class. You can read more about class files in the | ||
472 | ":ref:`ref-manual/ref-classes:Classes`" chapter of the Yocto | ||
473 | Reference Manual. | ||
474 | |||
475 | - *conf:* This area holds configuration files for the layer | ||
476 | (``conf/layer.conf``), the distribution | ||
477 | (``conf/distro/distro.conf``), and any distribution-wide include | ||
478 | files. | ||
479 | |||
480 | - *recipes-*:* Recipes and append files that affect common | ||
481 | functionality across the distribution. This area could include | ||
482 | recipes and append files to add distribution-specific configuration, | ||
483 | initialization scripts, custom image recipes, and so forth. Examples | ||
484 | of ``recipes-*`` directories are ``recipes-core`` and | ||
485 | ``recipes-extra``. Hierarchy and contents within a ``recipes-*`` | ||
486 | directory can vary. Generally, these directories contain recipe files | ||
487 | (``*.bb``), recipe append files (``*.bbappend``), directories that | ||
488 | are distro-specific for configuration files, and so forth. | ||
489 | |||
490 | BSP Layer | ||
491 | ~~~~~~~~~ | ||
492 | |||
493 | The BSP Layer provides machine configurations that target specific | ||
494 | hardware. Everything in this layer is specific to the machine for which | ||
495 | you are building the image or the SDK. A common structure or form is | ||
496 | defined for BSP layers. You can learn more about this structure in the | ||
497 | :doc:`../bsp-guide/bsp-guide`. | ||
498 | |||
499 | .. note:: | ||
500 | |||
501 | In order for a BSP layer to be considered compliant with the Yocto | ||
502 | Project, it must meet some structural requirements. | ||
503 | |||
504 | The BSP Layer's configuration directory contains configuration files for | ||
505 | the machine (``conf/machine/machine.conf``) and, of course, the layer | ||
506 | (``conf/layer.conf``). | ||
507 | |||
508 | The remainder of the layer is dedicated to specific recipes by function: | ||
509 | ``recipes-bsp``, ``recipes-core``, ``recipes-graphics``, | ||
510 | ``recipes-kernel``, and so forth. Metadata can exist for multiple | ||
511 | formfactors, graphics support systems, and so forth. | ||
512 | |||
513 | .. note:: | ||
514 | |||
515 | While the figure shows several | ||
516 | recipes-\* | ||
517 | directories, not all these directories appear in all BSP layers. | ||
518 | |||
519 | Software Layer | ||
520 | ~~~~~~~~~~~~~~ | ||
521 | |||
522 | The software layer provides the Metadata for additional software | ||
523 | packages used during the build. This layer does not include Metadata | ||
524 | that is specific to the distribution or the machine, which are found in | ||
525 | their respective layers. | ||
526 | |||
527 | This layer contains any recipes, append files, and patches, that your | ||
528 | project needs. | ||
529 | |||
530 | .. _sources-dev-environment: | ||
531 | |||
532 | Sources | ||
533 | ------- | ||
534 | |||
535 | In order for the OpenEmbedded build system to create an image or any | ||
536 | target, it must be able to access source files. The `general workflow | ||
537 | figure <#general-workflow-figure>`__ represents source files using the | ||
538 | "Upstream Project Releases", "Local Projects", and "SCMs (optional)" | ||
539 | boxes. The figure represents mirrors, which also play a role in locating | ||
540 | source files, with the "Source Materials" box. | ||
541 | |||
542 | The method by which source files are ultimately organized is a function | ||
543 | of the project. For example, for released software, projects tend to use | ||
544 | tarballs or other archived files that can capture the state of a release | ||
545 | guaranteeing that it is statically represented. On the other hand, for a | ||
546 | project that is more dynamic or experimental in nature, a project might | ||
547 | keep source files in a repository controlled by a Source Control Manager | ||
548 | (SCM) such as Git. Pulling source from a repository allows you to | ||
549 | control the point in the repository (the revision) from which you want | ||
550 | to build software. Finally, a combination of the two might exist, which | ||
551 | would give the consumer a choice when deciding where to get source | ||
552 | files. | ||
553 | |||
554 | BitBake uses the :term:`SRC_URI` | ||
555 | variable to point to source files regardless of their location. Each | ||
556 | recipe must have a ``SRC_URI`` variable that points to the source. | ||
557 | |||
558 | Another area that plays a significant role in where source files come | ||
559 | from is pointed to by the | ||
560 | :term:`DL_DIR` variable. This area is | ||
561 | a cache that can hold previously downloaded source. You can also | ||
562 | instruct the OpenEmbedded build system to create tarballs from Git | ||
563 | repositories, which is not the default behavior, and store them in the | ||
564 | ``DL_DIR`` by using the | ||
565 | :term:`BB_GENERATE_MIRROR_TARBALLS` | ||
566 | variable. | ||
567 | |||
568 | Judicious use of a ``DL_DIR`` directory can save the build system a trip | ||
569 | across the Internet when looking for files. A good method for using a | ||
570 | download directory is to have ``DL_DIR`` point to an area outside of | ||
571 | your Build Directory. Doing so allows you to safely delete the Build | ||
572 | Directory if needed without fear of removing any downloaded source file. | ||
573 | |||
574 | The remainder of this section provides a deeper look into the source | ||
575 | files and the mirrors. Here is a more detailed look at the source file | ||
576 | area of the `general workflow figure <#general-workflow-figure>`__: | ||
577 | |||
578 | .. image:: figures/source-input.png | ||
579 | :align: center | ||
580 | |||
581 | Upstream Project Releases | ||
582 | ~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
583 | |||
584 | Upstream project releases exist anywhere in the form of an archived file | ||
585 | (e.g. tarball or zip file). These files correspond to individual | ||
586 | recipes. For example, the figure uses specific releases each for | ||
587 | BusyBox, Qt, and Dbus. An archive file can be for any released product | ||
588 | that can be built using a recipe. | ||
589 | |||
590 | Local Projects | ||
591 | ~~~~~~~~~~~~~~ | ||
592 | |||
593 | Local projects are custom bits of software the user provides. These bits | ||
594 | reside somewhere local to a project - perhaps a directory into which the | ||
595 | user checks in items (e.g. a local directory containing a development | ||
596 | source tree used by the group). | ||
597 | |||
598 | The canonical method through which to include a local project is to use | ||
599 | the :ref:`externalsrc <ref-classes-externalsrc>` | ||
600 | class to include that local project. You use either the ``local.conf`` | ||
601 | or a recipe's append file to override or set the recipe to point to the | ||
602 | local directory on your disk to pull in the whole source tree. | ||
603 | |||
604 | .. _scms: | ||
605 | |||
606 | Source Control Managers (Optional) | ||
607 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
608 | |||
609 | Another place from which the build system can get source files is with | ||
610 | :ref:`fetchers <bitbake:bb-fetchers>` employing various Source | ||
611 | Control Managers (SCMs) such as Git or Subversion. In such cases, a | ||
612 | repository is cloned or checked out. The | ||
613 | :ref:`ref-tasks-fetch` task inside | ||
614 | BitBake uses the :term:`SRC_URI` | ||
615 | variable and the argument's prefix to determine the correct fetcher | ||
616 | module. | ||
617 | |||
618 | .. note:: | ||
619 | |||
620 | For information on how to have the OpenEmbedded build system generate | ||
621 | tarballs for Git repositories and place them in the | ||
622 | DL_DIR | ||
623 | directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS` | ||
624 | variable in the Yocto Project Reference Manual. | ||
625 | |||
626 | When fetching a repository, BitBake uses the | ||
627 | :term:`SRCREV` variable to determine | ||
628 | the specific revision from which to build. | ||
629 | |||
630 | Source Mirror(s) | ||
631 | ~~~~~~~~~~~~~~~~ | ||
632 | |||
633 | Two kinds of mirrors exist: pre-mirrors and regular mirrors. The | ||
634 | :term:`PREMIRRORS` and | ||
635 | :term:`MIRRORS` variables point to | ||
636 | these, respectively. BitBake checks pre-mirrors before looking upstream | ||
637 | for any source files. Pre-mirrors are appropriate when you have a shared | ||
638 | directory that is not a directory defined by the | ||
639 | :term:`DL_DIR` variable. A Pre-mirror | ||
640 | typically points to a shared directory that is local to your | ||
641 | organization. | ||
642 | |||
643 | Regular mirrors can be any site across the Internet that is used as an | ||
644 | alternative location for source code should the primary site not be | ||
645 | functioning for some reason or another. | ||
646 | |||
647 | .. _package-feeds-dev-environment: | ||
648 | |||
649 | Package Feeds | ||
650 | ------------- | ||
651 | |||
652 | When the OpenEmbedded build system generates an image or an SDK, it gets | ||
653 | the packages from a package feed area located in the | ||
654 | :term:`Build Directory`. The `general | ||
655 | workflow figure <#general-workflow-figure>`__ shows this package feeds | ||
656 | area in the upper-right corner. | ||
657 | |||
658 | This section looks a little closer into the package feeds area used by | ||
659 | the build system. Here is a more detailed look at the area: | ||
660 | |||
661 | .. image:: figures/package-feeds.png | ||
662 | :align: center | ||
663 | |||
664 | Package feeds are an intermediary step in the build process. The | ||
665 | OpenEmbedded build system provides classes to generate different package | ||
666 | types, and you specify which classes to enable through the | ||
667 | :term:`PACKAGE_CLASSES` | ||
668 | variable. Before placing the packages into package feeds, the build | ||
669 | process validates them with generated output quality assurance checks | ||
670 | through the :ref:`insane <ref-classes-insane>` | ||
671 | class. | ||
672 | |||
673 | The package feed area resides in the Build Directory. The directory the | ||
674 | build system uses to temporarily store packages is determined by a | ||
675 | combination of variables and the particular package manager in use. See | ||
676 | the "Package Feeds" box in the illustration and note the information to | ||
677 | the right of that area. In particular, the following defines where | ||
678 | package files are kept: | ||
679 | |||
680 | - :term:`DEPLOY_DIR`: Defined as | ||
681 | ``tmp/deploy`` in the Build Directory. | ||
682 | |||
683 | - ``DEPLOY_DIR_*``: Depending on the package manager used, the package | ||
684 | type sub-folder. Given RPM, IPK, or DEB packaging and tarball | ||
685 | creation, the | ||
686 | :term:`DEPLOY_DIR_RPM`, | ||
687 | :term:`DEPLOY_DIR_IPK`, | ||
688 | :term:`DEPLOY_DIR_DEB`, or | ||
689 | :term:`DEPLOY_DIR_TAR`, | ||
690 | variables are used, respectively. | ||
691 | |||
692 | - :term:`PACKAGE_ARCH`: Defines | ||
693 | architecture-specific sub-folders. For example, packages could exist | ||
694 | for the i586 or qemux86 architectures. | ||
695 | |||
696 | BitBake uses the | ||
697 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | ||
698 | tasks to generate packages and place them into the package holding area | ||
699 | (e.g. ``do_package_write_ipk`` for IPK packages). See the | ||
700 | ":ref:`ref-tasks-package_write_deb`", | ||
701 | ":ref:`ref-tasks-package_write_ipk`", | ||
702 | ":ref:`ref-tasks-package_write_rpm`", | ||
703 | and | ||
704 | ":ref:`ref-tasks-package_write_tar`" | ||
705 | sections in the Yocto Project Reference Manual for additional | ||
706 | information. As an example, consider a scenario where an IPK packaging | ||
707 | manager is being used and package architecture support for both i586 and | ||
708 | qemux86 exist. Packages for the i586 architecture are placed in | ||
709 | ``build/tmp/deploy/ipk/i586``, while packages for the qemux86 | ||
710 | architecture are placed in ``build/tmp/deploy/ipk/qemux86``. | ||
711 | |||
712 | .. _bitbake-dev-environment: | ||
713 | |||
714 | BitBake Tool | ||
715 | ------------ | ||
716 | |||
717 | The OpenEmbedded build system uses | ||
718 | :term:`BitBake` to produce images and | ||
719 | Software Development Kits (SDKs). You can see from the `general workflow | ||
720 | figure <#general-workflow-figure>`__, the BitBake area consists of | ||
721 | several functional areas. This section takes a closer look at each of | ||
722 | those areas. | ||
723 | |||
724 | .. note:: | ||
725 | |||
726 | Separate documentation exists for the BitBake tool. See the | ||
727 | BitBake User Manual | ||
728 | for reference material on BitBake. | ||
729 | |||
730 | .. _source-fetching-dev-environment: | ||
731 | |||
732 | Source Fetching | ||
733 | ~~~~~~~~~~~~~~~ | ||
734 | |||
735 | The first stages of building a recipe are to fetch and unpack the source | ||
736 | code: | ||
737 | |||
738 | .. image:: figures/source-fetching.png | ||
739 | :align: center | ||
740 | |||
741 | The :ref:`ref-tasks-fetch` and | ||
742 | :ref:`ref-tasks-unpack` tasks fetch | ||
743 | the source files and unpack them into the | ||
744 | :term:`Build Directory`. | ||
745 | |||
746 | .. note:: | ||
747 | |||
748 | For every local file (e.g. | ||
749 | file:// | ||
750 | ) that is part of a recipe's | ||
751 | SRC_URI | ||
752 | statement, the OpenEmbedded build system takes a checksum of the file | ||
753 | for the recipe and inserts the checksum into the signature for the | ||
754 | do_fetch | ||
755 | task. If any local file has been modified, the | ||
756 | do_fetch | ||
757 | task and all tasks that depend on it are re-executed. | ||
758 | |||
759 | By default, everything is accomplished in the Build Directory, which has | ||
760 | a defined structure. For additional general information on the Build | ||
761 | Directory, see the ":ref:`structure-core-build`" section in | ||
762 | the Yocto Project Reference Manual. | ||
763 | |||
764 | Each recipe has an area in the Build Directory where the unpacked source | ||
765 | code resides. The :term:`S` variable points | ||
766 | to this area for a recipe's unpacked source code. The name of that | ||
767 | directory for any given recipe is defined from several different | ||
768 | variables. The preceding figure and the following list describe the | ||
769 | Build Directory's hierarchy: | ||
770 | |||
771 | - :term:`TMPDIR`: The base directory | ||
772 | where the OpenEmbedded build system performs all its work during the | ||
773 | build. The default base directory is the ``tmp`` directory. | ||
774 | |||
775 | - :term:`PACKAGE_ARCH`: The | ||
776 | architecture of the built package or packages. Depending on the | ||
777 | eventual destination of the package or packages (i.e. machine | ||
778 | architecture, :term:`Build Host`, SDK, or | ||
779 | specific machine), ``PACKAGE_ARCH`` varies. See the variable's | ||
780 | description for details. | ||
781 | |||
782 | - :term:`TARGET_OS`: The operating | ||
783 | system of the target device. A typical value would be "linux" (e.g. | ||
784 | "qemux86-poky-linux"). | ||
785 | |||
786 | - :term:`PN`: The name of the recipe used | ||
787 | to build the package. This variable can have multiple meanings. | ||
788 | However, when used in the context of input files, ``PN`` represents | ||
789 | the name of the recipe. | ||
790 | |||
791 | - :term:`WORKDIR`: The location | ||
792 | where the OpenEmbedded build system builds a recipe (i.e. does the | ||
793 | work to create the package). | ||
794 | |||
795 | - :term:`PV`: The version of the | ||
796 | recipe used to build the package. | ||
797 | |||
798 | - :term:`PR`: The revision of the | ||
799 | recipe used to build the package. | ||
800 | |||
801 | - :term:`S`: Contains the unpacked source | ||
802 | files for a given recipe. | ||
803 | |||
804 | - :term:`BPN`: The name of the recipe | ||
805 | used to build the package. The ``BPN`` variable is a version of | ||
806 | the ``PN`` variable but with common prefixes and suffixes removed. | ||
807 | |||
808 | - :term:`PV`: The version of the | ||
809 | recipe used to build the package. | ||
810 | |||
811 | .. note:: | ||
812 | |||
813 | In the previous figure, notice that two sample hierarchies exist: one | ||
814 | based on package architecture (i.e. | ||
815 | PACKAGE_ARCH | ||
816 | ) and one based on a machine (i.e. | ||
817 | MACHINE | ||
818 | ). The underlying structures are identical. The differentiator being | ||
819 | what the OpenEmbedded build system is using as a build target (e.g. | ||
820 | general architecture, a build host, an SDK, or a specific machine). | ||
821 | |||
822 | .. _patching-dev-environment: | ||
823 | |||
824 | Patching | ||
825 | ~~~~~~~~ | ||
826 | |||
827 | Once source code is fetched and unpacked, BitBake locates patch files | ||
828 | and applies them to the source files: | ||
829 | |||
830 | .. image:: figures/patching.png | ||
831 | :align: center | ||
832 | |||
833 | The :ref:`ref-tasks-patch` task uses a | ||
834 | recipe's :term:`SRC_URI` statements | ||
835 | and the :term:`FILESPATH` variable | ||
836 | to locate applicable patch files. | ||
837 | |||
838 | Default processing for patch files assumes the files have either | ||
839 | ``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters | ||
840 | to change the way the build system recognizes patch files. See the | ||
841 | :ref:`ref-tasks-patch` task for more | ||
842 | information. | ||
843 | |||
844 | BitBake finds and applies multiple patches for a single recipe in the | ||
845 | order in which it locates the patches. The ``FILESPATH`` variable | ||
846 | defines the default set of directories that the build system uses to | ||
847 | search for patch files. Once found, patches are applied to the recipe's | ||
848 | source files, which are located in the | ||
849 | :term:`S` directory. | ||
850 | |||
851 | For more information on how the source directories are created, see the | ||
852 | "`Source Fetching <#source-fetching-dev-environment>`__" section. For | ||
853 | more information on how to create patches and how the build system | ||
854 | processes patches, see the | ||
855 | ":ref:`dev-manual/dev-manual-common-tasks:patching code`" | ||
856 | section in the | ||
857 | Yocto Project Development Tasks Manual. You can also see the | ||
858 | ":ref:`sdk-manual/sdk-extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" | ||
859 | section in the Yocto Project Application Development and the Extensible | ||
860 | Software Development Kit (SDK) manual and the | ||
861 | ":ref:`kernel-dev/kernel-dev-common:using traditional kernel development to patch the kernel`" | ||
862 | section in the Yocto Project Linux Kernel Development Manual. | ||
863 | |||
864 | .. _configuration-compilation-and-staging-dev-environment: | ||
865 | |||
866 | Configuration, Compilation, and Staging | ||
867 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
868 | |||
869 | After source code is patched, BitBake executes tasks that configure and | ||
870 | compile the source code. Once compilation occurs, the files are copied | ||
871 | to a holding area (staged) in preparation for packaging: | ||
872 | |||
873 | .. image:: figures/configuration-compile-autoreconf.png | ||
874 | :align: center | ||
875 | |||
876 | This step in the build process consists of the following tasks: | ||
877 | |||
878 | - :ref:`ref-tasks-prepare_recipe_sysroot`: | ||
879 | This task sets up the two sysroots in | ||
880 | ``${``\ :term:`WORKDIR`\ ``}`` | ||
881 | (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that | ||
882 | during the packaging phase the sysroots can contain the contents of | ||
883 | the | ||
884 | :ref:`ref-tasks-populate_sysroot` | ||
885 | tasks of the recipes on which the recipe containing the tasks | ||
886 | depends. A sysroot exists for both the target and for the native | ||
887 | binaries, which run on the host system. | ||
888 | |||
889 | - *do_configure*: This task configures the source by enabling and | ||
890 | disabling any build-time and configuration options for the software | ||
891 | being built. Configurations can come from the recipe itself as well | ||
892 | as from an inherited class. Additionally, the software itself might | ||
893 | configure itself depending on the target for which it is being built. | ||
894 | |||
895 | The configurations handled by the | ||
896 | :ref:`ref-tasks-configure` task | ||
897 | are specific to configurations for the source code being built by the | ||
898 | recipe. | ||
899 | |||
900 | If you are using the | ||
901 | :ref:`autotools <ref-classes-autotools>` class, | ||
902 | you can add additional configuration options by using the | ||
903 | :term:`EXTRA_OECONF` or | ||
904 | :term:`PACKAGECONFIG_CONFARGS` | ||
905 | variables. For information on how this variable works within that | ||
906 | class, see the | ||
907 | :ref:`autotools <ref-classes-autotools>` class | ||
908 | :yocto_git:`here </cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass>`. | ||
909 | |||
910 | - *do_compile*: Once a configuration task has been satisfied, | ||
911 | BitBake compiles the source using the | ||
912 | :ref:`ref-tasks-compile` task. | ||
913 | Compilation occurs in the directory pointed to by the | ||
914 | :term:`B` variable. Realize that the | ||
915 | ``B`` directory is, by default, the same as the | ||
916 | :term:`S` directory. | ||
917 | |||
918 | - *do_install*: After compilation completes, BitBake executes the | ||
919 | :ref:`ref-tasks-install` task. | ||
920 | This task copies files from the ``B`` directory and places them in a | ||
921 | holding area pointed to by the :term:`D` | ||
922 | variable. Packaging occurs later using files from this holding | ||
923 | directory. | ||
924 | |||
925 | .. _package-splitting-dev-environment: | ||
926 | |||
927 | Package Splitting | ||
928 | ~~~~~~~~~~~~~~~~~ | ||
929 | |||
930 | After source code is configured, compiled, and staged, the build system | ||
931 | analyzes the results and splits the output into packages: | ||
932 | |||
933 | .. image:: figures/analysis-for-package-splitting.png | ||
934 | :align: center | ||
935 | |||
936 | The :ref:`ref-tasks-package` and | ||
937 | :ref:`ref-tasks-packagedata` | ||
938 | tasks combine to analyze the files found in the | ||
939 | :term:`D` directory and split them into | ||
940 | subsets based on available packages and files. Analysis involves the | ||
941 | following as well as other items: splitting out debugging symbols, | ||
942 | looking at shared library dependencies between packages, and looking at | ||
943 | package relationships. | ||
944 | |||
945 | The ``do_packagedata`` task creates package metadata based on the | ||
946 | analysis such that the build system can generate the final packages. The | ||
947 | :ref:`ref-tasks-populate_sysroot` | ||
948 | task stages (copies) a subset of the files installed by the | ||
949 | :ref:`ref-tasks-install` task into | ||
950 | the appropriate sysroot. Working, staged, and intermediate results of | ||
951 | the analysis and package splitting process use several areas: | ||
952 | |||
953 | - :term:`PKGD`: The destination | ||
954 | directory (i.e. ``package``) for packages before they are split into | ||
955 | individual packages. | ||
956 | |||
957 | - :term:`PKGDESTWORK`: A | ||
958 | temporary work area (i.e. ``pkgdata``) used by the ``do_package`` | ||
959 | task to save package metadata. | ||
960 | |||
961 | - :term:`PKGDEST`: The parent | ||
962 | directory (i.e. ``packages-split``) for packages after they have been | ||
963 | split. | ||
964 | |||
965 | - :term:`PKGDATA_DIR`: A shared, | ||
966 | global-state directory that holds packaging metadata generated during | ||
967 | the packaging process. The packaging process copies metadata from | ||
968 | ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally | ||
969 | available. | ||
970 | |||
971 | - :term:`STAGING_DIR_HOST`: | ||
972 | The path for the sysroot for the system on which a component is built | ||
973 | to run (i.e. ``recipe-sysroot``). | ||
974 | |||
975 | - :term:`STAGING_DIR_NATIVE`: | ||
976 | The path for the sysroot used when building components for the build | ||
977 | host (i.e. ``recipe-sysroot-native``). | ||
978 | |||
979 | - :term:`STAGING_DIR_TARGET`: | ||
980 | The path for the sysroot used when a component that is built to | ||
981 | execute on a system and it generates code for yet another machine | ||
982 | (e.g. cross-canadian recipes). | ||
983 | |||
984 | The :term:`FILES` variable defines the | ||
985 | files that go into each package in | ||
986 | :term:`PACKAGES`. If you want | ||
987 | details on how this is accomplished, you can look at | ||
988 | :yocto_git:`package.bbclass </cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass>`. | ||
989 | |||
990 | Depending on the type of packages being created (RPM, DEB, or IPK), the | ||
991 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | ||
992 | task creates the actual packages and places them in the Package Feed | ||
993 | area, which is ``${TMPDIR}/deploy``. You can see the "`Package | ||
994 | Feeds <#package-feeds-dev-environment>`__" section for more detail on | ||
995 | that part of the build process. | ||
996 | |||
997 | .. note:: | ||
998 | |||
999 | Support for creating feeds directly from the | ||
1000 | deploy/\* | ||
1001 | directories does not exist. Creating such feeds usually requires some | ||
1002 | kind of feed maintenance mechanism that would upload the new packages | ||
1003 | into an official package feed (e.g. the Ångström distribution). This | ||
1004 | functionality is highly distribution-specific and thus is not | ||
1005 | provided out of the box. | ||
1006 | |||
1007 | .. _image-generation-dev-environment: | ||
1008 | |||
1009 | Image Generation | ||
1010 | ~~~~~~~~~~~~~~~~ | ||
1011 | |||
1012 | Once packages are split and stored in the Package Feeds area, the build | ||
1013 | system uses BitBake to generate the root filesystem image: | ||
1014 | |||
1015 | .. image:: figures/image-generation.png | ||
1016 | :align: center | ||
1017 | |||
1018 | The image generation process consists of several stages and depends on | ||
1019 | several tasks and variables. The | ||
1020 | :ref:`ref-tasks-rootfs` task creates | ||
1021 | the root filesystem (file and directory structure) for an image. This | ||
1022 | task uses several key variables to help create the list of packages to | ||
1023 | actually install: | ||
1024 | |||
1025 | - :term:`IMAGE_INSTALL`: Lists | ||
1026 | out the base set of packages from which to install from the Package | ||
1027 | Feeds area. | ||
1028 | |||
1029 | - :term:`PACKAGE_EXCLUDE`: | ||
1030 | Specifies packages that should not be installed into the image. | ||
1031 | |||
1032 | - :term:`IMAGE_FEATURES`: | ||
1033 | Specifies features to include in the image. Most of these features | ||
1034 | map to additional packages for installation. | ||
1035 | |||
1036 | - :term:`PACKAGE_CLASSES`: | ||
1037 | Specifies the package backend (e.g. RPM, DEB, or IPK) to use and | ||
1038 | consequently helps determine where to locate packages within the | ||
1039 | Package Feeds area. | ||
1040 | |||
1041 | - :term:`IMAGE_LINGUAS`: | ||
1042 | Determines the language(s) for which additional language support | ||
1043 | packages are installed. | ||
1044 | |||
1045 | - :term:`PACKAGE_INSTALL`: | ||
1046 | The final list of packages passed to the package manager for | ||
1047 | installation into the image. | ||
1048 | |||
1049 | With :term:`IMAGE_ROOTFS` | ||
1050 | pointing to the location of the filesystem under construction and the | ||
1051 | ``PACKAGE_INSTALL`` variable providing the final list of packages to | ||
1052 | install, the root file system is created. | ||
1053 | |||
1054 | Package installation is under control of the package manager (e.g. | ||
1055 | dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package | ||
1056 | management is enabled for the target. At the end of the process, if | ||
1057 | package management is not enabled for the target, the package manager's | ||
1058 | data files are deleted from the root filesystem. As part of the final | ||
1059 | stage of package installation, post installation scripts that are part | ||
1060 | of the packages are run. Any scripts that fail to run on the build host | ||
1061 | are run on the target when the target system is first booted. If you are | ||
1062 | using a | ||
1063 | :ref:`read-only root filesystem <dev-manual/dev-manual-common-tasks:creating a read-only root filesystem>`, | ||
1064 | all the post installation scripts must succeed on the build host during | ||
1065 | the package installation phase since the root filesystem on the target | ||
1066 | is read-only. | ||
1067 | |||
1068 | The final stages of the ``do_rootfs`` task handle post processing. Post | ||
1069 | processing includes creation of a manifest file and optimizations. | ||
1070 | |||
1071 | The manifest file (``.manifest``) resides in the same directory as the | ||
1072 | root filesystem image. This file lists out, line-by-line, the installed | ||
1073 | packages. The manifest file is useful for the | ||
1074 | :ref:`testimage <ref-classes-testimage*>` class, | ||
1075 | for example, to determine whether or not to run specific tests. See the | ||
1076 | :term:`IMAGE_MANIFEST` | ||
1077 | variable for additional information. | ||
1078 | |||
1079 | Optimizing processes that are run across the image include ``mklibs``, | ||
1080 | ``prelink``, and any other post-processing commands as defined by the | ||
1081 | :term:`ROOTFS_POSTPROCESS_COMMAND` | ||
1082 | variable. The ``mklibs`` process optimizes the size of the libraries, | ||
1083 | while the ``prelink`` process optimizes the dynamic linking of shared | ||
1084 | libraries to reduce start up time of executables. | ||
1085 | |||
1086 | After the root filesystem is built, processing begins on the image | ||
1087 | through the :ref:`ref-tasks-image` | ||
1088 | task. The build system runs any pre-processing commands as defined by | ||
1089 | the | ||
1090 | :term:`IMAGE_PREPROCESS_COMMAND` | ||
1091 | variable. This variable specifies a list of functions to call before the | ||
1092 | build system creates the final image output files. | ||
1093 | |||
1094 | The build system dynamically creates ``do_image_*`` tasks as needed, | ||
1095 | based on the image types specified in the | ||
1096 | :term:`IMAGE_FSTYPES` variable. | ||
1097 | The process turns everything into an image file or a set of image files | ||
1098 | and can compress the root filesystem image to reduce the overall size of | ||
1099 | the image. The formats used for the root filesystem depend on the | ||
1100 | ``IMAGE_FSTYPES`` variable. Compression depends on whether the formats | ||
1101 | support compression. | ||
1102 | |||
1103 | As an example, a dynamically created task when creating a particular | ||
1104 | image type would take the following form: | ||
1105 | :: | ||
1106 | |||
1107 | do_image_type | ||
1108 | |||
1109 | So, if the type | ||
1110 | as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically | ||
1111 | generated task would be as follows: | ||
1112 | :: | ||
1113 | |||
1114 | do_image_ext4 | ||
1115 | |||
1116 | The final task involved in image creation is the | ||
1117 | :ref:`do_image_complete <ref-tasks-image-complete>` | ||
1118 | task. This task completes the image by applying any image post | ||
1119 | processing as defined through the | ||
1120 | :term:`IMAGE_POSTPROCESS_COMMAND` | ||
1121 | variable. The variable specifies a list of functions to call once the | ||
1122 | build system has created the final image output files. | ||
1123 | |||
1124 | .. note:: | ||
1125 | |||
1126 | The entire image generation process is run under | ||
1127 | Pseudo. Running under Pseudo ensures that the files in the root filesystem | ||
1128 | have correct ownership. | ||
1129 | |||
1130 | .. _sdk-generation-dev-environment: | ||
1131 | |||
1132 | SDK Generation | ||
1133 | ~~~~~~~~~~~~~~ | ||
1134 | |||
1135 | The OpenEmbedded build system uses BitBake to generate the Software | ||
1136 | Development Kit (SDK) installer scripts for both the standard SDK and | ||
1137 | the extensible SDK (eSDK): | ||
1138 | |||
1139 | .. image:: figures/sdk-generation.png | ||
1140 | :align: center | ||
1141 | |||
1142 | .. note:: | ||
1143 | |||
1144 | For more information on the cross-development toolchain generation, | ||
1145 | see the ":ref:`overview-manual/overview-manual-concepts:cross-development toolchain generation`" | ||
1146 | section. For information on advantages gained when building a | ||
1147 | cross-development toolchain using the do_populate_sdk task, see the | ||
1148 | ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" section in | ||
1149 | the Yocto Project Application Development and the Extensible Software | ||
1150 | Development Kit (eSDK) manual. | ||
1151 | |||
1152 | Like image generation, the SDK script process consists of several stages | ||
1153 | and depends on many variables. The | ||
1154 | :ref:`ref-tasks-populate_sdk` | ||
1155 | and | ||
1156 | :ref:`ref-tasks-populate_sdk_ext` | ||
1157 | tasks use these key variables to help create the list of packages to | ||
1158 | actually install. For information on the variables listed in the figure, | ||
1159 | see the "`Application Development SDK <#sdk-dev-environment>`__" | ||
1160 | section. | ||
1161 | |||
1162 | The ``do_populate_sdk`` task helps create the standard SDK and handles | ||
1163 | two parts: a target part and a host part. The target part is the part | ||
1164 | built for the target hardware and includes libraries and headers. The | ||
1165 | host part is the part of the SDK that runs on the | ||
1166 | :term:`SDKMACHINE`. | ||
1167 | |||
1168 | The ``do_populate_sdk_ext`` task helps create the extensible SDK and | ||
1169 | handles host and target parts differently than its counter part does for | ||
1170 | the standard SDK. For the extensible SDK, the task encapsulates the | ||
1171 | build system, which includes everything needed (host and target) for the | ||
1172 | SDK. | ||
1173 | |||
1174 | Regardless of the type of SDK being constructed, the tasks perform some | ||
1175 | cleanup after which a cross-development environment setup script and any | ||
1176 | needed configuration files are created. The final output is the | ||
1177 | Cross-development toolchain installation script (``.sh`` file), which | ||
1178 | includes the environment setup script. | ||
1179 | |||
1180 | Stamp Files and the Rerunning of Tasks | ||
1181 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
1182 | |||
1183 | For each task that completes successfully, BitBake writes a stamp file | ||
1184 | into the :term:`STAMPS_DIR` | ||
1185 | directory. The beginning of the stamp file's filename is determined by | ||
1186 | the :term:`STAMP` variable, and the end | ||
1187 | of the name consists of the task's name and current `input | ||
1188 | checksum <#overview-checksums>`__. | ||
1189 | |||
1190 | .. note:: | ||
1191 | |||
1192 | This naming scheme assumes that | ||
1193 | BB_SIGNATURE_HANDLER | ||
1194 | is "OEBasicHash", which is almost always the case in current | ||
1195 | OpenEmbedded. | ||
1196 | |||
1197 | To determine if a task needs to be rerun, BitBake checks if a stamp file | ||
1198 | with a matching input checksum exists for the task. If such a stamp file | ||
1199 | exists, the task's output is assumed to exist and still be valid. If the | ||
1200 | file does not exist, the task is rerun. | ||
1201 | |||
1202 | .. note:: | ||
1203 | |||
1204 | The stamp mechanism is more general than the shared state (sstate) | ||
1205 | cache mechanism described in the "`Setscene Tasks and Shared | ||
1206 | State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids | ||
1207 | rerunning any task that has a valid stamp file, not just tasks that | ||
1208 | can be accelerated through the sstate cache. | ||
1209 | |||
1210 | However, you should realize that stamp files only serve as a marker | ||
1211 | that some work has been done and that these files do not record task | ||
1212 | output. The actual task output would usually be somewhere in | ||
1213 | :term:`TMPDIR` (e.g. in some | ||
1214 | recipe's :term:`WORKDIR`.) What | ||
1215 | the sstate cache mechanism adds is a way to cache task output that | ||
1216 | can then be shared between build machines. | ||
1217 | |||
1218 | Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing | ||
1219 | ``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will | ||
1220 | properly be rerun to repopulate ``TMPDIR``. | ||
1221 | |||
1222 | If you want some task to always be considered "out of date", you can | ||
1223 | mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>` | ||
1224 | varflag. If some other task depends on such a task, then that task will | ||
1225 | also always be considered out of date, which might not be what you want. | ||
1226 | |||
1227 | For details on how to view information about a task's signature, see the | ||
1228 | ":ref:`dev-manual/dev-manual-common-tasks:viewing task variable dependencies`" | ||
1229 | section in the Yocto Project Development Tasks Manual. | ||
1230 | |||
1231 | Setscene Tasks and Shared State | ||
1232 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
1233 | |||
1234 | The description of tasks so far assumes that BitBake needs to build | ||
1235 | everything and no available prebuilt objects exist. BitBake does support | ||
1236 | skipping tasks if prebuilt objects are available. These objects are | ||
1237 | usually made available in the form of a shared state (sstate) cache. | ||
1238 | |||
1239 | .. note:: | ||
1240 | |||
1241 | For information on variables affecting sstate, see the | ||
1242 | :term:`SSTATE_DIR` | ||
1243 | and | ||
1244 | :term:`SSTATE_MIRRORS` | ||
1245 | variables. | ||
1246 | |||
1247 | The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a | ||
1248 | version of the task where instead of building something, BitBake can | ||
1249 | skip to the end result and simply place a set of files into specific | ||
1250 | locations as needed. In some cases, it makes sense to have a setscene | ||
1251 | task variant (e.g. generating package files in the | ||
1252 | :ref:`do_package_write_* <ref-tasks-package_write_deb>` | ||
1253 | task). In other cases, it does not make sense (e.g. a | ||
1254 | :ref:`ref-tasks-patch` task or a | ||
1255 | :ref:`ref-tasks-unpack` task) since | ||
1256 | the work involved would be equal to or greater than the underlying task. | ||
1257 | |||
1258 | In the build system, the common tasks that have setscene variants are | ||
1259 | :ref:`ref-tasks-package`, | ||
1260 | ``do_package_write_*``, | ||
1261 | :ref:`ref-tasks-deploy`, | ||
1262 | :ref:`ref-tasks-packagedata`, and | ||
1263 | :ref:`ref-tasks-populate_sysroot`. | ||
1264 | Notice that these tasks represent most of the tasks whose output is an | ||
1265 | end result. | ||
1266 | |||
1267 | The build system has knowledge of the relationship between these tasks | ||
1268 | and other preceding tasks. For example, if BitBake runs | ||
1269 | ``do_populate_sysroot_setscene`` for something, it does not make sense | ||
1270 | to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``, | ||
1271 | ``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if | ||
1272 | ``do_package`` needs to be run, BitBake needs to run those other tasks. | ||
1273 | |||
1274 | It becomes more complicated if everything can come from an sstate cache | ||
1275 | because some objects are simply not required at all. For example, you do | ||
1276 | not need a compiler or native tools, such as quilt, if nothing exists to | ||
1277 | compile or patch. If the ``do_package_write_*`` packages are available | ||
1278 | from sstate, BitBake does not need the ``do_package`` task data. | ||
1279 | |||
1280 | To handle all these complexities, BitBake runs in two phases. The first | ||
1281 | is the "setscene" stage. During this stage, BitBake first checks the | ||
1282 | sstate cache for any targets it is planning to build. BitBake does a | ||
1283 | fast check to see if the object exists rather than a complete download. | ||
1284 | If nothing exists, the second phase, which is the setscene stage, | ||
1285 | completes and the main build proceeds. | ||
1286 | |||
1287 | If objects are found in the sstate cache, the build system works | ||
1288 | backwards from the end targets specified by the user. For example, if an | ||
1289 | image is being built, the build system first looks for the packages | ||
1290 | needed for that image and the tools needed to construct an image. If | ||
1291 | those are available, the compiler is not needed. Thus, the compiler is | ||
1292 | not even downloaded. If something was found to be unavailable, or the | ||
1293 | download or setscene task fails, the build system then tries to install | ||
1294 | dependencies, such as the compiler, from the cache. | ||
1295 | |||
1296 | The availability of objects in the sstate cache is handled by the | ||
1297 | function specified by the | ||
1298 | :term:`bitbake:BB_HASHCHECK_FUNCTION` | ||
1299 | variable and returns a list of available objects. The function specified | ||
1300 | by the | ||
1301 | :term:`bitbake:BB_SETSCENE_DEPVALID` | ||
1302 | variable is the function that determines whether a given dependency | ||
1303 | needs to be followed, and whether for any given relationship the | ||
1304 | function needs to be passed. The function returns a True or False value. | ||
1305 | |||
1306 | .. _images-dev-environment: | ||
1307 | |||
1308 | Images | ||
1309 | ------ | ||
1310 | |||
1311 | The images produced by the build system are compressed forms of the root | ||
1312 | filesystem and are ready to boot on a target device. You can see from | ||
1313 | the `general workflow figure <#general-workflow-figure>`__ that BitBake | ||
1314 | output, in part, consists of images. This section takes a closer look at | ||
1315 | this output: | ||
1316 | |||
1317 | .. image:: figures/images.png | ||
1318 | :align: center | ||
1319 | |||
1320 | .. note:: | ||
1321 | |||
1322 | For a list of example images that the Yocto Project provides, see the | ||
1323 | ":doc:`../ref-manual/ref-images`" chapter in the Yocto Project Reference | ||
1324 | Manual. | ||
1325 | |||
1326 | The build process writes images out to the :term:`Build Directory` | ||
1327 | inside the | ||
1328 | ``tmp/deploy/images/machine/`` folder as shown in the figure. This | ||
1329 | folder contains any files expected to be loaded on the target device. | ||
1330 | The :term:`DEPLOY_DIR` variable | ||
1331 | points to the ``deploy`` directory, while the | ||
1332 | :term:`DEPLOY_DIR_IMAGE` | ||
1333 | variable points to the appropriate directory containing images for the | ||
1334 | current configuration. | ||
1335 | |||
1336 | - kernel-image: A kernel binary file. The | ||
1337 | :term:`KERNEL_IMAGETYPE` | ||
1338 | variable determines the naming scheme for the kernel image file. | ||
1339 | Depending on this variable, the file could begin with a variety of | ||
1340 | naming strings. The ``deploy/images/``\ machine directory can contain | ||
1341 | multiple image files for the machine. | ||
1342 | |||
1343 | - root-filesystem-image: Root filesystems for the target device (e.g. | ||
1344 | ``*.ext3`` or ``*.bz2`` files). The | ||
1345 | :term:`IMAGE_FSTYPES` | ||
1346 | variable determines the root filesystem image type. The | ||
1347 | ``deploy/images/``\ machine directory can contain multiple root | ||
1348 | filesystems for the machine. | ||
1349 | |||
1350 | - kernel-modules: Tarballs that contain all the modules built for the | ||
1351 | kernel. Kernel module tarballs exist for legacy purposes and can be | ||
1352 | suppressed by setting the | ||
1353 | :term:`MODULE_TARBALL_DEPLOY` | ||
1354 | variable to "0". The ``deploy/images/``\ machine directory can | ||
1355 | contain multiple kernel module tarballs for the machine. | ||
1356 | |||
1357 | - bootloaders: If applicable to the target machine, bootloaders | ||
1358 | supporting the image. The ``deploy/images/``\ machine directory can | ||
1359 | contain multiple bootloaders for the machine. | ||
1360 | |||
1361 | - symlinks: The ``deploy/images/``\ machine folder contains a symbolic | ||
1362 | link that points to the most recently built file for each machine. | ||
1363 | These links might be useful for external scripts that need to obtain | ||
1364 | the latest version of each file. | ||
1365 | |||
1366 | .. _sdk-dev-environment: | ||
1367 | |||
1368 | Application Development SDK | ||
1369 | --------------------------- | ||
1370 | |||
1371 | In the `general workflow figure <#general-workflow-figure>`__, the | ||
1372 | output labeled "Application Development SDK" represents an SDK. The SDK | ||
1373 | generation process differs depending on whether you build an extensible | ||
1374 | SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK | ||
1375 | (e.g. ``bitbake -c populate_sdk`` imagename). This section takes a | ||
1376 | closer look at this output: | ||
1377 | |||
1378 | .. image:: figures/sdk.png | ||
1379 | :align: center | ||
1380 | |||
1381 | The specific form of this output is a set of files that includes a | ||
1382 | self-extracting SDK installer (``*.sh``), host and target manifest | ||
1383 | files, and files used for SDK testing. When the SDK installer file is | ||
1384 | run, it installs the SDK. The SDK consists of a cross-development | ||
1385 | toolchain, a set of libraries and headers, and an SDK environment setup | ||
1386 | script. Running this installer essentially sets up your | ||
1387 | cross-development environment. You can think of the cross-toolchain as | ||
1388 | the "host" part because it runs on the SDK machine. You can think of the | ||
1389 | libraries and headers as the "target" part because they are built for | ||
1390 | the target hardware. The environment setup script is added so that you | ||
1391 | can initialize the environment before using the tools. | ||
1392 | |||
1393 | .. note:: | ||
1394 | |||
1395 | - The Yocto Project supports several methods by which you can set up | ||
1396 | this cross-development environment. These methods include | ||
1397 | downloading pre-built SDK installers or building and installing | ||
1398 | your own SDK installer. | ||
1399 | |||
1400 | - For background information on cross-development toolchains in the | ||
1401 | Yocto Project development environment, see the "`Cross-Development | ||
1402 | Toolchain Generation <#cross-development-toolchain-generation>`__" | ||
1403 | section. | ||
1404 | |||
1405 | - For information on setting up a cross-development environment, see | ||
1406 | the :doc:`../sdk-manual/sdk-manual` manual. | ||
1407 | |||
1408 | All the output files for an SDK are written to the ``deploy/sdk`` folder | ||
1409 | inside the :term:`Build Directory` as | ||
1410 | shown in the previous figure. Depending on the type of SDK, several | ||
1411 | variables exist that help configure these files. The following list | ||
1412 | shows the variables associated with an extensible SDK: | ||
1413 | |||
1414 | - :term:`DEPLOY_DIR`: Points to | ||
1415 | the ``deploy`` directory. | ||
1416 | |||
1417 | - :term:`SDK_EXT_TYPE`: | ||
1418 | Controls whether or not shared state artifacts are copied into the | ||
1419 | extensible SDK. By default, all required shared state artifacts are | ||
1420 | copied into the SDK. | ||
1421 | |||
1422 | - :term:`SDK_INCLUDE_PKGDATA`: | ||
1423 | Specifies whether or not packagedata is included in the extensible | ||
1424 | SDK for all recipes in the "world" target. | ||
1425 | |||
1426 | - :term:`SDK_INCLUDE_TOOLCHAIN`: | ||
1427 | Specifies whether or not the toolchain is included when building the | ||
1428 | extensible SDK. | ||
1429 | |||
1430 | - :term:`SDK_LOCAL_CONF_WHITELIST`: | ||
1431 | A list of variables allowed through from the build system | ||
1432 | configuration into the extensible SDK configuration. | ||
1433 | |||
1434 | - :term:`SDK_LOCAL_CONF_BLACKLIST`: | ||
1435 | A list of variables not allowed through from the build system | ||
1436 | configuration into the extensible SDK configuration. | ||
1437 | |||
1438 | - :term:`SDK_INHERIT_BLACKLIST`: | ||
1439 | A list of classes to remove from the | ||
1440 | :term:`INHERIT` value globally | ||
1441 | within the extensible SDK configuration. | ||
1442 | |||
1443 | This next list, shows the variables associated with a standard SDK: | ||
1444 | |||
1445 | - :term:`DEPLOY_DIR`: Points to | ||
1446 | the ``deploy`` directory. | ||
1447 | |||
1448 | - :term:`SDKMACHINE`: Specifies | ||
1449 | the architecture of the machine on which the cross-development tools | ||
1450 | are run to create packages for the target hardware. | ||
1451 | |||
1452 | - :term:`SDKIMAGE_FEATURES`: | ||
1453 | Lists the features to include in the "target" part of the SDK. | ||
1454 | |||
1455 | - :term:`TOOLCHAIN_HOST_TASK`: | ||
1456 | Lists packages that make up the host part of the SDK (i.e. the part | ||
1457 | that runs on the ``SDKMACHINE``). When you use | ||
1458 | ``bitbake -c populate_sdk imagename`` to create the SDK, a set of | ||
1459 | default packages apply. This variable allows you to add more | ||
1460 | packages. | ||
1461 | |||
1462 | - :term:`TOOLCHAIN_TARGET_TASK`: | ||
1463 | Lists packages that make up the target part of the SDK (i.e. the part | ||
1464 | built for the target hardware). | ||
1465 | |||
1466 | - :term:`SDKPATH`: Defines the | ||
1467 | default SDK installation path offered by the installation script. | ||
1468 | |||
1469 | - :term:`SDK_HOST_MANIFEST`: | ||
1470 | Lists all the installed packages that make up the host part of the | ||
1471 | SDK. This variable also plays a minor role for extensible SDK | ||
1472 | development as well. However, it is mainly used for the standard SDK. | ||
1473 | |||
1474 | - :term:`SDK_TARGET_MANIFEST`: | ||
1475 | Lists all the installed packages that make up the target part of the | ||
1476 | SDK. This variable also plays a minor role for extensible SDK | ||
1477 | development as well. However, it is mainly used for the standard SDK. | ||
1478 | |||
1479 | Cross-Development Toolchain Generation | ||
1480 | ====================================== | ||
1481 | |||
1482 | The Yocto Project does most of the work for you when it comes to | ||
1483 | creating :ref:`sdk-manual/sdk-intro:the cross-development toolchain`. This | ||
1484 | section provides some technical background on how cross-development | ||
1485 | toolchains are created and used. For more information on toolchains, you | ||
1486 | can also see the :doc:`../sdk-manual/sdk-manual` manual. | ||
1487 | |||
1488 | In the Yocto Project development environment, cross-development | ||
1489 | toolchains are used to build images and applications that run on the | ||
1490 | target hardware. With just a few commands, the OpenEmbedded build system | ||
1491 | creates these necessary toolchains for you. | ||
1492 | |||
1493 | The following figure shows a high-level build environment regarding | ||
1494 | toolchain construction and use. | ||
1495 | |||
1496 | .. image:: figures/cross-development-toolchains.png | ||
1497 | :align: center | ||
1498 | |||
1499 | Most of the work occurs on the Build Host. This is the machine used to | ||
1500 | build images and generally work within the the Yocto Project | ||
1501 | environment. When you run | ||
1502 | :term:`BitBake` to create an image, the | ||
1503 | OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a | ||
1504 | cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what | ||
1505 | BitBake uses to compile source files when creating the target image. You | ||
1506 | can think of ``gcc-cross`` simply as an automatically generated | ||
1507 | cross-compiler that is used internally within BitBake only. | ||
1508 | |||
1509 | .. note:: | ||
1510 | |||
1511 | The extensible SDK does not use | ||
1512 | gcc-cross-canadian | ||
1513 | since this SDK ships a copy of the OpenEmbedded build system and the | ||
1514 | sysroot within it contains | ||
1515 | gcc-cross | ||
1516 | . | ||
1517 | |||
1518 | The chain of events that occurs when ``gcc-cross`` is bootstrapped is as | ||
1519 | follows: | ||
1520 | :: | ||
1521 | |||
1522 | gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime | ||
1523 | |||
1524 | - ``gcc``: The build host's GNU Compiler Collection (GCC). | ||
1525 | |||
1526 | - ``binutils-cross``: The bare minimum binary utilities needed in order | ||
1527 | to run the ``gcc-cross-initial`` phase of the bootstrap operation. | ||
1528 | |||
1529 | - ``gcc-cross-initial``: An early stage of the bootstrap process for | ||
1530 | creating the cross-compiler. This stage builds enough of the | ||
1531 | ``gcc-cross``, the C library, and other pieces needed to finish | ||
1532 | building the final cross-compiler in later stages. This tool is a | ||
1533 | "native" package (i.e. it is designed to run on the build host). | ||
1534 | |||
1535 | - ``linux-libc-headers``: Headers needed for the cross-compiler. | ||
1536 | |||
1537 | - ``glibc-initial``: An initial version of the Embedded GNU C Library | ||
1538 | (GLIBC) needed to bootstrap ``glibc``. | ||
1539 | |||
1540 | - ``glibc``: The GNU C Library. | ||
1541 | |||
1542 | - ``gcc-cross``: The final stage of the bootstrap process for the | ||
1543 | cross-compiler. This stage results in the actual cross-compiler that | ||
1544 | BitBake uses when it builds an image for a targeted device. | ||
1545 | |||
1546 | .. note:: | ||
1547 | |||
1548 | If you are replacing this cross compiler toolchain with a custom | ||
1549 | version, you must replace | ||
1550 | gcc-cross | ||
1551 | . | ||
1552 | |||
1553 | This tool is also a "native" package (i.e. it is designed to run on | ||
1554 | the build host). | ||
1555 | |||
1556 | - ``gcc-runtime``: Runtime libraries resulting from the toolchain | ||
1557 | bootstrapping process. This tool produces a binary that consists of | ||
1558 | the runtime libraries need for the targeted device. | ||
1559 | |||
1560 | You can use the OpenEmbedded build system to build an installer for the | ||
1561 | relocatable SDK used to develop applications. When you run the | ||
1562 | installer, it installs the toolchain, which contains the development | ||
1563 | tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and | ||
1564 | other ``nativesdk-*`` tools), which are tools native to the SDK (i.e. | ||
1565 | native to :term:`SDK_ARCH`), you | ||
1566 | need to cross-compile and test your software. The figure shows the | ||
1567 | commands you use to easily build out this toolchain. This | ||
1568 | cross-development toolchain is built to execute on the | ||
1569 | :term:`SDKMACHINE`, which might or | ||
1570 | might not be the same machine as the Build Host. | ||
1571 | |||
1572 | .. note:: | ||
1573 | |||
1574 | If your target architecture is supported by the Yocto Project, you | ||
1575 | can take advantage of pre-built images that ship with the Yocto | ||
1576 | Project and already contain cross-development toolchain installers. | ||
1577 | |||
1578 | Here is the bootstrap process for the relocatable toolchain: | ||
1579 | :: | ||
1580 | |||
1581 | gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian | ||
1582 | |||
1583 | - ``gcc``: The build host's GNU Compiler Collection (GCC). | ||
1584 | |||
1585 | - ``binutils-crosssdk``: The bare minimum binary utilities needed in | ||
1586 | order to run the ``gcc-crosssdk-initial`` phase of the bootstrap | ||
1587 | operation. | ||
1588 | |||
1589 | - ``gcc-crosssdk-initial``: An early stage of the bootstrap process for | ||
1590 | creating the cross-compiler. This stage builds enough of the | ||
1591 | ``gcc-crosssdk`` and supporting pieces so that the final stage of the | ||
1592 | bootstrap process can produce the finished cross-compiler. This tool | ||
1593 | is a "native" binary that runs on the build host. | ||
1594 | |||
1595 | - ``linux-libc-headers``: Headers needed for the cross-compiler. | ||
1596 | |||
1597 | - ``glibc-initial``: An initial version of the Embedded GLIBC needed to | ||
1598 | bootstrap ``nativesdk-glibc``. | ||
1599 | |||
1600 | - ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the | ||
1601 | ``gcc-crosssdk``. | ||
1602 | |||
1603 | - ``gcc-crosssdk``: The final stage of the bootstrap process for the | ||
1604 | relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory | ||
1605 | compiler and never leaves the build host. Its purpose is to help in | ||
1606 | the bootstrap process to create the eventual ``gcc-cross-canadian`` | ||
1607 | compiler, which is relocatable. This tool is also a "native" package | ||
1608 | (i.e. it is designed to run on the build host). | ||
1609 | |||
1610 | - ``gcc-cross-canadian``: The final relocatable cross-compiler. When | ||
1611 | run on the :term:`SDKMACHINE`, | ||
1612 | this tool produces executable code that runs on the target device. | ||
1613 | Only one cross-canadian compiler is produced per architecture since | ||
1614 | they can be targeted at different processor optimizations using | ||
1615 | configurations passed to the compiler through the compile commands. | ||
1616 | This circumvents the need for multiple compilers and thus reduces the | ||
1617 | size of the toolchains. | ||
1618 | |||
1619 | .. note:: | ||
1620 | |||
1621 | For information on advantages gained when building a | ||
1622 | cross-development toolchain installer, see the | ||
1623 | ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" appendix | ||
1624 | in the Yocto Project Application Development and the | ||
1625 | Extensible Software Development Kit (eSDK) manual. | ||
1626 | |||
1627 | Shared State Cache | ||
1628 | ================== | ||
1629 | |||
1630 | By design, the OpenEmbedded build system builds everything from scratch | ||
1631 | unless :term:`BitBake` can determine | ||
1632 | that parts do not need to be rebuilt. Fundamentally, building from | ||
1633 | scratch is attractive as it means all parts are built fresh and no | ||
1634 | possibility of stale data exists that can cause problems. When | ||
1635 | developers hit problems, they typically default back to building from | ||
1636 | scratch so they have a know state from the start. | ||
1637 | |||
1638 | Building an image from scratch is both an advantage and a disadvantage | ||
1639 | to the process. As mentioned in the previous paragraph, building from | ||
1640 | scratch ensures that everything is current and starts from a known | ||
1641 | state. However, building from scratch also takes much longer as it | ||
1642 | generally means rebuilding things that do not necessarily need to be | ||
1643 | rebuilt. | ||
1644 | |||
1645 | The Yocto Project implements shared state code that supports incremental | ||
1646 | builds. The implementation of the shared state code answers the | ||
1647 | following questions that were fundamental roadblocks within the | ||
1648 | OpenEmbedded incremental build support system: | ||
1649 | |||
1650 | - What pieces of the system have changed and what pieces have not | ||
1651 | changed? | ||
1652 | |||
1653 | - How are changed pieces of software removed and replaced? | ||
1654 | |||
1655 | - How are pre-built components that do not need to be rebuilt from | ||
1656 | scratch used when they are available? | ||
1657 | |||
1658 | For the first question, the build system detects changes in the "inputs" | ||
1659 | to a given task by creating a checksum (or signature) of the task's | ||
1660 | inputs. If the checksum changes, the system assumes the inputs have | ||
1661 | changed and the task needs to be rerun. For the second question, the | ||
1662 | shared state (sstate) code tracks which tasks add which output to the | ||
1663 | build process. This means the output from a given task can be removed, | ||
1664 | upgraded or otherwise manipulated. The third question is partly | ||
1665 | addressed by the solution for the second question assuming the build | ||
1666 | system can fetch the sstate objects from remote locations and install | ||
1667 | them if they are deemed to be valid. | ||
1668 | |||
1669 | .. note:: | ||
1670 | |||
1671 | - The build system does not maintain | ||
1672 | :term:`PR` information as part of | ||
1673 | the shared state packages. Consequently, considerations exist that | ||
1674 | affect maintaining shared state feeds. For information on how the | ||
1675 | build system works with packages and can track incrementing ``PR`` | ||
1676 | information, see the ":ref:`dev-manual/dev-manual-common-tasks:automatically incrementing a package version number`" | ||
1677 | section in the Yocto Project Development Tasks Manual. | ||
1678 | |||
1679 | - The code in the build system that supports incremental builds is | ||
1680 | not simple code. For techniques that help you work around issues | ||
1681 | related to shared state code, see the | ||
1682 | ":ref:`dev-manual/dev-manual-common-tasks:viewing metadata used to create the input signature of a shared state task`" | ||
1683 | and | ||
1684 | ":ref:`dev-manual/dev-manual-common-tasks:invalidating shared state to force a task to run`" | ||
1685 | sections both in the Yocto Project Development Tasks Manual. | ||
1686 | |||
1687 | The rest of this section goes into detail about the overall incremental | ||
1688 | build architecture, the checksums (signatures), and shared state. | ||
1689 | |||
1690 | .. _concepts-overall-architecture: | ||
1691 | |||
1692 | Overall Architecture | ||
1693 | -------------------- | ||
1694 | |||
1695 | When determining what parts of the system need to be built, BitBake | ||
1696 | works on a per-task basis rather than a per-recipe basis. You might | ||
1697 | wonder why using a per-task basis is preferred over a per-recipe basis. | ||
1698 | To help explain, consider having the IPK packaging backend enabled and | ||
1699 | then switching to DEB. In this case, the | ||
1700 | :ref:`ref-tasks-install` and | ||
1701 | :ref:`ref-tasks-package` task outputs | ||
1702 | are still valid. However, with a per-recipe approach, the build would | ||
1703 | not include the ``.deb`` files. Consequently, you would have to | ||
1704 | invalidate the whole build and rerun it. Rerunning everything is not the | ||
1705 | best solution. Also, in this case, the core must be "taught" much about | ||
1706 | specific tasks. This methodology does not scale well and does not allow | ||
1707 | users to easily add new tasks in layers or as external recipes without | ||
1708 | touching the packaged-staging core. | ||
1709 | |||
1710 | .. _overview-checksums: | ||
1711 | |||
1712 | Checksums (Signatures) | ||
1713 | ---------------------- | ||
1714 | |||
1715 | The shared state code uses a checksum, which is a unique signature of a | ||
1716 | task's inputs, to determine if a task needs to be run again. Because it | ||
1717 | is a change in a task's inputs that triggers a rerun, the process needs | ||
1718 | to detect all the inputs to a given task. For shell tasks, this turns | ||
1719 | out to be fairly easy because the build process generates a "run" shell | ||
1720 | script for each task and it is possible to create a checksum that gives | ||
1721 | you a good idea of when the task's data changes. | ||
1722 | |||
1723 | To complicate the problem, there are things that should not be included | ||
1724 | in the checksum. First, there is the actual specific build path of a | ||
1725 | given task - the :term:`WORKDIR`. It | ||
1726 | does not matter if the work directory changes because it should not | ||
1727 | affect the output for target packages. Also, the build process has the | ||
1728 | objective of making native or cross packages relocatable. | ||
1729 | |||
1730 | .. note:: | ||
1731 | |||
1732 | Both native and cross packages run on the | ||
1733 | build host. However, cross packages generate output for the target | ||
1734 | architecture. | ||
1735 | |||
1736 | The checksum therefore needs to exclude ``WORKDIR``. The simplistic | ||
1737 | approach for excluding the work directory is to set ``WORKDIR`` to some | ||
1738 | fixed value and create the checksum for the "run" script. | ||
1739 | |||
1740 | Another problem results from the "run" scripts containing functions that | ||
1741 | might or might not get called. The incremental build solution contains | ||
1742 | code that figures out dependencies between shell functions. This code is | ||
1743 | used to prune the "run" scripts down to the minimum set, thereby | ||
1744 | alleviating this problem and making the "run" scripts much more readable | ||
1745 | as a bonus. | ||
1746 | |||
1747 | So far, solutions for shell scripts exist. What about Python tasks? The | ||
1748 | same approach applies even though these tasks are more difficult. The | ||
1749 | process needs to figure out what variables a Python function accesses | ||
1750 | and what functions it calls. Again, the incremental build solution | ||
1751 | contains code that first figures out the variable and function | ||
1752 | dependencies, and then creates a checksum for the data used as the input | ||
1753 | to the task. | ||
1754 | |||
1755 | Like the ``WORKDIR`` case, situations exist where dependencies should be | ||
1756 | ignored. For these situations, you can instruct the build process to | ||
1757 | ignore a dependency by using a line like the following: | ||
1758 | :: | ||
1759 | |||
1760 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
1761 | |||
1762 | This example ensures that the :term:`PACKAGE_ARCHS` variable | ||
1763 | does not depend on the value of :term:`MACHINE`, even if it does | ||
1764 | reference it. | ||
1765 | |||
1766 | Equally, there are cases where you need to add dependencies BitBake is | ||
1767 | not able to find. You can accomplish this by using a line like the | ||
1768 | following: | ||
1769 | :: | ||
1770 | |||
1771 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
1772 | |||
1773 | This example explicitly | ||
1774 | adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. | ||
1775 | |||
1776 | As an example, consider a case with in-line Python where BitBake is not | ||
1777 | able to figure out dependencies. When running in debug mode (i.e. using | ||
1778 | ``-DDD``), BitBake produces output when it discovers something for which | ||
1779 | it cannot figure out dependencies. The Yocto Project team has currently | ||
1780 | not managed to cover those dependencies in detail and is aware of the | ||
1781 | need to fix this situation. | ||
1782 | |||
1783 | Thus far, this section has limited discussion to the direct inputs into | ||
1784 | a task. Information based on direct inputs is referred to as the | ||
1785 | "basehash" in the code. However, the question of a task's indirect | ||
1786 | inputs still exits - items already built and present in the | ||
1787 | :term:`Build Directory`. The checksum (or | ||
1788 | signature) for a particular task needs to add the hashes of all the | ||
1789 | tasks on which the particular task depends. Choosing which dependencies | ||
1790 | to add is a policy decision. However, the effect is to generate a master | ||
1791 | checksum that combines the basehash and the hashes of the task's | ||
1792 | dependencies. | ||
1793 | |||
1794 | At the code level, a variety of ways exist by which both the basehash | ||
1795 | and the dependent task hashes can be influenced. Within the BitBake | ||
1796 | configuration file, you can give BitBake some extra information to help | ||
1797 | it construct the basehash. The following statement effectively results | ||
1798 | in a list of global variable dependency excludes (i.e. variables never | ||
1799 | included in any checksum): | ||
1800 | :: | ||
1801 | |||
1802 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\ | ||
1803 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\ | ||
1804 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ | ||
1805 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ | ||
1806 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
1807 | |||
1808 | The | ||
1809 | previous example excludes | ||
1810 | :term:`WORKDIR` since that variable | ||
1811 | is actually constructed as a path within | ||
1812 | :term:`TMPDIR`, which is on the | ||
1813 | whitelist. | ||
1814 | |||
1815 | The rules for deciding which hashes of dependent tasks to include | ||
1816 | through dependency chains are more complex and are generally | ||
1817 | accomplished with a Python function. The code in | ||
1818 | ``meta/lib/oe/sstatesig.py`` shows two examples of this and also | ||
1819 | illustrates how you can insert your own policy into the system if so | ||
1820 | desired. This file defines the two basic signature generators | ||
1821 | :term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and | ||
1822 | "OEBasicHash". By default, a dummy "noop" signature handler is enabled | ||
1823 | in BitBake. This means that behavior is unchanged from previous | ||
1824 | versions. OE-Core uses the "OEBasicHash" signature handler by default | ||
1825 | through this setting in the ``bitbake.conf`` file: | ||
1826 | :: | ||
1827 | |||
1828 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
1829 | |||
1830 | The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same | ||
1831 | as the "OEBasic" version but adds the task hash to the `stamp | ||
1832 | files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any | ||
1833 | metadata change that changes the task hash, automatically causing the | ||
1834 | task to be run again. This removes the need to bump | ||
1835 | :term:`PR` values, and changes to metadata | ||
1836 | automatically ripple across the build. | ||
1837 | |||
1838 | It is also worth noting that the end result of these signature | ||
1839 | generators is to make some dependency and hash information available to | ||
1840 | the build. This information includes: | ||
1841 | |||
1842 | - ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the | ||
1843 | recipe. | ||
1844 | |||
1845 | - ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each | ||
1846 | dependent task. | ||
1847 | |||
1848 | - ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for | ||
1849 | each task. | ||
1850 | |||
1851 | - ``BB_TASKHASH``: The hash of the currently running task. | ||
1852 | |||
1853 | Shared State | ||
1854 | ------------ | ||
1855 | |||
1856 | Checksums and dependencies, as discussed in the previous section, solve | ||
1857 | half the problem of supporting a shared state. The other half of the | ||
1858 | problem is being able to use checksum information during the build and | ||
1859 | being able to reuse or rebuild specific components. | ||
1860 | |||
1861 | The :ref:`sstate <ref-classes-sstate>` class is a | ||
1862 | relatively generic implementation of how to "capture" a snapshot of a | ||
1863 | given task. The idea is that the build process does not care about the | ||
1864 | source of a task's output. Output could be freshly built or it could be | ||
1865 | downloaded and unpacked from somewhere. In other words, the build | ||
1866 | process does not need to worry about its origin. | ||
1867 | |||
1868 | Two types of output exist. One type is just about creating a directory | ||
1869 | in :term:`WORKDIR`. A good example is | ||
1870 | the output of either | ||
1871 | :ref:`ref-tasks-install` or | ||
1872 | :ref:`ref-tasks-package`. The other | ||
1873 | type of output occurs when a set of data is merged into a shared | ||
1874 | directory tree such as the sysroot. | ||
1875 | |||
1876 | The Yocto Project team has tried to keep the details of the | ||
1877 | implementation hidden in ``sstate`` class. From a user's perspective, | ||
1878 | adding shared state wrapping to a task is as simple as this | ||
1879 | :ref:`ref-tasks-deploy` example taken | ||
1880 | from the :ref:`deploy <ref-classes-deploy>` class: | ||
1881 | :: | ||
1882 | |||
1883 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" | ||
1884 | SSTATETASKS += "do_deploy" | ||
1885 | do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" | ||
1886 | do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" | ||
1887 | |||
1888 | python do_deploy_setscene () { | ||
1889 | sstate_setscene(d) | ||
1890 | } | ||
1891 | addtask do_deploy_setscene | ||
1892 | do_deploy[dirs] = "${DEPLOYDIR} ${B}" | ||
1893 | do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" | ||
1894 | |||
1895 | The following list explains the previous example: | ||
1896 | |||
1897 | - Adding "do_deploy" to ``SSTATETASKS`` adds some required | ||
1898 | sstate-related processing, which is implemented in the | ||
1899 | :ref:`sstate <ref-classes-sstate>` class, to | ||
1900 | before and after the | ||
1901 | :ref:`ref-tasks-deploy` task. | ||
1902 | |||
1903 | - The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that | ||
1904 | ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally | ||
1905 | (i.e. when not using the sstate cache). This output becomes the input | ||
1906 | to the shared state cache. | ||
1907 | |||
1908 | - The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line | ||
1909 | causes the contents of the shared state cache to be copied to | ||
1910 | ``${DEPLOY_DIR_IMAGE}``. | ||
1911 | |||
1912 | .. note:: | ||
1913 | |||
1914 | If ``do_deploy`` is not already in the shared state cache or if its input | ||
1915 | checksum (signature) has changed from when the output was cached, the task | ||
1916 | runs to populate the shared state cache, after which the contents of the | ||
1917 | shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If | ||
1918 | ``do_deploy`` is in the shared state cache and its signature indicates | ||
1919 | that the cached output is still valid (i.e. if no relevant task inputs | ||
1920 | have changed), then the contents of the shared state cache copies | ||
1921 | directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task | ||
1922 | instead, skipping the ``do_deploy`` task. | ||
1923 | |||
1924 | - The following task definition is glue logic needed to make the | ||
1925 | previous settings effective: | ||
1926 | :: | ||
1927 | |||
1928 | python do_deploy_setscene () { | ||
1929 | sstate_setscene(d) | ||
1930 | } | ||
1931 | addtask do_deploy_setscene | ||
1932 | |||
1933 | ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task | ||
1934 | through the shared state cache if possible. If the task was | ||
1935 | accelerated, ``sstate_setscene()`` returns True. Otherwise, it | ||
1936 | returns False, and the normal ``do_deploy`` task runs. For more | ||
1937 | information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`" | ||
1938 | section in the BitBake User Manual. | ||
1939 | |||
1940 | - The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates | ||
1941 | ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and | ||
1942 | also sets the current working directory of ``do_deploy`` to ``${B}``. | ||
1943 | For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" | ||
1944 | section in the BitBake | ||
1945 | User Manual. | ||
1946 | |||
1947 | .. note:: | ||
1948 | |||
1949 | In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be | ||
1950 | the same, you can use ``sstate-plaindirs``. For example, to preserve the | ||
1951 | ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package`` | ||
1952 | task, use the following: | ||
1953 | :: | ||
1954 | |||
1955 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | ||
1956 | |||
1957 | |||
1958 | - The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends | ||
1959 | extra metadata to the `stamp | ||
1960 | file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the | ||
1961 | metadata makes the task specific to a machine's architecture. See | ||
1962 | ":ref:`bitbake:ref-bitbake-tasklist`" | ||
1963 | section in the BitBake User Manual for more information on the | ||
1964 | ``stamp-extra-info`` flag. | ||
1965 | |||
1966 | - ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with | ||
1967 | multiple directories. For example, the following declares | ||
1968 | ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories, | ||
1969 | which populates the shared state cache, and ``PKGDATA_DIR`` and | ||
1970 | ``SHLIBSDIR`` as the corresponding shared state output directories: | ||
1971 | :: | ||
1972 | |||
1973 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" | ||
1974 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" | ||
1975 | |||
1976 | - These methods also include the ability to take a lockfile when | ||
1977 | manipulating shared state directory structures, for cases where file | ||
1978 | additions or removals are sensitive: | ||
1979 | :: | ||
1980 | |||
1981 | do_package[sstate-lockfile] = "${PACKAGELOCK}" | ||
1982 | |||
1983 | Behind the scenes, the shared state code works by looking in | ||
1984 | :term:`SSTATE_DIR` and | ||
1985 | :term:`SSTATE_MIRRORS` for | ||
1986 | shared state files. Here is an example: | ||
1987 | :: | ||
1988 | |||
1989 | SSTATE_MIRRORS ?= "\ | ||
1990 | file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ | ||
1991 | file://.\* file:///some/local/dir/sstate/PATH" | ||
1992 | |||
1993 | .. note:: | ||
1994 | |||
1995 | The shared state directory (``SSTATE_DIR``) is organized into two-character | ||
1996 | subdirectories, where the subdirectory names are based on the first two | ||
1997 | characters of the hash. | ||
1998 | If the shared state directory structure for a mirror has the same structure | ||
1999 | as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build | ||
2000 | system to map to the appropriate subdirectory. | ||
2001 | |||
2002 | The shared state package validity can be detected just by looking at the | ||
2003 | filename since the filename contains the task checksum (or signature) as | ||
2004 | described earlier in this section. If a valid shared state package is | ||
2005 | found, the build process downloads it and uses it to accelerate the | ||
2006 | task. | ||
2007 | |||
2008 | The build processes use the ``*_setscene`` tasks for the task | ||
2009 | acceleration phase. BitBake goes through this phase before the main | ||
2010 | execution code and tries to accelerate any tasks for which it can find | ||
2011 | shared state packages. If a shared state package for a task is | ||
2012 | available, the shared state package is used. This means the task and any | ||
2013 | tasks on which it is dependent are not executed. | ||
2014 | |||
2015 | As a real world example, the aim is when building an IPK-based image, | ||
2016 | only the | ||
2017 | :ref:`ref-tasks-package_write_ipk` | ||
2018 | tasks would have their shared state packages fetched and extracted. | ||
2019 | Since the sysroot is not used, it would never get extracted. This is | ||
2020 | another reason why a task-based approach is preferred over a | ||
2021 | recipe-based approach, which would have to install the output from every | ||
2022 | task. | ||
2023 | |||
2024 | Automatically Added Runtime Dependencies | ||
2025 | ======================================== | ||
2026 | |||
2027 | The OpenEmbedded build system automatically adds common types of runtime | ||
2028 | dependencies between packages, which means that you do not need to | ||
2029 | explicitly declare the packages using | ||
2030 | :term:`RDEPENDS`. Three automatic | ||
2031 | mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that | ||
2032 | handle shared libraries, package configuration (pkg-config) modules, and | ||
2033 | ``-dev`` and ``-dbg`` packages, respectively. For other types of runtime | ||
2034 | dependencies, you must manually declare the dependencies. | ||
2035 | |||
2036 | - ``shlibdeps``: During the | ||
2037 | :ref:`ref-tasks-package` task of | ||
2038 | each recipe, all shared libraries installed by the recipe are | ||
2039 | located. For each shared library, the package that contains the | ||
2040 | shared library is registered as providing the shared library. More | ||
2041 | specifically, the package is registered as providing the | ||
2042 | `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The | ||
2043 | resulting shared-library-to-package mapping is saved globally in | ||
2044 | :term:`PKGDATA_DIR` by the | ||
2045 | :ref:`ref-tasks-packagedata` | ||
2046 | task. | ||
2047 | |||
2048 | Simultaneously, all executables and shared libraries installed by the | ||
2049 | recipe are inspected to see what shared libraries they link against. | ||
2050 | For each shared library dependency that is found, ``PKGDATA_DIR`` is | ||
2051 | queried to see if some package (likely from a different recipe) | ||
2052 | contains the shared library. If such a package is found, a runtime | ||
2053 | dependency is added from the package that depends on the shared | ||
2054 | library to the package that contains the library. | ||
2055 | |||
2056 | The automatically added runtime dependency also includes a version | ||
2057 | restriction. This version restriction specifies that at least the | ||
2058 | current version of the package that provides the shared library must | ||
2059 | be used, as if "package (>= version)" had been added to ``RDEPENDS``. | ||
2060 | This forces an upgrade of the package containing the shared library | ||
2061 | when installing the package that depends on the library, if needed. | ||
2062 | |||
2063 | If you want to avoid a package being registered as providing a | ||
2064 | particular shared library (e.g. because the library is for internal | ||
2065 | use only), then add the library to | ||
2066 | :term:`PRIVATE_LIBS` inside | ||
2067 | the package's recipe. | ||
2068 | |||
2069 | - ``pcdeps``: During the ``do_package`` task of each recipe, all | ||
2070 | pkg-config modules (``*.pc`` files) installed by the recipe are | ||
2071 | located. For each module, the package that contains the module is | ||
2072 | registered as providing the module. The resulting module-to-package | ||
2073 | mapping is saved globally in ``PKGDATA_DIR`` by the | ||
2074 | ``do_packagedata`` task. | ||
2075 | |||
2076 | Simultaneously, all pkg-config modules installed by the recipe are | ||
2077 | inspected to see what other pkg-config modules they depend on. A | ||
2078 | module is seen as depending on another module if it contains a | ||
2079 | "Requires:" line that specifies the other module. For each module | ||
2080 | dependency, ``PKGDATA_DIR`` is queried to see if some package | ||
2081 | contains the module. If such a package is found, a runtime dependency | ||
2082 | is added from the package that depends on the module to the package | ||
2083 | that contains the module. | ||
2084 | |||
2085 | .. note:: | ||
2086 | |||
2087 | The | ||
2088 | pcdeps | ||
2089 | mechanism most often infers dependencies between | ||
2090 | -dev | ||
2091 | packages. | ||
2092 | |||
2093 | - ``depchains``: If a package ``foo`` depends on a package ``bar``, | ||
2094 | then ``foo-dev`` and ``foo-dbg`` are also made to depend on | ||
2095 | ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev`` | ||
2096 | packages as an example, the ``bar-dev`` package might provide headers | ||
2097 | and shared library symlinks needed by ``foo-dev``, which shows the | ||
2098 | need for a dependency between the packages. | ||
2099 | |||
2100 | The dependencies added by ``depchains`` are in the form of | ||
2101 | :term:`RRECOMMENDS`. | ||
2102 | |||
2103 | .. note:: | ||
2104 | |||
2105 | By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on | ||
2106 | ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in | ||
2107 | bitbake.conf) includes "${PN}". | ||
2108 | |||
2109 | To ensure that the dependency chain is never broken, ``-dev`` and | ||
2110 | ``-dbg`` packages are always generated by default, even if the | ||
2111 | packages turn out to be empty. See the | ||
2112 | :term:`ALLOW_EMPTY` variable | ||
2113 | for more information. | ||
2114 | |||
2115 | The ``do_package`` task depends on the ``do_packagedata`` task of each | ||
2116 | recipe in :term:`DEPENDS` through use | ||
2117 | of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` | ||
2118 | declaration, which guarantees that the required | ||
2119 | shared-library/module-to-package mapping information will be available | ||
2120 | when needed as long as ``DEPENDS`` has been correctly set. | ||
2121 | |||
2122 | Fakeroot and Pseudo | ||
2123 | =================== | ||
2124 | |||
2125 | Some tasks are easier to implement when allowed to perform certain | ||
2126 | operations that are normally reserved for the root user (e.g. | ||
2127 | :ref:`ref-tasks-install`, | ||
2128 | :ref:`do_package_write* <ref-tasks-package_write_deb>`, | ||
2129 | :ref:`ref-tasks-rootfs`, and | ||
2130 | :ref:`do_image* <ref-tasks-image>`). For example, | ||
2131 | the ``do_install`` task benefits from being able to set the UID and GID | ||
2132 | of installed files to arbitrary values. | ||
2133 | |||
2134 | One approach to allowing tasks to perform root-only operations would be | ||
2135 | to require :term:`BitBake` to run as | ||
2136 | root. However, this method is cumbersome and has security issues. The | ||
2137 | approach that is actually used is to run tasks that benefit from root | ||
2138 | privileges in a "fake" root environment. Within this environment, the | ||
2139 | task and its child processes believe that they are running as the root | ||
2140 | user, and see an internally consistent view of the filesystem. As long | ||
2141 | as generating the final output (e.g. a package or an image) does not | ||
2142 | require root privileges, the fact that some earlier steps ran in a fake | ||
2143 | root environment does not cause problems. | ||
2144 | |||
2145 | The capability to run tasks in a fake root environment is known as | ||
2146 | "`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from | ||
2147 | the BitBake keyword/variable flag that requests a fake root environment | ||
2148 | for a task. | ||
2149 | |||
2150 | In the :term:`OpenEmbedded Build System`, | ||
2151 | the program that | ||
2152 | implements fakeroot is known as | ||
2153 | `Pseudo <https://www.yoctoproject.org/software-item/pseudo/>`__. Pseudo | ||
2154 | overrides system calls by using the environment variable ``LD_PRELOAD``, | ||
2155 | which results in the illusion of running as root. To keep track of | ||
2156 | "fake" file ownership and permissions resulting from operations that | ||
2157 | require root permissions, Pseudo uses an SQLite 3 database. This | ||
2158 | database is stored in | ||
2159 | ``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db`` | ||
2160 | for individual recipes. Storing the database in a file as opposed to in | ||
2161 | memory gives persistence between tasks and builds, which is not | ||
2162 | accomplished using fakeroot. | ||
2163 | |||
2164 | .. note:: | ||
2165 | |||
2166 | If you add your own task that manipulates the same files or | ||
2167 | directories as a fakeroot task, then that task also needs to run | ||
2168 | under fakeroot. Otherwise, the task cannot run root-only operations, | ||
2169 | and cannot see the fake file ownership and permissions set by the | ||
2170 | other task. You need to also add a dependency on | ||
2171 | virtual/fakeroot-native:do_populate_sysroot | ||
2172 | , giving the following: | ||
2173 | :: | ||
2174 | |||
2175 | fakeroot do_mytask () { | ||
2176 | ... | ||
2177 | } | ||
2178 | do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" | ||
2179 | |||
2180 | |||
2181 | For more information, see the | ||
2182 | :term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the | ||
2183 | BitBake User Manual. You can also reference the "`Why Not | ||
2184 | Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__" | ||
2185 | article for background information on Fakeroot and Pseudo. | ||
diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml index f085dd710d..58b64bd269 100644 --- a/documentation/overview-manual/overview-manual-concepts.xml +++ b/documentation/overview-manual/overview-manual-concepts.xml | |||
@@ -1,6 +1,7 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
4 | 5 | ||
5 | <chapter id=' overview-manual-concepts'> | 6 | <chapter id=' overview-manual-concepts'> |
6 | <title>Yocto Project Concepts</title> | 7 | <title>Yocto Project Concepts</title> |
diff --git a/documentation/overview-manual/overview-manual-customization.xsl b/documentation/overview-manual/overview-manual-customization.xsl index 22360e7bab..1dd91bde80 100644 --- a/documentation/overview-manual/overview-manual-customization.xsl +++ b/documentation/overview-manual/overview-manual-customization.xsl | |||
@@ -1,4 +1,6 @@ | |||
1 | <?xml version='1.0'?> | 1 | <?xml version='1.0'?> |
2 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
3 | |||
2 | <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0"> | 4 | <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0"> |
3 | 5 | ||
4 | <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" /> | 6 | <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" /> |
diff --git a/documentation/overview-manual/overview-manual-development-environment.rst b/documentation/overview-manual/overview-manual-development-environment.rst new file mode 100644 index 0000000000..bb2c8e72e7 --- /dev/null +++ b/documentation/overview-manual/overview-manual-development-environment.rst | |||
@@ -0,0 +1,672 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ***************************************** | ||
4 | The Yocto Project Development Environment | ||
5 | ***************************************** | ||
6 | |||
7 | This chapter takes a look at the Yocto Project development environment. | ||
8 | The chapter provides Yocto Project Development environment concepts that | ||
9 | help you understand how work is accomplished in an open source | ||
10 | environment, which is very different as compared to work accomplished in | ||
11 | a closed, proprietary environment. | ||
12 | |||
13 | Specifically, this chapter addresses open source philosophy, source | ||
14 | repositories, workflows, Git, and licensing. | ||
15 | |||
16 | Open Source Philosophy | ||
17 | ====================== | ||
18 | |||
19 | Open source philosophy is characterized by software development directed | ||
20 | by peer production and collaboration through an active community of | ||
21 | developers. Contrast this to the more standard centralized development | ||
22 | models used by commercial software companies where a finite set of | ||
23 | developers produces a product for sale using a defined set of procedures | ||
24 | that ultimately result in an end product whose architecture and source | ||
25 | material are closed to the public. | ||
26 | |||
27 | Open source projects conceptually have differing concurrent agendas, | ||
28 | approaches, and production. These facets of the development process can | ||
29 | come from anyone in the public (community) who has a stake in the | ||
30 | software project. The open source environment contains new copyright, | ||
31 | licensing, domain, and consumer issues that differ from the more | ||
32 | traditional development environment. In an open source environment, the | ||
33 | end product, source material, and documentation are all available to the | ||
34 | public at no cost. | ||
35 | |||
36 | A benchmark example of an open source project is the Linux kernel, which | ||
37 | was initially conceived and created by Finnish computer science student | ||
38 | Linus Torvalds in 1991. Conversely, a good example of a non-open source | ||
39 | project is the Windows family of operating systems developed by | ||
40 | Microsoft Corporation. | ||
41 | |||
42 | Wikipedia has a good historical description of the Open Source | ||
43 | Philosophy `here <http://en.wikipedia.org/wiki/Open_source>`__. You can | ||
44 | also find helpful information on how to participate in the Linux | ||
45 | Community | ||
46 | `here <https://www.kernel.org/doc/html/latest/process/index.html>`__. | ||
47 | |||
48 | .. _gs-the-development-host: | ||
49 | |||
50 | The Development Host | ||
51 | ==================== | ||
52 | |||
53 | A development host or :term:`Build Host` is key to | ||
54 | using the Yocto Project. Because the goal of the Yocto Project is to | ||
55 | develop images or applications that run on embedded hardware, | ||
56 | development of those images and applications generally takes place on a | ||
57 | system not intended to run the software - the development host. | ||
58 | |||
59 | You need to set up a development host in order to use it with the Yocto | ||
60 | Project. Most find that it is best to have a native Linux machine | ||
61 | function as the development host. However, it is possible to use a | ||
62 | system that does not run Linux as its operating system as your | ||
63 | development host. When you have a Mac or Windows-based system, you can | ||
64 | set it up as the development host by using | ||
65 | `CROPS <https://github.com/crops/poky-container>`__, which leverages | ||
66 | `Docker Containers <https://www.docker.com/>`__. Once you take the steps | ||
67 | to set up a CROPS machine, you effectively have access to a shell | ||
68 | environment that is similar to what you see when using a Linux-based | ||
69 | development host. For the steps needed to set up a system using CROPS, | ||
70 | see the | ||
71 | ":ref:`dev-manual/dev-manual-start:setting up to use cross platforms (crops)`" | ||
72 | section in | ||
73 | the Yocto Project Development Tasks Manual. | ||
74 | |||
75 | If your development host is going to be a system that runs a Linux | ||
76 | distribution, steps still exist that you must take to prepare the system | ||
77 | for use with the Yocto Project. You need to be sure that the Linux | ||
78 | distribution on the system is one that supports the Yocto Project. You | ||
79 | also need to be sure that the correct set of host packages are installed | ||
80 | that allow development using the Yocto Project. For the steps needed to | ||
81 | set up a development host that runs Linux, see the | ||
82 | ":ref:`dev-manual/dev-manual-start:setting up a native linux host`" | ||
83 | section in the Yocto Project Development Tasks Manual. | ||
84 | |||
85 | Once your development host is set up to use the Yocto Project, several | ||
86 | methods exist for you to do work in the Yocto Project environment: | ||
87 | |||
88 | - *Command Lines, BitBake, and Shells:* Traditional development in the | ||
89 | Yocto Project involves using the :term:`OpenEmbedded Build System`, | ||
90 | which uses | ||
91 | BitBake, in a command-line environment from a shell on your | ||
92 | development host. You can accomplish this from a host that is a | ||
93 | native Linux machine or from a host that has been set up with CROPS. | ||
94 | Either way, you create, modify, and build images and applications all | ||
95 | within a shell-based environment using components and tools available | ||
96 | through your Linux distribution and the Yocto Project. | ||
97 | |||
98 | For a general flow of the build procedures, see the | ||
99 | ":ref:`dev-manual/dev-manual-common-tasks:building a simple image`" | ||
100 | section in the Yocto Project Development Tasks Manual. | ||
101 | |||
102 | - *Board Support Package (BSP) Development:* Development of BSPs | ||
103 | involves using the Yocto Project to create and test layers that allow | ||
104 | easy development of images and applications targeted for specific | ||
105 | hardware. To development BSPs, you need to take some additional steps | ||
106 | beyond what was described in setting up a development host. | ||
107 | |||
108 | The :doc:`../bsp-guide/bsp-guide` provides BSP-related development | ||
109 | information. For specifics on development host preparation, see the | ||
110 | ":ref:`bsp-guide/bsp:preparing your build host to work with bsp layers`" | ||
111 | section in the Yocto Project Board Support Package (BSP) Developer's | ||
112 | Guide. | ||
113 | |||
114 | - *Kernel Development:* If you are going to be developing kernels using | ||
115 | the Yocto Project you likely will be using ``devtool``. A workflow | ||
116 | using ``devtool`` makes kernel development quicker by reducing | ||
117 | iteration cycle times. | ||
118 | |||
119 | The :doc:`../kernel-dev/kernel-dev` provides kernel-related | ||
120 | development information. For specifics on development host | ||
121 | preparation, see the | ||
122 | ":ref:`kernel-dev/kernel-dev-common:preparing the build host to work on the kernel`" | ||
123 | section in the Yocto Project Linux Kernel Development Manual. | ||
124 | |||
125 | - *Using Toaster:* The other Yocto Project development method that | ||
126 | involves an interface that effectively puts the Yocto Project into | ||
127 | the background is Toaster. Toaster provides an interface to the | ||
128 | OpenEmbedded build system. The interface enables you to configure and | ||
129 | run your builds. Information about builds is collected and stored in | ||
130 | a database. You can use Toaster to configure and start builds on | ||
131 | multiple remote build servers. | ||
132 | |||
133 | For steps that show you how to set up your development host to use | ||
134 | Toaster and on how to use Toaster in general, see the | ||
135 | :doc:`../toaster-manual/toaster-manual`. | ||
136 | |||
137 | .. _yocto-project-repositories: | ||
138 | |||
139 | Yocto Project Source Repositories | ||
140 | ================================= | ||
141 | |||
142 | The Yocto Project team maintains complete source repositories for all | ||
143 | Yocto Project files at :yocto_git:`/`. This web-based source | ||
144 | code browser is organized into categories by function such as IDE | ||
145 | Plugins, Matchbox, Poky, Yocto Linux Kernel, and so forth. From the | ||
146 | interface, you can click on any particular item in the "Name" column and | ||
147 | see the URL at the bottom of the page that you need to clone a Git | ||
148 | repository for that particular item. Having a local Git repository of | ||
149 | the :term:`Source Directory`, which | ||
150 | is usually named "poky", allows you to make changes, contribute to the | ||
151 | history, and ultimately enhance the Yocto Project's tools, Board Support | ||
152 | Packages, and so forth. | ||
153 | |||
154 | For any supported release of Yocto Project, you can also go to the | ||
155 | :yocto_home:`Yocto Project Website <>` and select the "DOWNLOADS" | ||
156 | item from the "SOFTWARE" menu and get a released tarball of the ``poky`` | ||
157 | repository, any supported BSP tarball, or Yocto Project tools. Unpacking | ||
158 | these tarballs gives you a snapshot of the released files. | ||
159 | |||
160 | .. note:: | ||
161 | |||
162 | - The recommended method for setting up the Yocto Project | ||
163 | :term:`Source Directory` and the files | ||
164 | for supported BSPs (e.g., ``meta-intel``) is to use `Git <#git>`__ | ||
165 | to create a local copy of the upstream repositories. | ||
166 | |||
167 | - Be sure to always work in matching branches for both the selected | ||
168 | BSP repository and the Source Directory (i.e. ``poky``) | ||
169 | repository. For example, if you have checked out the "master" | ||
170 | branch of ``poky`` and you are going to use ``meta-intel``, be | ||
171 | sure to checkout the "master" branch of ``meta-intel``. | ||
172 | |||
173 | In summary, here is where you can get the project files needed for | ||
174 | development: | ||
175 | |||
176 | - :yocto_git:`Source Repositories: <>` This area contains IDE | ||
177 | Plugins, Matchbox, Poky, Poky Support, Tools, Yocto Linux Kernel, and | ||
178 | Yocto Metadata Layers. You can create local copies of Git | ||
179 | repositories for each of these areas. | ||
180 | |||
181 | .. image:: figures/source-repos.png | ||
182 | :align: center | ||
183 | |||
184 | For steps on how to view and access these upstream Git repositories, | ||
185 | see the ":ref:`dev-manual/dev-manual-start:accessing source repositories`" | ||
186 | Section in the Yocto Project Development Tasks Manual. | ||
187 | |||
188 | - :yocto_dl:`Index of /releases: </releases>` This is an index | ||
189 | of releases such as Poky, Pseudo, installers for cross-development | ||
190 | toolchains, miscellaneous support and all released versions of Yocto | ||
191 | Project in the form of images or tarballs. Downloading and extracting | ||
192 | these files does not produce a local copy of the Git repository but | ||
193 | rather a snapshot of a particular release or image. | ||
194 | |||
195 | .. image:: figures/index-downloads.png | ||
196 | :align: center | ||
197 | |||
198 | For steps on how to view and access these files, see the | ||
199 | ":ref:`dev-manual/dev-manual-start:accessing index of releases`" | ||
200 | section in the Yocto Project Development Tasks Manual. | ||
201 | |||
202 | - *"DOWNLOADS" page for the* :yocto_home:`Yocto Project Website <>` *:* | ||
203 | |||
204 | The Yocto Project website includes a "DOWNLOADS" page accessible | ||
205 | through the "SOFTWARE" menu that allows you to download any Yocto | ||
206 | Project release, tool, and Board Support Package (BSP) in tarball | ||
207 | form. The tarballs are similar to those found in the | ||
208 | :yocto_dl:`Index of /releases: </releases>` area. | ||
209 | |||
210 | .. image:: figures/yp-download.png | ||
211 | :align: center | ||
212 | |||
213 | For steps on how to use the "DOWNLOADS" page, see the | ||
214 | ":ref:`dev-manual/dev-manual-start:using the downloads page`" | ||
215 | section in the Yocto Project Development Tasks Manual. | ||
216 | |||
217 | .. _gs-git-workflows-and-the-yocto-project: | ||
218 | |||
219 | Git Workflows and the Yocto Project | ||
220 | =================================== | ||
221 | |||
222 | Developing using the Yocto Project likely requires the use of | ||
223 | `Git <#git>`__. Git is a free, open source distributed version control | ||
224 | system used as part of many collaborative design environments. This | ||
225 | section provides workflow concepts using the Yocto Project and Git. In | ||
226 | particular, the information covers basic practices that describe roles | ||
227 | and actions in a collaborative development environment. | ||
228 | |||
229 | .. note:: | ||
230 | |||
231 | If you are familiar with this type of development environment, you | ||
232 | might not want to read this section. | ||
233 | |||
234 | The Yocto Project files are maintained using Git in "branches" whose Git | ||
235 | histories track every change and whose structures provide branches for | ||
236 | all diverging functionality. Although there is no need to use Git, many | ||
237 | open source projects do so. | ||
238 | |||
239 | For the Yocto Project, a key individual called the "maintainer" is | ||
240 | responsible for the integrity of the "master" branch of a given Git | ||
241 | repository. The "master" branch is the "upstream" repository from which | ||
242 | final or most recent builds of a project occur. The maintainer is | ||
243 | responsible for accepting changes from other developers and for | ||
244 | organizing the underlying branch structure to reflect release strategies | ||
245 | and so forth. | ||
246 | |||
247 | .. note:: | ||
248 | |||
249 | For information on finding out who is responsible for (maintains) a | ||
250 | particular area of code in the Yocto Project, see the | ||
251 | ":ref:`dev-manual/dev-manual-common-tasks:submitting a change to the yocto project`" | ||
252 | section of the Yocto Project Development Tasks Manual. | ||
253 | |||
254 | The Yocto Project ``poky`` Git repository also has an upstream | ||
255 | contribution Git repository named ``poky-contrib``. You can see all the | ||
256 | branches in this repository using the web interface of the | ||
257 | :yocto_git:`Source Repositories <>` organized within the "Poky Support" | ||
258 | area. These branches hold changes (commits) to the project that have | ||
259 | been submitted or committed by the Yocto Project development team and by | ||
260 | community members who contribute to the project. The maintainer | ||
261 | determines if the changes are qualified to be moved from the "contrib" | ||
262 | branches into the "master" branch of the Git repository. | ||
263 | |||
264 | Developers (including contributing community members) create and | ||
265 | maintain cloned repositories of upstream branches. The cloned | ||
266 | repositories are local to their development platforms and are used to | ||
267 | develop changes. When a developer is satisfied with a particular feature | ||
268 | or change, they "push" the change to the appropriate "contrib" | ||
269 | repository. | ||
270 | |||
271 | Developers are responsible for keeping their local repository up-to-date | ||
272 | with whatever upstream branch they are working against. They are also | ||
273 | responsible for straightening out any conflicts that might arise within | ||
274 | files that are being worked on simultaneously by more than one person. | ||
275 | All this work is done locally on the development host before anything is | ||
276 | pushed to a "contrib" area and examined at the maintainer's level. | ||
277 | |||
278 | A somewhat formal method exists by which developers commit changes and | ||
279 | push them into the "contrib" area and subsequently request that the | ||
280 | maintainer include them into an upstream branch. This process is called | ||
281 | "submitting a patch" or "submitting a change." For information on | ||
282 | submitting patches and changes, see the | ||
283 | ":ref:`dev-manual/dev-manual-common-tasks:submitting a change to the yocto project`" | ||
284 | section in the Yocto Project Development Tasks Manual. | ||
285 | |||
286 | In summary, a single point of entry exists for changes into a "master" | ||
287 | or development branch of the Git repository, which is controlled by the | ||
288 | project's maintainer. And, a set of developers exist who independently | ||
289 | develop, test, and submit changes to "contrib" areas for the maintainer | ||
290 | to examine. The maintainer then chooses which changes are going to | ||
291 | become a permanent part of the project. | ||
292 | |||
293 | .. image:: figures/git-workflow.png | ||
294 | :align: center | ||
295 | |||
296 | While each development environment is unique, there are some best | ||
297 | practices or methods that help development run smoothly. The following | ||
298 | list describes some of these practices. For more information about Git | ||
299 | workflows, see the workflow topics in the `Git Community | ||
300 | Book <http://book.git-scm.com>`__. | ||
301 | |||
302 | - *Make Small Changes:* It is best to keep the changes you commit small | ||
303 | as compared to bundling many disparate changes into a single commit. | ||
304 | This practice not only keeps things manageable but also allows the | ||
305 | maintainer to more easily include or refuse changes. | ||
306 | |||
307 | - *Make Complete Changes:* It is also good practice to leave the | ||
308 | repository in a state that allows you to still successfully build | ||
309 | your project. In other words, do not commit half of a feature, then | ||
310 | add the other half as a separate, later commit. Each commit should | ||
311 | take you from one buildable project state to another buildable state. | ||
312 | |||
313 | - *Use Branches Liberally:* It is very easy to create, use, and delete | ||
314 | local branches in your working Git repository on the development | ||
315 | host. You can name these branches anything you like. It is helpful to | ||
316 | give them names associated with the particular feature or change on | ||
317 | which you are working. Once you are done with a feature or change and | ||
318 | have merged it into your local master branch, simply discard the | ||
319 | temporary branch. | ||
320 | |||
321 | - *Merge Changes:* The ``git merge`` command allows you to take the | ||
322 | changes from one branch and fold them into another branch. This | ||
323 | process is especially helpful when more than a single developer might | ||
324 | be working on different parts of the same feature. Merging changes | ||
325 | also automatically identifies any collisions or "conflicts" that | ||
326 | might happen as a result of the same lines of code being altered by | ||
327 | two different developers. | ||
328 | |||
329 | - *Manage Branches:* Because branches are easy to use, you should use a | ||
330 | system where branches indicate varying levels of code readiness. For | ||
331 | example, you can have a "work" branch to develop in, a "test" branch | ||
332 | where the code or change is tested, a "stage" branch where changes | ||
333 | are ready to be committed, and so forth. As your project develops, | ||
334 | you can merge code across the branches to reflect ever-increasing | ||
335 | stable states of the development. | ||
336 | |||
337 | - *Use Push and Pull:* The push-pull workflow is based on the concept | ||
338 | of developers "pushing" local commits to a remote repository, which | ||
339 | is usually a contribution repository. This workflow is also based on | ||
340 | developers "pulling" known states of the project down into their | ||
341 | local development repositories. The workflow easily allows you to | ||
342 | pull changes submitted by other developers from the upstream | ||
343 | repository into your work area ensuring that you have the most recent | ||
344 | software on which to develop. The Yocto Project has two scripts named | ||
345 | ``create-pull-request`` and ``send-pull-request`` that ship with the | ||
346 | release to facilitate this workflow. You can find these scripts in | ||
347 | the ``scripts`` folder of the | ||
348 | :term:`Source Directory`. For information | ||
349 | on how to use these scripts, see the | ||
350 | ":ref:`dev-manual/dev-manual-common-tasks:using scripts to push a change upstream and request a pull`" | ||
351 | section in the Yocto Project Development Tasks Manual. | ||
352 | |||
353 | - *Patch Workflow:* This workflow allows you to notify the maintainer | ||
354 | through an email that you have a change (or patch) you would like | ||
355 | considered for the "master" branch of the Git repository. To send | ||
356 | this type of change, you format the patch and then send the email | ||
357 | using the Git commands ``git format-patch`` and ``git send-email``. | ||
358 | For information on how to use these scripts, see the | ||
359 | ":ref:`dev-manual/dev-manual-common-tasks:submitting a change to the yocto project`" | ||
360 | section in the Yocto Project Development Tasks Manual. | ||
361 | |||
362 | Git | ||
363 | === | ||
364 | |||
365 | The Yocto Project makes extensive use of Git, which is a free, open | ||
366 | source distributed version control system. Git supports distributed | ||
367 | development, non-linear development, and can handle large projects. It | ||
368 | is best that you have some fundamental understanding of how Git tracks | ||
369 | projects and how to work with Git if you are going to use the Yocto | ||
370 | Project for development. This section provides a quick overview of how | ||
371 | Git works and provides you with a summary of some essential Git | ||
372 | commands. | ||
373 | |||
374 | .. note:: | ||
375 | |||
376 | - For more information on Git, see | ||
377 | http://git-scm.com/documentation. | ||
378 | |||
379 | - If you need to download Git, it is recommended that you add Git to | ||
380 | your system through your distribution's "software store" (e.g. for | ||
381 | Ubuntu, use the Ubuntu Software feature). For the Git download | ||
382 | page, see http://git-scm.com/download. | ||
383 | |||
384 | - For information beyond the introductory nature in this section, | ||
385 | see the ":ref:`dev-manual/dev-manual-start:locating yocto project source files`" | ||
386 | section in the Yocto Project Development Tasks Manual. | ||
387 | |||
388 | Repositories, Tags, and Branches | ||
389 | -------------------------------- | ||
390 | |||
391 | As mentioned briefly in the previous section and also in the "`Git | ||
392 | Workflows and the Yocto | ||
393 | Project <#gs-git-workflows-and-the-yocto-project>`__" section, the Yocto | ||
394 | Project maintains source repositories at :yocto_git:`/`. If you | ||
395 | look at this web-interface of the repositories, each item is a separate | ||
396 | Git repository. | ||
397 | |||
398 | Git repositories use branching techniques that track content change (not | ||
399 | files) within a project (e.g. a new feature or updated documentation). | ||
400 | Creating a tree-like structure based on project divergence allows for | ||
401 | excellent historical information over the life of a project. This | ||
402 | methodology also allows for an environment from which you can do lots of | ||
403 | local experimentation on projects as you develop changes or new | ||
404 | features. | ||
405 | |||
406 | A Git repository represents all development efforts for a given project. | ||
407 | For example, the Git repository ``poky`` contains all changes and | ||
408 | developments for that repository over the course of its entire life. | ||
409 | That means that all changes that make up all releases are captured. The | ||
410 | repository maintains a complete history of changes. | ||
411 | |||
412 | You can create a local copy of any repository by "cloning" it with the | ||
413 | ``git clone`` command. When you clone a Git repository, you end up with | ||
414 | an identical copy of the repository on your development system. Once you | ||
415 | have a local copy of a repository, you can take steps to develop | ||
416 | locally. For examples on how to clone Git repositories, see the | ||
417 | ":ref:`dev-manual/dev-manual-start:locating yocto project source files`" | ||
418 | section in the Yocto Project Development Tasks Manual. | ||
419 | |||
420 | It is important to understand that Git tracks content change and not | ||
421 | files. Git uses "branches" to organize different development efforts. | ||
422 | For example, the ``poky`` repository has several branches that include | ||
423 | the current "&DISTRO_NAME_NO_CAP;" branch, the "master" branch, and many | ||
424 | branches for past Yocto Project releases. You can see all the branches | ||
425 | by going to https://git.yoctoproject.org/cgit.cgi/poky/ and clicking on the | ||
426 | ``[...]`` link beneath the "Branch" heading. | ||
427 | |||
428 | Each of these branches represents a specific area of development. The | ||
429 | "master" branch represents the current or most recent development. All | ||
430 | other branches represent offshoots of the "master" branch. | ||
431 | |||
432 | When you create a local copy of a Git repository, the copy has the same | ||
433 | set of branches as the original. This means you can use Git to create a | ||
434 | local working area (also called a branch) that tracks a specific | ||
435 | development branch from the upstream source Git repository. in other | ||
436 | words, you can define your local Git environment to work on any | ||
437 | development branch in the repository. To help illustrate, consider the | ||
438 | following example Git commands: | ||
439 | :: | ||
440 | |||
441 | $ cd ~ | ||
442 | $ git clone git://git.yoctoproject.org/poky | ||
443 | $ cd poky | ||
444 | $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP; | ||
445 | |||
446 | In the previous example | ||
447 | after moving to the home directory, the ``git clone`` command creates a | ||
448 | local copy of the upstream ``poky`` Git repository. By default, Git | ||
449 | checks out the "master" branch for your work. After changing the working | ||
450 | directory to the new local repository (i.e. ``poky``), the | ||
451 | ``git checkout`` command creates and checks out a local branch named | ||
452 | "&DISTRO_NAME_NO_CAP;", which tracks the upstream | ||
453 | "origin/&DISTRO_NAME_NO_CAP;" branch. Changes you make while in this | ||
454 | branch would ultimately affect the upstream "&DISTRO_NAME_NO_CAP;" branch | ||
455 | of the ``poky`` repository. | ||
456 | |||
457 | It is important to understand that when you create and checkout a local | ||
458 | working branch based on a branch name, your local environment matches | ||
459 | the "tip" of that particular development branch at the time you created | ||
460 | your local branch, which could be different from the files in the | ||
461 | "master" branch of the upstream repository. In other words, creating and | ||
462 | checking out a local branch based on the "&DISTRO_NAME_NO_CAP;" branch | ||
463 | name is not the same as checking out the "master" branch in the | ||
464 | repository. Keep reading to see how you create a local snapshot of a | ||
465 | Yocto Project Release. | ||
466 | |||
467 | Git uses "tags" to mark specific changes in a repository branch | ||
468 | structure. Typically, a tag is used to mark a special point such as the | ||
469 | final change (or commit) before a project is released. You can see the | ||
470 | tags used with the ``poky`` Git repository by going to | ||
471 | https://git.yoctoproject.org/cgit.cgi/poky/ and clicking on the ``[...]`` link | ||
472 | beneath the "Tag" heading. | ||
473 | |||
474 | Some key tags for the ``poky`` repository are ``jethro-14.0.3``, | ||
475 | ``morty-16.0.1``, ``pyro-17.0.0``, and | ||
476 | ``&DISTRO_NAME_NO_CAP;-&POKYVERSION;``. These tags represent Yocto Project | ||
477 | releases. | ||
478 | |||
479 | When you create a local copy of the Git repository, you also have access | ||
480 | to all the tags in the upstream repository. Similar to branches, you can | ||
481 | create and checkout a local working Git branch based on a tag name. When | ||
482 | you do this, you get a snapshot of the Git repository that reflects the | ||
483 | state of the files when the change was made associated with that tag. | ||
484 | The most common use is to checkout a working branch that matches a | ||
485 | specific Yocto Project release. Here is an example: | ||
486 | :: | ||
487 | |||
488 | $ cd ~ | ||
489 | $ git clone git://git.yoctoproject.org/poky | ||
490 | $ cd poky | ||
491 | $ git fetch --tags | ||
492 | $ git checkout tags/rocko-18.0.0 -b my_rocko-18.0.0 | ||
493 | |||
494 | In this example, the name | ||
495 | of the top-level directory of your local Yocto Project repository is | ||
496 | ``poky``. After moving to the ``poky`` directory, the ``git fetch`` | ||
497 | command makes all the upstream tags available locally in your | ||
498 | repository. Finally, the ``git checkout`` command creates and checks out | ||
499 | a branch named "my-rocko-18.0.0" that is based on the upstream branch | ||
500 | whose "HEAD" matches the commit in the repository associated with the | ||
501 | "rocko-18.0.0" tag. The files in your repository now exactly match that | ||
502 | particular Yocto Project release as it is tagged in the upstream Git | ||
503 | repository. It is important to understand that when you create and | ||
504 | checkout a local working branch based on a tag, your environment matches | ||
505 | a specific point in time and not the entire development branch (i.e. | ||
506 | from the "tip" of the branch backwards). | ||
507 | |||
508 | Basic Commands | ||
509 | -------------- | ||
510 | |||
511 | Git has an extensive set of commands that lets you manage changes and | ||
512 | perform collaboration over the life of a project. Conveniently though, | ||
513 | you can manage with a small set of basic operations and workflows once | ||
514 | you understand the basic philosophy behind Git. You do not have to be an | ||
515 | expert in Git to be functional. A good place to look for instruction on | ||
516 | a minimal set of Git commands is | ||
517 | `here <http://git-scm.com/documentation>`__. | ||
518 | |||
519 | The following list of Git commands briefly describes some basic Git | ||
520 | operations as a way to get started. As with any set of commands, this | ||
521 | list (in most cases) simply shows the base command and omits the many | ||
522 | arguments it supports. See the Git documentation for complete | ||
523 | descriptions and strategies on how to use these commands: | ||
524 | |||
525 | - *git init:* Initializes an empty Git repository. You cannot use | ||
526 | Git commands unless you have a ``.git`` repository. | ||
527 | |||
528 | - *git clone:* Creates a local clone of a Git repository that is on | ||
529 | equal footing with a fellow developer's Git repository or an upstream | ||
530 | repository. | ||
531 | |||
532 | - *git add:* Locally stages updated file contents to the index that | ||
533 | Git uses to track changes. You must stage all files that have changed | ||
534 | before you can commit them. | ||
535 | |||
536 | - *git commit:* Creates a local "commit" that documents the changes | ||
537 | you made. Only changes that have been staged can be committed. | ||
538 | Commits are used for historical purposes, for determining if a | ||
539 | maintainer of a project will allow the change, and for ultimately | ||
540 | pushing the change from your local Git repository into the project's | ||
541 | upstream repository. | ||
542 | |||
543 | - *git status:* Reports any modified files that possibly need to be | ||
544 | staged and gives you a status of where you stand regarding local | ||
545 | commits as compared to the upstream repository. | ||
546 | |||
547 | - *git checkout branch-name:* Changes your local working branch and | ||
548 | in this form assumes the local branch already exists. This command is | ||
549 | analogous to "cd". | ||
550 | |||
551 | - *git checkout –b working-branch upstream-branch:* Creates and | ||
552 | checks out a working branch on your local machine. The local branch | ||
553 | tracks the upstream branch. You can use your local branch to isolate | ||
554 | your work. It is a good idea to use local branches when adding | ||
555 | specific features or changes. Using isolated branches facilitates | ||
556 | easy removal of changes if they do not work out. | ||
557 | |||
558 | - *git branch:* Displays the existing local branches associated | ||
559 | with your local repository. The branch that you have currently | ||
560 | checked out is noted with an asterisk character. | ||
561 | |||
562 | - *git branch -D branch-name:* Deletes an existing local branch. | ||
563 | You need to be in a local branch other than the one you are deleting | ||
564 | in order to delete branch-name. | ||
565 | |||
566 | - *git pull --rebase:* Retrieves information from an upstream Git | ||
567 | repository and places it in your local Git repository. You use this | ||
568 | command to make sure you are synchronized with the repository from | ||
569 | which you are basing changes (.e.g. the "master" branch). The | ||
570 | "--rebase" option ensures that any local commits you have in your | ||
571 | branch are preserved at the top of your local branch. | ||
572 | |||
573 | - *git push repo-name local-branch:upstream-branch:* Sends | ||
574 | all your committed local changes to the upstream Git repository that | ||
575 | your local repository is tracking (e.g. a contribution repository). | ||
576 | The maintainer of the project draws from these repositories to merge | ||
577 | changes (commits) into the appropriate branch of project's upstream | ||
578 | repository. | ||
579 | |||
580 | - *git merge:* Combines or adds changes from one local branch of | ||
581 | your repository with another branch. When you create a local Git | ||
582 | repository, the default branch is named "master". A typical workflow | ||
583 | is to create a temporary branch that is based off "master" that you | ||
584 | would use for isolated work. You would make your changes in that | ||
585 | isolated branch, stage and commit them locally, switch to the | ||
586 | "master" branch, and then use the ``git merge`` command to apply the | ||
587 | changes from your isolated branch into the currently checked out | ||
588 | branch (e.g. "master"). After the merge is complete and if you are | ||
589 | done with working in that isolated branch, you can safely delete the | ||
590 | isolated branch. | ||
591 | |||
592 | - *git cherry-pick commits:* Choose and apply specific commits from | ||
593 | one branch into another branch. There are times when you might not be | ||
594 | able to merge all the changes in one branch with another but need to | ||
595 | pick out certain ones. | ||
596 | |||
597 | - *gitk:* Provides a GUI view of the branches and changes in your | ||
598 | local Git repository. This command is a good way to graphically see | ||
599 | where things have diverged in your local repository. | ||
600 | |||
601 | .. note:: | ||
602 | |||
603 | You need to install the | ||
604 | gitk | ||
605 | package on your development system to use this command. | ||
606 | |||
607 | - *git log:* Reports a history of your commits to the repository. | ||
608 | This report lists all commits regardless of whether you have pushed | ||
609 | them upstream or not. | ||
610 | |||
611 | - *git diff:* Displays line-by-line differences between a local | ||
612 | working file and the same file as understood by Git. This command is | ||
613 | useful to see what you have changed in any given file. | ||
614 | |||
615 | Licensing | ||
616 | ========= | ||
617 | |||
618 | Because open source projects are open to the public, they have different | ||
619 | licensing structures in place. License evolution for both Open Source | ||
620 | and Free Software has an interesting history. If you are interested in | ||
621 | this history, you can find basic information here: | ||
622 | |||
623 | - `Open source license | ||
624 | history <http://en.wikipedia.org/wiki/Open-source_license>`__ | ||
625 | |||
626 | - `Free software license | ||
627 | history <http://en.wikipedia.org/wiki/Free_software_license>`__ | ||
628 | |||
629 | In general, the Yocto Project is broadly licensed under the | ||
630 | Massachusetts Institute of Technology (MIT) License. MIT licensing | ||
631 | permits the reuse of software within proprietary software as long as the | ||
632 | license is distributed with that software. MIT is also compatible with | ||
633 | the GNU General Public License (GPL). Patches to the Yocto Project | ||
634 | follow the upstream licensing scheme. You can find information on the | ||
635 | MIT license | ||
636 | `here <http://www.opensource.org/licenses/mit-license.php>`__. You can | ||
637 | find information on the GNU GPL | ||
638 | `here <http://www.opensource.org/licenses/LGPL-3.0>`__. | ||
639 | |||
640 | When you build an image using the Yocto Project, the build process uses | ||
641 | a known list of licenses to ensure compliance. You can find this list in | ||
642 | the :term:`Source Directory` at | ||
643 | ``meta/files/common-licenses``. Once the build completes, the list of | ||
644 | all licenses found and used during that build are kept in the | ||
645 | :term:`Build Directory` at | ||
646 | ``tmp/deploy/licenses``. | ||
647 | |||
648 | If a module requires a license that is not in the base list, the build | ||
649 | process generates a warning during the build. These tools make it easier | ||
650 | for a developer to be certain of the licenses with which their shipped | ||
651 | products must comply. However, even with these tools it is still up to | ||
652 | the developer to resolve potential licensing issues. | ||
653 | |||
654 | The base list of licenses used by the build process is a combination of | ||
655 | the Software Package Data Exchange (SPDX) list and the Open Source | ||
656 | Initiative (OSI) projects. `SPDX Group <http://spdx.org>`__ is a working | ||
657 | group of the Linux Foundation that maintains a specification for a | ||
658 | standard format for communicating the components, licenses, and | ||
659 | copyrights associated with a software package. | ||
660 | `OSI <http://opensource.org>`__ is a corporation dedicated to the Open | ||
661 | Source Definition and the effort for reviewing and approving licenses | ||
662 | that conform to the Open Source Definition (OSD). | ||
663 | |||
664 | You can find a list of the combined SPDX and OSI licenses that the Yocto | ||
665 | Project uses in the ``meta/files/common-licenses`` directory in your | ||
666 | :term:`Source Directory`. | ||
667 | |||
668 | For information that can help you maintain compliance with various open | ||
669 | source licensing during the lifecycle of a product created using the | ||
670 | Yocto Project, see the | ||
671 | ":ref:`dev-manual/dev-manual-common-tasks:maintaining open source license compliance during your product's lifecycle`" | ||
672 | section in the Yocto Project Development Tasks Manual. | ||
diff --git a/documentation/overview-manual/overview-manual-development-environment.xml b/documentation/overview-manual/overview-manual-development-environment.xml index 36ebf8a321..08ad071316 100644 --- a/documentation/overview-manual/overview-manual-development-environment.xml +++ b/documentation/overview-manual/overview-manual-development-environment.xml | |||
@@ -1,6 +1,7 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
4 | 5 | ||
5 | <chapter id='overview-development-environment'> | 6 | <chapter id='overview-development-environment'> |
6 | <title>The Yocto Project Development Environment</title> | 7 | <title>The Yocto Project Development Environment</title> |
@@ -326,7 +327,7 @@ | |||
326 | For the Yocto Project, a key individual called the "maintainer" is | 327 | For the Yocto Project, a key individual called the "maintainer" is |
327 | responsible for the integrity of the "master" branch of a given Git | 328 | responsible for the integrity of the "master" branch of a given Git |
328 | repository. | 329 | repository. |
329 | The "master" branch is the “upstream” repository from which final or | 330 | The "master" branch is the "upstream" repository from which final or |
330 | most recent builds of a project occur. | 331 | most recent builds of a project occur. |
331 | The maintainer is responsible for accepting changes from other | 332 | The maintainer is responsible for accepting changes from other |
332 | developers and for organizing the underlying branch structure to | 333 | developers and for organizing the underlying branch structure to |
@@ -371,7 +372,7 @@ | |||
371 | might arise within files that are being worked on simultaneously by | 372 | might arise within files that are being worked on simultaneously by |
372 | more than one person. | 373 | more than one person. |
373 | All this work is done locally on the development host before | 374 | All this work is done locally on the development host before |
374 | anything is pushed to a "contrib" area and examined at the maintainer’s | 375 | anything is pushed to a "contrib" area and examined at the maintainer's |
375 | level. | 376 | level. |
376 | </para> | 377 | </para> |
377 | 378 | ||
@@ -379,7 +380,7 @@ | |||
379 | A somewhat formal method exists by which developers commit changes | 380 | A somewhat formal method exists by which developers commit changes |
380 | and push them into the "contrib" area and subsequently request that | 381 | and push them into the "contrib" area and subsequently request that |
381 | the maintainer include them into an upstream branch. | 382 | the maintainer include them into an upstream branch. |
382 | This process is called “submitting a patch” or "submitting a change." | 383 | This process is called "submitting a patch" or "submitting a change." |
383 | For information on submitting patches and changes, see the | 384 | For information on submitting patches and changes, see the |
384 | "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>" | 385 | "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>" |
385 | section in the Yocto Project Development Tasks Manual. | 386 | section in the Yocto Project Development Tasks Manual. |
@@ -388,7 +389,7 @@ | |||
388 | <para> | 389 | <para> |
389 | In summary, a single point of entry | 390 | In summary, a single point of entry |
390 | exists for changes into a "master" or development branch of the | 391 | exists for changes into a "master" or development branch of the |
391 | Git repository, which is controlled by the project’s maintainer. | 392 | Git repository, which is controlled by the project's maintainer. |
392 | And, a set of developers exist who independently develop, test, and | 393 | And, a set of developers exist who independently develop, test, and |
393 | submit changes to "contrib" areas for the maintainer to examine. | 394 | submit changes to "contrib" areas for the maintainer to examine. |
394 | The maintainer then chooses which changes are going to become a | 395 | The maintainer then chooses which changes are going to become a |
@@ -733,7 +734,7 @@ | |||
733 | <listitem><para id='git-commands-clone'> | 734 | <listitem><para id='git-commands-clone'> |
734 | <emphasis><filename>git clone</filename>:</emphasis> | 735 | <emphasis><filename>git clone</filename>:</emphasis> |
735 | Creates a local clone of a Git repository that is on | 736 | Creates a local clone of a Git repository that is on |
736 | equal footing with a fellow developer’s Git repository | 737 | equal footing with a fellow developer's Git repository |
737 | or an upstream repository. | 738 | or an upstream repository. |
738 | </para></listitem> | 739 | </para></listitem> |
739 | <listitem><para> | 740 | <listitem><para> |
@@ -751,7 +752,7 @@ | |||
751 | Commits are used for historical purposes, for determining | 752 | Commits are used for historical purposes, for determining |
752 | if a maintainer of a project will allow the change, | 753 | if a maintainer of a project will allow the change, |
753 | and for ultimately pushing the change from your local | 754 | and for ultimately pushing the change from your local |
754 | Git repository into the project’s upstream repository. | 755 | Git repository into the project's upstream repository. |
755 | </para></listitem> | 756 | </para></listitem> |
756 | <listitem><para> | 757 | <listitem><para> |
757 | <emphasis><filename>git status</filename>:</emphasis> | 758 | <emphasis><filename>git status</filename>:</emphasis> |
diff --git a/documentation/overview-manual/overview-manual-intro.rst b/documentation/overview-manual/overview-manual-intro.rst new file mode 100644 index 0000000000..3f206fd54b --- /dev/null +++ b/documentation/overview-manual/overview-manual-intro.rst | |||
@@ -0,0 +1,74 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ********************************************** | ||
4 | The Yocto Project Overview and Concepts Manual | ||
5 | ********************************************** | ||
6 | |||
7 | .. _overview-manual-welcome: | ||
8 | |||
9 | Welcome | ||
10 | ======= | ||
11 | |||
12 | Welcome to the Yocto Project Overview and Concepts Manual! This manual | ||
13 | introduces the Yocto Project by providing concepts, software overviews, | ||
14 | best-known-methods (BKMs), and any other high-level introductory | ||
15 | information suitable for a new Yocto Project user. | ||
16 | |||
17 | The following list describes what you can get from this manual: | ||
18 | |||
19 | - `Introducing the Yocto Project <#overview-yp>`__\ *:* This chapter | ||
20 | provides an introduction to the Yocto Project. You will learn about | ||
21 | features and challenges of the Yocto Project, the layer model, | ||
22 | components and tools, development methods, the | ||
23 | :term:`Poky` reference distribution, the | ||
24 | OpenEmbedded build system workflow, and some basic Yocto terms. | ||
25 | |||
26 | - `The Yocto Project Development | ||
27 | Environment <#overview-development-environment>`__\ *:* This chapter | ||
28 | helps you get started understanding the Yocto Project development | ||
29 | environment. You will learn about open source, development hosts, | ||
30 | Yocto Project source repositories, workflows using Git and the Yocto | ||
31 | Project, a Git primer, and information about licensing. | ||
32 | |||
33 | - :doc:`overview-manual-concepts` *:* This | ||
34 | chapter presents various concepts regarding the Yocto Project. You | ||
35 | can find conceptual information about components, development, | ||
36 | cross-toolchains, and so forth. | ||
37 | |||
38 | This manual does not give you the following: | ||
39 | |||
40 | - *Step-by-step Instructions for Development Tasks:* Instructional | ||
41 | procedures reside in other manuals within the Yocto Project | ||
42 | documentation set. For example, the :doc:`../dev-manual/dev-manual` | ||
43 | provides examples on how to perform | ||
44 | various development tasks. As another example, the | ||
45 | :doc:`../sdk-manual/sdk-manual` manual contains detailed | ||
46 | instructions on how to install an SDK, which is used to develop | ||
47 | applications for target hardware. | ||
48 | |||
49 | - *Reference Material:* This type of material resides in an appropriate | ||
50 | reference manual. For example, system variables are documented in the | ||
51 | :doc:`../ref-manual/ref-manual`. As another | ||
52 | example, the :doc:`../bsp-guide/bsp-guide` contains reference information on | ||
53 | BSPs. | ||
54 | |||
55 | - *Detailed Public Information Not Specific to the Yocto Project:* For | ||
56 | example, exhaustive information on how to use the Source Control | ||
57 | Manager Git is better covered with Internet searches and official Git | ||
58 | Documentation than through the Yocto Project documentation. | ||
59 | |||
60 | .. _overview-manual-other-information: | ||
61 | |||
62 | Other Information | ||
63 | ================= | ||
64 | |||
65 | Because this manual presents information for many different topics, | ||
66 | supplemental information is recommended for full comprehension. For | ||
67 | additional introductory information on the Yocto Project, see the | ||
68 | :yocto_home:`Yocto Project Website <>`. If you want to build an image | ||
69 | with no knowledge of Yocto Project as a way of quickly testing it out, | ||
70 | see the :doc:`../brief-yoctoprojectqs/brief-yoctoprojectqs` document. | ||
71 | For a comprehensive list of links and other documentation, see the | ||
72 | ":ref:`Links and Related | ||
73 | Documentation <resources-links-and-related-documentation>`" | ||
74 | section in the Yocto Project Reference Manual. | ||
diff --git a/documentation/overview-manual/overview-manual-intro.xml b/documentation/overview-manual/overview-manual-intro.xml index 39433aa41b..0e0bfed6e5 100644 --- a/documentation/overview-manual/overview-manual-intro.xml +++ b/documentation/overview-manual/overview-manual-intro.xml | |||
@@ -1,6 +1,7 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
4 | 5 | ||
5 | <chapter id='overview-manual-intro'> | 6 | <chapter id='overview-manual-intro'> |
6 | 7 | ||
diff --git a/documentation/overview-manual/overview-manual-style.css b/documentation/overview-manual/overview-manual-style.css index 97a364b125..eec934161a 100644 --- a/documentation/overview-manual/overview-manual-style.css +++ b/documentation/overview-manual/overview-manual-style.css | |||
@@ -1,4 +1,6 @@ | |||
1 | /* | 1 | /* |
2 | SPDX-License-Identifier: CC-BY-2.0-UK | ||
3 | |||
2 | Generic XHTML / DocBook XHTML CSS Stylesheet. | 4 | Generic XHTML / DocBook XHTML CSS Stylesheet. |
3 | 5 | ||
4 | Browser wrangling and typographic design by | 6 | Browser wrangling and typographic design by |
diff --git a/documentation/overview-manual/overview-manual-yp-intro.rst b/documentation/overview-manual/overview-manual-yp-intro.rst new file mode 100644 index 0000000000..5cdab7ca4a --- /dev/null +++ b/documentation/overview-manual/overview-manual-yp-intro.rst | |||
@@ -0,0 +1,941 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ***************************** | ||
4 | Introducing the Yocto Project | ||
5 | ***************************** | ||
6 | |||
7 | What is the Yocto Project? | ||
8 | ========================== | ||
9 | |||
10 | The Yocto Project is an open source collaboration project that helps | ||
11 | developers create custom Linux-based systems that are designed for | ||
12 | embedded products regardless of the product's hardware architecture. | ||
13 | Yocto Project provides a flexible toolset and a development environment | ||
14 | that allows embedded device developers across the world to collaborate | ||
15 | through shared technologies, software stacks, configurations, and best | ||
16 | practices used to create these tailored Linux images. | ||
17 | |||
18 | Thousands of developers worldwide have discovered that Yocto Project | ||
19 | provides advantages in both systems and applications development, | ||
20 | archival and management benefits, and customizations used for speed, | ||
21 | footprint, and memory utilization. The project is a standard when it | ||
22 | comes to delivering embedded software stacks. The project allows | ||
23 | software customizations and build interchange for multiple hardware | ||
24 | platforms as well as software stacks that can be maintained and scaled. | ||
25 | |||
26 | .. image:: figures/key-dev-elements.png | ||
27 | :align: center | ||
28 | |||
29 | For further introductory information on the Yocto Project, you might be | ||
30 | interested in this | ||
31 | `article <https://www.embedded.com/electronics-blogs/say-what-/4458600/Why-the-Yocto-Project-for-my-IoT-Project->`__ | ||
32 | by Drew Moseley and in this short introductory | ||
33 | `video <https://www.youtube.com/watch?v=utZpKM7i5Z4>`__. | ||
34 | |||
35 | The remainder of this section overviews advantages and challenges tied | ||
36 | to the Yocto Project. | ||
37 | |||
38 | .. _gs-features: | ||
39 | |||
40 | Features | ||
41 | -------- | ||
42 | |||
43 | The following list describes features and advantages of the Yocto | ||
44 | Project: | ||
45 | |||
46 | - *Widely Adopted Across the Industry:* Semiconductor, operating | ||
47 | system, software, and service vendors exist whose products and | ||
48 | services adopt and support the Yocto Project. For a look at the Yocto | ||
49 | Project community and the companies involved with the Yocto Project, | ||
50 | see the "COMMUNITY" and "ECOSYSTEM" tabs on the | ||
51 | :yocto_home:`Yocto Project <>` home page. | ||
52 | |||
53 | - *Architecture Agnostic:* Yocto Project supports Intel, ARM, MIPS, | ||
54 | AMD, PPC and other architectures. Most ODMs, OSVs, and chip vendors | ||
55 | create and supply BSPs that support their hardware. If you have | ||
56 | custom silicon, you can create a BSP that supports that architecture. | ||
57 | |||
58 | Aside from lots of architecture support, the Yocto Project fully | ||
59 | supports a wide range of device emulation through the Quick EMUlator | ||
60 | (QEMU). | ||
61 | |||
62 | - *Images and Code Transfer Easily:* Yocto Project output can easily | ||
63 | move between architectures without moving to new development | ||
64 | environments. Additionally, if you have used the Yocto Project to | ||
65 | create an image or application and you find yourself not able to | ||
66 | support it, commercial Linux vendors such as Wind River, Mentor | ||
67 | Graphics, Timesys, and ENEA could take it and provide ongoing | ||
68 | support. These vendors have offerings that are built using the Yocto | ||
69 | Project. | ||
70 | |||
71 | - *Flexibility:* Corporations use the Yocto Project many different | ||
72 | ways. One example is to create an internal Linux distribution as a | ||
73 | code base the corporation can use across multiple product groups. | ||
74 | Through customization and layering, a project group can leverage the | ||
75 | base Linux distribution to create a distribution that works for their | ||
76 | product needs. | ||
77 | |||
78 | - *Ideal for Constrained Embedded and IoT devices:* Unlike a full Linux | ||
79 | distribution, you can use the Yocto Project to create exactly what | ||
80 | you need for embedded devices. You only add the feature support or | ||
81 | packages that you absolutely need for the device. For devices that | ||
82 | have display hardware, you can use available system components such | ||
83 | as X11, GTK+, Qt, Clutter, and SDL (among others) to create a rich | ||
84 | user experience. For devices that do not have a display or where you | ||
85 | want to use alternative UI frameworks, you can choose to not install | ||
86 | these components. | ||
87 | |||
88 | - *Comprehensive Toolchain Capabilities:* Toolchains for supported | ||
89 | architectures satisfy most use cases. However, if your hardware | ||
90 | supports features that are not part of a standard toolchain, you can | ||
91 | easily customize that toolchain through specification of | ||
92 | platform-specific tuning parameters. And, should you need to use a | ||
93 | third-party toolchain, mechanisms built into the Yocto Project allow | ||
94 | for that. | ||
95 | |||
96 | - *Mechanism Rules Over Policy:* Focusing on mechanism rather than | ||
97 | policy ensures that you are free to set policies based on the needs | ||
98 | of your design instead of adopting decisions enforced by some system | ||
99 | software provider. | ||
100 | |||
101 | - *Uses a Layer Model:* The Yocto Project `layer | ||
102 | infrastructure <#the-yocto-project-layer-model>`__ groups related | ||
103 | functionality into separate bundles. You can incrementally add these | ||
104 | grouped functionalities to your project as needed. Using layers to | ||
105 | isolate and group functionality reduces project complexity and | ||
106 | redundancy, allows you to easily extend the system, make | ||
107 | customizations, and keep functionality organized. | ||
108 | |||
109 | - *Supports Partial Builds:* You can build and rebuild individual | ||
110 | packages as needed. Yocto Project accomplishes this through its | ||
111 | `shared-state cache <#shared-state-cache>`__ (sstate) scheme. Being | ||
112 | able to build and debug components individually eases project | ||
113 | development. | ||
114 | |||
115 | - *Releases According to a Strict Schedule:* Major releases occur on a | ||
116 | :doc:`six-month cycle <../ref-manual/ref-release-process>` | ||
117 | predictably in October and April. The most recent two releases | ||
118 | support point releases to address common vulnerabilities and | ||
119 | exposures. This predictability is crucial for projects based on the | ||
120 | Yocto Project and allows development teams to plan activities. | ||
121 | |||
122 | - *Rich Ecosystem of Individuals and Organizations:* For open source | ||
123 | projects, the value of community is very important. Support forums, | ||
124 | expertise, and active developers who continue to push the Yocto | ||
125 | Project forward are readily available. | ||
126 | |||
127 | - *Binary Reproducibility:* The Yocto Project allows you to be very | ||
128 | specific about dependencies and achieves very high percentages of | ||
129 | binary reproducibility (e.g. 99.8% for ``core-image-minimal``). When | ||
130 | distributions are not specific about which packages are pulled in and | ||
131 | in what order to support dependencies, other build systems can | ||
132 | arbitrarily include packages. | ||
133 | |||
134 | - *License Manifest:* The Yocto Project provides a :ref:`license | ||
135 | manifest <dev-manual/dev-manual-common-tasks:maintaining open source license compliance during your product's lifecycle>` | ||
136 | for review by people who need to track the use of open source | ||
137 | licenses (e.g. legal teams). | ||
138 | |||
139 | .. _gs-challenges: | ||
140 | |||
141 | Challenges | ||
142 | ---------- | ||
143 | |||
144 | The following list presents challenges you might encounter when | ||
145 | developing using the Yocto Project: | ||
146 | |||
147 | - *Steep Learning Curve:* The Yocto Project has a steep learning curve | ||
148 | and has many different ways to accomplish similar tasks. It can be | ||
149 | difficult to choose how to proceed when varying methods exist by | ||
150 | which to accomplish a given task. | ||
151 | |||
152 | - *Understanding What Changes You Need to Make For Your Design Requires | ||
153 | Some Research:* Beyond the simple tutorial stage, understanding what | ||
154 | changes need to be made for your particular design can require a | ||
155 | significant amount of research and investigation. For information | ||
156 | that helps you transition from trying out the Yocto Project to using | ||
157 | it for your project, see the ":ref:`what-i-wish-id-known:what i wish i'd known about yocto project`" and | ||
158 | ":ref:`transitioning-to-a-custom-environment:transitioning to a custom environment for systems development`" | ||
159 | documents on the Yocto Project website. | ||
160 | |||
161 | - *Project Workflow Could Be Confusing:* The `Yocto Project | ||
162 | workflow <#overview-development-environment>`__ could be confusing if | ||
163 | you are used to traditional desktop and server software development. | ||
164 | In a desktop development environment, mechanisms exist to easily pull | ||
165 | and install new packages, which are typically pre-compiled binaries | ||
166 | from servers accessible over the Internet. Using the Yocto Project, | ||
167 | you must modify your configuration and rebuild to add additional | ||
168 | packages. | ||
169 | |||
170 | - *Working in a Cross-Build Environment Can Feel Unfamiliar:* When | ||
171 | developing code to run on a target, compilation, execution, and | ||
172 | testing done on the actual target can be faster than running a | ||
173 | BitBake build on a development host and then deploying binaries to | ||
174 | the target for test. While the Yocto Project does support development | ||
175 | tools on the target, the additional step of integrating your changes | ||
176 | back into the Yocto Project build environment would be required. | ||
177 | Yocto Project supports an intermediate approach that involves making | ||
178 | changes on the development system within the BitBake environment and | ||
179 | then deploying only the updated packages to the target. | ||
180 | |||
181 | The Yocto Project :term:`OpenEmbedded Build System` | ||
182 | produces packages | ||
183 | in standard formats (i.e. RPM, DEB, IPK, and TAR). You can deploy | ||
184 | these packages into the running system on the target by using | ||
185 | utilities on the target such as ``rpm`` or ``ipk``. | ||
186 | |||
187 | - *Initial Build Times Can be Significant:* Long initial build times | ||
188 | are unfortunately unavoidable due to the large number of packages | ||
189 | initially built from scratch for a fully functioning Linux system. | ||
190 | Once that initial build is completed, however, the shared-state | ||
191 | (sstate) cache mechanism Yocto Project uses keeps the system from | ||
192 | rebuilding packages that have not been "touched" since the last | ||
193 | build. The sstate mechanism significantly reduces times for | ||
194 | successive builds. | ||
195 | |||
196 | The Yocto Project Layer Model | ||
197 | ============================= | ||
198 | |||
199 | The Yocto Project's "Layer Model" is a development model for embedded | ||
200 | and IoT Linux creation that distinguishes the Yocto Project from other | ||
201 | simple build systems. The Layer Model simultaneously supports | ||
202 | collaboration and customization. Layers are repositories that contain | ||
203 | related sets of instructions that tell the :term:`OpenEmbedded Build System` | ||
204 | what to do. You can | ||
205 | collaborate, share, and reuse layers. | ||
206 | |||
207 | Layers can contain changes to previous instructions or settings at any | ||
208 | time. This powerful override capability is what allows you to customize | ||
209 | previously supplied collaborative or community layers to suit your | ||
210 | product requirements. | ||
211 | |||
212 | You use different layers to logically separate information in your | ||
213 | build. As an example, you could have BSP, GUI, distro configuration, | ||
214 | middleware, or application layers. Putting your entire build into one | ||
215 | layer limits and complicates future customization and reuse. Isolating | ||
216 | information into layers, on the other hand, helps simplify future | ||
217 | customizations and reuse. You might find it tempting to keep everything | ||
218 | in one layer when working on a single project. However, the more modular | ||
219 | your Metadata, the easier it is to cope with future changes. | ||
220 | |||
221 | .. note:: | ||
222 | |||
223 | - Use Board Support Package (BSP) layers from silicon vendors when | ||
224 | possible. | ||
225 | |||
226 | - Familiarize yourself with the `Yocto Project curated layer | ||
227 | index <https://www.yoctoproject.org/software-overview/layers/>`__ | ||
228 | or the `OpenEmbedded layer | ||
229 | index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__. | ||
230 | The latter contains more layers but they are less universally | ||
231 | validated. | ||
232 | |||
233 | - Layers support the inclusion of technologies, hardware components, | ||
234 | and software components. The :ref:`Yocto Project | ||
235 | Compatible <dev-manual/dev-manual-common-tasks:making sure your layer is compatible with yocto project>` | ||
236 | designation provides a minimum level of standardization that | ||
237 | contributes to a strong ecosystem. "YP Compatible" is applied to | ||
238 | appropriate products and software components such as BSPs, other | ||
239 | OE-compatible layers, and related open-source projects, allowing | ||
240 | the producer to use Yocto Project badges and branding assets. | ||
241 | |||
242 | To illustrate how layers are used to keep things modular, consider | ||
243 | machine customizations. These types of customizations typically reside | ||
244 | in a special layer, rather than a general layer, called a BSP Layer. | ||
245 | Furthermore, the machine customizations should be isolated from recipes | ||
246 | and Metadata that support a new GUI environment, for example. This | ||
247 | situation gives you a couple of layers: one for the machine | ||
248 | configurations, and one for the GUI environment. It is important to | ||
249 | understand, however, that the BSP layer can still make machine-specific | ||
250 | additions to recipes within the GUI environment layer without polluting | ||
251 | the GUI layer itself with those machine-specific changes. You can | ||
252 | accomplish this through a recipe that is a BitBake append | ||
253 | (``.bbappend``) file, which is described later in this section. | ||
254 | |||
255 | .. note:: | ||
256 | |||
257 | For general information on BSP layer structure, see the | ||
258 | :doc:`../bsp-guide/bsp-guide` | ||
259 | . | ||
260 | |||
261 | The :term:`Source Directory` | ||
262 | contains both general layers and BSP layers right out of the box. You | ||
263 | can easily identify layers that ship with a Yocto Project release in the | ||
264 | Source Directory by their names. Layers typically have names that begin | ||
265 | with the string ``meta-``. | ||
266 | |||
267 | .. note:: | ||
268 | |||
269 | It is not a requirement that a layer name begin with the prefix | ||
270 | meta- | ||
271 | , but it is a commonly accepted standard in the Yocto Project | ||
272 | community. | ||
273 | |||
274 | For example, if you were to examine the `tree | ||
275 | view <https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/>`__ of the | ||
276 | ``poky`` repository, you will see several layers: ``meta``, | ||
277 | ``meta-skeleton``, ``meta-selftest``, ``meta-poky``, and | ||
278 | ``meta-yocto-bsp``. Each of these repositories represents a distinct | ||
279 | layer. | ||
280 | |||
281 | For procedures on how to create layers, see the | ||
282 | ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" | ||
283 | section in the Yocto Project Development Tasks Manual. | ||
284 | |||
285 | Components and Tools | ||
286 | ==================== | ||
287 | |||
288 | The Yocto Project employs a collection of components and tools used by | ||
289 | the project itself, by project developers, and by those using the Yocto | ||
290 | Project. These components and tools are open source projects and | ||
291 | metadata that are separate from the reference distribution | ||
292 | (:term:`Poky`) and the | ||
293 | :term:`OpenEmbedded Build System`. Most of the | ||
294 | components and tools are downloaded separately. | ||
295 | |||
296 | This section provides brief overviews of the components and tools | ||
297 | associated with the Yocto Project. | ||
298 | |||
299 | .. _gs-development-tools: | ||
300 | |||
301 | Development Tools | ||
302 | ----------------- | ||
303 | |||
304 | The following list consists of tools that help you develop images and | ||
305 | applications using the Yocto Project: | ||
306 | |||
307 | - *CROPS:* `CROPS <https://github.com/crops/poky-container/>`__ is an | ||
308 | open source, cross-platform development framework that leverages | ||
309 | `Docker Containers <https://www.docker.com/>`__. CROPS provides an | ||
310 | easily managed, extensible environment that allows you to build | ||
311 | binaries for a variety of architectures on Windows, Linux and Mac OS | ||
312 | X hosts. | ||
313 | |||
314 | - *devtool:* This command-line tool is available as part of the | ||
315 | extensible SDK (eSDK) and is its cornerstone. You can use ``devtool`` | ||
316 | to help build, test, and package software within the eSDK. You can | ||
317 | use the tool to optionally integrate what you build into an image | ||
318 | built by the OpenEmbedded build system. | ||
319 | |||
320 | The ``devtool`` command employs a number of sub-commands that allow | ||
321 | you to add, modify, and upgrade recipes. As with the OpenEmbedded | ||
322 | build system, "recipes" represent software packages within | ||
323 | ``devtool``. When you use ``devtool add``, a recipe is automatically | ||
324 | created. When you use ``devtool modify``, the specified existing | ||
325 | recipe is used in order to determine where to get the source code and | ||
326 | how to patch it. In both cases, an environment is set up so that when | ||
327 | you build the recipe a source tree that is under your control is used | ||
328 | in order to allow you to make changes to the source as desired. By | ||
329 | default, both new recipes and the source go into a "workspace" | ||
330 | directory under the eSDK. The ``devtool upgrade`` command updates an | ||
331 | existing recipe so that you can build it for an updated set of source | ||
332 | files. | ||
333 | |||
334 | You can read about the ``devtool`` workflow in the Yocto Project | ||
335 | Application Development and Extensible Software Development Kit | ||
336 | (eSDK) Manual in the | ||
337 | ":ref:`sdk-manual/sdk-extensible:using \`\`devtool\`\` in your sdk workflow`" | ||
338 | section. | ||
339 | |||
340 | - *Extensible Software Development Kit (eSDK):* The eSDK provides a | ||
341 | cross-development toolchain and libraries tailored to the contents of | ||
342 | a specific image. The eSDK makes it easy to add new applications and | ||
343 | libraries to an image, modify the source for an existing component, | ||
344 | test changes on the target hardware, and integrate into the rest of | ||
345 | the OpenEmbedded build system. The eSDK gives you a toolchain | ||
346 | experience supplemented with the powerful set of ``devtool`` commands | ||
347 | tailored for the Yocto Project environment. | ||
348 | |||
349 | For information on the eSDK, see the :doc:`../sdk-manual/sdk-manual` Manual. | ||
350 | |||
351 | - *Toaster:* Toaster is a web interface to the Yocto Project | ||
352 | OpenEmbedded build system. Toaster allows you to configure, run, and | ||
353 | view information about builds. For information on Toaster, see the | ||
354 | :doc:`../toaster-manual/toaster-manual`. | ||
355 | |||
356 | .. _gs-production-tools: | ||
357 | |||
358 | Production Tools | ||
359 | ---------------- | ||
360 | |||
361 | The following list consists of tools that help production related | ||
362 | activities using the Yocto Project: | ||
363 | |||
364 | - *Auto Upgrade Helper:* This utility when used in conjunction with the | ||
365 | :term:`OpenEmbedded Build System` | ||
366 | (BitBake and | ||
367 | OE-Core) automatically generates upgrades for recipes that are based | ||
368 | on new versions of the recipes published upstream. See | ||
369 | :ref:`dev-manual/dev-manual-common-tasks:using the auto upgrade helper (auh)` | ||
370 | for how to set it up. | ||
371 | |||
372 | - *Recipe Reporting System:* The Recipe Reporting System tracks recipe | ||
373 | versions available for Yocto Project. The main purpose of the system | ||
374 | is to help you manage the recipes you maintain and to offer a dynamic | ||
375 | overview of the project. The Recipe Reporting System is built on top | ||
376 | of the `OpenEmbedded Layer | ||
377 | Index <http://layers.openembedded.org/layerindex/layers/>`__, which | ||
378 | is a website that indexes OpenEmbedded-Core layers. | ||
379 | |||
380 | - *Patchwork:* `Patchwork <http://jk.ozlabs.org/projects/patchwork/>`__ | ||
381 | is a fork of a project originally started by | ||
382 | `OzLabs <http://ozlabs.org/>`__. The project is a web-based tracking | ||
383 | system designed to streamline the process of bringing contributions | ||
384 | into a project. The Yocto Project uses Patchwork as an organizational | ||
385 | tool to handle patches, which number in the thousands for every | ||
386 | release. | ||
387 | |||
388 | - *AutoBuilder:* AutoBuilder is a project that automates build tests | ||
389 | and quality assurance (QA). By using the public AutoBuilder, anyone | ||
390 | can determine the status of the current "master" branch of Poky. | ||
391 | |||
392 | .. note:: | ||
393 | |||
394 | AutoBuilder is based on buildbot. | ||
395 | |||
396 | A goal of the Yocto Project is to lead the open source industry with | ||
397 | a project that automates testing and QA procedures. In doing so, the | ||
398 | project encourages a development community that publishes QA and test | ||
399 | plans, publicly demonstrates QA and test plans, and encourages | ||
400 | development of tools that automate and test and QA procedures for the | ||
401 | benefit of the development community. | ||
402 | |||
403 | You can learn more about the AutoBuilder used by the Yocto Project | ||
404 | Autobuilder :doc:`here <../test-manual/test-manual-understand-autobuilder>`. | ||
405 | |||
406 | - *Cross-Prelink:* Prelinking is the process of pre-computing the load | ||
407 | addresses and link tables generated by the dynamic linker as compared | ||
408 | to doing this at runtime. Doing this ahead of time results in | ||
409 | performance improvements when the application is launched and reduced | ||
410 | memory usage for libraries shared by many applications. | ||
411 | |||
412 | Historically, cross-prelink is a variant of prelink, which was | ||
413 | conceived by `Jakub | ||
414 | Jelínek <http://people.redhat.com/jakub/prelink.pdf>`__ a number of | ||
415 | years ago. Both prelink and cross-prelink are maintained in the same | ||
416 | repository albeit on separate branches. By providing an emulated | ||
417 | runtime dynamic linker (i.e. ``glibc``-derived ``ld.so`` emulation), | ||
418 | the cross-prelink project extends the prelink software's ability to | ||
419 | prelink a sysroot environment. Additionally, the cross-prelink | ||
420 | software enables the ability to work in sysroot style environments. | ||
421 | |||
422 | The dynamic linker determines standard load address calculations | ||
423 | based on a variety of factors such as mapping addresses, library | ||
424 | usage, and library function conflicts. The prelink tool uses this | ||
425 | information, from the dynamic linker, to determine unique load | ||
426 | addresses for executable and linkable format (ELF) binaries that are | ||
427 | shared libraries and dynamically linked. The prelink tool modifies | ||
428 | these ELF binaries with the pre-computed information. The result is | ||
429 | faster loading and often lower memory consumption because more of the | ||
430 | library code can be re-used from shared Copy-On-Write (COW) pages. | ||
431 | |||
432 | The original upstream prelink project only supports running prelink | ||
433 | on the end target device due to the reliance on the target device's | ||
434 | dynamic linker. This restriction causes issues when developing a | ||
435 | cross-compiled system. The cross-prelink adds a synthesized dynamic | ||
436 | loader that runs on the host, thus permitting cross-prelinking | ||
437 | without ever having to run on a read-write target filesystem. | ||
438 | |||
439 | - *Pseudo:* Pseudo is the Yocto Project implementation of | ||
440 | `fakeroot <http://man.he.net/man1/fakeroot>`__, which is used to run | ||
441 | commands in an environment that seemingly has root privileges. | ||
442 | |||
443 | During a build, it can be necessary to perform operations that | ||
444 | require system administrator privileges. For example, file ownership | ||
445 | or permissions might need definition. Pseudo is a tool that you can | ||
446 | either use directly or through the environment variable | ||
447 | ``LD_PRELOAD``. Either method allows these operations to succeed as | ||
448 | if system administrator privileges exist even when they do not. | ||
449 | |||
450 | You can read more about Pseudo in the "`Fakeroot and | ||
451 | Pseudo <#fakeroot-and-pseudo>`__" section. | ||
452 | |||
453 | .. _gs-openembedded-build-system: | ||
454 | |||
455 | Open-Embedded Build System Components | ||
456 | ------------------------------------- | ||
457 | |||
458 | The following list consists of components associated with the | ||
459 | :term:`OpenEmbedded Build System`: | ||
460 | |||
461 | - *BitBake:* BitBake is a core component of the Yocto Project and is | ||
462 | used by the OpenEmbedded build system to build images. While BitBake | ||
463 | is key to the build system, BitBake is maintained separately from the | ||
464 | Yocto Project. | ||
465 | |||
466 | BitBake is a generic task execution engine that allows shell and | ||
467 | Python tasks to be run efficiently and in parallel while working | ||
468 | within complex inter-task dependency constraints. In short, BitBake | ||
469 | is a build engine that works through recipes written in a specific | ||
470 | format in order to perform sets of tasks. | ||
471 | |||
472 | You can learn more about BitBake in the :doc:`BitBake User | ||
473 | Manual <bitbake:index>`. | ||
474 | |||
475 | - *OpenEmbedded-Core:* OpenEmbedded-Core (OE-Core) is a common layer of | ||
476 | metadata (i.e. recipes, classes, and associated files) used by | ||
477 | OpenEmbedded-derived systems, which includes the Yocto Project. The | ||
478 | Yocto Project and the OpenEmbedded Project both maintain the | ||
479 | OpenEmbedded-Core. You can find the OE-Core metadata in the Yocto | ||
480 | Project :yocto_git:`Source Repositories </cgit/cgit.cgi/poky/tree/meta>`. | ||
481 | |||
482 | Historically, the Yocto Project integrated the OE-Core metadata | ||
483 | throughout the Yocto Project source repository reference system | ||
484 | (Poky). After Yocto Project Version 1.0, the Yocto Project and | ||
485 | OpenEmbedded agreed to work together and share a common core set of | ||
486 | metadata (OE-Core), which contained much of the functionality | ||
487 | previously found in Poky. This collaboration achieved a long-standing | ||
488 | OpenEmbedded objective for having a more tightly controlled and | ||
489 | quality-assured core. The results also fit well with the Yocto | ||
490 | Project objective of achieving a smaller number of fully featured | ||
491 | tools as compared to many different ones. | ||
492 | |||
493 | Sharing a core set of metadata results in Poky as an integration | ||
494 | layer on top of OE-Core. You can see that in this | ||
495 | `figure <#yp-key-dev-elements>`__. The Yocto Project combines various | ||
496 | components such as BitBake, OE-Core, script "glue", and documentation | ||
497 | for its build system. | ||
498 | |||
499 | .. _gs-reference-distribution-poky: | ||
500 | |||
501 | Reference Distribution (Poky) | ||
502 | ----------------------------- | ||
503 | |||
504 | Poky is the Yocto Project reference distribution. It contains the | ||
505 | :term:`OpenEmbedded Build System` | ||
506 | (BitBake and OE-Core) as well as a set of metadata to get you started | ||
507 | building your own distribution. See the | ||
508 | `figure <#what-is-the-yocto-project>`__ in "What is the Yocto Project?" | ||
509 | section for an illustration that shows Poky and its relationship with | ||
510 | other parts of the Yocto Project. | ||
511 | |||
512 | To use the Yocto Project tools and components, you can download | ||
513 | (``clone``) Poky and use it to bootstrap your own distribution. | ||
514 | |||
515 | .. note:: | ||
516 | |||
517 | Poky does not contain binary files. It is a working example of how to | ||
518 | build your own custom Linux distribution from source. | ||
519 | |||
520 | You can read more about Poky in the "`Reference Embedded Distribution | ||
521 | (Poky) <#reference-embedded-distribution>`__" section. | ||
522 | |||
523 | .. _gs-packages-for-finished-targets: | ||
524 | |||
525 | Packages for Finished Targets | ||
526 | ----------------------------- | ||
527 | |||
528 | The following lists components associated with packages for finished | ||
529 | targets: | ||
530 | |||
531 | - *Matchbox:* Matchbox is an Open Source, base environment for the X | ||
532 | Window System running on non-desktop, embedded platforms such as | ||
533 | handhelds, set-top boxes, kiosks, and anything else for which screen | ||
534 | space, input mechanisms, or system resources are limited. | ||
535 | |||
536 | Matchbox consists of a number of interchangeable and optional | ||
537 | applications that you can tailor to a specific, non-desktop platform | ||
538 | to enhance usability in constrained environments. | ||
539 | |||
540 | You can find the Matchbox source in the Yocto Project | ||
541 | :yocto_git:`Source Repositories <>`. | ||
542 | |||
543 | - *Opkg:* Open PacKaGe management (opkg) is a lightweight package | ||
544 | management system based on the itsy package (ipkg) management system. | ||
545 | Opkg is written in C and resembles Advanced Package Tool (APT) and | ||
546 | Debian Package (dpkg) in operation. | ||
547 | |||
548 | Opkg is intended for use on embedded Linux devices and is used in | ||
549 | this capacity in the | ||
550 | `OpenEmbedded <http://www.openembedded.org/wiki/Main_Page>`__ and | ||
551 | `OpenWrt <https://openwrt.org/>`__ projects, as well as the Yocto | ||
552 | Project. | ||
553 | |||
554 | .. note:: | ||
555 | |||
556 | As best it can, opkg maintains backwards compatibility with ipkg | ||
557 | and conforms to a subset of Debian's policy manual regarding | ||
558 | control files. | ||
559 | |||
560 | You can find the opkg source in the Yocto Project | ||
561 | :yocto_git:`Source Repositories <>`. | ||
562 | |||
563 | .. _gs-archived-components: | ||
564 | |||
565 | Archived Components | ||
566 | ------------------- | ||
567 | |||
568 | The Build Appliance is a virtual machine image that enables you to build | ||
569 | and boot a custom embedded Linux image with the Yocto Project using a | ||
570 | non-Linux development system. | ||
571 | |||
572 | Historically, the Build Appliance was the second of three methods by | ||
573 | which you could use the Yocto Project on a system that was not native to | ||
574 | Linux. | ||
575 | |||
576 | 1. *Hob:* Hob, which is now deprecated and is no longer available since | ||
577 | the 2.1 release of the Yocto Project provided a rudimentary, | ||
578 | GUI-based interface to the Yocto Project. Toaster has fully replaced | ||
579 | Hob. | ||
580 | |||
581 | 2. *Build Appliance:* Post Hob, the Build Appliance became available. It | ||
582 | was never recommended that you use the Build Appliance as a | ||
583 | day-to-day production development environment with the Yocto Project. | ||
584 | Build Appliance was useful as a way to try out development in the | ||
585 | Yocto Project environment. | ||
586 | |||
587 | 3. *CROPS:* The final and best solution available now for developing | ||
588 | using the Yocto Project on a system not native to Linux is with | ||
589 | `CROPS <#gs-crops-overview>`__. | ||
590 | |||
591 | .. _gs-development-methods: | ||
592 | |||
593 | Development Methods | ||
594 | =================== | ||
595 | |||
596 | The Yocto Project development environment usually involves a | ||
597 | :term:`Build Host` and target | ||
598 | hardware. You use the Build Host to build images and develop | ||
599 | applications, while you use the target hardware to test deployed | ||
600 | software. | ||
601 | |||
602 | This section provides an introduction to the choices or development | ||
603 | methods you have when setting up your Build Host. Depending on the your | ||
604 | particular workflow preference and the type of operating system your | ||
605 | Build Host runs, several choices exist that allow you to use the Yocto | ||
606 | Project. | ||
607 | |||
608 | .. note:: | ||
609 | |||
610 | For additional detail about the Yocto Project development | ||
611 | environment, see the ":doc:`overview-manual-development-environment`" | ||
612 | chapter. | ||
613 | |||
614 | - *Native Linux Host:* By far the best option for a Build Host. A | ||
615 | system running Linux as its native operating system allows you to | ||
616 | develop software by directly using the | ||
617 | :term:`BitBake` tool. You can | ||
618 | accomplish all aspects of development from a familiar shell of a | ||
619 | supported Linux distribution. | ||
620 | |||
621 | For information on how to set up a Build Host on a system running | ||
622 | Linux as its native operating system, see the | ||
623 | ":ref:`dev-manual/dev-manual-start:setting up a native linux host`" | ||
624 | section in the Yocto Project Development Tasks Manual. | ||
625 | |||
626 | - *CROss PlatformS (CROPS):* Typically, you use | ||
627 | `CROPS <https://github.com/crops/poky-container/>`__, which leverages | ||
628 | `Docker Containers <https://www.docker.com/>`__, to set up a Build | ||
629 | Host that is not running Linux (e.g. Microsoft Windows or macOS). | ||
630 | |||
631 | .. note:: | ||
632 | |||
633 | You can, however, use CROPS on a Linux-based system. | ||
634 | |||
635 | CROPS is an open source, cross-platform development framework that | ||
636 | provides an easily managed, extensible environment for building | ||
637 | binaries targeted for a variety of architectures on Windows, macOS, | ||
638 | or Linux hosts. Once the Build Host is set up using CROPS, you can | ||
639 | prepare a shell environment to mimic that of a shell being used on a | ||
640 | system natively running Linux. | ||
641 | |||
642 | For information on how to set up a Build Host with CROPS, see the | ||
643 | ":ref:`dev-manual/dev-manual-start:setting up to use cross platforms (crops)`" | ||
644 | section in the Yocto Project Development Tasks Manual. | ||
645 | |||
646 | - *Windows Subsystem For Linux (WSLv2):* You may use Windows Subsystem | ||
647 | For Linux v2 to set up a build host using Windows 10. | ||
648 | |||
649 | .. note:: | ||
650 | |||
651 | The Yocto Project is not compatible with WSLv1, it is compatible | ||
652 | but not officially supported nor validated with WSLv2, if you | ||
653 | still decide to use WSL please upgrade to WSLv2. | ||
654 | |||
655 | The Windows Subsystem For Linux allows Windows 10 to run a real Linux | ||
656 | kernel inside of a lightweight utility virtual machine (VM) using | ||
657 | virtualization technology. | ||
658 | |||
659 | For information on how to set up a Build Host with WSLv2, see the | ||
660 | ":ref:`dev-manual/dev-manual-start:setting up to use windows subsystem for linux (wslv2)`" | ||
661 | section in the Yocto Project Development Tasks Manual. | ||
662 | |||
663 | - *Toaster:* Regardless of what your Build Host is running, you can use | ||
664 | Toaster to develop software using the Yocto Project. Toaster is a web | ||
665 | interface to the Yocto Project's :term:`OpenEmbedded Build System`. | ||
666 | The interface | ||
667 | enables you to configure and run your builds. Information about | ||
668 | builds is collected and stored in a database. You can use Toaster to | ||
669 | configure and start builds on multiple remote build servers. | ||
670 | |||
671 | For information about and how to use Toaster, see the | ||
672 | :doc:`../toaster-manual/toaster-manual`. | ||
673 | |||
674 | .. _reference-embedded-distribution: | ||
675 | |||
676 | Reference Embedded Distribution (Poky) | ||
677 | ====================================== | ||
678 | |||
679 | "Poky", which is pronounced *Pock*-ee, is the name of the Yocto | ||
680 | Project's reference distribution or Reference OS Kit. Poky contains the | ||
681 | :term:`OpenEmbedded Build System` | ||
682 | (:term:`BitBake` and | ||
683 | :term:`OpenEmbedded-Core (OE-Core)`) as well as a set | ||
684 | of :term:`Metadata` to get you started | ||
685 | building your own distro. In other words, Poky is a base specification | ||
686 | of the functionality needed for a typical embedded system as well as the | ||
687 | components from the Yocto Project that allow you to build a distribution | ||
688 | into a usable binary image. | ||
689 | |||
690 | Poky is a combined repository of BitBake, OpenEmbedded-Core (which is | ||
691 | found in ``meta``), ``meta-poky``, ``meta-yocto-bsp``, and documentation | ||
692 | provided all together and known to work well together. You can view | ||
693 | these items that make up the Poky repository in the | ||
694 | :yocto_git:`Source Repositories </cgit/cgit.cgi/poky/tree/>`. | ||
695 | |||
696 | .. note:: | ||
697 | |||
698 | If you are interested in all the contents of the | ||
699 | poky | ||
700 | Git repository, see the ":ref:`ref-manual/ref-structure:top-level core components`" | ||
701 | section in the Yocto Project Reference Manual. | ||
702 | |||
703 | The following figure illustrates what generally comprises Poky: | ||
704 | |||
705 | .. image:: figures/poky-reference-distribution.png | ||
706 | :align: center | ||
707 | |||
708 | - BitBake is a task executor and scheduler that is the heart of the | ||
709 | OpenEmbedded build system. | ||
710 | |||
711 | - ``meta-poky``, which is Poky-specific metadata. | ||
712 | |||
713 | - ``meta-yocto-bsp``, which are Yocto Project-specific Board Support | ||
714 | Packages (BSPs). | ||
715 | |||
716 | - OpenEmbedded-Core (OE-Core) metadata, which includes shared | ||
717 | configurations, global variable definitions, shared classes, | ||
718 | packaging, and recipes. Classes define the encapsulation and | ||
719 | inheritance of build logic. Recipes are the logical units of software | ||
720 | and images to be built. | ||
721 | |||
722 | - Documentation, which contains the Yocto Project source files used to | ||
723 | make the set of user manuals. | ||
724 | |||
725 | .. note:: | ||
726 | |||
727 | While Poky is a "complete" distribution specification and is tested | ||
728 | and put through QA, you cannot use it as a product "out of the box" | ||
729 | in its current form. | ||
730 | |||
731 | To use the Yocto Project tools, you can use Git to clone (download) the | ||
732 | Poky repository then use your local copy of the reference distribution | ||
733 | to bootstrap your own distribution. | ||
734 | |||
735 | .. note:: | ||
736 | |||
737 | Poky does not contain binary files. It is a working example of how to | ||
738 | build your own custom Linux distribution from source. | ||
739 | |||
740 | Poky has a regular, well established, six-month release cycle under its | ||
741 | own version. Major releases occur at the same time major releases (point | ||
742 | releases) occur for the Yocto Project, which are typically in the Spring | ||
743 | and Fall. For more information on the Yocto Project release schedule and | ||
744 | cadence, see the ":doc:`../ref-manual/ref-release-process`" chapter in the | ||
745 | Yocto Project Reference Manual. | ||
746 | |||
747 | Much has been said about Poky being a "default configuration". A default | ||
748 | configuration provides a starting image footprint. You can use Poky out | ||
749 | of the box to create an image ranging from a shell-accessible minimal | ||
750 | image all the way up to a Linux Standard Base-compliant image that uses | ||
751 | a GNOME Mobile and Embedded (GMAE) based reference user interface called | ||
752 | Sato. | ||
753 | |||
754 | One of the most powerful properties of Poky is that every aspect of a | ||
755 | build is controlled by the metadata. You can use metadata to augment | ||
756 | these base image types by adding metadata | ||
757 | `layers <#the-yocto-project-layer-model>`__ that extend functionality. | ||
758 | These layers can provide, for example, an additional software stack for | ||
759 | an image type, add a board support package (BSP) for additional | ||
760 | hardware, or even create a new image type. | ||
761 | |||
762 | Metadata is loosely grouped into configuration files or package recipes. | ||
763 | A recipe is a collection of non-executable metadata used by BitBake to | ||
764 | set variables or define additional build-time tasks. A recipe contains | ||
765 | fields such as the recipe description, the recipe version, the license | ||
766 | of the package and the upstream source repository. A recipe might also | ||
767 | indicate that the build process uses autotools, make, distutils or any | ||
768 | other build process, in which case the basic functionality can be | ||
769 | defined by the classes it inherits from the OE-Core layer's class | ||
770 | definitions in ``./meta/classes``. Within a recipe you can also define | ||
771 | additional tasks as well as task prerequisites. Recipe syntax through | ||
772 | BitBake also supports both ``_prepend`` and ``_append`` operators as a | ||
773 | method of extending task functionality. These operators inject code into | ||
774 | the beginning or end of a task. For information on these BitBake | ||
775 | operators, see the | ||
776 | ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:appending and prepending (override style syntax)`" | ||
777 | section in the BitBake User's Manual. | ||
778 | |||
779 | .. _openembedded-build-system-workflow: | ||
780 | |||
781 | The OpenEmbedded Build System Workflow | ||
782 | ====================================== | ||
783 | |||
784 | The :term:`OpenEmbedded Build System` uses a "workflow" to | ||
785 | accomplish image and SDK generation. The following figure overviews that | ||
786 | workflow: | ||
787 | |||
788 | .. image:: figures/YP-flow-diagram.png | ||
789 | :align: center | ||
790 | |||
791 | Following is a brief summary of the "workflow": | ||
792 | |||
793 | 1. Developers specify architecture, policies, patches and configuration | ||
794 | details. | ||
795 | |||
796 | 2. The build system fetches and downloads the source code from the | ||
797 | specified location. The build system supports standard methods such | ||
798 | as tarballs or source code repositories systems such as Git. | ||
799 | |||
800 | 3. Once source code is downloaded, the build system extracts the sources | ||
801 | into a local work area where patches are applied and common steps for | ||
802 | configuring and compiling the software are run. | ||
803 | |||
804 | 4. The build system then installs the software into a temporary staging | ||
805 | area where the binary package format you select (DEB, RPM, or IPK) is | ||
806 | used to roll up the software. | ||
807 | |||
808 | 5. Different QA and sanity checks run throughout entire build process. | ||
809 | |||
810 | 6. After the binaries are created, the build system generates a binary | ||
811 | package feed that is used to create the final root file image. | ||
812 | |||
813 | 7. The build system generates the file system image and a customized | ||
814 | Extensible SDK (eSDK) for application development in parallel. | ||
815 | |||
816 | For a very detailed look at this workflow, see the "`OpenEmbedded Build | ||
817 | System Concepts <#openembedded-build-system-build-concepts>`__" section. | ||
818 | |||
819 | Some Basic Terms | ||
820 | ================ | ||
821 | |||
822 | It helps to understand some basic fundamental terms when learning the | ||
823 | Yocto Project. Although a list of terms exists in the ":doc:`Yocto Project | ||
824 | Terms <../ref-manual/ref-terms>`" section of the Yocto Project | ||
825 | Reference Manual, this section provides the definitions of some terms | ||
826 | helpful for getting started: | ||
827 | |||
828 | - *Configuration Files:* Files that hold global definitions of | ||
829 | variables, user-defined variables, and hardware configuration | ||
830 | information. These files tell the :term:`OpenEmbedded Build System` | ||
831 | what to build and | ||
832 | what to put into the image to support a particular platform. | ||
833 | |||
834 | - *Extensible Software Development Kit (eSDK):* A custom SDK for | ||
835 | application developers. This eSDK allows developers to incorporate | ||
836 | their library and programming changes back into the image to make | ||
837 | their code available to other application developers. For information | ||
838 | on the eSDK, see the :doc:`../sdk-manual/sdk-manual` manual. | ||
839 | |||
840 | - *Layer:* A collection of related recipes. Layers allow you to | ||
841 | consolidate related metadata to customize your build. Layers also | ||
842 | isolate information used when building for multiple architectures. | ||
843 | Layers are hierarchical in their ability to override previous | ||
844 | specifications. You can include any number of available layers from | ||
845 | the Yocto Project and customize the build by adding your layers after | ||
846 | them. You can search the Layer Index for layers used within Yocto | ||
847 | Project. | ||
848 | |||
849 | For more detailed information on layers, see the | ||
850 | ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" | ||
851 | section in the Yocto Project Development Tasks Manual. For a | ||
852 | discussion specifically on BSP Layers, see the | ||
853 | ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto | ||
854 | Project Board Support Packages (BSP) Developer's Guide. | ||
855 | |||
856 | - *Metadata:* A key element of the Yocto Project is the Metadata that | ||
857 | is used to construct a Linux distribution and is contained in the | ||
858 | files that the OpenEmbedded build system parses when building an | ||
859 | image. In general, Metadata includes recipes, configuration files, | ||
860 | and other information that refers to the build instructions | ||
861 | themselves, as well as the data used to control what things get built | ||
862 | and the effects of the build. Metadata also includes commands and | ||
863 | data used to indicate what versions of software are used, from where | ||
864 | they are obtained, and changes or additions to the software itself | ||
865 | (patches or auxiliary files) that are used to fix bugs or customize | ||
866 | the software for use in a particular situation. OpenEmbedded-Core is | ||
867 | an important set of validated metadata. | ||
868 | |||
869 | - *OpenEmbedded Build System:* The terms "BitBake" and "build system" | ||
870 | are sometimes used for the OpenEmbedded Build System. | ||
871 | |||
872 | BitBake is a task scheduler and execution engine that parses | ||
873 | instructions (i.e. recipes) and configuration data. After a parsing | ||
874 | phase, BitBake creates a dependency tree to order the compilation, | ||
875 | schedules the compilation of the included code, and finally executes | ||
876 | the building of the specified custom Linux image (distribution). | ||
877 | BitBake is similar to the ``make`` tool. | ||
878 | |||
879 | During a build process, the build system tracks dependencies and | ||
880 | performs a native or cross-compilation of the package. As a first | ||
881 | step in a cross-build setup, the framework attempts to create a | ||
882 | cross-compiler toolchain (i.e. Extensible SDK) suited for the target | ||
883 | platform. | ||
884 | |||
885 | - *OpenEmbedded-Core (OE-Core):* OE-Core is metadata comprised of | ||
886 | foundation recipes, classes, and associated files that are meant to | ||
887 | be common among many different OpenEmbedded-derived systems, | ||
888 | including the Yocto Project. OE-Core is a curated subset of an | ||
889 | original repository developed by the OpenEmbedded community that has | ||
890 | been pared down into a smaller, core set of continuously validated | ||
891 | recipes. The result is a tightly controlled and quality-assured core | ||
892 | set of recipes. | ||
893 | |||
894 | You can see the Metadata in the ``meta`` directory of the Yocto | ||
895 | Project `Source | ||
896 | Repositories <http://git.yoctoproject.org/cgit/cgit.cgi>`__. | ||
897 | |||
898 | - *Packages:* In the context of the Yocto Project, this term refers to | ||
899 | a recipe's packaged output produced by BitBake (i.e. a "baked | ||
900 | recipe"). A package is generally the compiled binaries produced from | ||
901 | the recipe's sources. You "bake" something by running it through | ||
902 | BitBake. | ||
903 | |||
904 | It is worth noting that the term "package" can, in general, have | ||
905 | subtle meanings. For example, the packages referred to in the | ||
906 | ":ref:`ref-manual/ref-system-requirements:required packages for the build host`" | ||
907 | section in the Yocto Project Reference Manual are compiled binaries | ||
908 | that, when installed, add functionality to your Linux distribution. | ||
909 | |||
910 | Another point worth noting is that historically within the Yocto | ||
911 | Project, recipes were referred to as packages - thus, the existence | ||
912 | of several BitBake variables that are seemingly mis-named, (e.g. | ||
913 | :term:`PR`, | ||
914 | :term:`PV`, and | ||
915 | :term:`PE`). | ||
916 | |||
917 | - *Poky:* Poky is a reference embedded distribution and a reference | ||
918 | test configuration. Poky provides the following: | ||
919 | |||
920 | - A base-level functional distro used to illustrate how to customize | ||
921 | a distribution. | ||
922 | |||
923 | - A means by which to test the Yocto Project components (i.e. Poky | ||
924 | is used to validate the Yocto Project). | ||
925 | |||
926 | - A vehicle through which you can download the Yocto Project. | ||
927 | |||
928 | Poky is not a product level distro. Rather, it is a good starting | ||
929 | point for customization. | ||
930 | |||
931 | .. note:: | ||
932 | |||
933 | Poky is an integration layer on top of OE-Core. | ||
934 | |||
935 | - *Recipe:* The most common form of metadata. A recipe contains a list | ||
936 | of settings and tasks (i.e. instructions) for building packages that | ||
937 | are then used to build the binary image. A recipe describes where you | ||
938 | get source code and which patches to apply. Recipes describe | ||
939 | dependencies for libraries or for other recipes as well as | ||
940 | configuration and compilation options. Related recipes are | ||
941 | consolidated into a layer. | ||
diff --git a/documentation/overview-manual/overview-manual-yp-intro.xml b/documentation/overview-manual/overview-manual-yp-intro.xml index 1b60a30302..a2a1f494bb 100644 --- a/documentation/overview-manual/overview-manual-yp-intro.xml +++ b/documentation/overview-manual/overview-manual-yp-intro.xml | |||
@@ -1,6 +1,7 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
4 | 5 | ||
5 | <chapter id='overview-yp'> | 6 | <chapter id='overview-yp'> |
6 | <title>Introducing the Yocto Project</title> | 7 | <title>Introducing the Yocto Project</title> |
@@ -458,7 +459,7 @@ | |||
458 | <para>The <filename>devtool</filename> command employs | 459 | <para>The <filename>devtool</filename> command employs |
459 | a number of sub-commands that allow you to add, modify, | 460 | a number of sub-commands that allow you to add, modify, |
460 | and upgrade recipes. | 461 | and upgrade recipes. |
461 | As with the OpenEmbedded build system, “recipes” | 462 | As with the OpenEmbedded build system, "recipes" |
462 | represent software packages within | 463 | represent software packages within |
463 | <filename>devtool</filename>. | 464 | <filename>devtool</filename>. |
464 | When you use <filename>devtool add</filename>, a recipe | 465 | When you use <filename>devtool add</filename>, a recipe |
@@ -471,7 +472,7 @@ | |||
471 | control is used in order to allow you to make changes | 472 | control is used in order to allow you to make changes |
472 | to the source as desired. | 473 | to the source as desired. |
473 | By default, both new recipes and the source go into | 474 | By default, both new recipes and the source go into |
474 | a “workspace” directory under the eSDK. | 475 | a "workspace" directory under the eSDK. |
475 | The <filename>devtool upgrade</filename> command | 476 | The <filename>devtool upgrade</filename> command |
476 | updates an existing recipe so that you can build it | 477 | updates an existing recipe so that you can build it |
477 | for an updated set of source files.</para> | 478 | for an updated set of source files.</para> |
@@ -597,7 +598,7 @@ | |||
597 | By providing an emulated runtime dynamic linker | 598 | By providing an emulated runtime dynamic linker |
598 | (i.e. <filename>glibc</filename>-derived | 599 | (i.e. <filename>glibc</filename>-derived |
599 | <filename>ld.so</filename> emulation), the | 600 | <filename>ld.so</filename> emulation), the |
600 | cross-prelink project extends the prelink software’s | 601 | cross-prelink project extends the prelink software's |
601 | ability to prelink a sysroot environment. | 602 | ability to prelink a sysroot environment. |
602 | Additionally, the cross-prelink software enables the | 603 | Additionally, the cross-prelink software enables the |
603 | ability to work in sysroot style environments.</para> | 604 | ability to work in sysroot style environments.</para> |
@@ -619,7 +620,7 @@ | |||
619 | 620 | ||
620 | <para>The original upstream prelink project only | 621 | <para>The original upstream prelink project only |
621 | supports running prelink on the end target device | 622 | supports running prelink on the end target device |
622 | due to the reliance on the target device’s dynamic | 623 | due to the reliance on the target device's dynamic |
623 | linker. | 624 | linker. |
624 | This restriction causes issues when developing a | 625 | This restriction causes issues when developing a |
625 | cross-compiled system. | 626 | cross-compiled system. |
@@ -712,7 +713,7 @@ | |||
712 | You can see that in this | 713 | You can see that in this |
713 | <link linkend='yp-key-dev-elements'>figure</link>. | 714 | <link linkend='yp-key-dev-elements'>figure</link>. |
714 | The Yocto Project combines various components such as | 715 | The Yocto Project combines various components such as |
715 | BitBake, OE-Core, script “glue”, and documentation | 716 | BitBake, OE-Core, script "glue", and documentation |
716 | for its build system. | 717 | for its build system. |
717 | </para></listitem> | 718 | </para></listitem> |
718 | </itemizedlist> | 719 | </itemizedlist> |
@@ -790,7 +791,7 @@ | |||
790 | <note> | 791 | <note> |
791 | As best it can, opkg maintains backwards | 792 | As best it can, opkg maintains backwards |
792 | compatibility with ipkg and conforms to a subset | 793 | compatibility with ipkg and conforms to a subset |
793 | of Debian’s policy manual regarding control files. | 794 | of Debian's policy manual regarding control files. |
794 | </note> | 795 | </note> |
795 | </para></listitem> | 796 | </para></listitem> |
796 | </itemizedlist> | 797 | </itemizedlist> |
diff --git a/documentation/overview-manual/overview-manual.rst b/documentation/overview-manual/overview-manual.rst new file mode 100644 index 0000000000..80ce9aae76 --- /dev/null +++ b/documentation/overview-manual/overview-manual.rst | |||
@@ -0,0 +1,19 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ========================================== | ||
4 | Yocto Project Overview and Concepts Manual | ||
5 | ========================================== | ||
6 | |||
7 | | | ||
8 | |||
9 | .. toctree:: | ||
10 | :caption: Table of Contents | ||
11 | :numbered: | ||
12 | |||
13 | overview-manual-intro | ||
14 | overview-manual-yp-intro | ||
15 | overview-manual-development-environment | ||
16 | overview-manual-concepts | ||
17 | history | ||
18 | |||
19 | .. include:: /boilerplate.rst | ||
diff --git a/documentation/overview-manual/overview-manual.xml b/documentation/overview-manual/overview-manual.xml index 7c75e5086c..8021a2e95e 100755 --- a/documentation/overview-manual/overview-manual.xml +++ b/documentation/overview-manual/overview-manual.xml | |||
@@ -1,6 +1,7 @@ | |||
1 | <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | 1 | <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
4 | 5 | ||
5 | <book id='overview-manual' lang='en' | 6 | <book id='overview-manual' lang='en' |
6 | xmlns:xi="http://www.w3.org/2003/XInclude" | 7 | xmlns:xi="http://www.w3.org/2003/XInclude" |
@@ -52,28 +53,8 @@ | |||
52 | </revision> | 53 | </revision> |
53 | <revision> | 54 | <revision> |
54 | <revnumber>3.1</revnumber> | 55 | <revnumber>3.1</revnumber> |
55 | <date>April 2020</date> | ||
56 | <revremark>Released with the Yocto Project 3.1 Release.</revremark> | ||
57 | </revision> | ||
58 | <revision> | ||
59 | <revnumber>3.1.1</revnumber> | ||
60 | <date>June 2020</date> | ||
61 | <revremark>Released with the Yocto Project 3.1.1 Release.</revremark> | ||
62 | </revision> | ||
63 | <revision> | ||
64 | <revnumber>3.1.2</revnumber> | ||
65 | <date>August 2020</date> | ||
66 | <revremark>Released with the Yocto Project 3.1.2 Release.</revremark> | ||
67 | </revision> | ||
68 | <revision> | ||
69 | <revnumber>3.1.3</revnumber> | ||
70 | <date>October 2020</date> | ||
71 | <revremark>Released with the Yocto Project 3.1.3 Release.</revremark> | ||
72 | </revision> | ||
73 | <revision> | ||
74 | <revnumber>3.1.4</revnumber> | ||
75 | <date>&REL_MONTH_YEAR;</date> | 56 | <date>&REL_MONTH_YEAR;</date> |
76 | <revremark>Released with the Yocto Project 3.1.4 Release.</revremark> | 57 | <revremark>Released with the Yocto Project 3.1 Release.</revremark> |
77 | </revision> | 58 | </revision> |
78 | </revhistory> | 59 | </revhistory> |
79 | 60 | ||