summaryrefslogtreecommitdiffstats
path: root/documentation/overview-manual
diff options
context:
space:
mode:
authorNicolas Dechesne <nicolas.dechesne@linaro.org>2020-06-26 19:10:51 +0200
committerRichard Purdie <richard.purdie@linuxfoundation.org>2020-09-17 10:09:33 +0100
commit9bd69b1f1d71a9692189beeac75af9dfbad816cc (patch)
tree305347fca899074aed5610e0e82eaec180bf630c /documentation/overview-manual
parentc40a8d5904c29046f1cbbeb998e6cd7c24f9b206 (diff)
downloadpoky-9bd69b1f1d71a9692189beeac75af9dfbad816cc.tar.gz
sphinx: initial sphinx support
This commit is autogenerated pandoc to generate an inital set of reST files based on DocBook XML files. A .rst file is generated for each .xml files in all manuals with this command: cd <manual> for i in *.xml; do \ pandoc -f docbook -t rst --shift-heading-level-by=-1 \ $i -o $(basename $i .xml).rst \ done The conversion was done with: pandoc 2.9.2.1-91 (Arch Linux). Also created an initial top level index file for each document, and added all 'books' to the top leve index.rst file. The YP manuals layout is organized as: Book Chapter Section Section Section Sphinx uses section headers to create the document structure. ReStructuredText defines sections headers like that: To break longer text up into sections, you use section headers. These are a single line of text (one or more words) with adornment: an underline alone, or an underline and an overline together, in dashes "-----", equals "======", tildes "~~~~~~" or any of the non-alphanumeric characters = - ` : ' " ~ ^ _ * + # < > that you feel comfortable with. An underline-only adornment is distinct from an overline-and-underline adornment using the same character. The underline/overline must be at least as long as the title text. Be consistent, since all sections marked with the same adornment style are deemed to be at the same level: Let's define the following convention when converting from Docbook: Book => overline === (Title) Chapter => overline *** (1.) Section => ==== (1.1) Section => ---- (1.1.1) Section => ~~~~ (1.1.1.1) Section => ^^^^ (1.1.1.1.1) During the conversion with pandoc, we used --shift-heading-level=-1 to convert most of DocBook headings automatically. However with this setting, the Chapter header was removed, so I added it back manually. Without this setting all headings were off by one, which was more difficult to manually fix. At least with this change, we now have the same TOC with Sphinx and DocBook. (From yocto-docs rev: 3c73d64a476d4423ee4c6808c685fa94d88d7df8) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/overview-manual')
-rw-r--r--documentation/overview-manual/overview-manual-concepts.rst2133
-rw-r--r--documentation/overview-manual/overview-manual-development-environment.rst656
-rw-r--r--documentation/overview-manual/overview-manual-intro.rst74
-rw-r--r--documentation/overview-manual/overview-manual-yp-intro.rst941
-rw-r--r--documentation/overview-manual/overview-manual.rst12
5 files changed, 3816 insertions, 0 deletions
diff --git a/documentation/overview-manual/overview-manual-concepts.rst b/documentation/overview-manual/overview-manual-concepts.rst
new file mode 100644
index 0000000000..b37adbbba6
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-concepts.rst
@@ -0,0 +1,2133 @@
1**********************
2Yocto Project Concepts
3**********************
4
5This chapter provides explanations for Yocto Project concepts that go
6beyond the surface of "how-to" information and reference (or look-up)
7material. Concepts such as components, the `OpenEmbedded build
8system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ workflow,
9cross-development toolchains, shared state cache, and so forth are
10explained.
11
12Yocto Project Components
13========================
14
15The `BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__ task executor
16together with various types of configuration files form the
17`OpenEmbedded-Core <&YOCTO_DOCS_REF_URL;#oe-core>`__. This section
18overviews these components by describing their use and how they
19interact.
20
21BitBake handles the parsing and execution of the data files. The data
22itself is of various types:
23
24- *Recipes:* Provides details about particular pieces of software.
25
26- *Class Data:* Abstracts common build information (e.g. how to build a
27 Linux kernel).
28
29- *Configuration Data:* Defines machine-specific settings, policy
30 decisions, and so forth. Configuration data acts as the glue to bind
31 everything together.
32
33BitBake knows how to combine multiple data sources together and refers
34to each data source as a layer. For information on layers, see the
35"`Understanding and Creating
36Layers <&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers>`__"
37section of the Yocto Project Development Tasks Manual.
38
39Following are some brief details on these core components. For
40additional information on how these components interact during a build,
41see the "`OpenEmbedded Build System
42Concepts <#openembedded-build-system-build-concepts>`__" section.
43
44.. _usingpoky-components-bitbake:
45
46BitBake
47-------
48
49BitBake is the tool at the heart of the `OpenEmbedded build
50system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ and is responsible
51for parsing the `Metadata <&YOCTO_DOCS_REF_URL;#metadata>`__, generating
52a list of tasks from it, and then executing those tasks.
53
54This section briefly introduces BitBake. If you want more information on
55BitBake, see the `BitBake User Manual <&YOCTO_DOCS_BB_URL;>`__.
56
57To see a list of the options BitBake supports, use either of the
58following commands: $ bitbake -h $ bitbake --help
59
60The most common usage for BitBake is ``bitbake packagename``, where
61``packagename`` is the name of the package you want to build (referred
62to as the "target"). The target often equates to the first part of a
63recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
64So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
65type the following: $ bitbake matchbox-desktop Several different
66versions of ``matchbox-desktop`` might exist. BitBake chooses the one
67selected by the distribution configuration. You can get more details
68about how BitBake chooses between different target versions and
69providers in the
70"`Preferences <&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences>`__" section
71of the BitBake User Manual.
72
73BitBake also tries to execute any dependent tasks first. So for example,
74before building ``matchbox-desktop``, BitBake would build a cross
75compiler and ``glibc`` if they had not already been built.
76
77A useful BitBake option to consider is the ``-k`` or ``--continue``
78option. This option instructs BitBake to try and continue processing the
79job as long as possible even after encountering an error. When an error
80occurs, the target that failed and those that depend on it cannot be
81remade. However, when you use this option other dependencies can still
82be processed.
83
84.. _overview-components-recipes:
85
86Recipes
87-------
88
89Files that have the ``.bb`` suffix are "recipes" files. In general, a
90recipe contains information about a single piece of software. This
91information includes the location from which to download the unaltered
92source, any source patches to be applied to that source (if needed),
93which special configuration options to apply, how to compile the source
94files, and how to package the compiled output.
95
96The term "package" is sometimes used to refer to recipes. However, since
97the word "package" is used for the packaged output from the OpenEmbedded
98build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids
99using the term "package" when referring to recipes.
100
101.. _overview-components-classes:
102
103Classes
104-------
105
106Class files (``.bbclass``) contain information that is useful to share
107between recipes files. An example is the
108```autotools`` <&YOCTO_DOCS_REF_URL;#ref-classes-autotools>`__ class,
109which contains common settings for any application that Autotools uses.
110The "`Classes <&YOCTO_DOCS_REF_URL;#ref-classes>`__" chapter in the
111Yocto Project Reference Manual provides details about classes and how to
112use them.
113
114.. _overview-components-configurations:
115
116Configurations
117--------------
118
119The configuration files (``.conf``) define various configuration
120variables that govern the OpenEmbedded build process. These files fall
121into several areas that define machine configuration options,
122distribution configuration options, compiler tuning options, general
123common configuration options, and user configuration options in
124``conf/local.conf``, which is found in the `Build
125Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__.
126
127.. _overview-layers:
128
129Layers
130======
131
132Layers are repositories that contain related metadata (i.e. sets of
133instructions) that tell the OpenEmbedded build system how to build a
134target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__
135facilitates collaboration, sharing, customization, and reuse within the
136Yocto Project development environment. Layers logically separate
137information for your project. For example, you can use a layer to hold
138all the configurations for a particular piece of hardware. Isolating
139hardware-specific configurations allows you to share other metadata by
140using a different layer where that metadata might be common across
141several pieces of hardware.
142
143Many layers exist that work in the Yocto Project development
144environment. The `Yocto Project Curated Layer
145Index <https://caffelli-staging.yoctoproject.org/software-overview/layers/>`__
146and `OpenEmbedded Layer
147Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__
148both contain layers from which you can use or leverage.
149
150By convention, layers in the Yocto Project follow a specific form.
151Conforming to a known structure allows BitBake to make assumptions
152during builds on where to find types of metadata. You can find
153procedures and learn about tools (i.e. ``bitbake-layers``) for creating
154layers suitable for the Yocto Project in the "`Understanding and
155Creating
156Layers <&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers>`__"
157section of the Yocto Project Development Tasks Manual.
158
159.. _openembedded-build-system-build-concepts:
160
161OpenEmbedded Build System Concepts
162==================================
163
164This section takes a more detailed look inside the build process used by
165the `OpenEmbedded build
166system <&YOCTO_DOCS_REF_URL;#build-system-term>`__, which is the build
167system specific to the Yocto Project. At the heart of the build system
168is BitBake, the task executor.
169
170The following diagram represents the high-level workflow of a build. The
171remainder of this section expands on the fundamental input, output,
172process, and metadata logical blocks that make up the workflow.
173
174In general, the build's workflow consists of several functional areas:
175
176- *User Configuration:* metadata you can use to control the build
177 process.
178
179- *Metadata Layers:* Various layers that provide software, machine, and
180 distro metadata.
181
182- *Source Files:* Upstream releases, local projects, and SCMs.
183
184- *Build System:* Processes under the control of
185 `BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__. This block expands
186 on how BitBake fetches source, applies patches, completes
187 compilation, analyzes output for package generation, creates and
188 tests packages, generates images, and generates cross-development
189 tools.
190
191- *Package Feeds:* Directories containing output packages (RPM, DEB or
192 IPK), which are subsequently used in the construction of an image or
193 Software Development Kit (SDK), produced by the build system. These
194 feeds can also be copied and shared using a web server or other means
195 to facilitate extending or updating existing images on devices at
196 runtime if runtime package management is enabled.
197
198- *Images:* Images produced by the workflow.
199
200- *Application Development SDK:* Cross-development tools that are
201 produced along with an image or separately with BitBake.
202
203User Configuration
204------------------
205
206User configuration helps define the build. Through user configuration,
207you can tell BitBake the target architecture for which you are building
208the image, where to store downloaded source, and other build properties.
209
210The following figure shows an expanded representation of the "User
211Configuration" box of the `general workflow
212figure <#general-workflow-figure>`__:
213
214BitBake needs some basic configuration files in order to complete a
215build. These files are ``*.conf`` files. The minimally necessary ones
216reside as example files in the ``build/conf`` directory of the `Source
217Directory <&YOCTO_DOCS_REF_URL;#source-directory>`__. For simplicity,
218this section refers to the Source Directory as the "Poky Directory."
219
220When you clone the `Poky <&YOCTO_DOCS_REF_URL;#poky>`__ Git repository
221or you download and unpack a Yocto Project release, you can set up the
222Source Directory to be named anything you want. For this discussion, the
223cloned repository uses the default name ``poky``.
224
225.. note::
226
227 The Poky repository is primarily an aggregation of existing
228 repositories. It is not a canonical upstream source.
229
230The ``meta-poky`` layer inside Poky contains a ``conf`` directory that
231has example configuration files. These example files are used as a basis
232for creating actual configuration files when you source
233````` <&YOCTO_DOCS_REF_URL;#structure-core-script>`__, which is the
234build environment script.
235
236Sourcing the build environment script creates a `Build
237Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__ if one does not
238already exist. BitBake uses the Build Directory for all its work during
239builds. The Build Directory has a ``conf`` directory that contains
240default versions of your ``local.conf`` and ``bblayers.conf``
241configuration files. These default configuration files are created only
242if versions do not already exist in the Build Directory at the time you
243source the build environment setup script.
244
245Because the Poky repository is fundamentally an aggregation of existing
246repositories, some users might be familiar with running the ```` script
247in the context of separate
248`OpenEmbedded-Core <&YOCTO_DOCS_REF_URL;#oe-core>`__ and BitBake
249repositories rather than a single Poky repository. This discussion
250assumes the script is executed from within a cloned or unpacked version
251of Poky.
252
253Depending on where the script is sourced, different sub-scripts are
254called to set up the Build Directory (Yocto or OpenEmbedded).
255Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
256directory sets up the Build Directory and seeds the directory (if
257necessary) with configuration files appropriate for the Yocto Project
258development environment.
259
260.. note::
261
262 The
263 scripts/oe-setup-builddir
264 script uses the
265 $TEMPLATECONF
266 variable to determine which sample configuration files to locate.
267
268The ``local.conf`` file provides many basic variables that define a
269build environment. Here is a list of a few. To see the default
270configurations in a ``local.conf`` file created by the build environment
271script, see the
272```local.conf.sample`` <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample>`__
273in the ``meta-poky`` layer:
274
275- *Target Machine Selection:* Controlled by the
276 ```MACHINE`` <&YOCTO_DOCS_REF_URL;#var-MACHINE>`__ variable.
277
278- *Download Directory:* Controlled by the
279 ```DL_DIR`` <&YOCTO_DOCS_REF_URL;#var-DL_DIR>`__ variable.
280
281- *Shared State Directory:* Controlled by the
282 ```SSTATE_DIR`` <&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR>`__ variable.
283
284- *Build Output:* Controlled by the
285 ```TMPDIR`` <&YOCTO_DOCS_REF_URL;#var-TMPDIR>`__ variable.
286
287- *Distribution Policy:* Controlled by the
288 ```DISTRO`` <&YOCTO_DOCS_REF_URL;#var-DISTRO>`__ variable.
289
290- *Packaging Format:* Controlled by the
291 ```PACKAGE_CLASSES`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES>`__
292 variable.
293
294- *SDK Target Architecture:* Controlled by the
295 ```SDKMACHINE`` <&YOCTO_DOCS_REF_URL;#var-SDKMACHINE>`__ variable.
296
297- *Extra Image Packages:* Controlled by the
298 ```EXTRA_IMAGE_FEATURES`` <&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES>`__
299 variable.
300
301.. note::
302
303 Configurations set in the
304 conf/local.conf
305 file can also be set in the
306 conf/site.conf
307 and
308 conf/auto.conf
309 configuration files.
310
311The ``bblayers.conf`` file tells BitBake what layers you want considered
312during the build. By default, the layers listed in this file include
313layers minimally needed by the build system. However, you must manually
314add any custom layers you have created. You can find more information on
315working with the ``bblayers.conf`` file in the "`Enabling Your
316Layer <&YOCTO_DOCS_DEV_URL;#enabling-your-layer>`__" section in the
317Yocto Project Development Tasks Manual.
318
319The files ``site.conf`` and ``auto.conf`` are not created by the
320environment initialization script. If you want the ``site.conf`` file,
321you need to create that yourself. The ``auto.conf`` file is typically
322created by an autobuilder:
323
324- *``site.conf``:* You can use the ``conf/site.conf`` configuration
325 file to configure multiple build directories. For example, suppose
326 you had several build environments and they shared some common
327 features. You can set these default build properties here. A good
328 example is perhaps the packaging format to use through the
329 ```PACKAGE_CLASSES`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES>`__
330 variable.
331
332 One useful scenario for using the ``conf/site.conf`` file is to
333 extend your ```BBPATH`` <&YOCTO_DOCS_REF_URL;#var-BBPATH>`__ variable
334 to include the path to a ``conf/site.conf``. Then, when BitBake looks
335 for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file
336 and applies your common configurations found in the file. To override
337 configurations in a particular build directory, alter the similar
338 configurations within that build directory's ``conf/local.conf``
339 file.
340
341- *``auto.conf``:* The file is usually created and written to by an
342 autobuilder. The settings put into the file are typically the same as
343 you would find in the ``conf/local.conf`` or the ``conf/site.conf``
344 files.
345
346You can edit all configuration files to further define any particular
347build environment. This process is represented by the "User
348Configuration Edits" box in the figure.
349
350When you launch your build with the ``bitbake target`` command, BitBake
351sorts out the configurations to ultimately define your build
352environment. It is important to understand that the `OpenEmbedded build
353system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ reads the
354configuration files in a specific order: ``site.conf``, ``auto.conf``,
355and ``local.conf``. And, the build system applies the normal assignment
356statement rules as described in the "`Syntax and
357Operators <&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata>`__" chapter
358of the BitBake User Manual. Because the files are parsed in a specific
359order, variable assignments for the same variable could be affected. For
360example, if the ``auto.conf`` file and the ``local.conf`` set variable1
361to different values, because the build system parses ``local.conf``
362after ``auto.conf``, variable1 is assigned the value from the
363``local.conf`` file.
364
365Metadata, Machine Configuration, and Policy Configuration
366---------------------------------------------------------
367
368The previous section described the user configurations that define
369BitBake's global behavior. This section takes a closer look at the
370layers the build system uses to further control the build. These layers
371provide Metadata for the software, machine, and policies.
372
373In general, three types of layer input exists. You can see them below
374the "User Configuration" box in the `general workflow
375figure <#general-workflow-figure>`__:
376
377- *Metadata (``.bb`` + Patches):* Software layers containing
378 user-supplied recipe files, patches, and append files. A good example
379 of a software layer might be the
380 ```meta-qt5`` <https://github.com/meta-qt5/meta-qt5>`__ layer from
381 the `OpenEmbedded Layer
382 Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__.
383 This layer is for version 5.0 of the popular
384 `Qt <https://wiki.qt.io/About_Qt>`__ cross-platform application
385 development framework for desktop, embedded and mobile.
386
387- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
388 "BSP Layer" in the following figure) providing machine-specific
389 configurations. This type of information is specific to a particular
390 target architecture. A good example of a BSP layer from the `Poky
391 Reference Distribution <#gs-reference-distribution-poky>`__ is the
392 ```meta-yocto-bsp`` <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp>`__
393 layer.
394
395- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in
396 the following figure) providing top-level or general policies for the
397 images or SDKs being built for a particular distribution. For
398 example, in the Poky Reference Distribution the distro layer is the
399 ```meta-poky`` <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky>`__
400 layer. Within the distro layer is a ``conf/distro`` directory that
401 contains distro configuration files (e.g.
402 ```poky.conf`` <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf>`__
403 that contain many policy configurations for the Poky distribution.
404
405The following figure shows an expanded representation of these three
406layers from the `general workflow figure <#general-workflow-figure>`__:
407
408In general, all layers have a similar structure. They all contain a
409licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
410a ``README`` file as good practice and especially if the layer is to be
411distributed, a configuration directory, and recipe directories. You can
412learn about the general structure for layers used with the Yocto Project
413in the "`Creating Your Own
414Layer <&YOCTO_DOCS_DEV_URL;#creating-your-own-layer>`__" section in the
415Yocto Project Development Tasks Manual. For a general discussion on
416layers and the many layers from which you can draw, see the
417"`Layers <#overview-layers>`__" and "`The Yocto Project Layer
418Model <#the-yocto-project-layer-model>`__" sections both earlier in this
419manual.
420
421If you explored the previous links, you discovered some areas where many
422layers that work with the Yocto Project exist. The `Source
423Repositories <http://git.yoctoproject.org/>`__ also shows layers
424categorized under "Yocto Metadata Layers."
425
426.. note::
427
428 Layers exist in the Yocto Project Source Repositories that cannot be
429 found in the OpenEmbedded Layer Index. These layers are either
430 deprecated or experimental in nature.
431
432BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
433configuration, to find what layers it should be using as part of the
434build.
435
436Distro Layer
437~~~~~~~~~~~~
438
439The distribution layer provides policy configurations for your
440distribution. Best practices dictate that you isolate these types of
441configurations into their own layer. Settings you provide in
442``conf/distro/distro.conf`` override similar settings that BitBake finds
443in your ``conf/local.conf`` file in the Build Directory.
444
445The following list provides some explanation and references for what you
446typically find in the distribution layer:
447
448- *classes:* Class files (``.bbclass``) hold common functionality that
449 can be shared among recipes in the distribution. When your recipes
450 inherit a class, they take on the settings and functions for that
451 class. You can read more about class files in the
452 "`Classes <&YOCTO_DOCS_REF_URL;#ref-classes>`__" chapter of the Yocto
453 Reference Manual.
454
455- *conf:* This area holds configuration files for the layer
456 (``conf/layer.conf``), the distribution
457 (``conf/distro/distro.conf``), and any distribution-wide include
458 files.
459
460- *recipes-*:* Recipes and append files that affect common
461 functionality across the distribution. This area could include
462 recipes and append files to add distribution-specific configuration,
463 initialization scripts, custom image recipes, and so forth. Examples
464 of ``recipes-*`` directories are ``recipes-core`` and
465 ``recipes-extra``. Hierarchy and contents within a ``recipes-*``
466 directory can vary. Generally, these directories contain recipe files
467 (``*.bb``), recipe append files (``*.bbappend``), directories that
468 are distro-specific for configuration files, and so forth.
469
470BSP Layer
471~~~~~~~~~
472
473The BSP Layer provides machine configurations that target specific
474hardware. Everything in this layer is specific to the machine for which
475you are building the image or the SDK. A common structure or form is
476defined for BSP layers. You can learn more about this structure in the
477`Yocto Project Board Support Package (BSP) Developer's
478Guide <&YOCTO_DOCS_BSP_URL;>`__.
479
480.. note::
481
482 In order for a BSP layer to be considered compliant with the Yocto
483 Project, it must meet some structural requirements.
484
485The BSP Layer's configuration directory contains configuration files for
486the machine (``conf/machine/machine.conf``) and, of course, the layer
487(``conf/layer.conf``).
488
489The remainder of the layer is dedicated to specific recipes by function:
490``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
491``recipes-kernel``, and so forth. Metadata can exist for multiple
492formfactors, graphics support systems, and so forth.
493
494.. note::
495
496 While the figure shows several
497 recipes-\*
498 directories, not all these directories appear in all BSP layers.
499
500Software Layer
501~~~~~~~~~~~~~~
502
503The software layer provides the Metadata for additional software
504packages used during the build. This layer does not include Metadata
505that is specific to the distribution or the machine, which are found in
506their respective layers.
507
508This layer contains any recipes, append files, and patches, that your
509project needs.
510
511.. _sources-dev-environment:
512
513Sources
514-------
515
516In order for the OpenEmbedded build system to create an image or any
517target, it must be able to access source files. The `general workflow
518figure <#general-workflow-figure>`__ represents source files using the
519"Upstream Project Releases", "Local Projects", and "SCMs (optional)"
520boxes. The figure represents mirrors, which also play a role in locating
521source files, with the "Source Materials" box.
522
523The method by which source files are ultimately organized is a function
524of the project. For example, for released software, projects tend to use
525tarballs or other archived files that can capture the state of a release
526guaranteeing that it is statically represented. On the other hand, for a
527project that is more dynamic or experimental in nature, a project might
528keep source files in a repository controlled by a Source Control Manager
529(SCM) such as Git. Pulling source from a repository allows you to
530control the point in the repository (the revision) from which you want
531to build software. Finally, a combination of the two might exist, which
532would give the consumer a choice when deciding where to get source
533files.
534
535BitBake uses the ```SRC_URI`` <&YOCTO_DOCS_REF_URL;#var-SRC_URI>`__
536variable to point to source files regardless of their location. Each
537recipe must have a ``SRC_URI`` variable that points to the source.
538
539Another area that plays a significant role in where source files come
540from is pointed to by the
541```DL_DIR`` <&YOCTO_DOCS_REF_URL;#var-DL_DIR>`__ variable. This area is
542a cache that can hold previously downloaded source. You can also
543instruct the OpenEmbedded build system to create tarballs from Git
544repositories, which is not the default behavior, and store them in the
545``DL_DIR`` by using the
546```BB_GENERATE_MIRROR_TARBALLS`` <&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS>`__
547variable.
548
549Judicious use of a ``DL_DIR`` directory can save the build system a trip
550across the Internet when looking for files. A good method for using a
551download directory is to have ``DL_DIR`` point to an area outside of
552your Build Directory. Doing so allows you to safely delete the Build
553Directory if needed without fear of removing any downloaded source file.
554
555The remainder of this section provides a deeper look into the source
556files and the mirrors. Here is a more detailed look at the source file
557area of the `general workflow figure <#general-workflow-figure>`__:
558
559Upstream Project Releases
560~~~~~~~~~~~~~~~~~~~~~~~~~
561
562Upstream project releases exist anywhere in the form of an archived file
563(e.g. tarball or zip file). These files correspond to individual
564recipes. For example, the figure uses specific releases each for
565BusyBox, Qt, and Dbus. An archive file can be for any released product
566that can be built using a recipe.
567
568Local Projects
569~~~~~~~~~~~~~~
570
571Local projects are custom bits of software the user provides. These bits
572reside somewhere local to a project - perhaps a directory into which the
573user checks in items (e.g. a local directory containing a development
574source tree used by the group).
575
576The canonical method through which to include a local project is to use
577the ```externalsrc`` <&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc>`__
578class to include that local project. You use either the ``local.conf``
579or a recipe's append file to override or set the recipe to point to the
580local directory on your disk to pull in the whole source tree.
581
582.. _scms:
583
584Source Control Managers (Optional)
585~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
586
587Another place from which the build system can get source files is with
588`fetchers <&YOCTO_DOCS_BB_URL;#bb-fetchers>`__ employing various Source
589Control Managers (SCMs) such as Git or Subversion. In such cases, a
590repository is cloned or checked out. The
591```do_fetch`` <&YOCTO_DOCS_REF_URL;#ref-tasks-fetch>`__ task inside
592BitBake uses the ```SRC_URI`` <&YOCTO_DOCS_REF_URL;#var-SRC_URI>`__
593variable and the argument's prefix to determine the correct fetcher
594module.
595
596.. note::
597
598 For information on how to have the OpenEmbedded build system generate
599 tarballs for Git repositories and place them in the
600 DL_DIR
601 directory, see the
602 BB_GENERATE_MIRROR_TARBALLS
603 variable in the Yocto Project Reference Manual.
604
605When fetching a repository, BitBake uses the
606```SRCREV`` <&YOCTO_DOCS_REF_URL;#var-SRCREV>`__ variable to determine
607the specific revision from which to build.
608
609Source Mirror(s)
610~~~~~~~~~~~~~~~~
611
612Two kinds of mirrors exist: pre-mirrors and regular mirrors. The
613```PREMIRRORS`` <&YOCTO_DOCS_REF_URL;#var-PREMIRRORS>`__ and
614```MIRRORS`` <&YOCTO_DOCS_REF_URL;#var-MIRRORS>`__ variables point to
615these, respectively. BitBake checks pre-mirrors before looking upstream
616for any source files. Pre-mirrors are appropriate when you have a shared
617directory that is not a directory defined by the
618```DL_DIR`` <&YOCTO_DOCS_REF_URL;#var-DL_DIR>`__ variable. A Pre-mirror
619typically points to a shared directory that is local to your
620organization.
621
622Regular mirrors can be any site across the Internet that is used as an
623alternative location for source code should the primary site not be
624functioning for some reason or another.
625
626.. _package-feeds-dev-environment:
627
628Package Feeds
629-------------
630
631When the OpenEmbedded build system generates an image or an SDK, it gets
632the packages from a package feed area located in the `Build
633Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__. The `general
634workflow figure <#general-workflow-figure>`__ shows this package feeds
635area in the upper-right corner.
636
637This section looks a little closer into the package feeds area used by
638the build system. Here is a more detailed look at the area:
639
640Package feeds are an intermediary step in the build process. The
641OpenEmbedded build system provides classes to generate different package
642types, and you specify which classes to enable through the
643```PACKAGE_CLASSES`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES>`__
644variable. Before placing the packages into package feeds, the build
645process validates them with generated output quality assurance checks
646through the ```insane`` <&YOCTO_DOCS_REF_URL;#ref-classes-insane>`__
647class.
648
649The package feed area resides in the Build Directory. The directory the
650build system uses to temporarily store packages is determined by a
651combination of variables and the particular package manager in use. See
652the "Package Feeds" box in the illustration and note the information to
653the right of that area. In particular, the following defines where
654package files are kept:
655
656- ```DEPLOY_DIR`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR>`__: Defined as
657 ``tmp/deploy`` in the Build Directory.
658
659- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
660 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
661 creation, the
662 ```DEPLOY_DIR_RPM`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM>`__,
663 ```DEPLOY_DIR_IPK`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK>`__,
664 ```DEPLOY_DIR_DEB`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB>`__, or
665 ```DEPLOY_DIR_TAR`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR>`__,
666 variables are used, respectively.
667
668- ```PACKAGE_ARCH`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH>`__: Defines
669 architecture-specific sub-folders. For example, packages could exist
670 for the i586 or qemux86 architectures.
671
672BitBake uses the
673```do_package_write_*`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb>`__
674tasks to generate packages and place them into the package holding area
675(e.g. ``do_package_write_ipk`` for IPK packages). See the
676"```do_package_write_deb`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb>`__",
677"```do_package_write_ipk`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk>`__",
678"```do_package_write_rpm`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm>`__",
679and
680"```do_package_write_tar`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar>`__"
681sections in the Yocto Project Reference Manual for additional
682information. As an example, consider a scenario where an IPK packaging
683manager is being used and package architecture support for both i586 and
684qemux86 exist. Packages for the i586 architecture are placed in
685``build/tmp/deploy/ipk/i586``, while packages for the qemux86
686architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
687
688.. _bitbake-dev-environment:
689
690BitBake
691-------
692
693The OpenEmbedded build system uses
694`BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__ to produce images and
695Software Development Kits (SDKs). You can see from the `general workflow
696figure <#general-workflow-figure>`__, the BitBake area consists of
697several functional areas. This section takes a closer look at each of
698those areas.
699
700.. note::
701
702 Separate documentation exists for the BitBake tool. See the
703 BitBake User Manual
704 for reference material on BitBake.
705
706.. _source-fetching-dev-environment:
707
708Source Fetching
709~~~~~~~~~~~~~~~
710
711The first stages of building a recipe are to fetch and unpack the source
712code:
713
714The ```do_fetch`` <&YOCTO_DOCS_REF_URL;#ref-tasks-fetch>`__ and
715```do_unpack`` <&YOCTO_DOCS_REF_URL;#ref-tasks-unpack>`__ tasks fetch
716the source files and unpack them into the `Build
717Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__.
718
719.. note::
720
721 For every local file (e.g.
722 file://
723 ) that is part of a recipe's
724 SRC_URI
725 statement, the OpenEmbedded build system takes a checksum of the file
726 for the recipe and inserts the checksum into the signature for the
727 do_fetch
728 task. If any local file has been modified, the
729 do_fetch
730 task and all tasks that depend on it are re-executed.
731
732By default, everything is accomplished in the Build Directory, which has
733a defined structure. For additional general information on the Build
734Directory, see the
735"```build/`` <&YOCTO_DOCS_REF_URL;#structure-core-build>`__" section in
736the Yocto Project Reference Manual.
737
738Each recipe has an area in the Build Directory where the unpacked source
739code resides. The ```S`` <&YOCTO_DOCS_REF_URL;#var-S>`__ variable points
740to this area for a recipe's unpacked source code. The name of that
741directory for any given recipe is defined from several different
742variables. The preceding figure and the following list describe the
743Build Directory's hierarchy:
744
745- ```TMPDIR`` <&YOCTO_DOCS_REF_URL;#var-TMPDIR>`__: The base directory
746 where the OpenEmbedded build system performs all its work during the
747 build. The default base directory is the ``tmp`` directory.
748
749- ```PACKAGE_ARCH`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH>`__: The
750 architecture of the built package or packages. Depending on the
751 eventual destination of the package or packages (i.e. machine
752 architecture, `build
753 host <&YOCTO_DOCS_REF_URL;#hardware-build-system-term>`__, SDK, or
754 specific machine), ``PACKAGE_ARCH`` varies. See the variable's
755 description for details.
756
757- ```TARGET_OS`` <&YOCTO_DOCS_REF_URL;#var-TARGET_OS>`__: The operating
758 system of the target device. A typical value would be "linux" (e.g.
759 "qemux86-poky-linux").
760
761- ```PN`` <&YOCTO_DOCS_REF_URL;#var-PN>`__: The name of the recipe used
762 to build the package. This variable can have multiple meanings.
763 However, when used in the context of input files, ``PN`` represents
764 the the name of the recipe.
765
766- ```WORKDIR`` <&YOCTO_DOCS_REF_URL;#var-WORKDIR>`__: The location
767 where the OpenEmbedded build system builds a recipe (i.e. does the
768 work to create the package).
769
770 - ```PV`` <&YOCTO_DOCS_REF_URL;#var-PV>`__: The version of the
771 recipe used to build the package.
772
773 - ```PR`` <&YOCTO_DOCS_REF_URL;#var-PR>`__: The revision of the
774 recipe used to build the package.
775
776- ```S`` <&YOCTO_DOCS_REF_URL;#var-S>`__: Contains the unpacked source
777 files for a given recipe.
778
779 - ```BPN`` <&YOCTO_DOCS_REF_URL;#var-BPN>`__: The name of the recipe
780 used to build the package. The ``BPN`` variable is a version of
781 the ``PN`` variable but with common prefixes and suffixes removed.
782
783 - ```PV`` <&YOCTO_DOCS_REF_URL;#var-PV>`__: The version of the
784 recipe used to build the package.
785
786.. note::
787
788 In the previous figure, notice that two sample hierarchies exist: one
789 based on package architecture (i.e.
790 PACKAGE_ARCH
791 ) and one based on a machine (i.e.
792 MACHINE
793 ). The underlying structures are identical. The differentiator being
794 what the OpenEmbedded build system is using as a build target (e.g.
795 general architecture, a build host, an SDK, or a specific machine).
796
797.. _patching-dev-environment:
798
799Patching
800~~~~~~~~
801
802Once source code is fetched and unpacked, BitBake locates patch files
803and applies them to the source files:
804
805The ```do_patch`` <&YOCTO_DOCS_REF_URL;#ref-tasks-patch>`__ task uses a
806recipe's ```SRC_URI`` <&YOCTO_DOCS_REF_URL;#var-SRC_URI>`__ statements
807and the ```FILESPATH`` <&YOCTO_DOCS_REF_URL;#var-FILESPATH>`__ variable
808to locate applicable patch files.
809
810Default processing for patch files assumes the files have either
811``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters
812to change the way the build system recognizes patch files. See the
813```do_patch`` <&YOCTO_DOCS_REF_URL;#ref-tasks-patch>`__ task for more
814information.
815
816BitBake finds and applies multiple patches for a single recipe in the
817order in which it locates the patches. The ``FILESPATH`` variable
818defines the default set of directories that the build system uses to
819search for patch files. Once found, patches are applied to the recipe's
820source files, which are located in the
821```S`` <&YOCTO_DOCS_REF_URL;#var-S>`__ directory.
822
823For more information on how the source directories are created, see the
824"`Source Fetching <#source-fetching-dev-environment>`__" section. For
825more information on how to create patches and how the build system
826processes patches, see the "`Patching
827Code <&YOCTO_DOCS_DEV_URL;#new-recipe-patching-code>`__" section in the
828Yocto Project Development Tasks Manual. You can also see the "`Use
829``devtool modify`` to Modify the Source of an Existing
830Component <&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-modify-to-modify-the-source-of-an-existing-component>`__"
831section in the Yocto Project Application Development and the Extensible
832Software Development Kit (SDK) manual and the "`Using Traditional Kernel
833Development to Patch the
834Kernel <&YOCTO_DOCS_KERNEL_DEV_URL;#using-traditional-kernel-development-to-patch-the-kernel>`__"
835section in the Yocto Project Linux Kernel Development Manual.
836
837.. _configuration-compilation-and-staging-dev-environment:
838
839Configuration, Compilation, and Staging
840~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
841
842After source code is patched, BitBake executes tasks that configure and
843compile the source code. Once compilation occurs, the files are copied
844to a holding area (staged) in preparation for packaging:
845
846This step in the build process consists of the following tasks:
847
848- ```do_prepare_recipe_sysroot`` <&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot>`__:
849 This task sets up the two sysroots in
850 ``${``\ ```WORKDIR`` <&YOCTO_DOCS_REF_URL;#var-WORKDIR>`__\ ``}``
851 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that
852 during the packaging phase the sysroots can contain the contents of
853 the
854 ```do_populate_sysroot`` <&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot>`__
855 tasks of the recipes on which the recipe containing the tasks
856 depends. A sysroot exists for both the target and for the native
857 binaries, which run on the host system.
858
859- *``do_configure``*: This task configures the source by enabling and
860 disabling any build-time and configuration options for the software
861 being built. Configurations can come from the recipe itself as well
862 as from an inherited class. Additionally, the software itself might
863 configure itself depending on the target for which it is being built.
864
865 The configurations handled by the
866 ```do_configure`` <&YOCTO_DOCS_REF_URL;#ref-tasks-configure>`__ task
867 are specific to configurations for the source code being built by the
868 recipe.
869
870 If you are using the
871 ```autotools`` <&YOCTO_DOCS_REF_URL;#ref-classes-autotools>`__ class,
872 you can add additional configuration options by using the
873 ```EXTRA_OECONF`` <&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF>`__ or
874 ```PACKAGECONFIG_CONFARGS`` <&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS>`__
875 variables. For information on how this variable works within that
876 class, see the
877 ```autotools`` <&YOCTO_DOCS_REF_URL;#ref-classes-autotools>`__ class
878 `here <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass>`__.
879
880- *``do_compile``*: Once a configuration task has been satisfied,
881 BitBake compiles the source using the
882 ```do_compile`` <&YOCTO_DOCS_REF_URL;#ref-tasks-compile>`__ task.
883 Compilation occurs in the directory pointed to by the
884 ```B`` <&YOCTO_DOCS_REF_URL;#var-B>`__ variable. Realize that the
885 ``B`` directory is, by default, the same as the
886 ```S`` <&YOCTO_DOCS_REF_URL;#var-S>`__ directory.
887
888- *``do_install``*: After compilation completes, BitBake executes the
889 ```do_install`` <&YOCTO_DOCS_REF_URL;#ref-tasks-install>`__ task.
890 This task copies files from the ``B`` directory and places them in a
891 holding area pointed to by the ```D`` <&YOCTO_DOCS_REF_URL;#var-D>`__
892 variable. Packaging occurs later using files from this holding
893 directory.
894
895.. _package-splitting-dev-environment:
896
897Package Splitting
898~~~~~~~~~~~~~~~~~
899
900After source code is configured, compiled, and staged, the build system
901analyzes the results and splits the output into packages:
902
903The ```do_package`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package>`__ and
904```do_packagedata`` <&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata>`__
905tasks combine to analyze the files found in the
906```D`` <&YOCTO_DOCS_REF_URL;#var-D>`__ directory and split them into
907subsets based on available packages and files. Analysis involves the
908following as well as other items: splitting out debugging symbols,
909looking at shared library dependencies between packages, and looking at
910package relationships.
911
912The ``do_packagedata`` task creates package metadata based on the
913analysis such that the build system can generate the final packages. The
914```do_populate_sysroot`` <&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot>`__
915task stages (copies) a subset of the files installed by the
916```do_install`` <&YOCTO_DOCS_REF_URL;#ref-tasks-install>`__ task into
917the appropriate sysroot. Working, staged, and intermediate results of
918the analysis and package splitting process use several areas:
919
920- ```PKGD`` <&YOCTO_DOCS_REF_URL;#var-PKGD>`__: The destination
921 directory (i.e. ``package``) for packages before they are split into
922 individual packages.
923
924- ```PKGDESTWORK`` <&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK>`__: A
925 temporary work area (i.e. ``pkgdata``) used by the ``do_package``
926 task to save package metadata.
927
928- ```PKGDEST`` <&YOCTO_DOCS_REF_URL;#var-PKGDEST>`__: The parent
929 directory (i.e. ``packages-split``) for packages after they have been
930 split.
931
932- ```PKGDATA_DIR`` <&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR>`__: A shared,
933 global-state directory that holds packaging metadata generated during
934 the packaging process. The packaging process copies metadata from
935 ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally
936 available.
937
938- ```STAGING_DIR_HOST`` <&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_HOST>`__:
939 The path for the sysroot for the system on which a component is built
940 to run (i.e. ``recipe-sysroot``).
941
942- ```STAGING_DIR_NATIVE`` <&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_NATIVE>`__:
943 The path for the sysroot used when building components for the build
944 host (i.e. ``recipe-sysroot-native``).
945
946- ```STAGING_DIR_TARGET`` <&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_TARGET>`__:
947 The path for the sysroot used when a component that is built to
948 execute on a system and it generates code for yet another machine
949 (e.g. cross-canadian recipes).
950
951The ```FILES`` <&YOCTO_DOCS_REF_URL;#var-FILES>`__ variable defines the
952files that go into each package in
953```PACKAGES`` <&YOCTO_DOCS_REF_URL;#var-PACKAGES>`__. If you want
954details on how this is accomplished, you can look at
955```package.bbclass`` <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass>`__.
956
957Depending on the type of packages being created (RPM, DEB, or IPK), the
958```do_package_write_*`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb>`__
959task creates the actual packages and places them in the Package Feed
960area, which is ``${TMPDIR}/deploy``. You can see the "`Package
961Feeds <#package-feeds-dev-environment>`__" section for more detail on
962that part of the build process.
963
964.. note::
965
966 Support for creating feeds directly from the
967 deploy/\*
968 directories does not exist. Creating such feeds usually requires some
969 kind of feed maintenance mechanism that would upload the new packages
970 into an official package feed (e.g. the Ångström distribution). This
971 functionality is highly distribution-specific and thus is not
972 provided out of the box.
973
974.. _image-generation-dev-environment:
975
976Image Generation
977~~~~~~~~~~~~~~~~
978
979Once packages are split and stored in the Package Feeds area, the build
980system uses BitBake to generate the root filesystem image:
981
982The image generation process consists of several stages and depends on
983several tasks and variables. The
984```do_rootfs`` <&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs>`__ task creates
985the root filesystem (file and directory structure) for an image. This
986task uses several key variables to help create the list of packages to
987actually install:
988
989- ```IMAGE_INSTALL`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL>`__: Lists
990 out the base set of packages from which to install from the Package
991 Feeds area.
992
993- ```PACKAGE_EXCLUDE`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE>`__:
994 Specifies packages that should not be installed into the image.
995
996- ```IMAGE_FEATURES`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES>`__:
997 Specifies features to include in the image. Most of these features
998 map to additional packages for installation.
999
1000- ```PACKAGE_CLASSES`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES>`__:
1001 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and
1002 consequently helps determine where to locate packages within the
1003 Package Feeds area.
1004
1005- ```IMAGE_LINGUAS`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS>`__:
1006 Determines the language(s) for which additional language support
1007 packages are installed.
1008
1009- ```PACKAGE_INSTALL`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL>`__:
1010 The final list of packages passed to the package manager for
1011 installation into the image.
1012
1013With ```IMAGE_ROOTFS`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS>`__
1014pointing to the location of the filesystem under construction and the
1015``PACKAGE_INSTALL`` variable providing the final list of packages to
1016install, the root file system is created.
1017
1018Package installation is under control of the package manager (e.g.
1019dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package
1020management is enabled for the target. At the end of the process, if
1021package management is not enabled for the target, the package manager's
1022data files are deleted from the root filesystem. As part of the final
1023stage of package installation, post installation scripts that are part
1024of the packages are run. Any scripts that fail to run on the build host
1025are run on the target when the target system is first booted. If you are
1026using a `read-only root
1027filesystem <&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem>`__,
1028all the post installation scripts must succeed on the build host during
1029the package installation phase since the root filesystem on the target
1030is read-only.
1031
1032The final stages of the ``do_rootfs`` task handle post processing. Post
1033processing includes creation of a manifest file and optimizations.
1034
1035The manifest file (``.manifest``) resides in the same directory as the
1036root filesystem image. This file lists out, line-by-line, the installed
1037packages. The manifest file is useful for the
1038```testimage`` <&YOCTO_DOCS_REF_URL;#ref-classes-testimage*>`__ class,
1039for example, to determine whether or not to run specific tests. See the
1040```IMAGE_MANIFEST`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST>`__
1041variable for additional information.
1042
1043Optimizing processes that are run across the image include ``mklibs``,
1044``prelink``, and any other post-processing commands as defined by the
1045```ROOTFS_POSTPROCESS_COMMAND`` <&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND>`__
1046variable. The ``mklibs`` process optimizes the size of the libraries,
1047while the ``prelink`` process optimizes the dynamic linking of shared
1048libraries to reduce start up time of executables.
1049
1050After the root filesystem is built, processing begins on the image
1051through the ```do_image`` <&YOCTO_DOCS_REF_URL;#ref-tasks-image>`__
1052task. The build system runs any pre-processing commands as defined by
1053the
1054```IMAGE_PREPROCESS_COMMAND`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND>`__
1055variable. This variable specifies a list of functions to call before the
1056build system creates the final image output files.
1057
1058The build system dynamically creates ``do_image_*`` tasks as needed,
1059based on the image types specified in the
1060```IMAGE_FSTYPES`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES>`__ variable.
1061The process turns everything into an image file or a set of image files
1062and can compress the root filesystem image to reduce the overall size of
1063the image. The formats used for the root filesystem depend on the
1064``IMAGE_FSTYPES`` variable. Compression depends on whether the formats
1065support compression.
1066
1067As an example, a dynamically created task when creating a particular
1068image type would take the following form: do_image_type So, if the type
1069as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically
1070generated task would be as follows: do_image_ext4
1071
1072The final task involved in image creation is the
1073```do_image_complete`` <&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete>`__
1074task. This task completes the image by applying any image post
1075processing as defined through the
1076```IMAGE_POSTPROCESS_COMMAND`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND>`__
1077variable. The variable specifies a list of functions to call once the
1078build system has created the final image output files.
1079
1080.. note::
1081
1082 The entire image generation process is run under
1083 Pseudo
1084 . Running under Pseudo ensures that the files in the root filesystem
1085 have correct ownership.
1086
1087.. _sdk-generation-dev-environment:
1088
1089SDK Generation
1090~~~~~~~~~~~~~~
1091
1092The OpenEmbedded build system uses BitBake to generate the Software
1093Development Kit (SDK) installer scripts for both the standard SDK and
1094the extensible SDK (eSDK):
1095
1096.. note::
1097
1098 For more information on the cross-development toolchain generation,
1099 see the "
1100 Cross-Development Toolchain Generation
1101 " section. For information on advantages gained when building a
1102 cross-development toolchain using the
1103 do_populate_sdk
1104 task, see the "
1105 Building an SDK Installer
1106 " section in the Yocto Project Application Development and the
1107 Extensible Software Development Kit (eSDK) manual.
1108
1109Like image generation, the SDK script process consists of several stages
1110and depends on many variables. The
1111```do_populate_sdk`` <&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk>`__
1112and
1113```do_populate_sdk_ext`` <&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk_ext>`__
1114tasks use these key variables to help create the list of packages to
1115actually install. For information on the variables listed in the figure,
1116see the "`Application Development SDK <#sdk-dev-environment>`__"
1117section.
1118
1119The ``do_populate_sdk`` task helps create the standard SDK and handles
1120two parts: a target part and a host part. The target part is the part
1121built for the target hardware and includes libraries and headers. The
1122host part is the part of the SDK that runs on the
1123```SDKMACHINE`` <&YOCTO_DOCS_REF_URL;#var-SDKMACHINE>`__.
1124
1125The ``do_populate_sdk_ext`` task helps create the extensible SDK and
1126handles host and target parts differently than its counter part does for
1127the standard SDK. For the extensible SDK, the task encapsulates the
1128build system, which includes everything needed (host and target) for the
1129SDK.
1130
1131Regardless of the type of SDK being constructed, the tasks perform some
1132cleanup after which a cross-development environment setup script and any
1133needed configuration files are created. The final output is the
1134Cross-development toolchain installation script (``.sh`` file), which
1135includes the environment setup script.
1136
1137Stamp Files and the Rerunning of Tasks
1138~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1139
1140For each task that completes successfully, BitBake writes a stamp file
1141into the ```STAMPS_DIR`` <&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR>`__
1142directory. The beginning of the stamp file's filename is determined by
1143the ```STAMP`` <&YOCTO_DOCS_REF_URL;#var-STAMP>`__ variable, and the end
1144of the name consists of the task's name and current `input
1145checksum <#overview-checksums>`__.
1146
1147.. note::
1148
1149 This naming scheme assumes that
1150 BB_SIGNATURE_HANDLER
1151 is "OEBasicHash", which is almost always the case in current
1152 OpenEmbedded.
1153
1154To determine if a task needs to be rerun, BitBake checks if a stamp file
1155with a matching input checksum exists for the task. If such a stamp file
1156exists, the task's output is assumed to exist and still be valid. If the
1157file does not exist, the task is rerun.
1158
1159.. note::
1160
1161 The stamp mechanism is more general than the shared state (sstate)
1162 cache mechanism described in the "`Setscene Tasks and Shared
1163 State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids
1164 rerunning any task that has a valid stamp file, not just tasks that
1165 can be accelerated through the sstate cache.
1166
1167 However, you should realize that stamp files only serve as a marker
1168 that some work has been done and that these files do not record task
1169 output. The actual task output would usually be somewhere in
1170 ```TMPDIR`` <&YOCTO_DOCS_REF_URL;#var-TMPDIR>`__ (e.g. in some
1171 recipe's ```WORKDIR`` <&YOCTO_DOCS_REF_URL;#var-WORKDIR>`__.) What
1172 the sstate cache mechanism adds is a way to cache task output that
1173 can then be shared between build machines.
1174
1175Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing
1176``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will
1177properly be rerun to repopulate ``TMPDIR``.
1178
1179If you want some task to always be considered "out of date", you can
1180mark it with the ```nostamp`` <&YOCTO_DOCS_BB_URL;#variable-flags>`__
1181varflag. If some other task depends on such a task, then that task will
1182also always be considered out of date, which might not be what you want.
1183
1184For details on how to view information about a task's signature, see the
1185"`Viewing Task Variable
1186Dependencies <&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies>`__"
1187section in the Yocto Project Development Tasks Manual.
1188
1189Setscene Tasks and Shared State
1190~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1191
1192The description of tasks so far assumes that BitBake needs to build
1193everything and no available prebuilt objects exist. BitBake does support
1194skipping tasks if prebuilt objects are available. These objects are
1195usually made available in the form of a shared state (sstate) cache.
1196
1197.. note::
1198
1199 For information on variables affecting sstate, see the
1200 SSTATE_DIR
1201 and
1202 SSTATE_MIRRORS
1203 variables.
1204
1205The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a
1206version of the task where instead of building something, BitBake can
1207skip to the end result and simply place a set of files into specific
1208locations as needed. In some cases, it makes sense to have a setscene
1209task variant (e.g. generating package files in the
1210```do_package_write_*`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb>`__
1211task). In other cases, it does not make sense (e.g. a
1212```do_patch`` <&YOCTO_DOCS_REF_URL;#ref-tasks-patch>`__ task or a
1213```do_unpack`` <&YOCTO_DOCS_REF_URL;#ref-tasks-unpack>`__ task) since
1214the work involved would be equal to or greater than the underlying task.
1215
1216In the build system, the common tasks that have setscene variants are
1217```do_package`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package>`__,
1218``do_package_write_*``,
1219```do_deploy`` <&YOCTO_DOCS_REF_URL;#ref-tasks-deploy>`__,
1220```do_packagedata`` <&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata>`__, and
1221```do_populate_sysroot`` <&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot>`__.
1222Notice that these tasks represent most of the tasks whose output is an
1223end result.
1224
1225The build system has knowledge of the relationship between these tasks
1226and other preceding tasks. For example, if BitBake runs
1227``do_populate_sysroot_setscene`` for something, it does not make sense
1228to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``,
1229``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if
1230``do_package`` needs to be run, BitBake needs to run those other tasks.
1231
1232It becomes more complicated if everything can come from an sstate cache
1233because some objects are simply not required at all. For example, you do
1234not need a compiler or native tools, such as quilt, if nothing exists to
1235compile or patch. If the ``do_package_write_*`` packages are available
1236from sstate, BitBake does not need the ``do_package`` task data.
1237
1238To handle all these complexities, BitBake runs in two phases. The first
1239is the "setscene" stage. During this stage, BitBake first checks the
1240sstate cache for any targets it is planning to build. BitBake does a
1241fast check to see if the object exists rather than a complete download.
1242If nothing exists, the second phase, which is the setscene stage,
1243completes and the main build proceeds.
1244
1245If objects are found in the sstate cache, the build system works
1246backwards from the end targets specified by the user. For example, if an
1247image is being built, the build system first looks for the packages
1248needed for that image and the tools needed to construct an image. If
1249those are available, the compiler is not needed. Thus, the compiler is
1250not even downloaded. If something was found to be unavailable, or the
1251download or setscene task fails, the build system then tries to install
1252dependencies, such as the compiler, from the cache.
1253
1254The availability of objects in the sstate cache is handled by the
1255function specified by the
1256```BB_HASHCHECK_FUNCTION`` <&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION>`__
1257variable and returns a list of available objects. The function specified
1258by the
1259```BB_SETSCENE_DEPVALID`` <&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID>`__
1260variable is the function that determines whether a given dependency
1261needs to be followed, and whether for any given relationship the
1262function needs to be passed. The function returns a True or False value.
1263
1264.. _images-dev-environment:
1265
1266Images
1267------
1268
1269The images produced by the build system are compressed forms of the root
1270filesystem and are ready to boot on a target device. You can see from
1271the `general workflow figure <#general-workflow-figure>`__ that BitBake
1272output, in part, consists of images. This section takes a closer look at
1273this output:
1274
1275.. note::
1276
1277 For a list of example images that the Yocto Project provides, see the
1278 "
1279 Images
1280 " chapter in the Yocto Project Reference Manual.
1281
1282The build process writes images out to the `Build
1283Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__ inside the
1284``tmp/deploy/images/machine/`` folder as shown in the figure. This
1285folder contains any files expected to be loaded on the target device.
1286The ```DEPLOY_DIR`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR>`__ variable
1287points to the ``deploy`` directory, while the
1288```DEPLOY_DIR_IMAGE`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE>`__
1289variable points to the appropriate directory containing images for the
1290current configuration.
1291
1292- kernel-image: A kernel binary file. The
1293 ```KERNEL_IMAGETYPE`` <&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE>`__
1294 variable determines the naming scheme for the kernel image file.
1295 Depending on this variable, the file could begin with a variety of
1296 naming strings. The ``deploy/images/``\ machine directory can contain
1297 multiple image files for the machine.
1298
1299- root-filesystem-image: Root filesystems for the target device (e.g.
1300 ``*.ext3`` or ``*.bz2`` files). The
1301 ```IMAGE_FSTYPES`` <&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES>`__
1302 variable determines the root filesystem image type. The
1303 ``deploy/images/``\ machine directory can contain multiple root
1304 filesystems for the machine.
1305
1306- kernel-modules: Tarballs that contain all the modules built for the
1307 kernel. Kernel module tarballs exist for legacy purposes and can be
1308 suppressed by setting the
1309 ```MODULE_TARBALL_DEPLOY`` <&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY>`__
1310 variable to "0". The ``deploy/images/``\ machine directory can
1311 contain multiple kernel module tarballs for the machine.
1312
1313- bootloaders: If applicable to the target machine, bootloaders
1314 supporting the image. The ``deploy/images/``\ machine directory can
1315 contain multiple bootloaders for the machine.
1316
1317- symlinks: The ``deploy/images/``\ machine folder contains a symbolic
1318 link that points to the most recently built file for each machine.
1319 These links might be useful for external scripts that need to obtain
1320 the latest version of each file.
1321
1322.. _sdk-dev-environment:
1323
1324Application Development SDK
1325---------------------------
1326
1327In the `general workflow figure <#general-workflow-figure>`__, the
1328output labeled "Application Development SDK" represents an SDK. The SDK
1329generation process differs depending on whether you build an extensible
1330SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1331(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a
1332closer look at this output:
1333
1334The specific form of this output is a set of files that includes a
1335self-extracting SDK installer (``*.sh``), host and target manifest
1336files, and files used for SDK testing. When the SDK installer file is
1337run, it installs the SDK. The SDK consists of a cross-development
1338toolchain, a set of libraries and headers, and an SDK environment setup
1339script. Running this installer essentially sets up your
1340cross-development environment. You can think of the cross-toolchain as
1341the "host" part because it runs on the SDK machine. You can think of the
1342libraries and headers as the "target" part because they are built for
1343the target hardware. The environment setup script is added so that you
1344can initialize the environment before using the tools.
1345
1346.. note::
1347
1348 - The Yocto Project supports several methods by which you can set up
1349 this cross-development environment. These methods include
1350 downloading pre-built SDK installers or building and installing
1351 your own SDK installer.
1352
1353 - For background information on cross-development toolchains in the
1354 Yocto Project development environment, see the "`Cross-Development
1355 Toolchain Generation <#cross-development-toolchain-generation>`__"
1356 section.
1357
1358 - For information on setting up a cross-development environment, see
1359 the `Yocto Project Application Development and the Extensible
1360 Software Development Kit (eSDK) <&YOCTO_DOCS_SDK_URL;>`__ manual.
1361
1362All the output files for an SDK are written to the ``deploy/sdk`` folder
1363inside the `Build Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__ as
1364shown in the previous figure. Depending on the type of SDK, several
1365variables exist that help configure these files. The following list
1366shows the variables associated with an extensible SDK:
1367
1368- ```DEPLOY_DIR`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR>`__: Points to
1369 the ``deploy`` directory.
1370
1371- ```SDK_EXT_TYPE`` <&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE>`__:
1372 Controls whether or not shared state artifacts are copied into the
1373 extensible SDK. By default, all required shared state artifacts are
1374 copied into the SDK.
1375
1376- ```SDK_INCLUDE_PKGDATA`` <&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA>`__:
1377 Specifies whether or not packagedata is included in the extensible
1378 SDK for all recipes in the "world" target.
1379
1380- ```SDK_INCLUDE_TOOLCHAIN`` <&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN>`__:
1381 Specifies whether or not the toolchain is included when building the
1382 extensible SDK.
1383
1384- ```SDK_LOCAL_CONF_WHITELIST`` <&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST>`__:
1385 A list of variables allowed through from the build system
1386 configuration into the extensible SDK configuration.
1387
1388- ```SDK_LOCAL_CONF_BLACKLIST`` <&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST>`__:
1389 A list of variables not allowed through from the build system
1390 configuration into the extensible SDK configuration.
1391
1392- ```SDK_INHERIT_BLACKLIST`` <&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST>`__:
1393 A list of classes to remove from the
1394 ```INHERIT`` <&YOCTO_DOCS_REF_URL;#var-INHERIT>`__ value globally
1395 within the extensible SDK configuration.
1396
1397This next list, shows the variables associated with a standard SDK:
1398
1399- ```DEPLOY_DIR`` <&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR>`__: Points to
1400 the ``deploy`` directory.
1401
1402- ```SDKMACHINE`` <&YOCTO_DOCS_REF_URL;#var-SDKMACHINE>`__: Specifies
1403 the architecture of the machine on which the cross-development tools
1404 are run to create packages for the target hardware.
1405
1406- ```SDKIMAGE_FEATURES`` <&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES>`__:
1407 Lists the features to include in the "target" part of the SDK.
1408
1409- ```TOOLCHAIN_HOST_TASK`` <&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK>`__:
1410 Lists packages that make up the host part of the SDK (i.e. the part
1411 that runs on the ``SDKMACHINE``). When you use
1412 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1413 default packages apply. This variable allows you to add more
1414 packages.
1415
1416- ```TOOLCHAIN_TARGET_TASK`` <&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK>`__:
1417 Lists packages that make up the target part of the SDK (i.e. the part
1418 built for the target hardware).
1419
1420- ```SDKPATH`` <&YOCTO_DOCS_REF_URL;#var-SDKPATH>`__: Defines the
1421 default SDK installation path offered by the installation script.
1422
1423- ```SDK_HOST_MANIFEST`` <&YOCTO_DOCS_REF_URL;#var-SDK_HOST_MANIFEST>`__:
1424 Lists all the installed packages that make up the host part of the
1425 SDK. This variable also plays a minor role for extensible SDK
1426 development as well. However, it is mainly used for the standard SDK.
1427
1428- ```SDK_TARGET_MANIFEST`` <&YOCTO_DOCS_REF_URL;#var-SDK_TARGET_MANIFEST>`__:
1429 Lists all the installed packages that make up the target part of the
1430 SDK. This variable also plays a minor role for extensible SDK
1431 development as well. However, it is mainly used for the standard SDK.
1432
1433Cross-Development Toolchain Generation
1434======================================
1435
1436The Yocto Project does most of the work for you when it comes to
1437creating `cross-development
1438toolchains <&YOCTO_DOCS_REF_URL;#cross-development-toolchain>`__. This
1439section provides some technical background on how cross-development
1440toolchains are created and used. For more information on toolchains, you
1441can also see the `Yocto Project Application Development and the
1442Extensible Software Development Kit (eSDK) <&YOCTO_DOCS_SDK_URL;>`__
1443manual.
1444
1445In the Yocto Project development environment, cross-development
1446toolchains are used to build images and applications that run on the
1447target hardware. With just a few commands, the OpenEmbedded build system
1448creates these necessary toolchains for you.
1449
1450The following figure shows a high-level build environment regarding
1451toolchain construction and use.
1452
1453Most of the work occurs on the Build Host. This is the machine used to
1454build images and generally work within the the Yocto Project
1455environment. When you run
1456`BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__ to create an image, the
1457OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
1458cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what
1459BitBake uses to compile source files when creating the target image. You
1460can think of ``gcc-cross`` simply as an automatically generated
1461cross-compiler that is used internally within BitBake only.
1462
1463.. note::
1464
1465 The extensible SDK does not use
1466 gcc-cross-canadian
1467 since this SDK ships a copy of the OpenEmbedded build system and the
1468 sysroot within it contains
1469 gcc-cross
1470 .
1471
1472The chain of events that occurs when ``gcc-cross`` is bootstrapped is as
1473follows: gcc -> binutils-cross -> gcc-cross-initial ->
1474linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime
1475
1476- ``gcc``: The build host's GNU Compiler Collection (GCC).
1477
1478- ``binutils-cross``: The bare minimum binary utilities needed in order
1479 to run the ``gcc-cross-initial`` phase of the bootstrap operation.
1480
1481- ``gcc-cross-initial``: An early stage of the bootstrap process for
1482 creating the cross-compiler. This stage builds enough of the
1483 ``gcc-cross``, the C library, and other pieces needed to finish
1484 building the final cross-compiler in later stages. This tool is a
1485 "native" package (i.e. it is designed to run on the build host).
1486
1487- ``linux-libc-headers``: Headers needed for the cross-compiler.
1488
1489- ``glibc-initial``: An initial version of the Embedded GNU C Library
1490 (GLIBC) needed to bootstrap ``glibc``.
1491
1492- ``glibc``: The GNU C Library.
1493
1494- ``gcc-cross``: The final stage of the bootstrap process for the
1495 cross-compiler. This stage results in the actual cross-compiler that
1496 BitBake uses when it builds an image for a targeted device.
1497
1498 .. note::
1499
1500 If you are replacing this cross compiler toolchain with a custom
1501 version, you must replace
1502 gcc-cross
1503 .
1504
1505 This tool is also a "native" package (i.e. it is designed to run on
1506 the build host).
1507
1508- ``gcc-runtime``: Runtime libraries resulting from the toolchain
1509 bootstrapping process. This tool produces a binary that consists of
1510 the runtime libraries need for the targeted device.
1511
1512You can use the OpenEmbedded build system to build an installer for the
1513relocatable SDK used to develop applications. When you run the
1514installer, it installs the toolchain, which contains the development
1515tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1516other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1517native to ```SDK_ARCH`` <&YOCTO_DOCS_REF_URL;#var-SDK_ARCH>`__), you
1518need to cross-compile and test your software. The figure shows the
1519commands you use to easily build out this toolchain. This
1520cross-development toolchain is built to execute on the
1521```SDKMACHINE`` <&YOCTO_DOCS_REF_URL;#var-SDKMACHINE>`__, which might or
1522might not be the same machine as the Build Host.
1523
1524.. note::
1525
1526 If your target architecture is supported by the Yocto Project, you
1527 can take advantage of pre-built images that ship with the Yocto
1528 Project and already contain cross-development toolchain installers.
1529
1530Here is the bootstrap process for the relocatable toolchain: gcc ->
1531binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers ->
1532glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1533
1534- ``gcc``: The build host's GNU Compiler Collection (GCC).
1535
1536- ``binutils-crosssdk``: The bare minimum binary utilities needed in
1537 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap
1538 operation.
1539
1540- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for
1541 creating the cross-compiler. This stage builds enough of the
1542 ``gcc-crosssdk`` and supporting pieces so that the final stage of the
1543 bootstrap process can produce the finished cross-compiler. This tool
1544 is a "native" binary that runs on the build host.
1545
1546- ``linux-libc-headers``: Headers needed for the cross-compiler.
1547
1548- ``glibc-initial``: An initial version of the Embedded GLIBC needed to
1549 bootstrap ``nativesdk-glibc``.
1550
1551- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the
1552 ``gcc-crosssdk``.
1553
1554- ``gcc-crosssdk``: The final stage of the bootstrap process for the
1555 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory
1556 compiler and never leaves the build host. Its purpose is to help in
1557 the bootstrap process to create the eventual ``gcc-cross-canadian``
1558 compiler, which is relocatable. This tool is also a "native" package
1559 (i.e. it is designed to run on the build host).
1560
1561- ``gcc-cross-canadian``: The final relocatable cross-compiler. When
1562 run on the ```SDKMACHINE`` <&YOCTO_DOCS_REF_URL;#var-SDKMACHINE>`__,
1563 this tool produces executable code that runs on the target device.
1564 Only one cross-canadian compiler is produced per architecture since
1565 they can be targeted at different processor optimizations using
1566 configurations passed to the compiler through the compile commands.
1567 This circumvents the need for multiple compilers and thus reduces the
1568 size of the toolchains.
1569
1570.. note::
1571
1572 For information on advantages gained when building a
1573 cross-development toolchain installer, see the "
1574 Building an SDK Installer
1575 " appendix in the Yocto Project Application Development and the
1576 Extensible Software Development Kit (eSDK) manual.
1577
1578Shared State Cache
1579==================
1580
1581By design, the OpenEmbedded build system builds everything from scratch
1582unless `BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__ can determine
1583that parts do not need to be rebuilt. Fundamentally, building from
1584scratch is attractive as it means all parts are built fresh and no
1585possibility of stale data exists that can cause problems. When
1586developers hit problems, they typically default back to building from
1587scratch so they have a know state from the start.
1588
1589Building an image from scratch is both an advantage and a disadvantage
1590to the process. As mentioned in the previous paragraph, building from
1591scratch ensures that everything is current and starts from a known
1592state. However, building from scratch also takes much longer as it
1593generally means rebuilding things that do not necessarily need to be
1594rebuilt.
1595
1596The Yocto Project implements shared state code that supports incremental
1597builds. The implementation of the shared state code answers the
1598following questions that were fundamental roadblocks within the
1599OpenEmbedded incremental build support system:
1600
1601- What pieces of the system have changed and what pieces have not
1602 changed?
1603
1604- How are changed pieces of software removed and replaced?
1605
1606- How are pre-built components that do not need to be rebuilt from
1607 scratch used when they are available?
1608
1609For the first question, the build system detects changes in the "inputs"
1610to a given task by creating a checksum (or signature) of the task's
1611inputs. If the checksum changes, the system assumes the inputs have
1612changed and the task needs to be rerun. For the second question, the
1613shared state (sstate) code tracks which tasks add which output to the
1614build process. This means the output from a given task can be removed,
1615upgraded or otherwise manipulated. The third question is partly
1616addressed by the solution for the second question assuming the build
1617system can fetch the sstate objects from remote locations and install
1618them if they are deemed to be valid.
1619
1620.. note::
1621
1622 - The build system does not maintain
1623 ```PR`` <&YOCTO_DOCS_REF_URL;#var-PR>`__ information as part of
1624 the shared state packages. Consequently, considerations exist that
1625 affect maintaining shared state feeds. For information on how the
1626 build system works with packages and can track incrementing ``PR``
1627 information, see the "`Automatically Incrementing a Binary Package
1628 Revision
1629 Number <&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number>`__"
1630 section in the Yocto Project Development Tasks Manual.
1631
1632 - The code in the build system that supports incremental builds is
1633 not simple code. For techniques that help you work around issues
1634 related to shared state code, see the "`Viewing Metadata Used to
1635 Create the Input Signature of a Shared State
1636 Task <&YOCTO_DOCS_DEV_URL;#dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task>`__"
1637 and "`Invalidating Shared State to Force a Task to
1638 Run <&YOCTO_DOCS_DEV_URL;#dev-invalidating-shared-state-to-force-a-task-to-run>`__"
1639 sections both in the Yocto Project Development Tasks Manual.
1640
1641The rest of this section goes into detail about the overall incremental
1642build architecture, the checksums (signatures), and shared state.
1643
1644.. _concepts-overall-architecture:
1645
1646Overall Architecture
1647--------------------
1648
1649When determining what parts of the system need to be built, BitBake
1650works on a per-task basis rather than a per-recipe basis. You might
1651wonder why using a per-task basis is preferred over a per-recipe basis.
1652To help explain, consider having the IPK packaging backend enabled and
1653then switching to DEB. In this case, the
1654```do_install`` <&YOCTO_DOCS_REF_URL;#ref-tasks-install>`__ and
1655```do_package`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package>`__ task outputs
1656are still valid. However, with a per-recipe approach, the build would
1657not include the ``.deb`` files. Consequently, you would have to
1658invalidate the whole build and rerun it. Rerunning everything is not the
1659best solution. Also, in this case, the core must be "taught" much about
1660specific tasks. This methodology does not scale well and does not allow
1661users to easily add new tasks in layers or as external recipes without
1662touching the packaged-staging core.
1663
1664.. _overview-checksums:
1665
1666Checksums (Signatures)
1667----------------------
1668
1669The shared state code uses a checksum, which is a unique signature of a
1670task's inputs, to determine if a task needs to be run again. Because it
1671is a change in a task's inputs that triggers a rerun, the process needs
1672to detect all the inputs to a given task. For shell tasks, this turns
1673out to be fairly easy because the build process generates a "run" shell
1674script for each task and it is possible to create a checksum that gives
1675you a good idea of when the task's data changes.
1676
1677To complicate the problem, there are things that should not be included
1678in the checksum. First, there is the actual specific build path of a
1679given task - the ```WORKDIR`` <&YOCTO_DOCS_REF_URL;#var-WORKDIR>`__. It
1680does not matter if the work directory changes because it should not
1681affect the output for target packages. Also, the build process has the
1682objective of making native or cross packages relocatable.
1683
1684.. note::
1685
1686 Both native and cross packages run on the
1687 build host
1688 . However, cross packages generate output for the target
1689 architecture.
1690
1691The checksum therefore needs to exclude ``WORKDIR``. The simplistic
1692approach for excluding the work directory is to set ``WORKDIR`` to some
1693fixed value and create the checksum for the "run" script.
1694
1695Another problem results from the "run" scripts containing functions that
1696might or might not get called. The incremental build solution contains
1697code that figures out dependencies between shell functions. This code is
1698used to prune the "run" scripts down to the minimum set, thereby
1699alleviating this problem and making the "run" scripts much more readable
1700as a bonus.
1701
1702So far, solutions for shell scripts exist. What about Python tasks? The
1703same approach applies even though these tasks are more difficult. The
1704process needs to figure out what variables a Python function accesses
1705and what functions it calls. Again, the incremental build solution
1706contains code that first figures out the variable and function
1707dependencies, and then creates a checksum for the data used as the input
1708to the task.
1709
1710Like the ``WORKDIR`` case, situations exist where dependencies should be
1711ignored. For these situations, you can instruct the build process to
1712ignore a dependency by using a line like the following:
1713PACKAGE_ARCHS[vardepsexclude] = "MACHINE" This example ensures that the
1714```PACKAGE_ARCHS`` <&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS>`__ variable
1715does not depend on the value of
1716```MACHINE`` <&YOCTO_DOCS_REF_URL;#var-MACHINE>`__, even if it does
1717reference it.
1718
1719Equally, there are cases where you need to add dependencies BitBake is
1720not able to find. You can accomplish this by using a line like the
1721following: PACKAGE_ARCHS[vardeps] = "MACHINE" This example explicitly
1722adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
1723
1724As an example, consider a case with in-line Python where BitBake is not
1725able to figure out dependencies. When running in debug mode (i.e. using
1726``-DDD``), BitBake produces output when it discovers something for which
1727it cannot figure out dependencies. The Yocto Project team has currently
1728not managed to cover those dependencies in detail and is aware of the
1729need to fix this situation.
1730
1731Thus far, this section has limited discussion to the direct inputs into
1732a task. Information based on direct inputs is referred to as the
1733"basehash" in the code. However, the question of a task's indirect
1734inputs still exits - items already built and present in the `Build
1735Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__. The checksum (or
1736signature) for a particular task needs to add the hashes of all the
1737tasks on which the particular task depends. Choosing which dependencies
1738to add is a policy decision. However, the effect is to generate a master
1739checksum that combines the basehash and the hashes of the task's
1740dependencies.
1741
1742At the code level, a variety of ways exist by which both the basehash
1743and the dependent task hashes can be influenced. Within the BitBake
1744configuration file, you can give BitBake some extra information to help
1745it construct the basehash. The following statement effectively results
1746in a list of global variable dependency excludes (i.e. variables never
1747included in any checksum): BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH
1748PWD BB_TASKHASH BBPATH DL_DIR \\ SSTATE_DIR THISDIR FILESEXTRAPATHS
1749FILE_DIRNAME HOME LOGNAME SHELL TERM \\ USER FILESPATH STAGING_DIR_HOST
1750STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ PRSERV_DUMPDIR
1751PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ CCACHE_DIR
1752EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" The
1753previous example excludes
1754```WORKDIR`` <&YOCTO_DOCS_REF_URL;#var-WORKDIR>`__ since that variable
1755is actually constructed as a path within
1756```TMPDIR`` <&YOCTO_DOCS_REF_URL;#var-TMPDIR>`__, which is on the
1757whitelist.
1758
1759The rules for deciding which hashes of dependent tasks to include
1760through dependency chains are more complex and are generally
1761accomplished with a Python function. The code in
1762``meta/lib/oe/sstatesig.py`` shows two examples of this and also
1763illustrates how you can insert your own policy into the system if so
1764desired. This file defines the two basic signature generators
1765`OE-Core <&YOCTO_DOCS_REF_URL;#oe-core>`__ uses: "OEBasic" and
1766"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1767in BitBake. This means that behavior is unchanged from previous
1768versions. OE-Core uses the "OEBasicHash" signature handler by default
1769through this setting in the ``bitbake.conf`` file: BB_SIGNATURE_HANDLER
1770?= "OEBasicHash" The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same
1771as the "OEBasic" version but adds the task hash to the `stamp
1772files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any
1773metadata change that changes the task hash, automatically causing the
1774task to be run again. This removes the need to bump
1775```PR`` <&YOCTO_DOCS_REF_URL;#var-PR>`__ values, and changes to metadata
1776automatically ripple across the build.
1777
1778It is also worth noting that the end result of these signature
1779generators is to make some dependency and hash information available to
1780the build. This information includes:
1781
1782- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the
1783 recipe.
1784
1785- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1786 dependent task.
1787
1788- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for
1789 each task.
1790
1791- ``BB_TASKHASH``: The hash of the currently running task.
1792
1793Shared State
1794------------
1795
1796Checksums and dependencies, as discussed in the previous section, solve
1797half the problem of supporting a shared state. The other half of the
1798problem is being able to use checksum information during the build and
1799being able to reuse or rebuild specific components.
1800
1801The ```sstate`` <&YOCTO_DOCS_REF_URL;#ref-classes-sstate>`__ class is a
1802relatively generic implementation of how to "capture" a snapshot of a
1803given task. The idea is that the build process does not care about the
1804source of a task's output. Output could be freshly built or it could be
1805downloaded and unpacked from somewhere. In other words, the build
1806process does not need to worry about its origin.
1807
1808Two types of output exist. One type is just about creating a directory
1809in ```WORKDIR`` <&YOCTO_DOCS_REF_URL;#var-WORKDIR>`__. A good example is
1810the output of either
1811```do_install`` <&YOCTO_DOCS_REF_URL;#ref-tasks-install>`__ or
1812```do_package`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package>`__. The other
1813type of output occurs when a set of data is merged into a shared
1814directory tree such as the sysroot.
1815
1816The Yocto Project team has tried to keep the details of the
1817implementation hidden in ``sstate`` class. From a user's perspective,
1818adding shared state wrapping to a task is as simple as this
1819```do_deploy`` <&YOCTO_DOCS_REF_URL;#ref-tasks-deploy>`__ example taken
1820from the ```deploy`` <&YOCTO_DOCS_REF_URL;#ref-classes-deploy>`__ class:
1821DEPLOYDIR = "${WORKDIR}/deploy-${PN}" SSTATETASKS += "do_deploy"
1822do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
1823do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" python
1824do_deploy_setscene () { sstate_setscene(d) } addtask do_deploy_setscene
1825do_deploy[dirs] = "${DEPLOYDIR} ${B}" do_deploy[stamp-extra-info] =
1826"${MACHINE_ARCH}" The following list explains the previous example:
1827
1828- Adding "do_deploy" to ``SSTATETASKS`` adds some required
1829 sstate-related processing, which is implemented in the
1830 ```sstate`` <&YOCTO_DOCS_REF_URL;#ref-classes-sstate>`__ class, to
1831 before and after the
1832 ```do_deploy`` <&YOCTO_DOCS_REF_URL;#ref-tasks-deploy>`__ task.
1833
1834- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1835 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally
1836 (i.e. when not using the sstate cache). This output becomes the input
1837 to the shared state cache.
1838
1839- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line
1840 causes the contents of the shared state cache to be copied to
1841 ``${DEPLOY_DIR_IMAGE}``.
1842
1843 .. note::
1844
1845 If
1846 do_deploy
1847 is not already in the shared state cache or if its input checksum
1848 (signature) has changed from when the output was cached, the task
1849 runs to populate the shared state cache, after which the contents
1850 of the shared state cache is copied to
1851 ${DEPLOY_DIR_IMAGE}
1852 . If
1853 do_deploy
1854 is in the shared state cache and its signature indicates that the
1855 cached output is still valid (i.e. if no relevant task inputs have
1856 changed), then the contents of the shared state cache copies
1857 directly to
1858 ${DEPLOY_DIR_IMAGE}
1859 by the
1860 do_deploy_setscene
1861 task instead, skipping the
1862 do_deploy
1863 task.
1864
1865- The following task definition is glue logic needed to make the
1866 previous settings effective: python do_deploy_setscene () {
1867 sstate_setscene(d) } addtask do_deploy_setscene ``sstate_setscene()``
1868 takes the flags above as input and accelerates the ``do_deploy`` task
1869 through the shared state cache if possible. If the task was
1870 accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1871 returns False, and the normal ``do_deploy`` task runs. For more
1872 information, see the "`setscene <&YOCTO_DOCS_BB_URL;#setscene>`__"
1873 section in the BitBake User Manual.
1874
1875- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates
1876 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and
1877 also sets the current working directory of ``do_deploy`` to ``${B}``.
1878 For more information, see the "`Variable
1879 Flags <&YOCTO_DOCS_BB_URL;#variable-flags>`__" section in the BitBake
1880 User Manual.
1881
1882 .. note::
1883
1884 In cases where
1885 sstate-inputdirs
1886 and
1887 sstate-outputdirs
1888 would be the same, you can use
1889 sstate-plaindirs
1890 . For example, to preserve the
1891 ${PKGD}
1892 and
1893 ${PKGDEST}
1894 output from the
1895 do_package
1896 task, use the following:
1897 ::
1898
1899 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1900
1901
1902- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends
1903 extra metadata to the `stamp
1904 file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the
1905 metadata makes the task specific to a machine's architecture. See
1906 "`The Task List <&YOCTO_DOCS_BB_URL;#ref-bitbake-tasklist>`__"
1907 section in the BitBake User Manual for more information on the
1908 ``stamp-extra-info`` flag.
1909
1910- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1911 multiple directories. For example, the following declares
1912 ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories,
1913 which populates the shared state cache, and ``PKGDATA_DIR`` and
1914 ``SHLIBSDIR`` as the corresponding shared state output directories:
1915 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1916 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1917
1918- These methods also include the ability to take a lockfile when
1919 manipulating shared state directory structures, for cases where file
1920 additions or removals are sensitive: do_package[sstate-lockfile] =
1921 "${PACKAGELOCK}"
1922
1923Behind the scenes, the shared state code works by looking in
1924```SSTATE_DIR`` <&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR>`__ and
1925```SSTATE_MIRRORS`` <&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS>`__ for
1926shared state files. Here is an example: SSTATE_MIRRORS ?= "\\ file://.\*
1927http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \\n \\
1928file://.\* file:///some/local/dir/sstate/PATH"
1929
1930.. note::
1931
1932 The shared state directory (
1933 SSTATE_DIR
1934 ) is organized into two-character subdirectories, where the
1935 subdirectory names are based on the first two characters of the hash.
1936 If the shared state directory structure for a mirror has the same
1937 structure as
1938 SSTATE_DIR
1939 , you must specify "PATH" as part of the URI to enable the build
1940 system to map to the appropriate subdirectory.
1941
1942The shared state package validity can be detected just by looking at the
1943filename since the filename contains the task checksum (or signature) as
1944described earlier in this section. If a valid shared state package is
1945found, the build process downloads it and uses it to accelerate the
1946task.
1947
1948The build processes use the ``*_setscene`` tasks for the task
1949acceleration phase. BitBake goes through this phase before the main
1950execution code and tries to accelerate any tasks for which it can find
1951shared state packages. If a shared state package for a task is
1952available, the shared state package is used. This means the task and any
1953tasks on which it is dependent are not executed.
1954
1955As a real world example, the aim is when building an IPK-based image,
1956only the
1957```do_package_write_ipk`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk>`__
1958tasks would have their shared state packages fetched and extracted.
1959Since the sysroot is not used, it would never get extracted. This is
1960another reason why a task-based approach is preferred over a
1961recipe-based approach, which would have to install the output from every
1962task.
1963
1964Automatically Added Runtime Dependencies
1965========================================
1966
1967The OpenEmbedded build system automatically adds common types of runtime
1968dependencies between packages, which means that you do not need to
1969explicitly declare the packages using
1970```RDEPENDS`` <&YOCTO_DOCS_REF_URL;#var-RDEPENDS>`__. Three automatic
1971mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that
1972handle shared libraries, package configuration (pkg-config) modules, and
1973``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
1974dependencies, you must manually declare the dependencies.
1975
1976- ``shlibdeps``: During the
1977 ```do_package`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package>`__ task of
1978 each recipe, all shared libraries installed by the recipe are
1979 located. For each shared library, the package that contains the
1980 shared library is registered as providing the shared library. More
1981 specifically, the package is registered as providing the
1982 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The
1983 resulting shared-library-to-package mapping is saved globally in
1984 ```PKGDATA_DIR`` <&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR>`__ by the
1985 ```do_packagedata`` <&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata>`__
1986 task.
1987
1988 Simultaneously, all executables and shared libraries installed by the
1989 recipe are inspected to see what shared libraries they link against.
1990 For each shared library dependency that is found, ``PKGDATA_DIR`` is
1991 queried to see if some package (likely from a different recipe)
1992 contains the shared library. If such a package is found, a runtime
1993 dependency is added from the package that depends on the shared
1994 library to the package that contains the library.
1995
1996 The automatically added runtime dependency also includes a version
1997 restriction. This version restriction specifies that at least the
1998 current version of the package that provides the shared library must
1999 be used, as if "package (>= version)" had been added to ``RDEPENDS``.
2000 This forces an upgrade of the package containing the shared library
2001 when installing the package that depends on the library, if needed.
2002
2003 If you want to avoid a package being registered as providing a
2004 particular shared library (e.g. because the library is for internal
2005 use only), then add the library to
2006 ```PRIVATE_LIBS`` <&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS>`__ inside
2007 the package's recipe.
2008
2009- ``pcdeps``: During the ``do_package`` task of each recipe, all
2010 pkg-config modules (``*.pc`` files) installed by the recipe are
2011 located. For each module, the package that contains the module is
2012 registered as providing the module. The resulting module-to-package
2013 mapping is saved globally in ``PKGDATA_DIR`` by the
2014 ``do_packagedata`` task.
2015
2016 Simultaneously, all pkg-config modules installed by the recipe are
2017 inspected to see what other pkg-config modules they depend on. A
2018 module is seen as depending on another module if it contains a
2019 "Requires:" line that specifies the other module. For each module
2020 dependency, ``PKGDATA_DIR`` is queried to see if some package
2021 contains the module. If such a package is found, a runtime dependency
2022 is added from the package that depends on the module to the package
2023 that contains the module.
2024
2025 .. note::
2026
2027 The
2028 pcdeps
2029 mechanism most often infers dependencies between
2030 -dev
2031 packages.
2032
2033- ``depchains``: If a package ``foo`` depends on a package ``bar``,
2034 then ``foo-dev`` and ``foo-dbg`` are also made to depend on
2035 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev``
2036 packages as an example, the ``bar-dev`` package might provide headers
2037 and shared library symlinks needed by ``foo-dev``, which shows the
2038 need for a dependency between the packages.
2039
2040 The dependencies added by ``depchains`` are in the form of
2041 ```RRECOMMENDS`` <&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS>`__.
2042
2043 .. note::
2044
2045 By default,
2046 foo-dev
2047 also has an
2048 RDEPENDS
2049 -style dependency on
2050 foo
2051 , because the default value of
2052 RDEPENDS_${PN}-dev
2053 (set in
2054 bitbake.conf
2055 ) includes "${PN}".
2056
2057 To ensure that the dependency chain is never broken, ``-dev`` and
2058 ``-dbg`` packages are always generated by default, even if the
2059 packages turn out to be empty. See the
2060 ```ALLOW_EMPTY`` <&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY>`__ variable
2061 for more information.
2062
2063The ``do_package`` task depends on the ``do_packagedata`` task of each
2064recipe in ```DEPENDS`` <&YOCTO_DOCS_REF_URL;#var-DEPENDS>`__ through use
2065of a ``[``\ ```deptask`` <&YOCTO_DOCS_BB_URL;#variable-flags>`__\ ``]``
2066declaration, which guarantees that the required
2067shared-library/module-to-package mapping information will be available
2068when needed as long as ``DEPENDS`` has been correctly set.
2069
2070Fakeroot and Pseudo
2071===================
2072
2073Some tasks are easier to implement when allowed to perform certain
2074operations that are normally reserved for the root user (e.g.
2075```do_install`` <&YOCTO_DOCS_REF_URL;#ref-tasks-install>`__,
2076```do_package_write*`` <&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb>`__,
2077```do_rootfs`` <&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs>`__, and
2078```do_image*`` <&YOCTO_DOCS_REF_URL;#ref-tasks-image>`__). For example,
2079the ``do_install`` task benefits from being able to set the UID and GID
2080of installed files to arbitrary values.
2081
2082One approach to allowing tasks to perform root-only operations would be
2083to require `BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__ to run as
2084root. However, this method is cumbersome and has security issues. The
2085approach that is actually used is to run tasks that benefit from root
2086privileges in a "fake" root environment. Within this environment, the
2087task and its child processes believe that they are running as the root
2088user, and see an internally consistent view of the filesystem. As long
2089as generating the final output (e.g. a package or an image) does not
2090require root privileges, the fact that some earlier steps ran in a fake
2091root environment does not cause problems.
2092
2093The capability to run tasks in a fake root environment is known as
2094"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from
2095the BitBake keyword/variable flag that requests a fake root environment
2096for a task.
2097
2098In the `OpenEmbedded build
2099system <&YOCTO_DOCS_REF_URL;#build-system-term>`__, the program that
2100implements fakeroot is known as
2101`Pseudo <https://www.yoctoproject.org/software-item/pseudo/>`__. Pseudo
2102overrides system calls by using the environment variable ``LD_PRELOAD``,
2103which results in the illusion of running as root. To keep track of
2104"fake" file ownership and permissions resulting from operations that
2105require root permissions, Pseudo uses an SQLite 3 database. This
2106database is stored in
2107``${``\ ```WORKDIR`` <&YOCTO_DOCS_REF_URL;#var-WORKDIR>`__\ ``}/pseudo/files.db``
2108for individual recipes. Storing the database in a file as opposed to in
2109memory gives persistence between tasks and builds, which is not
2110accomplished using fakeroot.
2111
2112.. note::
2113
2114 If you add your own task that manipulates the same files or
2115 directories as a fakeroot task, then that task also needs to run
2116 under fakeroot. Otherwise, the task cannot run root-only operations,
2117 and cannot see the fake file ownership and permissions set by the
2118 other task. You need to also add a dependency on
2119 virtual/fakeroot-native:do_populate_sysroot
2120 , giving the following:
2121 ::
2122
2123 fakeroot do_mytask () {
2124 ...
2125 }
2126 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
2127
2128
2129For more information, see the
2130```FAKEROOT*`` <&YOCTO_DOCS_BB_URL;#var-FAKEROOT>`__ variables in the
2131BitBake User Manual. You can also reference the "`Why Not
2132Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2133article for background information on Fakeroot and Pseudo.
diff --git a/documentation/overview-manual/overview-manual-development-environment.rst b/documentation/overview-manual/overview-manual-development-environment.rst
new file mode 100644
index 0000000000..4e6770c4f4
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-development-environment.rst
@@ -0,0 +1,656 @@
1*****************************************
2The Yocto Project Development Environment
3*****************************************
4
5This chapter takes a look at the Yocto Project development environment.
6The chapter provides Yocto Project Development environment concepts that
7help you understand how work is accomplished in an open source
8environment, which is very different as compared to work accomplished in
9a closed, proprietary environment.
10
11Specifically, this chapter addresses open source philosophy, source
12repositories, workflows, Git, and licensing.
13
14Open Source Philosophy
15======================
16
17Open source philosophy is characterized by software development directed
18by peer production and collaboration through an active community of
19developers. Contrast this to the more standard centralized development
20models used by commercial software companies where a finite set of
21developers produces a product for sale using a defined set of procedures
22that ultimately result in an end product whose architecture and source
23material are closed to the public.
24
25Open source projects conceptually have differing concurrent agendas,
26approaches, and production. These facets of the development process can
27come from anyone in the public (community) who has a stake in the
28software project. The open source environment contains new copyright,
29licensing, domain, and consumer issues that differ from the more
30traditional development environment. In an open source environment, the
31end product, source material, and documentation are all available to the
32public at no cost.
33
34A benchmark example of an open source project is the Linux kernel, which
35was initially conceived and created by Finnish computer science student
36Linus Torvalds in 1991. Conversely, a good example of a non-open source
37project is the Windows family of operating systems developed by
38Microsoft Corporation.
39
40Wikipedia has a good historical description of the Open Source
41Philosophy `here <http://en.wikipedia.org/wiki/Open_source>`__. You can
42also find helpful information on how to participate in the Linux
43Community
44`here <http://ldn.linuxfoundation.org/book/how-participate-linux-community>`__.
45
46.. _gs-the-development-host:
47
48The Development Host
49====================
50
51A development host or `build
52host <&YOCTO_DOCS_REF_URL;#hardware-build-system-term>`__ is key to
53using the Yocto Project. Because the goal of the Yocto Project is to
54develop images or applications that run on embedded hardware,
55development of those images and applications generally takes place on a
56system not intended to run the software - the development host.
57
58You need to set up a development host in order to use it with the Yocto
59Project. Most find that it is best to have a native Linux machine
60function as the development host. However, it is possible to use a
61system that does not run Linux as its operating system as your
62development host. When you have a Mac or Windows-based system, you can
63set it up as the development host by using
64`CROPS <https://github.com/crops/poky-container>`__, which leverages
65`Docker Containers <https://www.docker.com/>`__. Once you take the steps
66to set up a CROPS machine, you effectively have access to a shell
67environment that is similar to what you see when using a Linux-based
68development host. For the steps needed to set up a system using CROPS,
69see the "`Setting Up to Use CROss PlatformS
70(CROPS) <&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops>`__" section in
71the Yocto Project Development Tasks Manual.
72
73If your development host is going to be a system that runs a Linux
74distribution, steps still exist that you must take to prepare the system
75for use with the Yocto Project. You need to be sure that the Linux
76distribution on the system is one that supports the Yocto Project. You
77also need to be sure that the correct set of host packages are installed
78that allow development using the Yocto Project. For the steps needed to
79set up a development host that runs Linux, see the "`Setting Up a Native
80Linux Host <&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host>`__"
81section in the Yocto Project Development Tasks Manual.
82
83Once your development host is set up to use the Yocto Project, several
84methods exist for you to do work in the Yocto Project environment:
85
86- *Command Lines, BitBake, and Shells:* Traditional development in the
87 Yocto Project involves using the `OpenEmbedded build
88 system <&YOCTO_DOCS_REF_URL;#build-system-term>`__, which uses
89 BitBake, in a command-line environment from a shell on your
90 development host. You can accomplish this from a host that is a
91 native Linux machine or from a host that has been set up with CROPS.
92 Either way, you create, modify, and build images and applications all
93 within a shell-based environment using components and tools available
94 through your Linux distribution and the Yocto Project.
95
96 For a general flow of the build procedures, see the "`Building a
97 Simple Image <&YOCTO_DOCS_DEV_URL;#dev-building-a-simple-image>`__"
98 section in the Yocto Project Development Tasks Manual.
99
100- *Board Support Package (BSP) Development:* Development of BSPs
101 involves using the Yocto Project to create and test layers that allow
102 easy development of images and applications targeted for specific
103 hardware. To development BSPs, you need to take some additional steps
104 beyond what was described in setting up a development host.
105
106 The `Yocto Project Board Support Package (BSP) Developer's
107 Guide <&YOCTO_DOCS_BSP_URL;>`__ provides BSP-related development
108 information. For specifics on development host preparation, see the
109 "`Preparing Your Build Host to Work With BSP
110 Layers <&YOCTO_DOCS_BSP_URL;#preparing-your-build-host-to-work-with-bsp-layers>`__"
111 section in the Yocto Project Board Support Package (BSP) Developer's
112 Guide.
113
114- *Kernel Development:* If you are going to be developing kernels using
115 the Yocto Project you likely will be using ``devtool``. A workflow
116 using ``devtool`` makes kernel development quicker by reducing
117 iteration cycle times.
118
119 The `Yocto Project Linux Kernel Development
120 Manual <&YOCTO_DOCS_KERNEL_DEV_URL;>`__ provides kernel-related
121 development information. For specifics on development host
122 preparation, see the "`Preparing the Build Host to Work on the
123 Kernel <&YOCTO_DOCS_KERNEL_DEV_URL;#preparing-the-build-host-to-work-on-the-kernel>`__"
124 section in the Yocto Project Linux Kernel Development Manual.
125
126- *Using Toaster:* The other Yocto Project development method that
127 involves an interface that effectively puts the Yocto Project into
128 the background is Toaster. Toaster provides an interface to the
129 OpenEmbedded build system. The interface enables you to configure and
130 run your builds. Information about builds is collected and stored in
131 a database. You can use Toaster to configure and start builds on
132 multiple remote build servers.
133
134 For steps that show you how to set up your development host to use
135 Toaster and on how to use Toaster in general, see the `Toaster User
136 Manual <&YOCTO_DOCS_TOAST_URL;>`__.
137
138.. _yocto-project-repositories:
139
140Yocto Project Source Repositories
141=================================
142
143The Yocto Project team maintains complete source repositories for all
144Yocto Project files at ` <&YOCTO_GIT_URL;>`__. This web-based source
145code browser is organized into categories by function such as IDE
146Plugins, Matchbox, Poky, Yocto Linux Kernel, and so forth. From the
147interface, you can click on any particular item in the "Name" column and
148see the URL at the bottom of the page that you need to clone a Git
149repository for that particular item. Having a local Git repository of
150the `Source Directory <&YOCTO_DOCS_REF_URL;#source-directory>`__, which
151is usually named "poky", allows you to make changes, contribute to the
152history, and ultimately enhance the Yocto Project's tools, Board Support
153Packages, and so forth.
154
155For any supported release of Yocto Project, you can also go to the
156`Yocto Project Website <&YOCTO_HOME_URL;>`__ and select the "DOWNLOADS"
157item from the "SOFTWARE" menu and get a released tarball of the ``poky``
158repository, any supported BSP tarball, or Yocto Project tools. Unpacking
159these tarballs gives you a snapshot of the released files.
160
161.. note::
162
163 - The recommended method for setting up the Yocto Project `Source
164 Directory <&YOCTO_DOCS_REF_URL;#source-directory>`__ and the files
165 for supported BSPs (e.g., ``meta-intel``) is to use `Git <#git>`__
166 to create a local copy of the upstream repositories.
167
168 - Be sure to always work in matching branches for both the selected
169 BSP repository and the Source Directory (i.e. ``poky``)
170 repository. For example, if you have checked out the "master"
171 branch of ``poky`` and you are going to use ``meta-intel``, be
172 sure to checkout the "master" branch of ``meta-intel``.
173
174In summary, here is where you can get the project files needed for
175development:
176
177- `Source Repositories: <&YOCTO_GIT_URL;>`__ This area contains IDE
178 Plugins, Matchbox, Poky, Poky Support, Tools, Yocto Linux Kernel, and
179 Yocto Metadata Layers. You can create local copies of Git
180 repositories for each of these areas.
181
182 For steps on how to view and access these upstream Git repositories,
183 see the "`Accessing Source
184 Repositories <&YOCTO_DOCS_DEV_URL;#accessing-source-repositories>`__"
185 Section in the Yocto Project Development Tasks Manual.
186
187- `Index of /releases: <&YOCTO_DL_URL;/releases/>`__ This is an index
188 of releases such as Poky, Pseudo, installers for cross-development
189 toolchains, miscellaneous support and all released versions of Yocto
190 Project in the form of images or tarballs. Downloading and extracting
191 these files does not produce a local copy of the Git repository but
192 rather a snapshot of a particular release or image.
193
194 For steps on how to view and access these files, see the "`Accessing
195 Index of
196 Releases <&YOCTO_DOCS_DEV_URL;#accessing-index-of-releases>`__"
197 section in the Yocto Project Development Tasks Manual.
198
199- *"DOWNLOADS" page for the*\ `Yocto Project
200 Website <&YOCTO_HOME_URL;>`__\ *:*
201
202 The Yocto Project website includes a "DOWNLOADS" page accessible
203 through the "SOFTWARE" menu that allows you to download any Yocto
204 Project release, tool, and Board Support Package (BSP) in tarball
205 form. The tarballs are similar to those found in the `Index of
206 /releases: <&YOCTO_DL_URL;/releases/>`__ area.
207
208 For steps on how to use the "DOWNLOADS" page, see the "`Using the
209 Downloads Page <&YOCTO_DOCS_DEV_URL;#using-the-downloads-page>`__"
210 section in the Yocto Project Development Tasks Manual.
211
212.. _gs-git-workflows-and-the-yocto-project:
213
214Git Workflows and the Yocto Project
215===================================
216
217Developing using the Yocto Project likely requires the use of
218`Git <#git>`__. Git is a free, open source distributed version control
219system used as part of many collaborative design environments. This
220section provides workflow concepts using the Yocto Project and Git. In
221particular, the information covers basic practices that describe roles
222and actions in a collaborative development environment.
223
224.. note::
225
226 If you are familiar with this type of development environment, you
227 might not want to read this section.
228
229The Yocto Project files are maintained using Git in "branches" whose Git
230histories track every change and whose structures provide branches for
231all diverging functionality. Although there is no need to use Git, many
232open source projects do so.
233
234For the Yocto Project, a key individual called the "maintainer" is
235responsible for the integrity of the "master" branch of a given Git
236repository. The "master" branch is the “upstream” repository from which
237final or most recent builds of a project occur. The maintainer is
238responsible for accepting changes from other developers and for
239organizing the underlying branch structure to reflect release strategies
240and so forth.
241
242.. note::
243
244 For information on finding out who is responsible for (maintains) a
245 particular area of code in the Yocto Project, see the "
246 Submitting a Change to the Yocto Project
247 " section of the Yocto Project Development Tasks Manual.
248
249The Yocto Project ``poky`` Git repository also has an upstream
250contribution Git repository named ``poky-contrib``. You can see all the
251branches in this repository using the web interface of the `Source
252Repositories <&YOCTO_GIT_URL;>`__ organized within the "Poky Support"
253area. These branches hold changes (commits) to the project that have
254been submitted or committed by the Yocto Project development team and by
255community members who contribute to the project. The maintainer
256determines if the changes are qualified to be moved from the "contrib"
257branches into the "master" branch of the Git repository.
258
259Developers (including contributing community members) create and
260maintain cloned repositories of upstream branches. The cloned
261repositories are local to their development platforms and are used to
262develop changes. When a developer is satisfied with a particular feature
263or change, they "push" the change to the appropriate "contrib"
264repository.
265
266Developers are responsible for keeping their local repository up-to-date
267with whatever upstream branch they are working against. They are also
268responsible for straightening out any conflicts that might arise within
269files that are being worked on simultaneously by more than one person.
270All this work is done locally on the development host before anything is
271pushed to a "contrib" area and examined at the maintainer’s level.
272
273A somewhat formal method exists by which developers commit changes and
274push them into the "contrib" area and subsequently request that the
275maintainer include them into an upstream branch. This process is called
276“submitting a patch” or "submitting a change." For information on
277submitting patches and changes, see the "`Submitting a Change to the
278Yocto Project <&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change>`__" section
279in the Yocto Project Development Tasks Manual.
280
281In summary, a single point of entry exists for changes into a "master"
282or development branch of the Git repository, which is controlled by the
283project’s maintainer. And, a set of developers exist who independently
284develop, test, and submit changes to "contrib" areas for the maintainer
285to examine. The maintainer then chooses which changes are going to
286become a permanent part of the project.
287
288While each development environment is unique, there are some best
289practices or methods that help development run smoothly. The following
290list describes some of these practices. For more information about Git
291workflows, see the workflow topics in the `Git Community
292Book <http://book.git-scm.com>`__.
293
294- *Make Small Changes:* It is best to keep the changes you commit small
295 as compared to bundling many disparate changes into a single commit.
296 This practice not only keeps things manageable but also allows the
297 maintainer to more easily include or refuse changes.
298
299- *Make Complete Changes:* It is also good practice to leave the
300 repository in a state that allows you to still successfully build
301 your project. In other words, do not commit half of a feature, then
302 add the other half as a separate, later commit. Each commit should
303 take you from one buildable project state to another buildable state.
304
305- *Use Branches Liberally:* It is very easy to create, use, and delete
306 local branches in your working Git repository on the development
307 host. You can name these branches anything you like. It is helpful to
308 give them names associated with the particular feature or change on
309 which you are working. Once you are done with a feature or change and
310 have merged it into your local master branch, simply discard the
311 temporary branch.
312
313- *Merge Changes:* The ``git merge`` command allows you to take the
314 changes from one branch and fold them into another branch. This
315 process is especially helpful when more than a single developer might
316 be working on different parts of the same feature. Merging changes
317 also automatically identifies any collisions or "conflicts" that
318 might happen as a result of the same lines of code being altered by
319 two different developers.
320
321- *Manage Branches:* Because branches are easy to use, you should use a
322 system where branches indicate varying levels of code readiness. For
323 example, you can have a "work" branch to develop in, a "test" branch
324 where the code or change is tested, a "stage" branch where changes
325 are ready to be committed, and so forth. As your project develops,
326 you can merge code across the branches to reflect ever-increasing
327 stable states of the development.
328
329- *Use Push and Pull:* The push-pull workflow is based on the concept
330 of developers "pushing" local commits to a remote repository, which
331 is usually a contribution repository. This workflow is also based on
332 developers "pulling" known states of the project down into their
333 local development repositories. The workflow easily allows you to
334 pull changes submitted by other developers from the upstream
335 repository into your work area ensuring that you have the most recent
336 software on which to develop. The Yocto Project has two scripts named
337 ``create-pull-request`` and ``send-pull-request`` that ship with the
338 release to facilitate this workflow. You can find these scripts in
339 the ``scripts`` folder of the `Source
340 Directory <&YOCTO_DOCS_REF_URL;#source-directory>`__. For information
341 on how to use these scripts, see the "`Using Scripts to Push a Change
342 Upstream and Request a
343 Pull <&YOCTO_DOCS_DEV_URL;#pushing-a-change-upstream>`__" section in
344 the Yocto Project Development Tasks Manual.
345
346- *Patch Workflow:* This workflow allows you to notify the maintainer
347 through an email that you have a change (or patch) you would like
348 considered for the "master" branch of the Git repository. To send
349 this type of change, you format the patch and then send the email
350 using the Git commands ``git format-patch`` and ``git send-email``.
351 For information on how to use these scripts, see the "`Submitting a
352 Change to the Yocto
353 Project <&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change>`__" section in
354 the Yocto Project Development Tasks Manual.
355
356Git
357===
358
359The Yocto Project makes extensive use of Git, which is a free, open
360source distributed version control system. Git supports distributed
361development, non-linear development, and can handle large projects. It
362is best that you have some fundamental understanding of how Git tracks
363projects and how to work with Git if you are going to use the Yocto
364Project for development. This section provides a quick overview of how
365Git works and provides you with a summary of some essential Git
366commands.
367
368.. note::
369
370 - For more information on Git, see
371 ` <http://git-scm.com/documentation>`__.
372
373 - If you need to download Git, it is recommended that you add Git to
374 your system through your distribution's "software store" (e.g. for
375 Ubuntu, use the Ubuntu Software feature). For the Git download
376 page, see ` <http://git-scm.com/download>`__.
377
378 - For information beyond the introductory nature in this section,
379 see the "`Locating Yocto Project Source
380 Files <&YOCTO_DOCS_DEV_URL;#locating-yocto-project-source-files>`__"
381 section in the Yocto Project Development Tasks Manual.
382
383Repositories, Tags, and Branches
384--------------------------------
385
386As mentioned briefly in the previous section and also in the "`Git
387Workflows and the Yocto
388Project <#gs-git-workflows-and-the-yocto-project>`__" section, the Yocto
389Project maintains source repositories at ` <&YOCTO_GIT_URL;>`__. If you
390look at this web-interface of the repositories, each item is a separate
391Git repository.
392
393Git repositories use branching techniques that track content change (not
394files) within a project (e.g. a new feature or updated documentation).
395Creating a tree-like structure based on project divergence allows for
396excellent historical information over the life of a project. This
397methodology also allows for an environment from which you can do lots of
398local experimentation on projects as you develop changes or new
399features.
400
401A Git repository represents all development efforts for a given project.
402For example, the Git repository ``poky`` contains all changes and
403developments for that repository over the course of its entire life.
404That means that all changes that make up all releases are captured. The
405repository maintains a complete history of changes.
406
407You can create a local copy of any repository by "cloning" it with the
408``git clone`` command. When you clone a Git repository, you end up with
409an identical copy of the repository on your development system. Once you
410have a local copy of a repository, you can take steps to develop
411locally. For examples on how to clone Git repositories, see the
412"`Locating Yocto Project Source
413Files <&YOCTO_DOCS_DEV_URL;#locating-yocto-project-source-files>`__"
414section in the Yocto Project Development Tasks Manual.
415
416It is important to understand that Git tracks content change and not
417files. Git uses "branches" to organize different development efforts.
418For example, the ``poky`` repository has several branches that include
419the current "DISTRO_NAME_NO_CAP" branch, the "master" branch, and many
420branches for past Yocto Project releases. You can see all the branches
421by going to ` <&YOCTO_GIT_URL;/cgit.cgi/poky/>`__ and clicking on the
422``[...]`` link beneath the "Branch" heading.
423
424Each of these branches represents a specific area of development. The
425"master" branch represents the current or most recent development. All
426other branches represent offshoots of the "master" branch.
427
428When you create a local copy of a Git repository, the copy has the same
429set of branches as the original. This means you can use Git to create a
430local working area (also called a branch) that tracks a specific
431development branch from the upstream source Git repository. in other
432words, you can define your local Git environment to work on any
433development branch in the repository. To help illustrate, consider the
434following example Git commands: $ cd ~ $ git clone
435git://git.yoctoproject.org/poky $ cd poky $ git checkout -b
436DISTRO_NAME_NO_CAP origin/DISTRO_NAME_NO_CAP In the previous example
437after moving to the home directory, the ``git clone`` command creates a
438local copy of the upstream ``poky`` Git repository. By default, Git
439checks out the "master" branch for your work. After changing the working
440directory to the new local repository (i.e. ``poky``), the
441``git checkout`` command creates and checks out a local branch named
442"DISTRO_NAME_NO_CAP", which tracks the upstream
443"origin/DISTRO_NAME_NO_CAP" branch. Changes you make while in this
444branch would ultimately affect the upstream "DISTRO_NAME_NO_CAP" branch
445of the ``poky`` repository.
446
447It is important to understand that when you create and checkout a local
448working branch based on a branch name, your local environment matches
449the "tip" of that particular development branch at the time you created
450your local branch, which could be different from the files in the
451"master" branch of the upstream repository. In other words, creating and
452checking out a local branch based on the "DISTRO_NAME_NO_CAP" branch
453name is not the same as checking out the "master" branch in the
454repository. Keep reading to see how you create a local snapshot of a
455Yocto Project Release.
456
457Git uses "tags" to mark specific changes in a repository branch
458structure. Typically, a tag is used to mark a special point such as the
459final change (or commit) before a project is released. You can see the
460tags used with the ``poky`` Git repository by going to
461` <&YOCTO_GIT_URL;/cgit.cgi/poky/>`__ and clicking on the ``[...]`` link
462beneath the "Tag" heading.
463
464Some key tags for the ``poky`` repository are ``jethro-14.0.3``,
465``morty-16.0.1``, ``pyro-17.0.0``, and
466``DISTRO_NAME_NO_CAP-POKYVERSION``. These tags represent Yocto Project
467releases.
468
469When you create a local copy of the Git repository, you also have access
470to all the tags in the upstream repository. Similar to branches, you can
471create and checkout a local working Git branch based on a tag name. When
472you do this, you get a snapshot of the Git repository that reflects the
473state of the files when the change was made associated with that tag.
474The most common use is to checkout a working branch that matches a
475specific Yocto Project release. Here is an example: $ cd ~ $ git clone
476git://git.yoctoproject.org/poky $ cd poky $ git fetch --tags $ git
477checkout tags/rocko-18.0.0 -b my_rocko-18.0.0 In this example, the name
478of the top-level directory of your local Yocto Project repository is
479``poky``. After moving to the ``poky`` directory, the ``git fetch``
480command makes all the upstream tags available locally in your
481repository. Finally, the ``git checkout`` command creates and checks out
482a branch named "my-rocko-18.0.0" that is based on the upstream branch
483whose "HEAD" matches the commit in the repository associated with the
484"rocko-18.0.0" tag. The files in your repository now exactly match that
485particular Yocto Project release as it is tagged in the upstream Git
486repository. It is important to understand that when you create and
487checkout a local working branch based on a tag, your environment matches
488a specific point in time and not the entire development branch (i.e.
489from the "tip" of the branch backwards).
490
491Basic Commands
492--------------
493
494Git has an extensive set of commands that lets you manage changes and
495perform collaboration over the life of a project. Conveniently though,
496you can manage with a small set of basic operations and workflows once
497you understand the basic philosophy behind Git. You do not have to be an
498expert in Git to be functional. A good place to look for instruction on
499a minimal set of Git commands is
500`here <http://git-scm.com/documentation>`__.
501
502The following list of Git commands briefly describes some basic Git
503operations as a way to get started. As with any set of commands, this
504list (in most cases) simply shows the base command and omits the many
505arguments it supports. See the Git documentation for complete
506descriptions and strategies on how to use these commands:
507
508- *``git init``:* Initializes an empty Git repository. You cannot use
509 Git commands unless you have a ``.git`` repository.
510
511- *``git clone``:* Creates a local clone of a Git repository that is on
512 equal footing with a fellow developer’s Git repository or an upstream
513 repository.
514
515- *``git add``:* Locally stages updated file contents to the index that
516 Git uses to track changes. You must stage all files that have changed
517 before you can commit them.
518
519- *``git commit``:* Creates a local "commit" that documents the changes
520 you made. Only changes that have been staged can be committed.
521 Commits are used for historical purposes, for determining if a
522 maintainer of a project will allow the change, and for ultimately
523 pushing the change from your local Git repository into the project’s
524 upstream repository.
525
526- *``git status``:* Reports any modified files that possibly need to be
527 staged and gives you a status of where you stand regarding local
528 commits as compared to the upstream repository.
529
530- *``git checkout`` branch-name:* Changes your local working branch and
531 in this form assumes the local branch already exists. This command is
532 analogous to "cd".
533
534- *``git checkout –b`` working-branch upstream-branch:* Creates and
535 checks out a working branch on your local machine. The local branch
536 tracks the upstream branch. You can use your local branch to isolate
537 your work. It is a good idea to use local branches when adding
538 specific features or changes. Using isolated branches facilitates
539 easy removal of changes if they do not work out.
540
541- *``git branch``:* Displays the existing local branches associated
542 with your local repository. The branch that you have currently
543 checked out is noted with an asterisk character.
544
545- *``git branch -D`` branch-name:* Deletes an existing local branch.
546 You need to be in a local branch other than the one you are deleting
547 in order to delete branch-name.
548
549- *``git pull --rebase``:* Retrieves information from an upstream Git
550 repository and places it in your local Git repository. You use this
551 command to make sure you are synchronized with the repository from
552 which you are basing changes (.e.g. the "master" branch). The
553 "--rebase" option ensures that any local commits you have in your
554 branch are preserved at the top of your local branch.
555
556- *``git push`` repo-name local-branch\ ``:``\ upstream-branch:* Sends
557 all your committed local changes to the upstream Git repository that
558 your local repository is tracking (e.g. a contribution repository).
559 The maintainer of the project draws from these repositories to merge
560 changes (commits) into the appropriate branch of project's upstream
561 repository.
562
563- *``git merge``:* Combines or adds changes from one local branch of
564 your repository with another branch. When you create a local Git
565 repository, the default branch is named "master". A typical workflow
566 is to create a temporary branch that is based off "master" that you
567 would use for isolated work. You would make your changes in that
568 isolated branch, stage and commit them locally, switch to the
569 "master" branch, and then use the ``git merge`` command to apply the
570 changes from your isolated branch into the currently checked out
571 branch (e.g. "master"). After the merge is complete and if you are
572 done with working in that isolated branch, you can safely delete the
573 isolated branch.
574
575- *``git cherry-pick`` commits:* Choose and apply specific commits from
576 one branch into another branch. There are times when you might not be
577 able to merge all the changes in one branch with another but need to
578 pick out certain ones.
579
580- *``gitk``:* Provides a GUI view of the branches and changes in your
581 local Git repository. This command is a good way to graphically see
582 where things have diverged in your local repository.
583
584 .. note::
585
586 You need to install the
587 gitk
588 package on your development system to use this command.
589
590- *``git log``:* Reports a history of your commits to the repository.
591 This report lists all commits regardless of whether you have pushed
592 them upstream or not.
593
594- *``git diff``:* Displays line-by-line differences between a local
595 working file and the same file as understood by Git. This command is
596 useful to see what you have changed in any given file.
597
598Licensing
599=========
600
601Because open source projects are open to the public, they have different
602licensing structures in place. License evolution for both Open Source
603and Free Software has an interesting history. If you are interested in
604this history, you can find basic information here:
605
606- `Open source license
607 history <http://en.wikipedia.org/wiki/Open-source_license>`__
608
609- `Free software license
610 history <http://en.wikipedia.org/wiki/Free_software_license>`__
611
612In general, the Yocto Project is broadly licensed under the
613Massachusetts Institute of Technology (MIT) License. MIT licensing
614permits the reuse of software within proprietary software as long as the
615license is distributed with that software. MIT is also compatible with
616the GNU General Public License (GPL). Patches to the Yocto Project
617follow the upstream licensing scheme. You can find information on the
618MIT license
619`here <http://www.opensource.org/licenses/mit-license.php>`__. You can
620find information on the GNU GPL
621`here <http://www.opensource.org/licenses/LGPL-3.0>`__.
622
623When you build an image using the Yocto Project, the build process uses
624a known list of licenses to ensure compliance. You can find this list in
625the `Source Directory <&YOCTO_DOCS_REF_URL;#source-directory>`__ at
626``meta/files/common-licenses``. Once the build completes, the list of
627all licenses found and used during that build are kept in the `Build
628Directory <&YOCTO_DOCS_REF_URL;#build-directory>`__ at
629``tmp/deploy/licenses``.
630
631If a module requires a license that is not in the base list, the build
632process generates a warning during the build. These tools make it easier
633for a developer to be certain of the licenses with which their shipped
634products must comply. However, even with these tools it is still up to
635the developer to resolve potential licensing issues.
636
637The base list of licenses used by the build process is a combination of
638the Software Package Data Exchange (SPDX) list and the Open Source
639Initiative (OSI) projects. `SPDX Group <http://spdx.org>`__ is a working
640group of the Linux Foundation that maintains a specification for a
641standard format for communicating the components, licenses, and
642copyrights associated with a software package.
643`OSI <http://opensource.org>`__ is a corporation dedicated to the Open
644Source Definition and the effort for reviewing and approving licenses
645that conform to the Open Source Definition (OSD).
646
647You can find a list of the combined SPDX and OSI licenses that the Yocto
648Project uses in the ``meta/files/common-licenses`` directory in your
649`Source Directory <&YOCTO_DOCS_REF_URL;#source-directory>`__.
650
651For information that can help you maintain compliance with various open
652source licensing during the lifecycle of a product created using the
653Yocto Project, see the "`Maintaining Open Source License Compliance
654During Your Product's
655Lifecycle <&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle>`__"
656section in the Yocto Project Development Tasks Manual.
diff --git a/documentation/overview-manual/overview-manual-intro.rst b/documentation/overview-manual/overview-manual-intro.rst
new file mode 100644
index 0000000000..82c0051c47
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-intro.rst
@@ -0,0 +1,74 @@
1**********************************************
2The Yocto Project Overview and Concepts Manual
3**********************************************
4
5.. _overview-manual-welcome:
6
7Welcome
8=======
9
10Welcome to the Yocto Project Overview and Concepts Manual! This manual
11introduces the Yocto Project by providing concepts, software overviews,
12best-known-methods (BKMs), and any other high-level introductory
13information suitable for a new Yocto Project user.
14
15The following list describes what you can get from this manual:
16
17- `Introducing the Yocto Project <#overview-yp>`__\ *:* This chapter
18 provides an introduction to the Yocto Project. You will learn about
19 features and challenges of the Yocto Project, the layer model,
20 components and tools, development methods, the
21 `Poky <&YOCTO_DOCS_REF_URL;#poky>`__ reference distribution, the
22 OpenEmbedded build system workflow, and some basic Yocto terms.
23
24- `The Yocto Project Development
25 Environment <#overview-development-environment>`__\ *:* This chapter
26 helps you get started understanding the Yocto Project development
27 environment. You will learn about open source, development hosts,
28 Yocto Project source repositories, workflows using Git and the Yocto
29 Project, a Git primer, and information about licensing.
30
31- `Yocto Project Concepts <#overview-manual-concepts>`__\ *:* This
32 chapter presents various concepts regarding the Yocto Project. You
33 can find conceptual information about components, development,
34 cross-toolchains, and so forth.
35
36This manual does not give you the following:
37
38- *Step-by-step Instructions for Development Tasks:* Instructional
39 procedures reside in other manuals within the Yocto Project
40 documentation set. For example, the `Yocto Project Development Tasks
41 Manual <&YOCTO_DOCS_DEV_URL;>`__ provides examples on how to perform
42 various development tasks. As another example, the `Yocto Project
43 Application Development and the Extensible Software Development Kit
44 (eSDK) <&YOCTO_DOCS_SDK_URL;>`__ manual contains detailed
45 instructions on how to install an SDK, which is used to develop
46 applications for target hardware.
47
48- *Reference Material:* This type of material resides in an appropriate
49 reference manual. For example, system variables are documented in the
50 `Yocto Project Reference Manual <&YOCTO_DOCS_REF_URL;>`__. As another
51 example, the `Yocto Project Board Support Package (BSP) Developer's
52 Guide <&YOCTO_DOCS_BSP_URL;>`__ contains reference information on
53 BSPs.
54
55- *Detailed Public Information Not Specific to the Yocto Project:* For
56 example, exhaustive information on how to use the Source Control
57 Manager Git is better covered with Internet searches and official Git
58 Documentation than through the Yocto Project documentation.
59
60.. _overview-manual-other-information:
61
62Other Information
63=================
64
65Because this manual presents information for many different topics,
66supplemental information is recommended for full comprehension. For
67additional introductory information on the Yocto Project, see the `Yocto
68Project Website <&YOCTO_HOME_URL;>`__. If you want to build an image
69with no knowledge of Yocto Project as a way of quickly testing it out,
70see the `Yocto Project Quick Build <&YOCTO_DOCS_BRIEF_URL;>`__ document.
71For a comprehensive list of links and other documentation, see the
72"`Links and Related
73Documentation <&YOCTO_DOCS_REF_URL;#resources-links-and-related-documentation>`__"
74section in the Yocto Project Reference Manual.
diff --git a/documentation/overview-manual/overview-manual-yp-intro.rst b/documentation/overview-manual/overview-manual-yp-intro.rst
new file mode 100644
index 0000000000..62257c26b9
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-yp-intro.rst
@@ -0,0 +1,941 @@
1*****************************
2Introducing the Yocto Project
3*****************************
4
5What is the Yocto Project?
6==========================
7
8The Yocto Project is an open source collaboration project that helps
9developers create custom Linux-based systems that are designed for
10embedded products regardless of the product's hardware architecture.
11Yocto Project provides a flexible toolset and a development environment
12that allows embedded device developers across the world to collaborate
13through shared technologies, software stacks, configurations, and best
14practices used to create these tailored Linux images.
15
16Thousands of developers worldwide have discovered that Yocto Project
17provides advantages in both systems and applications development,
18archival and management benefits, and customizations used for speed,
19footprint, and memory utilization. The project is a standard when it
20comes to delivering embedded software stacks. The project allows
21software customizations and build interchange for multiple hardware
22platforms as well as software stacks that can be maintained and scaled.
23
24For further introductory information on the Yocto Project, you might be
25interested in this
26`article <https://www.embedded.com/electronics-blogs/say-what-/4458600/Why-the-Yocto-Project-for-my-IoT-Project->`__
27by Drew Moseley and in this short introductory
28`video <https://www.youtube.com/watch?v=utZpKM7i5Z4>`__.
29
30The remainder of this section overviews advantages and challenges tied
31to the Yocto Project.
32
33.. _gs-features:
34
35Features
36--------
37
38The following list describes features and advantages of the Yocto
39Project:
40
41- *Widely Adopted Across the Industry:* Semiconductor, operating
42 system, software, and service vendors exist whose products and
43 services adopt and support the Yocto Project. For a look at the Yocto
44 Project community and the companies involved with the Yocto Project,
45 see the "COMMUNITY" and "ECOSYSTEM" tabs on the `Yocto
46 Project <&YOCTO_HOME_URL;>`__ home page.
47
48- *Architecture Agnostic:* Yocto Project supports Intel, ARM, MIPS,
49 AMD, PPC and other architectures. Most ODMs, OSVs, and chip vendors
50 create and supply BSPs that support their hardware. If you have
51 custom silicon, you can create a BSP that supports that architecture.
52
53 Aside from lots of architecture support, the Yocto Project fully
54 supports a wide range of device emulation through the Quick EMUlator
55 (QEMU).
56
57- *Images and Code Transfer Easily:* Yocto Project output can easily
58 move between architectures without moving to new development
59 environments. Additionally, if you have used the Yocto Project to
60 create an image or application and you find yourself not able to
61 support it, commercial Linux vendors such as Wind River, Mentor
62 Graphics, Timesys, and ENEA could take it and provide ongoing
63 support. These vendors have offerings that are built using the Yocto
64 Project.
65
66- *Flexibility:* Corporations use the Yocto Project many different
67 ways. One example is to create an internal Linux distribution as a
68 code base the corporation can use across multiple product groups.
69 Through customization and layering, a project group can leverage the
70 base Linux distribution to create a distribution that works for their
71 product needs.
72
73- *Ideal for Constrained Embedded and IoT devices:* Unlike a full Linux
74 distribution, you can use the Yocto Project to create exactly what
75 you need for embedded devices. You only add the feature support or
76 packages that you absolutely need for the device. For devices that
77 have display hardware, you can use available system components such
78 as X11, GTK+, Qt, Clutter, and SDL (among others) to create a rich
79 user experience. For devices that do not have a display or where you
80 want to use alternative UI frameworks, you can choose to not install
81 these components.
82
83- *Comprehensive Toolchain Capabilities:* Toolchains for supported
84 architectures satisfy most use cases. However, if your hardware
85 supports features that are not part of a standard toolchain, you can
86 easily customize that toolchain through specification of
87 platform-specific tuning parameters. And, should you need to use a
88 third-party toolchain, mechanisms built into the Yocto Project allow
89 for that.
90
91- *Mechanism Rules Over Policy:* Focusing on mechanism rather than
92 policy ensures that you are free to set policies based on the needs
93 of your design instead of adopting decisions enforced by some system
94 software provider.
95
96- *Uses a Layer Model:* The Yocto Project `layer
97 infrastructure <#the-yocto-project-layer-model>`__ groups related
98 functionality into separate bundles. You can incrementally add these
99 grouped functionalities to your project as needed. Using layers to
100 isolate and group functionality reduces project complexity and
101 redundancy, allows you to easily extend the system, make
102 customizations, and keep functionality organized.
103
104- *Supports Partial Builds:* You can build and rebuild individual
105 packages as needed. Yocto Project accomplishes this through its
106 `shared-state cache <#shared-state-cache>`__ (sstate) scheme. Being
107 able to build and debug components individually eases project
108 development.
109
110- *Releases According to a Strict Schedule:* Major releases occur on a
111 `six-month cycle <&YOCTO_DOCS_REF_URL;#ref-release-process>`__
112 predictably in October and April. The most recent two releases
113 support point releases to address common vulnerabilities and
114 exposures. This predictability is crucial for projects based on the
115 Yocto Project and allows development teams to plan activities.
116
117- *Rich Ecosystem of Individuals and Organizations:* For open source
118 projects, the value of community is very important. Support forums,
119 expertise, and active developers who continue to push the Yocto
120 Project forward are readily available.
121
122- *Binary Reproducibility:* The Yocto Project allows you to be very
123 specific about dependencies and achieves very high percentages of
124 binary reproducibility (e.g. 99.8% for ``core-image-minimal``). When
125 distributions are not specific about which packages are pulled in and
126 in what order to support dependencies, other build systems can
127 arbitrarily include packages.
128
129- *License Manifest:* The Yocto Project provides a `license
130 manifest <&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle>`__
131 for review by people who need to track the use of open source
132 licenses (e.g.legal teams).
133
134.. _gs-challenges:
135
136Challenges
137----------
138
139The following list presents challenges you might encounter when
140developing using the Yocto Project:
141
142- *Steep Learning Curve:* The Yocto Project has a steep learning curve
143 and has many different ways to accomplish similar tasks. It can be
144 difficult to choose how to proceed when varying methods exist by
145 which to accomplish a given task.
146
147- *Understanding What Changes You Need to Make For Your Design Requires
148 Some Research:* Beyond the simple tutorial stage, understanding what
149 changes need to be made for your particular design can require a
150 significant amount of research and investigation. For information
151 that helps you transition from trying out the Yocto Project to using
152 it for your project, see the "`What I wish I'd
153 Known <&YOCTO_DOCS_URL;/what-i-wish-id-known/>`__" and
154 "`Transitioning to a Custom Environment for Systems
155 Development <&YOCTO_DOCS_URL;/transitioning-to-a-custom-environment/>`__"
156 documents on the Yocto Project website.
157
158- *Project Workflow Could Be Confusing:* The `Yocto Project
159 workflow <#overview-development-environment>`__ could be confusing if
160 you are used to traditional desktop and server software development.
161 In a desktop development environment, mechanisms exist to easily pull
162 and install new packages, which are typically pre-compiled binaries
163 from servers accessible over the Internet. Using the Yocto Project,
164 you must modify your configuration and rebuild to add additional
165 packages.
166
167- *Working in a Cross-Build Environment Can Feel Unfamiliar:* When
168 developing code to run on a target, compilation, execution, and
169 testing done on the actual target can be faster than running a
170 BitBake build on a development host and then deploying binaries to
171 the target for test. While the Yocto Project does support development
172 tools on the target, the additional step of integrating your changes
173 back into the Yocto Project build environment would be required.
174 Yocto Project supports an intermediate approach that involves making
175 changes on the development system within the BitBake environment and
176 then deploying only the updated packages to the target.
177
178 The Yocto Project `OpenEmbedded build
179 system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ produces packages
180 in standard formats (i.e. RPM, DEB, IPK, and TAR). You can deploy
181 these packages into the running system on the target by using
182 utilities on the target such as ``rpm`` or ``ipk``.
183
184- *Initial Build Times Can be Significant:* Long initial build times
185 are unfortunately unavoidable due to the large number of packages
186 initially built from scratch for a fully functioning Linux system.
187 Once that initial build is completed, however, the shared-state
188 (sstate) cache mechanism Yocto Project uses keeps the system from
189 rebuilding packages that have not been "touched" since the last
190 build. The sstate mechanism significantly reduces times for
191 successive builds.
192
193The Yocto Project Layer Model
194=============================
195
196The Yocto Project's "Layer Model" is a development model for embedded
197and IoT Linux creation that distinguishes the Yocto Project from other
198simple build systems. The Layer Model simultaneously supports
199collaboration and customization. Layers are repositories that contain
200related sets of instructions that tell the `OpenEmbedded build
201system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ what to do. You can
202collaborate, share, and reuse layers.
203
204Layers can contain changes to previous instructions or settings at any
205time. This powerful override capability is what allows you to customize
206previously supplied collaborative or community layers to suit your
207product requirements.
208
209You use different layers to logically separate information in your
210build. As an example, you could have BSP, GUI, distro configuration,
211middleware, or application layers. Putting your entire build into one
212layer limits and complicates future customization and reuse. Isolating
213information into layers, on the other hand, helps simplify future
214customizations and reuse. You might find it tempting to keep everything
215in one layer when working on a single project. However, the more modular
216your Metadata, the easier it is to cope with future changes.
217
218.. note::
219
220 - Use Board Support Package (BSP) layers from silicon vendors when
221 possible.
222
223 - Familiarize yourself with the `Yocto Project curated layer
224 index <https://caffelli-staging.yoctoproject.org/software-overview/layers/>`__
225 or the `OpenEmbedded layer
226 index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__.
227 The latter contains more layers but they are less universally
228 validated.
229
230 - Layers support the inclusion of technologies, hardware components,
231 and software components. The `Yocto Project
232 Compatible <&YOCTO_DOCS_DEV_URL;#making-sure-your-layer-is-compatible-with-yocto-project>`__
233 designation provides a minimum level of standardization that
234 contributes to a strong ecosystem. "YP Compatible" is applied to
235 appropriate products and software components such as BSPs, other
236 OE-compatible layers, and related open-source projects, allowing
237 the producer to use Yocto Project badges and branding assets.
238
239To illustrate how layers are used to keep things modular, consider
240machine customizations. These types of customizations typically reside
241in a special layer, rather than a general layer, called a BSP Layer.
242Furthermore, the machine customizations should be isolated from recipes
243and Metadata that support a new GUI environment, for example. This
244situation gives you a couple of layers: one for the machine
245configurations, and one for the GUI environment. It is important to
246understand, however, that the BSP layer can still make machine-specific
247additions to recipes within the GUI environment layer without polluting
248the GUI layer itself with those machine-specific changes. You can
249accomplish this through a recipe that is a BitBake append
250(``.bbappend``) file, which is described later in this section.
251
252.. note::
253
254 For general information on BSP layer structure, see the
255 Yocto Project Board Support Packages (BSP) Developer's Guide
256 .
257
258The `Source Directory <&YOCTO_DOCS_REF_URL;#source-directory>`__
259contains both general layers and BSP layers right out of the box. You
260can easily identify layers that ship with a Yocto Project release in the
261Source Directory by their names. Layers typically have names that begin
262with the string ``meta-``.
263
264.. note::
265
266 It is not a requirement that a layer name begin with the prefix
267 meta-
268 , but it is a commonly accepted standard in the Yocto Project
269 community.
270
271For example, if you were to examine the `tree
272view <https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/>`__ of the
273``poky`` repository, you will see several layers: ``meta``,
274``meta-skeleton``, ``meta-selftest``, ``meta-poky``, and
275``meta-yocto-bsp``. Each of these repositories represents a distinct
276layer.
277
278For procedures on how to create layers, see the "`Understanding and
279Creating
280Layers <&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers>`__"
281section in the Yocto Project Development Tasks Manual.
282
283Components and Tools
284====================
285
286The Yocto Project employs a collection of components and tools used by
287the project itself, by project developers, and by those using the Yocto
288Project. These components and tools are open source projects and
289metadata that are separate from the reference distribution
290(`Poky <&YOCTO_DOCS_REF_URL;#poky>`__) and the `OpenEmbedded build
291system <&YOCTO_DOCS_REF_URL;#build-system-term>`__. Most of the
292components and tools are downloaded separately.
293
294This section provides brief overviews of the components and tools
295associated with the Yocto Project.
296
297.. _gs-development-tools:
298
299Development Tools
300-----------------
301
302The following list consists of tools that help you develop images and
303applications using the Yocto Project:
304
305- *CROPS:* `CROPS <https://github.com/crops/poky-container/>`__ is an
306 open source, cross-platform development framework that leverages
307 `Docker Containers <https://www.docker.com/>`__. CROPS provides an
308 easily managed, extensible environment that allows you to build
309 binaries for a variety of architectures on Windows, Linux and Mac OS
310 X hosts.
311
312- *``devtool``:* This command-line tool is available as part of the
313 extensible SDK (eSDK) and is its cornerstone. You can use ``devtool``
314 to help build, test, and package software within the eSDK. You can
315 use the tool to optionally integrate what you build into an image
316 built by the OpenEmbedded build system.
317
318 The ``devtool`` command employs a number of sub-commands that allow
319 you to add, modify, and upgrade recipes. As with the OpenEmbedded
320 build system, “recipes” represent software packages within
321 ``devtool``. When you use ``devtool add``, a recipe is automatically
322 created. When you use ``devtool modify``, the specified existing
323 recipe is used in order to determine where to get the source code and
324 how to patch it. In both cases, an environment is set up so that when
325 you build the recipe a source tree that is under your control is used
326 in order to allow you to make changes to the source as desired. By
327 default, both new recipes and the source go into a “workspace”
328 directory under the eSDK. The ``devtool upgrade`` command updates an
329 existing recipe so that you can build it for an updated set of source
330 files.
331
332 You can read about the ``devtool`` workflow in the Yocto Project
333 Application Development and Extensible Software Development Kit
334 (eSDK) Manual in the "`Using ``devtool`` in Your SDK
335 Workflow' <&YOCTO_DOCS_SDK_URL;#using-devtool-in-your-sdk-workflow>`__"
336 section.
337
338- *Extensible Software Development Kit (eSDK):* The eSDK provides a
339 cross-development toolchain and libraries tailored to the contents of
340 a specific image. The eSDK makes it easy to add new applications and
341 libraries to an image, modify the source for an existing component,
342 test changes on the target hardware, and integrate into the rest of
343 the OpenEmbedded build system. The eSDK gives you a toolchain
344 experience supplemented with the powerful set of ``devtool`` commands
345 tailored for the Yocto Project environment.
346
347 For information on the eSDK, see the `Yocto Project Application
348 Development and the Extensible Software Development Kit
349 (eSDK) <&YOCTO_DOCS_SDK_URL;>`__ Manual.
350
351- *Toaster:* Toaster is a web interface to the Yocto Project
352 OpenEmbedded build system. Toaster allows you to configure, run, and
353 view information about builds. For information on Toaster, see the
354 `Toaster User Manual <&YOCTO_DOCS_TOAST_URL;>`__.
355
356.. _gs-production-tools:
357
358Production Tools
359----------------
360
361The following list consists of tools that help production related
362activities using the Yocto Project:
363
364- *Auto Upgrade Helper:* This utility when used in conjunction with the
365 `OpenEmbedded build
366 system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ (BitBake and
367 OE-Core) automatically generates upgrades for recipes that are based
368 on new versions of the recipes published upstream.
369
370- *Recipe Reporting System:* The Recipe Reporting System tracks recipe
371 versions available for Yocto Project. The main purpose of the system
372 is to help you manage the recipes you maintain and to offer a dynamic
373 overview of the project. The Recipe Reporting System is built on top
374 of the `OpenEmbedded Layer
375 Index <http://layers.openembedded.org/layerindex/layers/>`__, which
376 is a website that indexes OpenEmbedded-Core layers.
377
378- *Patchwork:* `Patchwork <http://jk.ozlabs.org/projects/patchwork/>`__
379 is a fork of a project originally started by
380 `OzLabs <http://ozlabs.org/>`__. The project is a web-based tracking
381 system designed to streamline the process of bringing contributions
382 into a project. The Yocto Project uses Patchwork as an organizational
383 tool to handle patches, which number in the thousands for every
384 release.
385
386- *AutoBuilder:* AutoBuilder is a project that automates build tests
387 and quality assurance (QA). By using the public AutoBuilder, anyone
388 can determine the status of the current "master" branch of Poky.
389
390 .. note::
391
392 AutoBuilder is based on
393 buildbot
394 .
395
396 A goal of the Yocto Project is to lead the open source industry with
397 a project that automates testing and QA procedures. In doing so, the
398 project encourages a development community that publishes QA and test
399 plans, publicly demonstrates QA and test plans, and encourages
400 development of tools that automate and test and QA procedures for the
401 benefit of the development community.
402
403 You can learn more about the AutoBuilder used by the Yocto Project
404 `here <&YOCTO_AB_URL;>`__.
405
406- *Cross-Prelink:* Prelinking is the process of pre-computing the load
407 addresses and link tables generated by the dynamic linker as compared
408 to doing this at runtime. Doing this ahead of time results in
409 performance improvements when the application is launched and reduced
410 memory usage for libraries shared by many applications.
411
412 Historically, cross-prelink is a variant of prelink, which was
413 conceived by `Jakub
414 Jelínek <http://people.redhat.com/jakub/prelink.pdf>`__ a number of
415 years ago. Both prelink and cross-prelink are maintained in the same
416 repository albeit on separate branches. By providing an emulated
417 runtime dynamic linker (i.e. ``glibc``-derived ``ld.so`` emulation),
418 the cross-prelink project extends the prelink software’s ability to
419 prelink a sysroot environment. Additionally, the cross-prelink
420 software enables the ability to work in sysroot style environments.
421
422 The dynamic linker determines standard load address calculations
423 based on a variety of factors such as mapping addresses, library
424 usage, and library function conflicts. The prelink tool uses this
425 information, from the dynamic linker, to determine unique load
426 addresses for executable and linkable format (ELF) binaries that are
427 shared libraries and dynamically linked. The prelink tool modifies
428 these ELF binaries with the pre-computed information. The result is
429 faster loading and often lower memory consumption because more of the
430 library code can be re-used from shared Copy-On-Write (COW) pages.
431
432 The original upstream prelink project only supports running prelink
433 on the end target device due to the reliance on the target device’s
434 dynamic linker. This restriction causes issues when developing a
435 cross-compiled system. The cross-prelink adds a synthesized dynamic
436 loader that runs on the host, thus permitting cross-prelinking
437 without ever having to run on a read-write target filesystem.
438
439- *Pseudo:* Pseudo is the Yocto Project implementation of
440 `fakeroot <http://man.he.net/man1/fakeroot>`__, which is used to run
441 commands in an environment that seemingly has root privileges.
442
443 During a build, it can be necessary to perform operations that
444 require system administrator privileges. For example, file ownership
445 or permissions might need definition. Pseudo is a tool that you can
446 either use directly or through the environment variable
447 ``LD_PRELOAD``. Either method allows these operations to succeed as
448 if system administrator privileges exist even when they do not.
449
450 You can read more about Pseudo in the "`Fakeroot and
451 Pseudo <#fakeroot-and-pseudo>`__" section.
452
453.. _gs-openembedded-build-system:
454
455Open-Embedded Build System Components
456-------------------------------------
457
458The following list consists of components associated with the
459`OpenEmbedded build system <&YOCTO_DOCS_REF_URL;#build-system-term>`__:
460
461- *BitBake:* BitBake is a core component of the Yocto Project and is
462 used by the OpenEmbedded build system to build images. While BitBake
463 is key to the build system, BitBake is maintained separately from the
464 Yocto Project.
465
466 BitBake is a generic task execution engine that allows shell and
467 Python tasks to be run efficiently and in parallel while working
468 within complex inter-task dependency constraints. In short, BitBake
469 is a build engine that works through recipes written in a specific
470 format in order to perform sets of tasks.
471
472 You can learn more about BitBake in the `BitBake User
473 Manual <&YOCTO_DOCS_BB_URL;>`__.
474
475- *OpenEmbedded-Core:* OpenEmbedded-Core (OE-Core) is a common layer of
476 metadata (i.e. recipes, classes, and associated files) used by
477 OpenEmbedded-derived systems, which includes the Yocto Project. The
478 Yocto Project and the OpenEmbedded Project both maintain the
479 OpenEmbedded-Core. You can find the OE-Core metadata in the Yocto
480 Project `Source
481 Repositories <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta>`__.
482
483 Historically, the Yocto Project integrated the OE-Core metadata
484 throughout the Yocto Project source repository reference system
485 (Poky). After Yocto Project Version 1.0, the Yocto Project and
486 OpenEmbedded agreed to work together and share a common core set of
487 metadata (OE-Core), which contained much of the functionality
488 previously found in Poky. This collaboration achieved a long-standing
489 OpenEmbedded objective for having a more tightly controlled and
490 quality-assured core. The results also fit well with the Yocto
491 Project objective of achieving a smaller number of fully featured
492 tools as compared to many different ones.
493
494 Sharing a core set of metadata results in Poky as an integration
495 layer on top of OE-Core. You can see that in this
496 `figure <#yp-key-dev-elements>`__. The Yocto Project combines various
497 components such as BitBake, OE-Core, script “glue”, and documentation
498 for its build system.
499
500.. _gs-reference-distribution-poky:
501
502Reference Distribution (Poky)
503-----------------------------
504
505Poky is the Yocto Project reference distribution. It contains the
506`Open-Embedded build system <&YOCTO_DOCS_REF_URL;#build-system-term>`__
507(BitBake and OE-Core) as well as a set of metadata to get you started
508building your own distribution. See the
509`figure <#what-is-the-yocto-project>`__ in "What is the Yocto Project?"
510section for an illustration that shows Poky and its relationship with
511other parts of the Yocto Project.
512
513To use the Yocto Project tools and components, you can download
514(``clone``) Poky and use it to bootstrap your own distribution.
515
516.. note::
517
518 Poky does not contain binary files. It is a working example of how to
519 build your own custom Linux distribution from source.
520
521You can read more about Poky in the "`Reference Embedded Distribution
522(Poky) <#reference-embedded-distribution>`__" section.
523
524.. _gs-packages-for-finished-targets:
525
526Packages for Finished Targets
527-----------------------------
528
529The following lists components associated with packages for finished
530targets:
531
532- *Matchbox:* Matchbox is an Open Source, base environment for the X
533 Window System running on non-desktop, embedded platforms such as
534 handhelds, set-top boxes, kiosks, and anything else for which screen
535 space, input mechanisms, or system resources are limited.
536
537 Matchbox consists of a number of interchangeable and optional
538 applications that you can tailor to a specific, non-desktop platform
539 to enhance usability in constrained environments.
540
541 You can find the Matchbox source in the Yocto Project `Source
542 Repositories <&YOCTO_GIT_URL;>`__.
543
544- *Opkg* Open PacKaGe management (opkg) is a lightweight package
545 management system based on the itsy package (ipkg) management system.
546 Opkg is written in C and resembles Advanced Package Tool (APT) and
547 Debian Package (dpkg) in operation.
548
549 Opkg is intended for use on embedded Linux devices and is used in
550 this capacity in the
551 `OpenEmbedded <http://www.openembedded.org/wiki/Main_Page>`__ and
552 `OpenWrt <https://openwrt.org/>`__ projects, as well as the Yocto
553 Project.
554
555 .. note::
556
557 As best it can, opkg maintains backwards compatibility with ipkg
558 and conforms to a subset of Debian’s policy manual regarding
559 control files.
560
561.. _gs-archived-components:
562
563Archived Components
564-------------------
565
566The Build Appliance is a virtual machine image that enables you to build
567and boot a custom embedded Linux image with the Yocto Project using a
568non-Linux development system.
569
570Historically, the Build Appliance was the second of three methods by
571which you could use the Yocto Project on a system that was not native to
572Linux.
573
5741. *Hob:* Hob, which is now deprecated and is no longer available since
575 the 2.1 release of the Yocto Project provided a rudimentary,
576 GUI-based interface to the Yocto Project. Toaster has fully replaced
577 Hob.
578
5792. *Build Appliance:* Post Hob, the Build Appliance became available. It
580 was never recommended that you use the Build Appliance as a
581 day-to-day production development environment with the Yocto Project.
582 Build Appliance was useful as a way to try out development in the
583 Yocto Project environment.
584
5853. *CROPS:* The final and best solution available now for developing
586 using the Yocto Project on a system not native to Linux is with
587 `CROPS <#gs-crops-overview>`__.
588
589.. _gs-development-methods:
590
591Development Methods
592===================
593
594The Yocto Project development environment usually involves a `Build
595Host <&YOCTO_DOCS_REF_URL;#hardware-build-system-term>`__ and target
596hardware. You use the Build Host to build images and develop
597applications, while you use the target hardware to test deployed
598software.
599
600This section provides an introduction to the choices or development
601methods you have when setting up your Build Host. Depending on the your
602particular workflow preference and the type of operating system your
603Build Host runs, several choices exist that allow you to use the Yocto
604Project.
605
606.. note::
607
608 For additional detail about the Yocto Project development
609 environment, see the "
610 The Yocto Project Development Environment
611 " chapter.
612
613- *Native Linux Host:* By far the best option for a Build Host. A
614 system running Linux as its native operating system allows you to
615 develop software by directly using the
616 `BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__ tool. You can
617 accomplish all aspects of development from a familiar shell of a
618 supported Linux distribution.
619
620 For information on how to set up a Build Host on a system running
621 Linux as its native operating system, see the "`Setting Up a Native
622 Linux Host <&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host>`__"
623 section in the Yocto Project Development Tasks Manual.
624
625- *CROss PlatformS (CROPS):* Typically, you use
626 `CROPS <https://github.com/crops/poky-container/>`__, which leverages
627 `Docker Containers <https://www.docker.com/>`__, to set up a Build
628 Host that is not running Linux (e.g. Microsoft Windows or macOS).
629
630 .. note::
631
632 You can, however, use CROPS on a Linux-based system.
633
634 CROPS is an open source, cross-platform development framework that
635 provides an easily managed, extensible environment for building
636 binaries targeted for a variety of architectures on Windows, macOS,
637 or Linux hosts. Once the Build Host is set up using CROPS, you can
638 prepare a shell environment to mimic that of a shell being used on a
639 system natively running Linux.
640
641 For information on how to set up a Build Host with CROPS, see the
642 "`Setting Up to Use CROss PlatformS
643 (CROPS) <&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops>`__" section in
644 the Yocto Project Development Tasks Manual.
645
646- *Windows Subsystem For Linux (WSLv2):* You may use Windows Subsystem
647 For Linux v2 to set up a build host using Windows 10.
648
649 .. note::
650
651 The Yocto Project is not compatible with WSLv1, it is compatible
652 but not officially supported nor validated with WSLv2, if you
653 still decide to use WSL please upgrade to WSLv2.
654
655 The Windows Subsystem For Linux allows Windows 10 to run a real Linux
656 kernel inside of a lightweight utility virtual machine (VM) using
657 virtualization technology.
658
659 For information on how to set up a Build Host with WSLv2, see the
660 "`Setting Up to Use Windows Subsystem For
661 Linux <&YOCTO_DOCS_DEV_URL;#setting-up-to-use-wsl>`__" section in the
662 Yocto Project Development Tasks Manual.
663
664- *Toaster:* Regardless of what your Build Host is running, you can use
665 Toaster to develop software using the Yocto Project. Toaster is a web
666 interface to the Yocto Project's `Open-Embedded build
667 system <&YOCTO_DOCS_REF_URL;#build-system-term>`__. The interface
668 enables you to configure and run your builds. Information about
669 builds is collected and stored in a database. You can use Toaster to
670 configure and start builds on multiple remote build servers.
671
672 For information about and how to use Toaster, see the `Toaster User
673 Manual <&YOCTO_DOCS_TOAST_URL;>`__.
674
675.. _reference-embedded-distribution:
676
677Reference Embedded Distribution (Poky)
678======================================
679
680"Poky", which is pronounced *Pock*-ee, is the name of the Yocto
681Project's reference distribution or Reference OS Kit. Poky contains the
682`OpenEmbedded Build System <&YOCTO_DOCS_REF_URL;#build-system-term>`__
683(`BitBake <&YOCTO_DOCS_REF_URL;#bitbake-term>`__ and
684`OpenEmbedded-Core <&YOCTO_DOCS_REF_URL;#oe-core>`__) as well as a set
685of `metadata <&YOCTO_DOCS_REF_URL;#metadata>`__ to get you started
686building your own distro. In other words, Poky is a base specification
687of the functionality needed for a typical embedded system as well as the
688components from the Yocto Project that allow you to build a distribution
689into a usable binary image.
690
691Poky is a combined repository of BitBake, OpenEmbedded-Core (which is
692found in ``meta``), ``meta-poky``, ``meta-yocto-bsp``, and documentation
693provided all together and known to work well together. You can view
694these items that make up the Poky repository in the `Source
695Repositories <&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/>`__.
696
697.. note::
698
699 If you are interested in all the contents of the
700 poky
701 Git repository, see the "
702 Top-Level Core Components
703 " section in the Yocto Project Reference Manual.
704
705The following figure illustrates what generally comprises Poky:
706
707- BitBake is a task executor and scheduler that is the heart of the
708 OpenEmbedded build system.
709
710- ``meta-poky``, which is Poky-specific metadata.
711
712- ``meta-yocto-bsp``, which are Yocto Project-specific Board Support
713 Packages (BSPs).
714
715- OpenEmbedded-Core (OE-Core) metadata, which includes shared
716 configurations, global variable definitions, shared classes,
717 packaging, and recipes. Classes define the encapsulation and
718 inheritance of build logic. Recipes are the logical units of software
719 and images to be built.
720
721- Documentation, which contains the Yocto Project source files used to
722 make the set of user manuals.
723
724.. note::
725
726 While Poky is a "complete" distribution specification and is tested
727 and put through QA, you cannot use it as a product "out of the box"
728 in its current form.
729
730To use the Yocto Project tools, you can use Git to clone (download) the
731Poky repository then use your local copy of the reference distribution
732to bootstrap your own distribution.
733
734.. note::
735
736 Poky does not contain binary files. It is a working example of how to
737 build your own custom Linux distribution from source.
738
739Poky has a regular, well established, six-month release cycle under its
740own version. Major releases occur at the same time major releases (point
741releases) occur for the Yocto Project, which are typically in the Spring
742and Fall. For more information on the Yocto Project release schedule and
743cadence, see the "`Yocto Project Releases and the Stable Release
744Process <&YOCTO_DOCS_REF_URL;#ref-release-process>`__" chapter in the
745Yocto Project Reference Manual.
746
747Much has been said about Poky being a "default configuration." A default
748configuration provides a starting image footprint. You can use Poky out
749of the box to create an image ranging from a shell-accessible minimal
750image all the way up to a Linux Standard Base-compliant image that uses
751a GNOME Mobile and Embedded (GMAE) based reference user interface called
752Sato.
753
754One of the most powerful properties of Poky is that every aspect of a
755build is controlled by the metadata. You can use metadata to augment
756these base image types by adding metadata
757`layers <#the-yocto-project-layer-model>`__ that extend functionality.
758These layers can provide, for example, an additional software stack for
759an image type, add a board support package (BSP) for additional
760hardware, or even create a new image type.
761
762Metadata is loosely grouped into configuration files or package recipes.
763A recipe is a collection of non-executable metadata used by BitBake to
764set variables or define additional build-time tasks. A recipe contains
765fields such as the recipe description, the recipe version, the license
766of the package and the upstream source repository. A recipe might also
767indicate that the build process uses autotools, make, distutils or any
768other build process, in which case the basic functionality can be
769defined by the classes it inherits from the OE-Core layer's class
770definitions in ``./meta/classes``. Within a recipe you can also define
771additional tasks as well as task prerequisites. Recipe syntax through
772BitBake also supports both ``_prepend`` and ``_append`` operators as a
773method of extending task functionality. These operators inject code into
774the beginning or end of a task. For information on these BitBake
775operators, see the "`Appending and Prepending (Override Style
776Syntax) <&YOCTO_DOCS_BB_URL;#appending-and-prepending-override-style-syntax>`__"
777section in the BitBake User's Manual.
778
779.. _openembedded-build-system-workflow:
780
781The OpenEmbedded Build System Workflow
782======================================
783
784The `OpenEmbedded build
785system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ uses a "workflow" to
786accomplish image and SDK generation. The following figure overviews that
787workflow: Following is a brief summary of the "workflow":
788
7891. Developers specify architecture, policies, patches and configuration
790 details.
791
7922. The build system fetches and downloads the source code from the
793 specified location. The build system supports standard methods such
794 as tarballs or source code repositories systems such as Git.
795
7963. Once source code is downloaded, the build system extracts the sources
797 into a local work area where patches are applied and common steps for
798 configuring and compiling the software are run.
799
8004. The build system then installs the software into a temporary staging
801 area where the binary package format you select (DEB, RPM, or IPK) is
802 used to roll up the software.
803
8045. Different QA and sanity checks run throughout entire build process.
805
8066. After the binaries are created, the build system generates a binary
807 package feed that is used to create the final root file image.
808
8097. The build system generates the file system image and a customized
810 Extensible SDK (eSDK) for application development in parallel.
811
812For a very detailed look at this workflow, see the "`OpenEmbedded Build
813System Concepts <#openembedded-build-system-build-concepts>`__" section.
814
815Some Basic Terms
816================
817
818It helps to understand some basic fundamental terms when learning the
819Yocto Project. Although a list of terms exists in the "`Yocto Project
820Terms <&YOCTO_DOCS_REF_URL;#ref-terms>`__" section of the Yocto Project
821Reference Manual, this section provides the definitions of some terms
822helpful for getting started:
823
824- *Configuration Files:* Files that hold global definitions of
825 variables, user-defined variables, and hardware configuration
826 information. These files tell the `Open-Embedded build
827 system <&YOCTO_DOCS_REF_URL;#build-system-term>`__ what to build and
828 what to put into the image to support a particular platform.
829
830- *Extensible Software Development Kit (eSDK):* A custom SDK for
831 application developers. This eSDK allows developers to incorporate
832 their library and programming changes back into the image to make
833 their code available to other application developers. For information
834 on the eSDK, see the `Yocto Project Application Development and the
835 Extensible Software Development Kit (eSDK) <&YOCTO_DOCS_SDK_URL;>`__
836 manual.
837
838- *Layer:* A collection of related recipes. Layers allow you to
839 consolidate related metadata to customize your build. Layers also
840 isolate information used when building for multiple architectures.
841 Layers are hierarchical in their ability to override previous
842 specifications. You can include any number of available layers from
843 the Yocto Project and customize the build by adding your layers after
844 them. You can search the Layer Index for layers used within Yocto
845 Project.
846
847 For more detailed information on layers, see the "`Understanding and
848 Creating
849 Layers <&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers>`__"
850 section in the Yocto Project Development Tasks Manual. For a
851 discussion specifically on BSP Layers, see the "`BSP
852 Layers <&YOCTO_DOCS_BSP_URL;#bsp-layers>`__" section in the Yocto
853 Project Board Support Packages (BSP) Developer's Guide.
854
855- *Metadata:* A key element of the Yocto Project is the Metadata that
856 is used to construct a Linux distribution and is contained in the
857 files that the OpenEmbedded build system parses when building an
858 image. In general, Metadata includes recipes, configuration files,
859 and other information that refers to the build instructions
860 themselves, as well as the data used to control what things get built
861 and the effects of the build. Metadata also includes commands and
862 data used to indicate what versions of software are used, from where
863 they are obtained, and changes or additions to the software itself
864 (patches or auxiliary files) that are used to fix bugs or customize
865 the software for use in a particular situation. OpenEmbedded-Core is
866 an important set of validated metadata.
867
868- *OpenEmbedded Build System:* The terms "BitBake" and "build system"
869 are sometimes used for the OpenEmbedded Build System.
870
871 BitBake is a task scheduler and execution engine that parses
872 instructions (i.e. recipes) and configuration data. After a parsing
873 phase, BitBake creates a dependency tree to order the compilation,
874 schedules the compilation of the included code, and finally executes
875 the building of the specified custom Linux image (distribution).
876 BitBake is similar to the ``make`` tool.
877
878 During a build process, the build system tracks dependencies and
879 performs a native or cross-compilation of the package. As a first
880 step in a cross-build setup, the framework attempts to create a
881 cross-compiler toolchain (i.e. Extensible SDK) suited for the target
882 platform.
883
884- *OpenEmbedded-Core (OE-Core):* OE-Core is metadata comprised of
885 foundation recipes, classes, and associated files that are meant to
886 be common among many different OpenEmbedded-derived systems,
887 including the Yocto Project. OE-Core is a curated subset of an
888 original repository developed by the OpenEmbedded community that has
889 been pared down into a smaller, core set of continuously validated
890 recipes. The result is a tightly controlled and quality-assured core
891 set of recipes.
892
893 You can see the Metadata in the ``meta`` directory of the Yocto
894 Project `Source
895 Repositories <http://git.yoctoproject.org/cgit/cgit.cgi>`__.
896
897- *Packages:* In the context of the Yocto Project, this term refers to
898 a recipe's packaged output produced by BitBake (i.e. a "baked
899 recipe"). A package is generally the compiled binaries produced from
900 the recipe's sources. You "bake" something by running it through
901 BitBake.
902
903 It is worth noting that the term "package" can, in general, have
904 subtle meanings. For example, the packages referred to in the
905 "`Required Packages for the Build
906 Host <&YOCTO_DOCS_REF_URL;#required-packages-for-the-build-host>`__"
907 section in the Yocto Project Reference Manual are compiled binaries
908 that, when installed, add functionality to your Linux distribution.
909
910 Another point worth noting is that historically within the Yocto
911 Project, recipes were referred to as packages - thus, the existence
912 of several BitBake variables that are seemingly mis-named, (e.g.
913 ```PR`` <&YOCTO_DOCS_REF_URL;#var-PR>`__,
914 ```PV`` <&YOCTO_DOCS_REF_URL;#var-PV>`__, and
915 ```PE`` <&YOCTO_DOCS_REF_URL;#var-PE>`__).
916
917- *Poky:* Poky is a reference embedded distribution and a reference
918 test configuration. Poky provides the following:
919
920 - A base-level functional distro used to illustrate how to customize
921 a distribution.
922
923 - A means by which to test the Yocto Project components (i.e. Poky
924 is used to validate the Yocto Project).
925
926 - A vehicle through which you can download the Yocto Project.
927
928 Poky is not a product level distro. Rather, it is a good starting
929 point for customization.
930
931 .. note::
932
933 Poky is an integration layer on top of OE-Core.
934
935- *Recipe:* The most common form of metadata. A recipe contains a list
936 of settings and tasks (i.e. instructions) for building packages that
937 are then used to build the binary image. A recipe describes where you
938 get source code and which patches to apply. Recipes describe
939 dependencies for libraries or for other recipes as well as
940 configuration and compilation options. Related recipes are
941 consolidated into a layer.
diff --git a/documentation/overview-manual/overview-manual.rst b/documentation/overview-manual/overview-manual.rst
new file mode 100644
index 0000000000..e6cd07d34d
--- /dev/null
+++ b/documentation/overview-manual/overview-manual.rst
@@ -0,0 +1,12 @@
1==========================================
2Yocto Project Overview and Concepts Manual
3==========================================
4
5.. toctree::
6 :caption: Table of Contents
7 :numbered:
8
9 overview-manual-intro
10 overview-manual-yp-intro
11 overview-manual-development-environment
12 overview-manual-concepts