summaryrefslogtreecommitdiffstats
path: root/documentation/overview-manual
diff options
context:
space:
mode:
authorScott Rifenbark <srifenbark@gmail.com>2018-04-12 10:52:45 -0700
committerRichard Purdie <richard.purdie@linuxfoundation.org>2018-05-24 17:16:12 +0100
commitc555881ac8e62e1bc1ddd9e0337e0c70a560b3b6 (patch)
tree80925d3cd31d9fe7e0b6692cd9ea39fb3e947146 /documentation/overview-manual
parent4fd8b39b243f1b0d49971c233a5db2b09f1a2048 (diff)
downloadpoky-c555881ac8e62e1bc1ddd9e0337e0c70a560b3b6.tar.gz
poky.ent: Added YOCTO_DOCS_OM_URL entity
The variabe for the "getting-started" manual goes away and is replaced by this one for the new "overview-manual." (From yocto-docs rev: 45fc9beac6db4c40c3660fc9e54cc11e9c1f96c4) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/overview-manual')
-rw-r--r--documentation/overview-manual/figures/YP-flow-diagram.pngbin0 -> 190715 bytes
-rw-r--r--documentation/overview-manual/figures/analysis-for-package-splitting.pngbin0 -> 63291 bytes
-rw-r--r--documentation/overview-manual/figures/configuration-compile-autoreconf.pngbin0 -> 46080 bytes
-rw-r--r--documentation/overview-manual/figures/cross-development-toolchains.pngbin0 -> 82633 bytes
-rw-r--r--documentation/overview-manual/figures/git-workflow.pngbin0 -> 26586 bytes
-rw-r--r--documentation/overview-manual/figures/image-generation.pngbin0 -> 112991 bytes
-rw-r--r--documentation/overview-manual/figures/images.pngbin0 -> 22926 bytes
-rw-r--r--documentation/overview-manual/figures/index-downloads.pngbin0 -> 36362 bytes
-rw-r--r--documentation/overview-manual/figures/key-dev-elements.pngbin0 -> 20424 bytes
-rw-r--r--documentation/overview-manual/figures/layer-input.pngbin0 -> 45856 bytes
-rw-r--r--documentation/overview-manual/figures/overview-manual-title.pngbin0 -> 21648 bytes
-rw-r--r--documentation/overview-manual/figures/package-feeds.pngbin0 -> 30112 bytes
-rw-r--r--documentation/overview-manual/figures/patching.pngbin0 -> 42523 bytes
-rw-r--r--documentation/overview-manual/figures/poky-reference-distribution.pngbin0 -> 23784 bytes
-rwxr-xr-xdocumentation/overview-manual/figures/sdk-generation.pngbin0 -> 115643 bytes
-rw-r--r--documentation/overview-manual/figures/sdk.pngbin0 -> 67478 bytes
-rw-r--r--documentation/overview-manual/figures/source-fetching.pngbin0 -> 38860 bytes
-rw-r--r--documentation/overview-manual/figures/source-input.pngbin0 -> 39872 bytes
-rw-r--r--documentation/overview-manual/figures/source-repos.pngbin0 -> 298757 bytes
-rw-r--r--documentation/overview-manual/figures/user-configuration.pngbin0 -> 74109 bytes
-rw-r--r--documentation/overview-manual/figures/yocto-environment-ref.pngbin0 -> 83206 bytes
-rw-r--r--documentation/overview-manual/figures/yp-download.pngbin0 -> 82939 bytes
-rw-r--r--documentation/overview-manual/overview-manual-concepts.xml3612
-rw-r--r--documentation/overview-manual/overview-manual-customization.xsl27
-rw-r--r--documentation/overview-manual/overview-manual-development-environment.xml972
-rw-r--r--documentation/overview-manual/overview-manual-eclipse-customization.xsl35
-rw-r--r--documentation/overview-manual/overview-manual-intro.xml111
-rw-r--r--documentation/overview-manual/overview-manual-style.css988
-rw-r--r--documentation/overview-manual/overview-manual-yp-intro.xml1335
-rw-r--r--documentation/overview-manual/overview-manual.xml108
30 files changed, 7188 insertions, 0 deletions
diff --git a/documentation/overview-manual/figures/YP-flow-diagram.png b/documentation/overview-manual/figures/YP-flow-diagram.png
new file mode 100644
index 0000000000..8264410504
--- /dev/null
+++ b/documentation/overview-manual/figures/YP-flow-diagram.png
Binary files differ
diff --git a/documentation/overview-manual/figures/analysis-for-package-splitting.png b/documentation/overview-manual/figures/analysis-for-package-splitting.png
new file mode 100644
index 0000000000..04f2794ea9
--- /dev/null
+++ b/documentation/overview-manual/figures/analysis-for-package-splitting.png
Binary files differ
diff --git a/documentation/overview-manual/figures/configuration-compile-autoreconf.png b/documentation/overview-manual/figures/configuration-compile-autoreconf.png
new file mode 100644
index 0000000000..a07464f04c
--- /dev/null
+++ b/documentation/overview-manual/figures/configuration-compile-autoreconf.png
Binary files differ
diff --git a/documentation/overview-manual/figures/cross-development-toolchains.png b/documentation/overview-manual/figures/cross-development-toolchains.png
new file mode 100644
index 0000000000..cbe8371c05
--- /dev/null
+++ b/documentation/overview-manual/figures/cross-development-toolchains.png
Binary files differ
diff --git a/documentation/overview-manual/figures/git-workflow.png b/documentation/overview-manual/figures/git-workflow.png
new file mode 100644
index 0000000000..e401330a12
--- /dev/null
+++ b/documentation/overview-manual/figures/git-workflow.png
Binary files differ
diff --git a/documentation/overview-manual/figures/image-generation.png b/documentation/overview-manual/figures/image-generation.png
new file mode 100644
index 0000000000..71a48dc6f4
--- /dev/null
+++ b/documentation/overview-manual/figures/image-generation.png
Binary files differ
diff --git a/documentation/overview-manual/figures/images.png b/documentation/overview-manual/figures/images.png
new file mode 100644
index 0000000000..d99eac1fbf
--- /dev/null
+++ b/documentation/overview-manual/figures/images.png
Binary files differ
diff --git a/documentation/overview-manual/figures/index-downloads.png b/documentation/overview-manual/figures/index-downloads.png
new file mode 100644
index 0000000000..96303b8781
--- /dev/null
+++ b/documentation/overview-manual/figures/index-downloads.png
Binary files differ
diff --git a/documentation/overview-manual/figures/key-dev-elements.png b/documentation/overview-manual/figures/key-dev-elements.png
new file mode 100644
index 0000000000..76c44050fd
--- /dev/null
+++ b/documentation/overview-manual/figures/key-dev-elements.png
Binary files differ
diff --git a/documentation/overview-manual/figures/layer-input.png b/documentation/overview-manual/figures/layer-input.png
new file mode 100644
index 0000000000..0a4f2e74f3
--- /dev/null
+++ b/documentation/overview-manual/figures/layer-input.png
Binary files differ
diff --git a/documentation/overview-manual/figures/overview-manual-title.png b/documentation/overview-manual/figures/overview-manual-title.png
new file mode 100644
index 0000000000..698f0f3d11
--- /dev/null
+++ b/documentation/overview-manual/figures/overview-manual-title.png
Binary files differ
diff --git a/documentation/overview-manual/figures/package-feeds.png b/documentation/overview-manual/figures/package-feeds.png
new file mode 100644
index 0000000000..37c9c32506
--- /dev/null
+++ b/documentation/overview-manual/figures/package-feeds.png
Binary files differ
diff --git a/documentation/overview-manual/figures/patching.png b/documentation/overview-manual/figures/patching.png
new file mode 100644
index 0000000000..8ecd018502
--- /dev/null
+++ b/documentation/overview-manual/figures/patching.png
Binary files differ
diff --git a/documentation/overview-manual/figures/poky-reference-distribution.png b/documentation/overview-manual/figures/poky-reference-distribution.png
new file mode 100644
index 0000000000..1be89ae68e
--- /dev/null
+++ b/documentation/overview-manual/figures/poky-reference-distribution.png
Binary files differ
diff --git a/documentation/overview-manual/figures/sdk-generation.png b/documentation/overview-manual/figures/sdk-generation.png
new file mode 100755
index 0000000000..adbe1f4acf
--- /dev/null
+++ b/documentation/overview-manual/figures/sdk-generation.png
Binary files differ
diff --git a/documentation/overview-manual/figures/sdk.png b/documentation/overview-manual/figures/sdk.png
new file mode 100644
index 0000000000..5c36b7548b
--- /dev/null
+++ b/documentation/overview-manual/figures/sdk.png
Binary files differ
diff --git a/documentation/overview-manual/figures/source-fetching.png b/documentation/overview-manual/figures/source-fetching.png
new file mode 100644
index 0000000000..26aefb50c2
--- /dev/null
+++ b/documentation/overview-manual/figures/source-fetching.png
Binary files differ
diff --git a/documentation/overview-manual/figures/source-input.png b/documentation/overview-manual/figures/source-input.png
new file mode 100644
index 0000000000..f7515058ef
--- /dev/null
+++ b/documentation/overview-manual/figures/source-input.png
Binary files differ
diff --git a/documentation/overview-manual/figures/source-repos.png b/documentation/overview-manual/figures/source-repos.png
new file mode 100644
index 0000000000..603300b6d2
--- /dev/null
+++ b/documentation/overview-manual/figures/source-repos.png
Binary files differ
diff --git a/documentation/overview-manual/figures/user-configuration.png b/documentation/overview-manual/figures/user-configuration.png
new file mode 100644
index 0000000000..c298401fc3
--- /dev/null
+++ b/documentation/overview-manual/figures/user-configuration.png
Binary files differ
diff --git a/documentation/overview-manual/figures/yocto-environment-ref.png b/documentation/overview-manual/figures/yocto-environment-ref.png
new file mode 100644
index 0000000000..650c6c8030
--- /dev/null
+++ b/documentation/overview-manual/figures/yocto-environment-ref.png
Binary files differ
diff --git a/documentation/overview-manual/figures/yp-download.png b/documentation/overview-manual/figures/yp-download.png
new file mode 100644
index 0000000000..bfd12b678a
--- /dev/null
+++ b/documentation/overview-manual/figures/yp-download.png
Binary files differ
diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml
new file mode 100644
index 0000000000..123cf15d19
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-concepts.xml
@@ -0,0 +1,3612 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id=' overview-manual-concepts'>
6<title>Yocto Project Concepts</title>
7
8 <section id='yocto-project-components'>
9 <title>Yocto Project Components</title>
10
11 <para>
12 The
13 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
14 task executor together with various types of configuration files
15 form the OpenEmbedded-Core.
16 This section overviews these components by describing their use and
17 how they interact.
18 </para>
19
20 <para>
21 BitBake handles the parsing and execution of the data files.
22 The data itself is of various types:
23 <itemizedlist>
24 <listitem><para>
25 <emphasis>Recipes:</emphasis>
26 Provides details about particular pieces of software.
27 </para></listitem>
28 <listitem><para>
29 <emphasis>Class Data:</emphasis>
30 Abstracts common build information (e.g. how to build a
31 Linux kernel).
32 </para></listitem>
33 <listitem><para>
34 <emphasis>Configuration Data:</emphasis>
35 Defines machine-specific settings, policy decisions, and
36 so forth.
37 Configuration data acts as the glue to bind everything
38 together.
39 </para></listitem>
40 </itemizedlist>
41 </para>
42
43 <para>
44 BitBake knows how to combine multiple data sources together and
45 refers to each data source as a layer.
46 For information on layers, see the
47 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
48 section of the Yocto Project Development Tasks Manual.
49 </para>
50
51 <para>
52 Following are some brief details on these core components.
53 For additional information on how these components interact during
54 a build, see the
55 "<link linkend='development-concepts'>Development Concepts</link>"
56 section.
57 </para>
58
59 <section id='usingpoky-components-bitbake'>
60 <title>BitBake</title>
61
62 <para>
63 BitBake is the tool at the heart of the
64 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
65 and is responsible for parsing the
66 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>,
67 generating a list of tasks from it, and then executing those
68 tasks.
69 </para>
70
71 <para>
72 This section briefly introduces BitBake.
73 If you want more information on BitBake, see the
74 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>.
75 </para>
76
77 <para>
78 To see a list of the options BitBake supports, use either of
79 the following commands:
80 <literallayout class='monospaced'>
81 $ bitbake -h
82 $ bitbake --help
83 </literallayout>
84 </para>
85
86 <para>
87 The most common usage for BitBake is
88 <filename>bitbake <replaceable>packagename</replaceable></filename>,
89 where <filename>packagename</filename> is the name of the
90 package you want to build (referred to as the "target").
91 The target often equates to the first part of a recipe's
92 filename (e.g. "foo" for a recipe named
93 <filename>foo_1.3.0-r0.bb</filename>).
94 So, to process the
95 <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you
96 might type the following:
97 <literallayout class='monospaced'>
98 $ bitbake matchbox-desktop
99 </literallayout>
100 Several different versions of
101 <filename>matchbox-desktop</filename> might exist.
102 BitBake chooses the one selected by the distribution
103 configuration.
104 You can get more details about how BitBake chooses between
105 different target versions and providers in the
106 "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>"
107 section of the BitBake User Manual.
108 </para>
109
110 <para>
111 BitBake also tries to execute any dependent tasks first.
112 So for example, before building
113 <filename>matchbox-desktop</filename>, BitBake would build a
114 cross compiler and <filename>glibc</filename> if they had not
115 already been built.
116 </para>
117
118 <para>
119 A useful BitBake option to consider is the
120 <filename>-k</filename> or <filename>--continue</filename>
121 option.
122 This option instructs BitBake to try and continue processing
123 the job as long as possible even after encountering an error.
124 When an error occurs, the target that failed and those that
125 depend on it cannot be remade.
126 However, when you use this option other dependencies can
127 still be processed.
128 </para>
129 </section>
130
131 <section id='concepts-components-metadata'>
132 <title>Metadata (Recipes)</title>
133
134 <para>
135 Files that have the <filename>.bb</filename> suffix are
136 "recipes" files.
137 In general, a recipe contains information about a single piece
138 of software.
139 This information includes the location from which to download
140 the unaltered source, any source patches to be applied to that
141 source (if needed), which special configuration options to
142 apply, how to compile the source files, and how to package the
143 compiled output.
144 </para>
145
146 <para>
147 The term "package" is sometimes used to refer to recipes.
148 However, since the word "package" is used for the packaged
149 output from the OpenEmbedded build system (i.e.
150 <filename>.ipk</filename> or <filename>.deb</filename> files),
151 this document avoids using the term "package" when referring
152 to recipes.
153 </para>
154 </section>
155
156 <section id='concepts-components-classes'>
157 <title>Classes</title>
158
159 <para>
160 Class files (<filename>.bbclass</filename>) contain information
161 that is useful to share between Metadata files.
162 An example is the
163 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
164 class, which contains common settings for any application that
165 Autotools uses.
166 The
167 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
168 chapter in the Yocto Project Reference Manual provides
169 details about classes and how to use them.
170 </para>
171 </section>
172
173 <section id='concepts-components-configuration'>
174 <title>Configuration</title>
175
176 <para>
177 The configuration files (<filename>.conf</filename>) define
178 various configuration variables that govern the OpenEmbedded
179 build process.
180 These files fall into several areas that define machine
181 configuration options, distribution configuration options,
182 compiler tuning options, general common configuration options,
183 and user configuration options in
184 <filename>local.conf</filename>, which is found in the
185 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
186 </para>
187 </section>
188 </section>
189
190 <section id='cm-layers'>
191 <title>Layers</title>
192
193 <para>
194 Layers are repositories that contain related sets of instructions
195 that tell the OpenEmbedded build system what to do.
196 You use different layers to logically separate information in your
197 build.
198 You can collaborate, share, and reuse layers.
199 The Layer Model simultaneously supports collaboration and
200 customization.
201 </para>
202
203 <para>
204 For more introductory information on the Yocto Project's layer
205 model, see the
206 "<ulink url='&YOCTO_DOCS_GS_URL;#the-yocto-project-layer-model'>The Yocto Project Layer Model</ulink>"
207 section in the Yocto Project Overview and Concepts Manual.
208 For procedures on how to create layers, see the
209 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
210 section in the Yocto Project Development Tasks Manual.
211 </para>
212 </section>
213
214 <section id="development-concepts">
215 <title>Development Concepts</title>
216
217 <para>
218 This section takes a more detailed look inside the build
219 process used by the OpenEmbedded build system.
220 The following diagram represents the build at a high level.
221 The remainder of this section expands on the fundamental input,
222 output, process, and
223 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
224 blocks that make up the build process.
225 </para>
226
227 <para id='general-yocto-environment-figure'>
228 <imagedata fileref="figures/yocto-environment-ref.png" align="center" width="8in" depth="4.25in" />
229 </para>
230
231 <para>
232 In general, the build process consists of several functional areas:
233 <itemizedlist>
234 <listitem><para>
235 <emphasis>User Configuration:</emphasis>
236 Metadata you can use to control the build process.
237 </para></listitem>
238 <listitem><para>
239 <emphasis>Metadata Layers:</emphasis>
240 Various layers that provide software, machine, and
241 distro Metadata.
242 </para></listitem>
243 <listitem><para>
244 <emphasis>Source Files:</emphasis>
245 Upstream releases, local projects, and SCMs.
246 </para></listitem>
247 <listitem><para>
248 <emphasis>Build System:</emphasis>
249 Processes under the control of
250 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>.
251 This block expands on how BitBake fetches source, applies
252 patches, completes compilation, analyzes output for package
253 generation, creates and tests packages, generates images,
254 and generates cross-development tools.
255 </para></listitem>
256 <listitem><para>
257 <emphasis>Package Feeds:</emphasis>
258 Directories containing output packages (RPM, DEB or IPK),
259 which are subsequently used in the construction of an
260 image or SDK, produced by the build system.
261 These feeds can also be copied and shared using a web
262 server or other means to facilitate extending or updating
263 existing images on devices at runtime if runtime package
264 management is enabled.
265 </para></listitem>
266 <listitem><para>
267 <emphasis>Images:</emphasis>
268 Images produced by the development process.
269 </para></listitem>
270 <listitem><para>
271 <emphasis>Application Development SDK:</emphasis>
272 Cross-development tools that are produced along with
273 an image or separately with BitBake.
274 </para></listitem>
275 </itemizedlist>
276 </para>
277
278 <section id="user-configuration">
279 <title>User Configuration</title>
280
281 <para>
282 User configuration helps define the build.
283 Through user configuration, you can tell BitBake the
284 target architecture for which you are building the image,
285 where to store downloaded source, and other build properties.
286 </para>
287
288 <para>
289 The following figure shows an expanded representation of the
290 "User Configuration" box of the
291 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>:
292 </para>
293
294 <para>
295 <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" />
296 </para>
297
298 <para>
299 BitBake needs some basic configuration files in order to
300 complete a build.
301 These files are <filename>*.conf</filename> files.
302 The minimally necessary ones reside as example files in the
303 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
304 For simplicity, this section refers to the Source Directory as
305 the "Poky Directory."
306 </para>
307
308 <para>
309 When you clone the <filename>poky</filename> Git repository
310 or you download and unpack a Yocto Project release, you
311 can set up the Source Directory to be named anything you want.
312 For this discussion, the cloned repository uses the default
313 name <filename>poky</filename>.
314 <note>
315 The
316 <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>
317 repository is primarily an aggregation of existing
318 repositories.
319 It is not a canonical upstream source.
320 </note>
321 </para>
322
323 <para>
324 The <filename>meta-poky</filename> layer inside Poky contains
325 a <filename>conf</filename> directory that has example
326 configuration files.
327 These example files are used as a basis for creating actual
328 configuration files when you source the build environment
329 script
330 (i.e.
331 <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>).
332 </para>
333
334 <para>
335 Sourcing the build environment script creates a
336 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
337 if one does not already exist.
338 BitBake uses the Build Directory for all its work during
339 builds.
340 The Build Directory has a <filename>conf</filename> directory
341 that contains default versions of your
342 <filename>local.conf</filename> and
343 <filename>bblayers.conf</filename> configuration files.
344 These default configuration files are created only if versions
345 do not already exist in the Build Directory at the time you
346 source the build environment setup script.
347 </para>
348
349 <para>
350 Because the Poky repository is fundamentally an aggregation of
351 existing repositories, some users might be familiar with
352 running the <filename>&OE_INIT_FILE;</filename> script
353 in the context of separate
354 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>
355 and BitBake repositories rather than a single Poky repository.
356 This discussion assumes the script is executed from
357 within a cloned or unpacked version of Poky.
358 </para>
359
360 <para>
361 Depending on where the script is sourced, different
362 sub-scripts are called to set up the Build Directory
363 (Yocto or OpenEmbedded).
364 Specifically, the script
365 <filename>scripts/oe-setup-builddir</filename> inside the
366 poky directory sets up the Build Directory and seeds the
367 directory (if necessary) with configuration files appropriate
368 for the Yocto Project development environment.
369 <note>
370 The <filename>scripts/oe-setup-builddir</filename> script
371 uses the <filename>$TEMPLATECONF</filename> variable to
372 determine which sample configuration files to locate.
373 </note>
374 </para>
375
376 <para>
377 The <filename>local.conf</filename> file provides many
378 basic variables that define a build environment.
379 Here is a list of a few.
380 To see the default configurations in a
381 <filename>local.conf</filename> file created by the build
382 environment script, see the
383 <filename>local.conf.sample</filename> in the
384 <filename>meta-poky</filename> layer:
385 <itemizedlist>
386 <listitem><para>
387 <emphasis>Parallelism Options:</emphasis>
388 Controlled by the
389 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></ulink>,
390 <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink>,
391 and
392 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></ulink>
393 variables.
394 </para></listitem>
395 <listitem><para>
396 <emphasis>Target Machine Selection:</emphasis>
397 Controlled by the
398 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>
399 variable.
400 </para></listitem>
401 <listitem><para>
402 <emphasis>Download Directory:</emphasis>
403 Controlled by the
404 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
405 variable.
406 </para></listitem>
407 <listitem><para>
408 <emphasis>Shared State Directory:</emphasis>
409 Controlled by the
410 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
411 variable.
412 </para></listitem>
413 <listitem><para>
414 <emphasis>Build Output:</emphasis>
415 Controlled by the
416 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
417 variable.
418 </para></listitem>
419 </itemizedlist>
420 <note>
421 Configurations set in the
422 <filename>conf/local.conf</filename> file can also be set
423 in the <filename>conf/site.conf</filename> and
424 <filename>conf/auto.conf</filename> configuration files.
425 </note>
426 </para>
427
428 <para>
429 The <filename>bblayers.conf</filename> file tells BitBake what
430 layers you want considered during the build.
431 By default, the layers listed in this file include layers
432 minimally needed by the build system.
433 However, you must manually add any custom layers you have
434 created.
435 You can find more information on working with the
436 <filename>bblayers.conf</filename> file in the
437 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
438 section in the Yocto Project Development Tasks Manual.
439 </para>
440
441 <para>
442 The files <filename>site.conf</filename> and
443 <filename>auto.conf</filename> are not created by the
444 environment initialization script.
445 If you want the <filename>site.conf</filename> file, you
446 need to create that yourself.
447 The <filename>auto.conf</filename> file is typically created by
448 an autobuilder:
449 <itemizedlist>
450 <listitem><para>
451 <emphasis><filename>site.conf</filename>:</emphasis>
452 You can use the <filename>conf/site.conf</filename>
453 configuration file to configure multiple
454 build directories.
455 For example, suppose you had several build environments
456 and they shared some common features.
457 You can set these default build properties here.
458 A good example is perhaps the packaging format to use
459 through the
460 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
461 variable.</para>
462
463 <para>One useful scenario for using the
464 <filename>conf/site.conf</filename> file is to extend
465 your
466 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
467 variable to include the path to a
468 <filename>conf/site.conf</filename>.
469 Then, when BitBake looks for Metadata using
470 <filename>BBPATH</filename>, it finds the
471 <filename>conf/site.conf</filename> file and applies
472 your common configurations found in the file.
473 To override configurations in a particular build
474 directory, alter the similar configurations within
475 that build directory's
476 <filename>conf/local.conf</filename> file.
477 </para></listitem>
478 <listitem><para>
479 <emphasis><filename>auto.conf</filename>:</emphasis>
480 The file is usually created and written to by
481 an autobuilder.
482 The settings put into the file are typically the
483 same as you would find in the
484 <filename>conf/local.conf</filename> or the
485 <filename>conf/site.conf</filename> files.
486 </para></listitem>
487 </itemizedlist>
488 </para>
489
490 <para>
491 You can edit all configuration files to further define
492 any particular build environment.
493 This process is represented by the "User Configuration Edits"
494 box in the figure.
495 </para>
496
497 <para>
498 When you launch your build with the
499 <filename>bitbake <replaceable>target</replaceable></filename>
500 command, BitBake sorts out the configurations to ultimately
501 define your build environment.
502 It is important to understand that the
503 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
504 reads the configuration files in a specific order:
505 <filename>site.conf</filename>, <filename>auto.conf</filename>,
506 and <filename>local.conf</filename>.
507 And, the build system applies the normal assignment statement
508 rules.
509 Because the files are parsed in a specific order, variable
510 assignments for the same variable could be affected.
511 For example, if the <filename>auto.conf</filename> file and
512 the <filename>local.conf</filename> set
513 <replaceable>variable1</replaceable> to different values,
514 because the build system parses <filename>local.conf</filename>
515 after <filename>auto.conf</filename>,
516 <replaceable>variable1</replaceable> is assigned the value from
517 the <filename>local.conf</filename> file.
518 </para>
519 </section>
520
521 <section id="metadata-machine-configuration-and-policy-configuration">
522 <title>Metadata, Machine Configuration, and Policy Configuration</title>
523
524 <para>
525 The previous section described the user configurations that
526 define BitBake's global behavior.
527 This section takes a closer look at the layers the build system
528 uses to further control the build.
529 These layers provide Metadata for the software, machine, and
530 policy.
531 </para>
532
533 <para>
534 In general, three types of layer input exist:
535 <itemizedlist>
536 <listitem><para>
537 <emphasis>Policy Configuration:</emphasis>
538 Distribution Layers provide top-level or general
539 policies for the image or SDK being built.
540 For example, this layer would dictate whether BitBake
541 produces RPM or IPK packages.
542 </para></listitem>
543 <listitem><para>
544 <emphasis>Machine Configuration:</emphasis>
545 Board Support Package (BSP) layers provide machine
546 configurations.
547 This type of information is specific to a particular
548 target architecture.
549 </para></listitem>
550 <listitem><para>
551 <emphasis>Metadata:</emphasis>
552 Software layers contain user-supplied recipe files,
553 patches, and append files.
554 </para></listitem>
555 </itemizedlist>
556 </para>
557
558 <para>
559 The following figure shows an expanded representation of the
560 Metadata, Machine Configuration, and Policy Configuration input
561 (layers) boxes of the
562 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>:
563 </para>
564
565 <para>
566 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="7.5in" />
567 </para>
568
569 <para>
570 In general, all layers have a similar structure.
571 They all contain a licensing file
572 (e.g. <filename>COPYING</filename>) if the layer is to be
573 distributed, a <filename>README</filename> file as good
574 practice and especially if the layer is to be distributed, a
575 configuration directory, and recipe directories.
576 </para>
577
578 <para>
579 The Yocto Project has many layers that can be used.
580 You can see a web-interface listing of them on the
581 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
582 page.
583 The layers appear at the bottom categorized under
584 "Yocto Metadata Layers."
585 These layers are fundamentally a subset of the
586 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Metadata Index</ulink>,
587 which lists all layers provided by the OpenEmbedded community.
588 <note>
589 Layers exist in the Yocto Project Source Repositories that
590 cannot be found in the OpenEmbedded Metadata Index.
591 These layers are either deprecated or experimental
592 in nature.
593 </note>
594 </para>
595
596 <para>
597 BitBake uses the <filename>conf/bblayers.conf</filename> file,
598 which is part of the user configuration, to find what layers it
599 should be using as part of the build.
600 </para>
601
602 <para>
603 For more information on layers, see the
604 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
605 section in the Yocto Project Development Tasks Manual.
606 </para>
607
608 <section id="distro-layer">
609 <title>Distro Layer</title>
610
611 <para>
612 The distribution layer provides policy configurations for
613 your distribution.
614 Best practices dictate that you isolate these types of
615 configurations into their own layer.
616 Settings you provide in
617 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
618 similar settings that BitBake finds in your
619 <filename>conf/local.conf</filename> file in the Build
620 Directory.
621 </para>
622
623 <para>
624 The following list provides some explanation and references
625 for what you typically find in the distribution layer:
626 <itemizedlist>
627 <listitem><para>
628 <emphasis>classes:</emphasis>
629 Class files (<filename>.bbclass</filename>) hold
630 common functionality that can be shared among
631 recipes in the distribution.
632 When your recipes inherit a class, they take on the
633 settings and functions for that class.
634 You can read more about class files in the
635 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
636 chapter of the Yocto Reference Manual.
637 </para></listitem>
638 <listitem><para>
639 <emphasis>conf:</emphasis>
640 This area holds configuration files for the
641 layer (<filename>conf/layer.conf</filename>),
642 the distribution
643 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
644 and any distribution-wide include files.
645 </para></listitem>
646 <listitem><para>
647 <emphasis>recipes-*:</emphasis>
648 Recipes and append files that affect common
649 functionality across the distribution.
650 This area could include recipes and append files
651 to add distribution-specific configuration,
652 initialization scripts, custom image recipes,
653 and so forth.
654 </para></listitem>
655 </itemizedlist>
656 </para>
657 </section>
658
659 <section id="bsp-layer">
660 <title>BSP Layer</title>
661
662 <para>
663 The BSP Layer provides machine configurations.
664 Everything in this layer is specific to the machine for
665 which you are building the image or the SDK.
666 A common structure or form is defined for BSP layers.
667 You can learn more about this structure in the
668 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
669 <note>
670 In order for a BSP layer to be considered compliant
671 with the Yocto Project, it must meet some structural
672 requirements.
673 </note>
674 </para>
675
676 <para>
677 The BSP Layer's configuration directory contains
678 configuration files for the machine
679 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>)
680 and, of course, the layer
681 (<filename>conf/layer.conf</filename>).
682 </para>
683
684 <para>
685 The remainder of the layer is dedicated to specific recipes
686 by function: <filename>recipes-bsp</filename>,
687 <filename>recipes-core</filename>,
688 <filename>recipes-graphics</filename>, and
689 <filename>recipes-kernel</filename>.
690 Metadata can exist for multiple formfactors, graphics
691 support systems, and so forth.
692 <note>
693 While the figure shows several
694 <filename>recipes-*</filename> directories, not all
695 these directories appear in all BSP layers.
696 </note>
697 </para>
698 </section>
699
700 <section id="software-layer">
701 <title>Software Layer</title>
702
703 <para>
704 The software layer provides the Metadata for additional
705 software packages used during the build.
706 This layer does not include Metadata that is specific to
707 the distribution or the machine, which are found in their
708 respective layers.
709 </para>
710
711 <para>
712 This layer contains any new recipes that your project
713 needs in the form of recipe files.
714 </para>
715 </section>
716 </section>
717
718 <section id="sources-dev-environment">
719 <title>Sources</title>
720
721 <para>
722 In order for the OpenEmbedded build system to create an
723 image or any target, it must be able to access source files.
724 The
725 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>
726 represents source files using the "Upstream Project Releases",
727 "Local Projects", and "SCMs (optional)" boxes.
728 The figure represents mirrors, which also play a role in
729 locating source files, with the "Source Mirror(s)" box.
730 </para>
731
732 <para>
733 The method by which source files are ultimately organized is
734 a function of the project.
735 For example, for released software, projects tend to use
736 tarballs or other archived files that can capture the
737 state of a release guaranteeing that it is statically
738 represented.
739 On the other hand, for a project that is more dynamic or
740 experimental in nature, a project might keep source files in a
741 repository controlled by a Source Control Manager (SCM) such as
742 Git.
743 Pulling source from a repository allows you to control
744 the point in the repository (the revision) from which you
745 want to build software.
746 Finally, a combination of the two might exist, which would
747 give the consumer a choice when deciding where to get
748 source files.
749 </para>
750
751 <para>
752 BitBake uses the
753 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
754 variable to point to source files regardless of their location.
755 Each recipe must have a <filename>SRC_URI</filename> variable
756 that points to the source.
757 </para>
758
759 <para>
760 Another area that plays a significant role in where source
761 files come from is pointed to by the
762 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
763 variable.
764 This area is a cache that can hold previously downloaded
765 source.
766 You can also instruct the OpenEmbedded build system to create
767 tarballs from Git repositories, which is not the default
768 behavior, and store them in the <filename>DL_DIR</filename>
769 by using the
770 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
771 variable.
772 </para>
773
774 <para>
775 Judicious use of a <filename>DL_DIR</filename> directory can
776 save the build system a trip across the Internet when looking
777 for files.
778 A good method for using a download directory is to have
779 <filename>DL_DIR</filename> point to an area outside of your
780 Build Directory.
781 Doing so allows you to safely delete the Build Directory
782 if needed without fear of removing any downloaded source file.
783 </para>
784
785 <para>
786 The remainder of this section provides a deeper look into the
787 source files and the mirrors.
788 Here is a more detailed look at the source file area of the
789 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>:
790 <imagedata fileref="figures/source-input.png" align="center" width="7in" depth="7.5in" />
791 </para>
792
793 <section id='upstream-project-releases'>
794 <title>Upstream Project Releases</title>
795
796 <para>
797 Upstream project releases exist anywhere in the form of an
798 archived file (e.g. tarball or zip file).
799 These files correspond to individual recipes.
800 For example, the figure uses specific releases each for
801 BusyBox, Qt, and Dbus.
802 An archive file can be for any released product that can be
803 built using a recipe.
804 </para>
805 </section>
806
807 <section id='local-projects'>
808 <title>Local Projects</title>
809
810 <para>
811 Local projects are custom bits of software the user
812 provides.
813 These bits reside somewhere local to a project - perhaps
814 a directory into which the user checks in items (e.g.
815 a local directory containing a development source tree
816 used by the group).
817 </para>
818
819 <para>
820 The canonical method through which to include a local
821 project is to use the
822 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
823 class to include that local project.
824 You use either the <filename>local.conf</filename> or a
825 recipe's append file to override or set the
826 recipe to point to the local directory on your disk to pull
827 in the whole source tree.
828 </para>
829
830 <para>
831 For information on how to use the
832 <filename>externalsrc</filename> class, see the
833 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></ulink>"
834 section.
835 </para>
836 </section>
837
838 <section id='scms'>
839 <title>Source Control Managers (Optional)</title>
840
841 <para>
842 Another place the build system can get source files from is
843 through an SCM such as Git or Subversion.
844 In this case, a repository is cloned or checked out.
845 The
846 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
847 task inside BitBake uses
848 the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
849 variable and the argument's prefix to determine the correct
850 fetcher module.
851 <note>
852 For information on how to have the OpenEmbedded build
853 system generate tarballs for Git repositories and place
854 them in the
855 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
856 directory, see the
857 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
858 variable.
859 </note>
860 </para>
861
862 <para>
863 When fetching a repository, BitBake uses the
864 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
865 variable to determine the specific revision from which to
866 build.
867 </para>
868 </section>
869
870 <section id='source-mirrors'>
871 <title>Source Mirror(s)</title>
872
873 <para>
874 Two kinds of mirrors exist: pre-mirrors and regular
875 mirrors.
876 The
877 <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink>
878 and
879 <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink>
880 variables point to these, respectively.
881 BitBake checks pre-mirrors before looking upstream for any
882 source files.
883 Pre-mirrors are appropriate when you have a shared
884 directory that is not a directory defined by the
885 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
886 variable.
887 A Pre-mirror typically points to a shared directory that is
888 local to your organization.
889 </para>
890
891 <para>
892 Regular mirrors can be any site across the Internet
893 that is used as an alternative location for source
894 code should the primary site not be functioning for
895 some reason or another.
896 </para>
897 </section>
898 </section>
899
900 <section id="package-feeds-dev-environment">
901 <title>Package Feeds</title>
902
903 <para>
904 When the OpenEmbedded build system generates an image or an
905 SDK, it gets the packages from a package feed area located
906 in the
907 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
908 The
909 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>
910 shows this package feeds area in the upper-right corner.
911 </para>
912
913 <para>
914 This section looks a little closer into the package feeds
915 area used by the build system.
916 Here is a more detailed look at the area:
917 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
918 </para>
919
920 <para>
921 Package feeds are an intermediary step in the build process.
922 The OpenEmbedded build system provides classes to generate
923 different package types, and you specify which classes to
924 enable through the
925 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
926 variable.
927 Before placing the packages into package feeds,
928 the build process validates them with generated output quality
929 assurance checks through the
930 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink>
931 class.
932 </para>
933
934 <para>
935 The package feed area resides in the Build Directory.
936 The directory the build system uses to temporarily store
937 packages is determined by a combination of variables and the
938 particular package manager in use.
939 See the "Package Feeds" box in the illustration and note the
940 information to the right of that area.
941 In particular, the following defines where package files are
942 kept:
943 <itemizedlist>
944 <listitem><para>
945 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
946 Defined as <filename>tmp/deploy</filename> in the Build
947 Directory.
948 </para></listitem>
949 <listitem><para>
950 <filename>DEPLOY_DIR_*</filename>:
951 Depending on the package manager used, the package type
952 sub-folder.
953 Given RPM, IPK, or DEB packaging and tarball creation,
954 the
955 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>,
956 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>,
957 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>,
958 or
959 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>,
960 variables are used, respectively.
961 </para></listitem>
962 <listitem><para>
963 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
964 Defines architecture-specific sub-folders.
965 For example, packages could exist for the i586 or
966 qemux86 architectures.
967 </para></listitem>
968 </itemizedlist>
969 </para>
970
971 <para>
972 BitBake uses the
973 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
974 tasks to generate packages and place them into the package
975 holding area (e.g. <filename>do_package_write_ipk</filename>
976 for IPK packages).
977 See the
978 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>",
979 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>",
980 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>",
981 and
982 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>"
983 sections in the Yocto Project Reference Manual
984 for additional information.
985 As an example, consider a scenario where an IPK packaging
986 manager is being used and package architecture support for
987 both i586 and qemux86 exist.
988 Packages for the i586 architecture are placed in
989 <filename>build/tmp/deploy/ipk/i586</filename>, while packages
990 for the qemux86 architecture are placed in
991 <filename>build/tmp/deploy/ipk/qemux86</filename>.
992 </para>
993 </section>
994
995 <section id='bitbake-dev-environment'>
996 <title>BitBake</title>
997
998 <para>
999 The OpenEmbedded build system uses
1000 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
1001 to produce images.
1002 You can see from the
1003 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>,
1004 the BitBake area consists of several functional areas.
1005 This section takes a closer look at each of those areas.
1006 </para>
1007
1008 <para>
1009 Separate documentation exists for the BitBake tool.
1010 See the
1011 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>
1012 for reference material on BitBake.
1013 </para>
1014
1015 <section id='source-fetching-dev-environment'>
1016 <title>Source Fetching</title>
1017
1018 <para>
1019 The first stages of building a recipe are to fetch and
1020 unpack the source code:
1021 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
1022 </para>
1023
1024 <para>
1025 The
1026 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
1027 and
1028 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1029 tasks fetch the source files and unpack them into the work
1030 directory.
1031 <note>
1032 For every local file (e.g. <filename>file://</filename>)
1033 that is part of a recipe's
1034 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1035 statement, the OpenEmbedded build system takes a
1036 checksum of the file for the recipe and inserts the
1037 checksum into the signature for the
1038 <filename>do_fetch</filename>.
1039 If any local file has been modified, the
1040 <filename>do_fetch</filename> task and all tasks that
1041 depend on it are re-executed.
1042 </note>
1043 By default, everything is accomplished in the
1044 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>,
1045 which has a defined structure.
1046 For additional general information on the Build Directory,
1047 see the
1048 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>"
1049 section in the Yocto Project Reference Manual.
1050 </para>
1051
1052 <para>
1053 Unpacked source files are pointed to by the
1054 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1055 variable.
1056 Each recipe has an area in the Build Directory where the
1057 unpacked source code resides.
1058 The name of that directory for any given recipe is defined
1059 from several different variables.
1060 You can see the variables that define these directories
1061 by looking at the figure:
1062 <itemizedlist>
1063 <listitem><para>
1064 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>:
1065 The base directory where the OpenEmbedded build
1066 system performs all its work during the build.
1067 </para></listitem>
1068 <listitem><para>
1069 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1070 The architecture of the built package or packages.
1071 </para></listitem>
1072 <listitem><para>
1073 <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>:
1074 The operating system of the target device.
1075 </para></listitem>
1076 <listitem><para>
1077 <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>:
1078 The name of the built package.
1079 </para></listitem>
1080 <listitem><para>
1081 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1082 The version of the recipe used to build the
1083 package.
1084 </para></listitem>
1085 <listitem><para>
1086 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>:
1087 The revision of the recipe used to build the
1088 package.
1089 </para></listitem>
1090 <listitem><para>
1091 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>:
1092 The location within <filename>TMPDIR</filename>
1093 where a specific package is built.
1094 </para></listitem>
1095 <listitem><para>
1096 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>:
1097 Contains the unpacked source files for a given
1098 recipe.
1099 </para></listitem>
1100 </itemizedlist>
1101 </para>
1102 </section>
1103
1104 <section id='patching-dev-environment'>
1105 <title>Patching</title>
1106
1107 <para>
1108 Once source code is fetched and unpacked, BitBake locates
1109 patch files and applies them to the source files:
1110 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
1111 </para>
1112
1113 <para>
1114 The
1115 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1116 task processes recipes by using the
1117 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1118 variable to locate applicable patch files, which by default
1119 are <filename>*.patch</filename> or
1120 <filename>*.diff</filename> files, or any file if
1121 "apply=yes" is specified for the file in
1122 <filename>SRC_URI</filename>.
1123 </para>
1124
1125 <para>
1126 BitBake finds and applies multiple patches for a single
1127 recipe in the order in which it finds the patches.
1128 Patches are applied to the recipe's source files located
1129 in the
1130 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1131 directory.
1132 </para>
1133
1134 <para>
1135 For more information on how the source directories are
1136 created, see the
1137 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
1138 section.
1139 </para>
1140 </section>
1141
1142 <section id='configuration-and-compilation-dev-environment'>
1143 <title>Configuration and Compilation</title>
1144
1145 <para>
1146 After source code is patched, BitBake executes tasks that
1147 configure and compile the source code:
1148 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
1149 </para>
1150
1151 <para>
1152 This step in the build process consists of three tasks:
1153 <itemizedlist>
1154 <listitem><para>
1155 <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>:
1156 This task sets up the two sysroots in
1157 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>
1158 (i.e. <filename>recipe-sysroot</filename> and
1159 <filename>recipe-sysroot-native</filename>) so that
1160 the sysroots contain the contents of the
1161 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1162 tasks of the recipes on which the recipe
1163 containing the tasks depends.
1164 A sysroot exists for both the target and for the
1165 native binaries, which run on the host system.
1166 </para></listitem>
1167 <listitem><para>
1168 <emphasis><filename>do_configure</filename></emphasis>:
1169 This task configures the source by enabling and
1170 disabling any build-time and configuration options
1171 for the software being built.
1172 Configurations can come from the recipe itself as
1173 well as from an inherited class.
1174 Additionally, the software itself might configure
1175 itself depending on the target for which it is
1176 being built.</para>
1177
1178 <para>The configurations handled by the
1179 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink>
1180 task are specific to source code configuration for
1181 the source code being built by the recipe.</para>
1182
1183 <para>If you are using the
1184 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1185 class, you can add additional configuration options
1186 by using the
1187 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
1188 or
1189 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
1190 variables.
1191 For information on how this variable works within
1192 that class, see the
1193 <filename>meta/classes/autotools.bbclass</filename>
1194 file.
1195 </para></listitem>
1196 <listitem><para>
1197 <emphasis><filename>do_compile</filename></emphasis>:
1198 Once a configuration task has been satisfied, BitBake
1199 compiles the source using the
1200 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink>
1201 task.
1202 Compilation occurs in the directory pointed to by
1203 the
1204 <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink>
1205 variable.
1206 Realize that the <filename>B</filename> directory
1207 is, by default, the same as the
1208 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1209 directory.
1210 </para></listitem>
1211 <listitem><para>
1212 <emphasis><filename>do_install</filename></emphasis>:
1213 Once compilation is done, BitBake executes the
1214 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1215 task.
1216 This task copies files from the
1217 <filename>B</filename> directory and places them
1218 in a holding area pointed to by the
1219 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1220 variable.
1221 </para></listitem>
1222 </itemizedlist>
1223 </para>
1224 </section>
1225
1226 <section id='package-splitting-dev-environment'>
1227 <title>Package Splitting</title>
1228
1229 <para>
1230 After source code is configured and compiled, the
1231 OpenEmbedded build system analyzes
1232 the results and splits the output into packages:
1233 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
1234 </para>
1235
1236 <para>
1237 The
1238 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
1239 and
1240 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
1241 tasks combine to analyze the files found in the
1242 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1243 directory and split them into subsets based on available
1244 packages and files.
1245 The analyzing process involves the following as well as
1246 other items: splitting out debugging symbols, looking at
1247 shared library dependencies between packages, and looking
1248 at package relationships.
1249 The <filename>do_packagedata</filename> task creates
1250 package metadata based on the analysis such that the
1251 OpenEmbedded build system can generate the final packages.
1252 Working, staged, and intermediate results of the analysis
1253 and package splitting process use these areas:
1254 <itemizedlist>
1255 <listitem><para>
1256 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>:
1257 The destination directory for packages before they
1258 are split.
1259 </para></listitem>
1260 <listitem><para>
1261 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>:
1262 A shared, global-state directory that holds data
1263 generated during the packaging process.
1264 </para></listitem>
1265 <listitem><para>
1266 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>:
1267 A temporary work area used by the
1268 <filename>do_package</filename> task.
1269 </para></listitem>
1270 <listitem><para>
1271 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>:
1272 The parent directory for packages after they have
1273 been split.
1274 </para></listitem>
1275 </itemizedlist>
1276 The
1277 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>
1278 variable defines the files that go into each package in
1279 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>.
1280 If you want details on how this is accomplished, you can
1281 look at the
1282 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-package'><filename>package</filename></ulink>
1283 class.
1284 </para>
1285
1286 <para>
1287 Depending on the type of packages being created (RPM, DEB,
1288 or IPK), the
1289 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1290 task creates the actual packages and places them in the
1291 Package Feed area, which is
1292 <filename>${TMPDIR}/deploy</filename>.
1293 You can see the
1294 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
1295 section for more detail on that part of the build process.
1296 <note>
1297 Support for creating feeds directly from the
1298 <filename>deploy/*</filename> directories does not
1299 exist.
1300 Creating such feeds usually requires some kind of feed
1301 maintenance mechanism that would upload the new
1302 packages into an official package feed (e.g. the
1303 Ångström distribution).
1304 This functionality is highly distribution-specific
1305 and thus is not provided out of the box.
1306 </note>
1307 </para>
1308 </section>
1309
1310 <section id='image-generation-dev-environment'>
1311 <title>Image Generation</title>
1312
1313 <para>
1314 Once packages are split and stored in the Package Feeds
1315 area, the OpenEmbedded build system uses BitBake to
1316 generate the root filesystem image:
1317 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
1318 </para>
1319
1320 <para>
1321 The image generation process consists of several stages and
1322 depends on several tasks and variables.
1323 The
1324 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>
1325 task creates the root filesystem (file and directory
1326 structure) for an image.
1327 This task uses several key variables to help create the
1328 list of packages to actually install:
1329 <itemizedlist>
1330 <listitem><para>
1331 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>:
1332 Lists out the base set of packages to install from
1333 the Package Feeds area.
1334 </para></listitem>
1335 <listitem><para>
1336 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>:
1337 Specifies packages that should not be installed.
1338 </para></listitem>
1339 <listitem><para>
1340 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>:
1341 Specifies features to include in the image.
1342 Most of these features map to additional packages
1343 for installation.
1344 </para></listitem>
1345 <listitem><para>
1346 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>:
1347 Specifies the package backend to use and
1348 consequently helps determine where to locate
1349 packages within the Package Feeds area.
1350 </para></listitem>
1351 <listitem><para>
1352 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>:
1353 Determines the language(s) for which additional
1354 language support packages are installed.
1355 </para></listitem>
1356 <listitem><para>
1357 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>:
1358 The final list of packages passed to the package manager
1359 for installation into the image.
1360 </para></listitem>
1361 </itemizedlist>
1362 </para>
1363
1364 <para>
1365 With
1366 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink>
1367 pointing to the location of the filesystem under
1368 construction and the <filename>PACKAGE_INSTALL</filename>
1369 variable providing the final list of packages to install,
1370 the root file system is created.
1371 </para>
1372
1373 <para>
1374 Package installation is under control of the package
1375 manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of
1376 whether or not package management is enabled for the
1377 target.
1378 At the end of the process, if package management is not
1379 enabled for the target, the package manager's data files
1380 are deleted from the root filesystem.
1381 As part of the final stage of package installation,
1382 postinstall scripts that are part of the packages are run.
1383 Any scripts that fail to run
1384 on the build host are run on the target when the target
1385 system is first booted.
1386 If you are using a
1387 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
1388 all the post installation scripts must succeed during the
1389 package installation phase since the root filesystem is
1390 read-only.
1391 </para>
1392
1393 <para>
1394 The final stages of the <filename>do_rootfs</filename> task
1395 handle post processing.
1396 Post processing includes creation of a manifest file and
1397 optimizations.
1398 </para>
1399
1400 <para>
1401 The manifest file (<filename>.manifest</filename>) resides
1402 in the same directory as the root filesystem image.
1403 This file lists out, line-by-line, the installed packages.
1404 The manifest file is useful for the
1405 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink>
1406 class, for example, to determine whether or not to run
1407 specific tests.
1408 See the
1409 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink>
1410 variable for additional information.
1411 </para>
1412
1413 <para>
1414 Optimizing processes run across the image include
1415 <filename>mklibs</filename>, <filename>prelink</filename>,
1416 and any other post-processing commands as defined by the
1417 <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink>
1418 variable.
1419 The <filename>mklibs</filename> process optimizes the size
1420 of the libraries, while the <filename>prelink</filename>
1421 process optimizes the dynamic linking of shared libraries
1422 to reduce start up time of executables.
1423 </para>
1424
1425 <para>
1426 After the root filesystem is built, processing begins on
1427 the image through the
1428 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink>
1429 task.
1430 The build system runs any pre-processing commands as
1431 defined by the
1432 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink>
1433 variable.
1434 This variable specifies a list of functions to call before
1435 the OpenEmbedded build system creates the final image
1436 output files.
1437 </para>
1438
1439 <para>
1440 The OpenEmbedded build system dynamically creates
1441 <filename>do_image_*</filename> tasks as needed, based
1442 on the image types specified in the
1443 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
1444 variable.
1445 The process turns everything into an image file or a set of
1446 image files and can compress the root filesystem image to
1447 reduce the overall size of the image.
1448 The formats used for the root filesystem depend on the
1449 <filename>IMAGE_FSTYPES</filename> variable.
1450 Compression depends on whether the formats support
1451 compression.
1452 </para>
1453
1454 <para>
1455 As an example, a dynamically created task when creating a
1456 particular image <replaceable>type</replaceable> would
1457 take the following form:
1458 <literallayout class='monospaced'>
1459 do_image_<replaceable>type</replaceable>
1460 </literallayout>
1461 So, if the <replaceable>type</replaceable> as specified by
1462 the <filename>IMAGE_FSTYPES</filename> were
1463 <filename>ext4</filename>, the dynamically generated task
1464 would be as follows:
1465 <literallayout class='monospaced'>
1466 do_image_ext4
1467 </literallayout>
1468 </para>
1469
1470 <para>
1471 The final task involved in image creation is the
1472 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink>
1473 task.
1474 This task completes the image by applying any image
1475 post processing as defined through the
1476 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink>
1477 variable.
1478 The variable specifies a list of functions to call once the
1479 OpenEmbedded build system has created the final image
1480 output files.
1481 <note>
1482 The entire image generation process is run under
1483 Pseudo.
1484 Running under Pseudo ensures that the files in the
1485 root filesystem have correct ownership.
1486 </note>
1487 </para>
1488 </section>
1489
1490 <section id='sdk-generation-dev-environment'>
1491 <title>SDK Generation</title>
1492
1493 <para>
1494 The OpenEmbedded build system uses BitBake to generate the
1495 Software Development Kit (SDK) installer script for both
1496 the standard and extensible SDKs:
1497 <imagedata fileref="figures/sdk-generation.png" align="center" />
1498 <note>
1499 For more information on the cross-development toolchain
1500 generation, see the
1501 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1502 section.
1503 For information on advantages gained when building a
1504 cross-development toolchain using the
1505 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1506 task, see the
1507 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
1508 section in the Yocto Project Application Development
1509 and the Extensible Software Development Kit (SDK)
1510 manual.
1511 </note>
1512 </para>
1513
1514 <para>
1515 Like image generation, the SDK script process consists of
1516 several stages and depends on many variables.
1517 The <filename>do_populate_sdk</filename> and
1518 <filename>do_populate_sdk_ext</filename> tasks use these
1519 key variables to help create the list of packages to
1520 actually install.
1521 For information on the variables listed in the figure,
1522 see the
1523 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1524 section.
1525 </para>
1526
1527 <para>
1528 The <filename>do_populate_sdk</filename> task helps create
1529 the standard SDK and handles two parts: a target part and a
1530 host part.
1531 The target part is the part built for the target hardware
1532 and includes libraries and headers.
1533 The host part is the part of the SDK that runs on the
1534 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>.
1535 </para>
1536
1537 <para>
1538 The <filename>do_populate_sdk_ext</filename> task helps
1539 create the extensible SDK and handles host and target parts
1540 differently than its counter part does for the standard SDK.
1541 For the extensible SDK, the task encapsulates the build
1542 system, which includes everything needed (host and target)
1543 for the SDK.
1544 </para>
1545
1546 <para>
1547 Regardless of the type of SDK being constructed, the
1548 tasks perform some cleanup after which a cross-development
1549 environment setup script and any needed configuration files
1550 are created.
1551 The final output is the Cross-development
1552 toolchain installation script (<filename>.sh</filename>
1553 file), which includes the environment setup script.
1554 </para>
1555 </section>
1556
1557 <section id='stamp-files-and-the-rerunning-of-tasks'>
1558 <title>Stamp Files and the Rerunning of Tasks</title>
1559
1560 <para>
1561 For each task that completes successfully, BitBake writes a
1562 stamp file into the
1563 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink>
1564 directory.
1565 The beginning of the stamp file's filename is determined
1566 by the
1567 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
1568 variable, and the end of the name consists of the task's
1569 name and current
1570 <ulink url='&YOCTO_DOCS_GS_URL;#overview-checksums'>input checksum</ulink>.
1571 <note>
1572 This naming scheme assumes that
1573 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
1574 is "OEBasicHash", which is almost always the case in
1575 current OpenEmbedded.
1576 </note>
1577 To determine if a task needs to be rerun, BitBake checks
1578 if a stamp file with a matching input checksum exists
1579 for the task.
1580 If such a stamp file exists, the task's output is
1581 assumed to exist and still be valid.
1582 If the file does not exist, the task is rerun.
1583 <note>
1584 <para>The stamp mechanism is more general than the
1585 shared state (sstate) cache mechanism described in the
1586 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
1587 section.
1588 BitBake avoids rerunning any task that has a valid
1589 stamp file, not just tasks that can be accelerated
1590 through the sstate cache.</para>
1591
1592 <para>However, you should realize that stamp files only
1593 serve as a marker that some work has been done and that
1594 these files do not record task output.
1595 The actual task output would usually be somewhere in
1596 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
1597 (e.g. in some recipe's
1598 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.)
1599 What the sstate cache mechanism adds is a way to cache
1600 task output that can then be shared between build
1601 machines.</para>
1602 </note>
1603 Since <filename>STAMPS_DIR</filename> is usually a
1604 subdirectory of <filename>TMPDIR</filename>, removing
1605 <filename>TMPDIR</filename> will also remove
1606 <filename>STAMPS_DIR</filename>, which means tasks will
1607 properly be rerun to repopulate
1608 <filename>TMPDIR</filename>.
1609 </para>
1610
1611 <para>
1612 If you want some task to always be considered "out of
1613 date", you can mark it with the
1614 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
1615 varflag.
1616 If some other task depends on such a task, then that
1617 task will also always be considered out of date, which
1618 might not be what you want.
1619 </para>
1620
1621 <para>
1622 For details on how to view information about a task's
1623 signature, see the
1624 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
1625 section in the Yocto Project Development Tasks Manual.
1626 </para>
1627 </section>
1628
1629 <section id='setscene-tasks-and-shared-state'>
1630 <title>Setscene Tasks and Shared State</title>
1631
1632 <para>
1633 The description of tasks so far assumes that BitBake needs
1634 to build everything and there are no prebuilt objects
1635 available.
1636 BitBake does support skipping tasks if prebuilt objects are
1637 available.
1638 These objects are usually made available in the form of a
1639 shared state (sstate) cache.
1640 <note>
1641 For information on variables affecting sstate, see the
1642 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
1643 and
1644 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
1645 variables.
1646 </note>
1647 </para>
1648
1649 <para>
1650 The idea of a setscene task (i.e
1651 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
1652 is a version of the task where
1653 instead of building something, BitBake can skip to the end
1654 result and simply place a set of files into specific
1655 locations as needed.
1656 In some cases, it makes sense to have a setscene task
1657 variant (e.g. generating package files in the
1658 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1659 task).
1660 In other cases, it does not make sense, (e.g. a
1661 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1662 task or
1663 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1664 task) since the work involved would be equal to or greater
1665 than the underlying task.
1666 </para>
1667
1668 <para>
1669 In the OpenEmbedded build system, the common tasks that
1670 have setscene variants are
1671 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>,
1672 <filename>do_package_write_*</filename>,
1673 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>,
1674 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>,
1675 and
1676 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>.
1677 Notice that these are most of the tasks whose output is an
1678 end result.
1679 </para>
1680
1681 <para>
1682 The OpenEmbedded build system has knowledge of the
1683 relationship between these tasks and other tasks that
1684 precede them.
1685 For example, if BitBake runs
1686 <filename>do_populate_sysroot_setscene</filename> for
1687 something, there is little point in running any of the
1688 <filename>do_fetch</filename>,
1689 <filename>do_unpack</filename>,
1690 <filename>do_patch</filename>,
1691 <filename>do_configure</filename>,
1692 <filename>do_compile</filename>, and
1693 <filename>do_install</filename> tasks.
1694 However, if <filename>do_package</filename> needs to be
1695 run, BitBake would need to run those other tasks.
1696 </para>
1697
1698 <para>
1699 It becomes more complicated if everything can come
1700 from an sstate cache because some objects are simply
1701 not required at all.
1702 For example, you do not need a compiler or native tools,
1703 such as quilt, if there is nothing to compile or patch.
1704 If the <filename>do_package_write_*</filename> packages
1705 are available from sstate, BitBake does not need the
1706 <filename>do_package</filename> task data.
1707 </para>
1708
1709 <para>
1710 To handle all these complexities, BitBake runs in two
1711 phases.
1712 The first is the "setscene" stage.
1713 During this stage, BitBake first checks the sstate cache
1714 for any targets it is planning to build.
1715 BitBake does a fast check to see if the object exists
1716 rather than a complete download.
1717 If nothing exists, the second phase, which is the setscene
1718 stage, completes and the main build proceeds.
1719 </para>
1720
1721 <para>
1722 If objects are found in the sstate cache, the OpenEmbedded
1723 build system works backwards from the end targets specified
1724 by the user.
1725 For example, if an image is being built, the OpenEmbedded
1726 build system first looks for the packages needed for
1727 that image and the tools needed to construct an image.
1728 If those are available, the compiler is not needed.
1729 Thus, the compiler is not even downloaded.
1730 If something was found to be unavailable, or the
1731 download or setscene task fails, the OpenEmbedded build
1732 system then tries to install dependencies, such as the
1733 compiler, from the cache.
1734 </para>
1735
1736 <para>
1737 The availability of objects in the sstate cache is
1738 handled by the function specified by the
1739 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
1740 variable and returns a list of the objects that are
1741 available.
1742 The function specified by the
1743 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
1744 variable is the function that determines whether a given
1745 dependency needs to be followed, and whether for any given
1746 relationship the function needs to be passed.
1747 The function returns a True or False value.
1748 </para>
1749 </section>
1750 </section>
1751
1752 <section id='images-dev-environment'>
1753 <title>Images</title>
1754
1755 <para>
1756 The images produced by the OpenEmbedded build system
1757 are compressed forms of the
1758 root filesystem that are ready to boot on a target device.
1759 You can see from the
1760 <link linkend='general-yocto-environment-figure'>general Build Process figure</link>
1761 that BitBake output, in part, consists of images.
1762 This section is going to look more closely at this output:
1763 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1764 </para>
1765
1766 <para>
1767 For a list of example images that the Yocto Project provides,
1768 see the
1769 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>"
1770 chapter in the Yocto Project Reference Manual.
1771 </para>
1772
1773 <para>
1774 Images are written out to the
1775 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1776 inside the
1777 <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
1778 folder as shown in the figure.
1779 This folder contains any files expected to be loaded on the
1780 target device.
1781 The
1782 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>
1783 variable points to the <filename>deploy</filename> directory,
1784 while the
1785 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink>
1786 variable points to the appropriate directory containing images
1787 for the current configuration.
1788 <itemizedlist>
1789 <listitem><para>
1790 <filename><replaceable>kernel-image</replaceable></filename>:
1791 A kernel binary file.
1792 The
1793 <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink>
1794 variable setting determines the naming scheme for the
1795 kernel image file.
1796 Depending on that variable, the file could begin with
1797 a variety of naming strings.
1798 The
1799 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1800 directory can contain multiple image files for the
1801 machine.
1802 </para></listitem>
1803 <listitem><para>
1804 <filename><replaceable>root-filesystem-image</replaceable></filename>:
1805 Root filesystems for the target device (e.g.
1806 <filename>*.ext3</filename> or
1807 <filename>*.bz2</filename> files).
1808 The
1809 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
1810 variable setting determines the root filesystem image
1811 type.
1812 The
1813 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1814 directory can contain multiple root filesystems for the
1815 machine.
1816 </para></listitem>
1817 <listitem><para>
1818 <filename><replaceable>kernel-modules</replaceable></filename>:
1819 Tarballs that contain all the modules built for the
1820 kernel.
1821 Kernel module tarballs exist for legacy purposes and
1822 can be suppressed by setting the
1823 <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink>
1824 variable to "0".
1825 The
1826 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1827 directory can contain multiple kernel module tarballs
1828 for the machine.
1829 </para></listitem>
1830 <listitem><para>
1831 <filename><replaceable>bootloaders</replaceable></filename>:
1832 Bootloaders supporting the image, if applicable to the
1833 target machine.
1834 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
1835 directory can contain multiple bootloaders for the
1836 machine.
1837 </para></listitem>
1838 <listitem><para>
1839 <filename><replaceable>symlinks</replaceable></filename>:
1840 The
1841 <filename>deploy/images/<replaceable>machine</replaceable></filename>
1842 folder contains a symbolic link that points to the
1843 most recently built file for each machine.
1844 These links might be useful for external scripts that
1845 need to obtain the latest version of each file.
1846 </para></listitem>
1847 </itemizedlist>
1848 </para>
1849 </section>
1850
1851 <section id='sdk-dev-environment'>
1852 <title>Application Development SDK</title>
1853
1854 <para>
1855 In the
1856 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
1857 the output labeled "Application Development SDK" represents an
1858 SDK.
1859 The SDK generation process differs depending on whether you
1860 build a standard SDK (e.g.
1861 <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>)
1862 or an extensible SDK (e.g.
1863 <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>).
1864 This section is going to take a closer look at this output:
1865 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
1866 </para>
1867
1868 <para>
1869 The specific form of this output is a self-extracting
1870 SDK installer (<filename>*.sh</filename>) that, when run,
1871 installs the SDK, which consists of a cross-development
1872 toolchain, a set of libraries and headers, and an SDK
1873 environment setup script.
1874 Running this installer essentially sets up your
1875 cross-development environment.
1876 You can think of the cross-toolchain as the "host"
1877 part because it runs on the SDK machine.
1878 You can think of the libraries and headers as the "target"
1879 part because they are built for the target hardware.
1880 The environment setup script is added so that you can
1881 initialize the environment before using the tools.
1882 </para>
1883
1884 <note><title>Notes</title>
1885 <itemizedlist>
1886 <listitem><para>
1887 The Yocto Project supports several methods by which
1888 you can set up this cross-development environment.
1889 These methods include downloading pre-built SDK
1890 installers or building and installing your own SDK
1891 installer.
1892 </para></listitem>
1893 <listitem><para>
1894 For background information on cross-development
1895 toolchains in the Yocto Project development
1896 environment, see the
1897 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1898 section.
1899 </para></listitem>
1900 <listitem><para>
1901 For information on setting up a cross-development
1902 environment, see the
1903 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
1904 manual.
1905 </para></listitem>
1906 </itemizedlist>
1907 </note>
1908
1909 <para>
1910 Once built, the SDK installers are written out to the
1911 <filename>deploy/sdk</filename> folder inside the
1912 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1913 as shown in the figure at the beginning of this section.
1914 Depending on the type of SDK, several variables exist that help
1915 configure these files.
1916 The following list shows the variables associated with
1917 a standard SDK:
1918 <itemizedlist>
1919 <listitem><para>
1920 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1921 Points to the <filename>deploy</filename>
1922 directory.
1923 </para></listitem>
1924 <listitem><para>
1925 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>:
1926 Specifies the architecture of the machine
1927 on which the cross-development tools are run to
1928 create packages for the target hardware.
1929 </para></listitem>
1930 <listitem><para>
1931 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>:
1932 Lists the features to include in the "target" part
1933 of the SDK.
1934 </para></listitem>
1935 <listitem><para>
1936 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>:
1937 Lists packages that make up the host
1938 part of the SDK (i.e. the part that runs on
1939 the <filename>SDKMACHINE</filename>).
1940 When you use
1941 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
1942 to create the SDK, a set of default packages
1943 apply.
1944 This variable allows you to add more packages.
1945 </para></listitem>
1946 <listitem><para>
1947 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>:
1948 Lists packages that make up the target part
1949 of the SDK (i.e. the part built for the
1950 target hardware).
1951 </para></listitem>
1952 <listitem><para>
1953 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>:
1954 Defines the default SDK installation path offered
1955 by the installation script.
1956 </para></listitem>
1957 </itemizedlist>
1958 This next list, shows the variables associated with an
1959 extensible SDK:
1960 <itemizedlist>
1961 <listitem><para>
1962 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1963 Points to the <filename>deploy</filename> directory.
1964 </para></listitem>
1965 <listitem><para>
1966 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>:
1967 Controls whether or not shared state artifacts are
1968 copied into the extensible SDK.
1969 By default, all required shared state artifacts are
1970 copied into the SDK.
1971 </para></listitem>
1972 <listitem><para>
1973 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>:
1974 Specifies whether or not packagedata will be
1975 included in the extensible SDK for all recipes in
1976 the "world" target.
1977 </para></listitem>
1978 <listitem><para>
1979 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>:
1980 Specifies whether or not the toolchain will be included
1981 when building the extensible SDK.
1982 </para></listitem>
1983 <listitem><para>
1984 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>:
1985 A list of variables allowed through from the build
1986 system configuration into the extensible SDK
1987 configuration.
1988 </para></listitem>
1989 <listitem><para>
1990 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>:
1991 A list of variables not allowed through from the build
1992 system configuration into the extensible SDK
1993 configuration.
1994 </para></listitem>
1995 <listitem><para>
1996 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>:
1997 A list of classes to remove from the
1998 <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink>
1999 value globally within the extensible SDK configuration.
2000 </para></listitem>
2001 </itemizedlist>
2002 </para>
2003 </section>
2004 </section>
2005
2006 <section id="cross-development-toolchain-generation">
2007 <title>Cross-Development Toolchain Generation</title>
2008
2009 <para>
2010 The Yocto Project does most of the work for you when it comes to
2011 creating
2012 <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>.
2013 This section provides some technical background on how
2014 cross-development toolchains are created and used.
2015 For more information on toolchains, you can also see the
2016 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2017 manual.
2018 </para>
2019
2020 <para>
2021 In the Yocto Project development environment, cross-development
2022 toolchains are used to build the image and applications that run
2023 on the target hardware.
2024 With just a few commands, the OpenEmbedded build system creates
2025 these necessary toolchains for you.
2026 </para>
2027
2028 <para>
2029 The following figure shows a high-level build environment regarding
2030 toolchain construction and use.
2031 </para>
2032
2033 <para>
2034 <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" />
2035 </para>
2036
2037 <para>
2038 Most of the work occurs on the Build Host.
2039 This is the machine used to build images and generally work within
2040 the the Yocto Project environment.
2041 When you run
2042 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2043 to create an image, the OpenEmbedded build system
2044 uses the host <filename>gcc</filename> compiler to bootstrap a
2045 cross-compiler named <filename>gcc-cross</filename>.
2046 The <filename>gcc-cross</filename> compiler is what BitBake uses to
2047 compile source files when creating the target image.
2048 You can think of <filename>gcc-cross</filename> simply as an
2049 automatically generated cross-compiler that is used internally
2050 within BitBake only.
2051 <note>
2052 The extensible SDK does not use
2053 <filename>gcc-cross-canadian</filename> since this SDK
2054 ships a copy of the OpenEmbedded build system and the sysroot
2055 within it contains <filename>gcc-cross</filename>.
2056 </note>
2057 </para>
2058
2059 <para>
2060 The chain of events that occurs when <filename>gcc-cross</filename> is
2061 bootstrapped is as follows:
2062 <literallayout class='monospaced'>
2063 gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime
2064 </literallayout>
2065 <itemizedlist>
2066 <listitem><para>
2067 <filename>gcc</filename>:
2068 The build host's GNU Compiler Collection (GCC).
2069 </para></listitem>
2070 <listitem><para>
2071 <filename>binutils-cross</filename>:
2072 The bare minimum binary utilities needed in order to run
2073 the <filename>gcc-cross-initial</filename> phase of the
2074 bootstrap operation.
2075 </para></listitem>
2076 <listitem><para>
2077 <filename>gcc-cross-initial</filename>:
2078 An early stage of the bootstrap process for creating
2079 the cross-compiler.
2080 This stage builds enough of the <filename>gcc-cross</filename>,
2081 the C library, and other pieces needed to finish building the
2082 final cross-compiler in later stages.
2083 This tool is a "native" package (i.e. it is designed to run on
2084 the build host).
2085 </para></listitem>
2086 <listitem><para>
2087 <filename>linux-libc-headers</filename>:
2088 Headers needed for the cross-compiler.
2089 </para></listitem>
2090 <listitem><para>
2091 <filename>glibc-initial</filename>:
2092 An initial version of the Embedded GLIBC needed to bootstrap
2093 <filename>glibc</filename>.
2094 </para></listitem>
2095 <listitem><para>
2096 <filename>gcc-cross</filename>:
2097 The final stage of the bootstrap process for the
2098 cross-compiler.
2099 This stage results in the actual cross-compiler that
2100 BitBake uses when it builds an image for a targeted
2101 device.
2102 <note>
2103 If you are replacing this cross compiler toolchain
2104 with a custom version, you must replace
2105 <filename>gcc-cross</filename>.
2106 </note>
2107 This tool is also a "native" package (i.e. it is
2108 designed to run on the build host).
2109 </para></listitem>
2110 <listitem><para>
2111 <filename>gcc-runtime</filename>:
2112 Runtime libraries resulting from the toolchain bootstrapping
2113 process.
2114 This tool produces a binary that consists of the
2115 runtime libraries need for the targeted device.
2116 </para></listitem>
2117 </itemizedlist>
2118 </para>
2119
2120 <para>
2121 You can use the OpenEmbedded build system to build an installer for
2122 the relocatable SDK used to develop applications.
2123 When you run the installer, it installs the toolchain, which
2124 contains the development tools (e.g.,
2125 <filename>gcc-cross-canadian</filename>,
2126 <filename>binutils-cross-canadian</filename>, and other
2127 <filename>nativesdk-*</filename> tools),
2128 which are tools native to the SDK (i.e. native to
2129 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>),
2130 you need to cross-compile and test your software.
2131 The figure shows the commands you use to easily build out this
2132 toolchain.
2133 This cross-development toolchain is built to execute on the
2134 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2135 which might or might not be the same
2136 machine as the Build Host.
2137 <note>
2138 If your target architecture is supported by the Yocto Project,
2139 you can take advantage of pre-built images that ship with the
2140 Yocto Project and already contain cross-development toolchain
2141 installers.
2142 </note>
2143 </para>
2144
2145 <para>
2146 Here is the bootstrap process for the relocatable toolchain:
2147 <literallayout class='monospaced'>
2148 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers ->
2149 glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
2150 </literallayout>
2151 <itemizedlist>
2152 <listitem><para>
2153 <filename>gcc</filename>:
2154 The build host's GNU Compiler Collection (GCC).
2155 </para></listitem>
2156 <listitem><para>
2157 <filename>binutils-crosssdk</filename>:
2158 The bare minimum binary utilities needed in order to run
2159 the <filename>gcc-crosssdk-initial</filename> phase of the
2160 bootstrap operation.
2161 </para></listitem>
2162 <listitem><para>
2163 <filename>gcc-crosssdk-initial</filename>:
2164 An early stage of the bootstrap process for creating
2165 the cross-compiler.
2166 This stage builds enough of the
2167 <filename>gcc-crosssdk</filename> and supporting pieces so that
2168 the final stage of the bootstrap process can produce the
2169 finished cross-compiler.
2170 This tool is a "native" binary that runs on the build host.
2171 </para></listitem>
2172 <listitem><para>
2173 <filename>linux-libc-headers</filename>:
2174 Headers needed for the cross-compiler.
2175 </para></listitem>
2176 <listitem><para>
2177 <filename>glibc-initial</filename>:
2178 An initial version of the Embedded GLIBC needed to bootstrap
2179 <filename>nativesdk-glibc</filename>.
2180 </para></listitem>
2181 <listitem><para>
2182 <filename>nativesdk-glibc</filename>:
2183 The Embedded GLIBC needed to bootstrap the
2184 <filename>gcc-crosssdk</filename>.
2185 </para></listitem>
2186 <listitem><para>
2187 <filename>gcc-crosssdk</filename>:
2188 The final stage of the bootstrap process for the
2189 relocatable cross-compiler.
2190 The <filename>gcc-crosssdk</filename> is a transitory compiler
2191 and never leaves the build host.
2192 Its purpose is to help in the bootstrap process to create the
2193 eventual relocatable <filename>gcc-cross-canadian</filename>
2194 compiler, which is relocatable.
2195 This tool is also a "native" package (i.e. it is
2196 designed to run on the build host).
2197 </para></listitem>
2198 <listitem><para>
2199 <filename>gcc-cross-canadian</filename>:
2200 The final relocatable cross-compiler.
2201 When run on the
2202 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2203 this tool
2204 produces executable code that runs on the target device.
2205 Only one cross-canadian compiler is produced per architecture
2206 since they can be targeted at different processor optimizations
2207 using configurations passed to the compiler through the
2208 compile commands.
2209 This circumvents the need for multiple compilers and thus
2210 reduces the size of the toolchains.
2211 </para></listitem>
2212 </itemizedlist>
2213 </para>
2214
2215 <note>
2216 For information on advantages gained when building a
2217 cross-development toolchain installer, see the
2218 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
2219 section in the Yocto Project Application Development and the
2220 Extensible Software Development Kit (eSDK) manual.
2221 </note>
2222 </section>
2223
2224 <section id="shared-state-cache">
2225 <title>Shared State Cache</title>
2226
2227 <para>
2228 By design, the OpenEmbedded build system builds everything from
2229 scratch unless
2230 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2231 can determine that parts do not need to be rebuilt.
2232 Fundamentally, building from scratch is attractive as it means all
2233 parts are built fresh and there is no possibility of stale data
2234 causing problems.
2235 When developers hit problems, they typically default back to
2236 building from scratch so they know the state of things from the
2237 start.
2238 </para>
2239
2240 <para>
2241 Building an image from scratch is both an advantage and a
2242 disadvantage to the process.
2243 As mentioned in the previous paragraph, building from scratch
2244 ensures that everything is current and starts from a known state.
2245 However, building from scratch also takes much longer as it
2246 generally means rebuilding things that do not necessarily need
2247 to be rebuilt.
2248 </para>
2249
2250 <para>
2251 The Yocto Project implements shared state code that supports
2252 incremental builds.
2253 The implementation of the shared state code answers the following
2254 questions that were fundamental roadblocks within the OpenEmbedded
2255 incremental build support system:
2256 <itemizedlist>
2257 <listitem><para>
2258 What pieces of the system have changed and what pieces have
2259 not changed?
2260 </para></listitem>
2261 <listitem><para>
2262 How are changed pieces of software removed and replaced?
2263 </para></listitem>
2264 <listitem><para>
2265 How are pre-built components that do not need to be rebuilt
2266 from scratch used when they are available?
2267 </para></listitem>
2268 </itemizedlist>
2269 </para>
2270
2271 <para>
2272 For the first question, the
2273 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
2274 detects changes in the "inputs" to a given task by creating a
2275 checksum (or signature) of the task's inputs.
2276 If the checksum changes, the system assumes the inputs have changed
2277 and the task needs to be rerun.
2278 For the second question, the shared state (sstate) code tracks
2279 which tasks add which output to the build process.
2280 This means the output from a given task can be removed, upgraded
2281 or otherwise manipulated.
2282 The third question is partly addressed by the solution for the
2283 second question assuming the build system can fetch the sstate
2284 objects from remote locations and install them if they are deemed
2285 to be valid.
2286 <note>
2287 The OpenEmbedded build system does not maintain
2288 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2289 information as part of the shared state packages.
2290 Consequently, considerations exist that affect maintaining
2291 shared state feeds.
2292 For information on how the OpenEmbedded build system
2293 works with packages and can track incrementing
2294 <filename>PR</filename> information, see the
2295 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
2296 section in the Yocto Project Development Tasks Manual.
2297 </note>
2298 </para>
2299
2300 <para>
2301 The rest of this section goes into detail about the overall
2302 incremental build architecture, the checksums (signatures), shared
2303 state, and some tips and tricks.
2304 </para>
2305
2306 <section id='concepts-overall-architecture'>
2307 <title>Overall Architecture</title>
2308
2309 <para>
2310 When determining what parts of the system need to be built,
2311 BitBake works on a per-task basis rather than a per-recipe
2312 basis.
2313 You might wonder why using a per-task basis is preferred over
2314 a per-recipe basis.
2315 To help explain, consider having the IPK packaging backend
2316 enabled and then switching to DEB.
2317 In this case, the
2318 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2319 and
2320 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2321 task outputs are still valid.
2322 However, with a per-recipe approach, the build would not
2323 include the <filename>.deb</filename> files.
2324 Consequently, you would have to invalidate the whole build and
2325 rerun it.
2326 Rerunning everything is not the best solution.
2327 Also, in this case, the core must be "taught" much about
2328 specific tasks.
2329 This methodology does not scale well and does not allow users
2330 to easily add new tasks in layers or as external recipes
2331 without touching the packaged-staging core.
2332 </para>
2333 </section>
2334
2335 <section id='overview-checksums'>
2336 <title>Checksums (Signatures)</title>
2337
2338 <para>
2339 The shared state code uses a checksum, which is a unique
2340 signature of a task's inputs, to determine if a task needs to
2341 be run again.
2342 Because it is a change in a task's inputs that triggers a
2343 rerun, the process needs to detect all the inputs to a given
2344 task.
2345 For shell tasks, this turns out to be fairly easy because
2346 the build process generates a "run" shell script for each task
2347 and it is possible to create a checksum that gives you a good
2348 idea of when the task's data changes.
2349 </para>
2350
2351 <para>
2352 To complicate the problem, there are things that should not be
2353 included in the checksum.
2354 First, there is the actual specific build path of a given
2355 task - the
2356 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2357 It does not matter if the work directory changes because it
2358 should not affect the output for target packages.
2359 Also, the build process has the objective of making native
2360 or cross packages relocatable.
2361 <note>
2362 Both native and cross packages run on the
2363 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>.
2364 However, cross packages generate output for the target
2365 architecture.
2366 </note>
2367 The checksum therefore needs to exclude
2368 <filename>WORKDIR</filename>.
2369 The simplistic approach for excluding the work directory is to
2370 set <filename>WORKDIR</filename> to some fixed value and
2371 create the checksum for the "run" script.
2372 </para>
2373
2374 <para>
2375 Another problem results from the "run" scripts containing
2376 functions that might or might not get called.
2377 The incremental build solution contains code that figures out
2378 dependencies between shell functions.
2379 This code is used to prune the "run" scripts down to the
2380 minimum set, thereby alleviating this problem and making the
2381 "run" scripts much more readable as a bonus.
2382 </para>
2383
2384 <para>
2385 So far, solutions for shell scripts exist.
2386 What about Python tasks?
2387 The same approach applies even though these tasks are more
2388 difficult.
2389 The process needs to figure out what variables a Python
2390 function accesses and what functions it calls.
2391 Again, the incremental build solution contains code that first
2392 figures out the variable and function dependencies, and then
2393 creates a checksum for the data used as the input to the task.
2394 </para>
2395
2396 <para>
2397 Like the <filename>WORKDIR</filename> case, situations exist
2398 where dependencies should be ignored.
2399 For these situations, you can instruct the build process to
2400 ignore a dependency by using a line like the following:
2401 <literallayout class='monospaced'>
2402 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
2403 </literallayout>
2404 This example ensures that the
2405 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
2406 variable does not depend on the value of
2407 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
2408 even if it does reference it.
2409 </para>
2410
2411 <para>
2412 Equally, there are cases where you need to add dependencies
2413 BitBake is not able to find.
2414 You can accomplish this by using a line like the following:
2415 <literallayout class='monospaced'>
2416 PACKAGE_ARCHS[vardeps] = "MACHINE"
2417 </literallayout>
2418 This example explicitly adds the <filename>MACHINE</filename>
2419 variable as a dependency for
2420 <filename>PACKAGE_ARCHS</filename>.
2421 </para>
2422
2423 <para>
2424 As an example, consider a case with in-line Python where
2425 BitBake is not able to figure out dependencies.
2426 When running in debug mode (i.e. using
2427 <filename>-DDD</filename>), BitBake produces output when it
2428 discovers something for which it cannot figure out dependencies.
2429 The Yocto Project team has currently not managed to cover
2430 those dependencies in detail and is aware of the need to fix
2431 this situation.
2432 </para>
2433
2434 <para>
2435 Thus far, this section has limited discussion to the direct
2436 inputs into a task.
2437 Information based on direct inputs is referred to as the
2438 "basehash" in the code.
2439 However, there is still the question of a task's indirect
2440 inputs - the things that were already built and present in the
2441 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
2442 The checksum (or signature) for a particular task needs to add
2443 the hashes of all the tasks on which the particular task
2444 depends.
2445 Choosing which dependencies to add is a policy decision.
2446 However, the effect is to generate a master checksum that
2447 combines the basehash and the hashes of the task's
2448 dependencies.
2449 </para>
2450
2451 <para>
2452 At the code level, a variety of ways exist by which both the
2453 basehash and the dependent task hashes can be influenced.
2454 Within the BitBake configuration file, you can give BitBake
2455 some extra information to help it construct the basehash.
2456 The following statement effectively results in a list of
2457 global variable dependency excludes - variables never
2458 included in any checksum:
2459 <literallayout class='monospaced'>
2460 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
2461 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
2462 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
2463 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
2464 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
2465 </literallayout>
2466 The previous example excludes
2467 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
2468 since that variable is actually constructed as a path within
2469 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
2470 which is on the whitelist.
2471 </para>
2472
2473 <para>
2474 The rules for deciding which hashes of dependent tasks to
2475 include through dependency chains are more complex and are
2476 generally accomplished with a Python function.
2477 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
2478 two examples of this and also illustrates how you can insert
2479 your own policy into the system if so desired.
2480 This file defines the two basic signature generators
2481 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
2482 uses: "OEBasic" and "OEBasicHash".
2483 By default, there is a dummy "noop" signature handler enabled
2484 in BitBake.
2485 This means that behavior is unchanged from previous versions.
2486 OE-Core uses the "OEBasicHash" signature handler by default
2487 through this setting in the <filename>bitbake.conf</filename>
2488 file:
2489 <literallayout class='monospaced'>
2490 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
2491 </literallayout>
2492 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
2493 is the same as the "OEBasic" version but adds the task hash to
2494 the stamp files.
2495 This results in any
2496 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
2497 change that changes the task hash, automatically
2498 causing the task to be run again.
2499 This removes the need to bump
2500 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2501 values, and changes to Metadata automatically ripple across
2502 the build.
2503 </para>
2504
2505 <para>
2506 It is also worth noting that the end result of these
2507 signature generators is to make some dependency and hash
2508 information available to the build.
2509 This information includes:
2510 <itemizedlist>
2511 <listitem><para>
2512 <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
2513 The base hashes for each task in the recipe.
2514 </para></listitem>
2515 <listitem><para>
2516 <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2517 The base hashes for each dependent task.
2518 </para></listitem>
2519 <listitem><para>
2520 <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2521 The task dependencies for each task.
2522 </para></listitem>
2523 <listitem><para>
2524 <filename>BB_TASKHASH</filename>:
2525 The hash of the currently running task.
2526 </para></listitem>
2527 </itemizedlist>
2528 </para>
2529 </section>
2530
2531 <section id='shared-state'>
2532 <title>Shared State</title>
2533
2534 <para>
2535 Checksums and dependencies, as discussed in the previous
2536 section, solve half the problem of supporting a shared state.
2537 The other part of the problem is being able to use checksum
2538 information during the build and being able to reuse or rebuild
2539 specific components.
2540 </para>
2541
2542 <para>
2543 The
2544 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2545 class is a relatively generic implementation of how to
2546 "capture" a snapshot of a given task.
2547 The idea is that the build process does not care about the
2548 source of a task's output.
2549 Output could be freshly built or it could be downloaded and
2550 unpacked from somewhere - the build process does not need to
2551 worry about its origin.
2552 </para>
2553
2554 <para>
2555 Two types of output exist.
2556 One type is just about creating a directory in
2557 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2558 A good example is the output of either
2559 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2560 or
2561 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
2562 The other type of output occurs when a set of data is merged
2563 into a shared directory tree such as the sysroot.
2564 </para>
2565
2566 <para>
2567 The Yocto Project team has tried to keep the details of the
2568 implementation hidden in <filename>sstate</filename> class.
2569 From a user's perspective, adding shared state wrapping to a task
2570 is as simple as this
2571 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2572 example taken from the
2573 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
2574 class:
2575 <literallayout class='monospaced'>
2576 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
2577 SSTATETASKS += "do_deploy"
2578 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
2579 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
2580
2581 python do_deploy_setscene () {
2582 sstate_setscene(d)
2583 }
2584 addtask do_deploy_setscene
2585 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
2586 </literallayout>
2587 The following list explains the previous example:
2588 <itemizedlist>
2589 <listitem><para>
2590 Adding "do_deploy" to <filename>SSTATETASKS</filename>
2591 adds some required sstate-related processing, which is
2592 implemented in the
2593 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2594 class, to before and after the
2595 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2596 task.
2597 </para></listitem>
2598 <listitem><para>
2599 The
2600 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
2601 declares that <filename>do_deploy</filename> places its
2602 output in <filename>${DEPLOYDIR}</filename> when run
2603 normally (i.e. when not using the sstate cache).
2604 This output becomes the input to the shared state cache.
2605 </para></listitem>
2606 <listitem><para>
2607 The
2608 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
2609 line causes the contents of the shared state cache to be
2610 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
2611 <note>
2612 If <filename>do_deploy</filename> is not already in
2613 the shared state cache or if its input checksum
2614 (signature) has changed from when the output was
2615 cached, the task will be run to populate the shared
2616 state cache, after which the contents of the shared
2617 state cache is copied to
2618 <filename>${DEPLOY_DIR_IMAGE}</filename>.
2619 If <filename>do_deploy</filename> is in the shared
2620 state cache and its signature indicates that the
2621 cached output is still valid (i.e. if no
2622 relevant task inputs have changed), then the
2623 contents of the shared state cache will be copied
2624 directly to
2625 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
2626 <filename>do_deploy_setscene</filename> task
2627 instead, skipping the
2628 <filename>do_deploy</filename> task.
2629 </note>
2630 </para></listitem>
2631 <listitem><para>
2632 The following task definition is glue logic needed to
2633 make the previous settings effective:
2634 <literallayout class='monospaced'>
2635 python do_deploy_setscene () {
2636 sstate_setscene(d)
2637 }
2638 addtask do_deploy_setscene
2639 </literallayout>
2640 <filename>sstate_setscene()</filename> takes the flags
2641 above as input and accelerates the
2642 <filename>do_deploy</filename> task through the
2643 shared state cache if possible.
2644 If the task was accelerated,
2645 <filename>sstate_setscene()</filename> returns True.
2646 Otherwise, it returns False, and the normal
2647 <filename>do_deploy</filename> task runs.
2648 For more information, see the
2649 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
2650 section in the BitBake User Manual.
2651 </para></listitem>
2652 <listitem><para>
2653 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
2654 line creates <filename>${DEPLOYDIR}</filename> and
2655 <filename>${B}</filename> before the
2656 <filename>do_deploy</filename> task runs, and also sets
2657 the current working directory of
2658 <filename>do_deploy</filename> to
2659 <filename>${B}</filename>.
2660 For more information, see the
2661 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
2662 section in the BitBake User Manual.
2663 <note>
2664 In cases where
2665 <filename>sstate-inputdirs</filename> and
2666 <filename>sstate-outputdirs</filename> would be the
2667 same, you can use
2668 <filename>sstate-plaindirs</filename>.
2669 For example, to preserve the
2670 <filename>${PKGD}</filename> and
2671 <filename>${PKGDEST}</filename> output from the
2672 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2673 task, use the following:
2674 <literallayout class='monospaced'>
2675 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
2676 </literallayout>
2677 </note>
2678 </para></listitem>
2679 <listitem><para>
2680 <filename>sstate-inputdirs</filename> and
2681 <filename>sstate-outputdirs</filename> can also be used
2682 with multiple directories.
2683 For example, the following declares
2684 <filename>PKGDESTWORK</filename> and
2685 <filename>SHLIBWORK</filename> as shared state
2686 input directories, which populates the shared state
2687 cache, and <filename>PKGDATA_DIR</filename> and
2688 <filename>SHLIBSDIR</filename> as the corresponding
2689 shared state output directories:
2690 <literallayout class='monospaced'>
2691 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
2692 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
2693 </literallayout>
2694 </para></listitem>
2695 <listitem><para>
2696 These methods also include the ability to take a
2697 lockfile when manipulating shared state directory
2698 structures, for cases where file additions or removals
2699 are sensitive:
2700 <literallayout class='monospaced'>
2701 do_package[sstate-lockfile] = "${PACKAGELOCK}"
2702 </literallayout>
2703 </para></listitem>
2704 </itemizedlist>
2705 </para>
2706
2707 <para>
2708 Behind the scenes, the shared state code works by looking in
2709 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
2710 and
2711 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
2712 for shared state files.
2713 Here is an example:
2714 <literallayout class='monospaced'>
2715 SSTATE_MIRRORS ?= "\
2716 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
2717 file://.* file:///some/local/dir/sstate/PATH"
2718 </literallayout>
2719 <note>
2720 The shared state directory
2721 (<filename>SSTATE_DIR</filename>) is organized into
2722 two-character subdirectories, where the subdirectory
2723 names are based on the first two characters of the hash.
2724 If the shared state directory structure for a mirror has the
2725 same structure as <filename>SSTATE_DIR</filename>, you must
2726 specify "PATH" as part of the URI to enable the build system
2727 to map to the appropriate subdirectory.
2728 </note>
2729 </para>
2730
2731 <para>
2732 The shared state package validity can be detected just by
2733 looking at the filename since the filename contains the task
2734 checksum (or signature) as described earlier in this section.
2735 If a valid shared state package is found, the build process
2736 downloads it and uses it to accelerate the task.
2737 </para>
2738
2739 <para>
2740 The build processes use the <filename>*_setscene</filename>
2741 tasks for the task acceleration phase.
2742 BitBake goes through this phase before the main execution
2743 code and tries to accelerate any tasks for which it can find
2744 shared state packages.
2745 If a shared state package for a task is available, the
2746 shared state package is used.
2747 This means the task and any tasks on which it is dependent
2748 are not executed.
2749 </para>
2750
2751 <para>
2752 As a real world example, the aim is when building an IPK-based
2753 image, only the
2754 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
2755 tasks would have their shared state packages fetched and
2756 extracted.
2757 Since the sysroot is not used, it would never get extracted.
2758 This is another reason why a task-based approach is preferred
2759 over a recipe-based approach, which would have to install the
2760 output from every task.n
2761 </para>
2762 </section>
2763
2764 <section id='concepts-tips-and-tricks'>
2765 <title>Tips and Tricks</title>
2766
2767 <para>
2768 The code in the build system that supports incremental builds
2769 is not simple code.
2770 This section presents some tips and tricks that help you work
2771 around issues related to shared state code.
2772 </para>
2773
2774 <section id='concepts-overview-debugging'>
2775 <title>Debugging</title>
2776
2777 <para>
2778 Seeing what metadata went into creating the input signature
2779 of a shared state (sstate) task can be a useful debugging
2780 aid.
2781 This information is available in signature information
2782 (<filename>siginfo</filename>) files in
2783 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>.
2784 For information on how to view and interpret information in
2785 <filename>siginfo</filename> files, see the
2786 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
2787 section in the Yocto Project Development Tasks Manual.
2788 </para>
2789 </section>
2790
2791 <section id='concepts-invalidating-shared-state'>
2792 <title>Invalidating Shared State</title>
2793
2794 <para>
2795 The OpenEmbedded build system uses checksums and shared
2796 state cache to avoid unnecessarily rebuilding tasks.
2797 Collectively, this scheme is known as "shared state code."
2798 </para>
2799
2800 <para>
2801 As with all schemes, this one has some drawbacks.
2802 It is possible that you could make implicit changes to your
2803 code that the checksum calculations do not take into
2804 account.
2805 These implicit changes affect a task's output but do not
2806 trigger the shared state code into rebuilding a recipe.
2807 Consider an example during which a tool changes its output.
2808 Assume that the output of <filename>rpmdeps</filename>
2809 changes.
2810 The result of the change should be that all the
2811 <filename>package</filename> and
2812 <filename>package_write_rpm</filename> shared state cache
2813 items become invalid.
2814 However, because the change to the output is
2815 external to the code and therefore implicit,
2816 the associated shared state cache items do not become
2817 invalidated.
2818 In this case, the build process uses the cached items
2819 rather than running the task again.
2820 Obviously, these types of implicit changes can cause
2821 problems.
2822 </para>
2823
2824 <para>
2825 To avoid these problems during the build, you need to
2826 understand the effects of any changes you make.
2827 Realize that changes you make directly to a function
2828 are automatically factored into the checksum calculation.
2829 Thus, these explicit changes invalidate the associated
2830 area of shared state cache.
2831 However, you need to be aware of any implicit changes that
2832 are not obvious changes to the code and could affect
2833 the output of a given task.
2834 </para>
2835
2836 <para>
2837 When you identify an implicit change, you can easily
2838 take steps to invalidate the cache and force the tasks
2839 to run.
2840 The steps you can take are as simple as changing a
2841 function's comments in the source code.
2842 For example, to invalidate package shared state files,
2843 change the comment statements of
2844 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2845 or the comments of one of the functions it calls.
2846 Even though the change is purely cosmetic, it causes the
2847 checksum to be recalculated and forces the OpenEmbedded
2848 build system to run the task again.
2849 <note>
2850 For an example of a commit that makes a cosmetic
2851 change to invalidate shared state, see this
2852 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
2853 </note>
2854 </para>
2855 </section>
2856 </section>
2857 </section>
2858
2859 <section id='automatically-added-runtime-dependencies'>
2860 <title>Automatically Added Runtime Dependencies</title>
2861
2862 <para>
2863 The OpenEmbedded build system automatically adds common types of
2864 runtime dependencies between packages, which means that you do not
2865 need to explicitly declare the packages using
2866 <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>.
2867 Three automatic mechanisms exist (<filename>shlibdeps</filename>,
2868 <filename>pcdeps</filename>, and <filename>depchains</filename>)
2869 that handle shared libraries, package configuration (pkg-config)
2870 modules, and <filename>-dev</filename> and
2871 <filename>-dbg</filename> packages, respectively.
2872 For other types of runtime dependencies, you must manually declare
2873 the dependencies.
2874 <itemizedlist>
2875 <listitem><para>
2876 <filename>shlibdeps</filename>:
2877 During the
2878 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2879 task of each recipe, all shared libraries installed by the
2880 recipe are located.
2881 For each shared library, the package that contains the
2882 shared library is registered as providing the shared
2883 library.
2884 More specifically, the package is registered as providing
2885 the
2886 <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>
2887 of the library.
2888 The resulting shared-library-to-package mapping
2889 is saved globally in
2890 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>
2891 by the
2892 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
2893 task.</para>
2894
2895 <para>Simultaneously, all executables and shared libraries
2896 installed by the recipe are inspected to see what shared
2897 libraries they link against.
2898 For each shared library dependency that is found,
2899 <filename>PKGDATA_DIR</filename> is queried to
2900 see if some package (likely from a different recipe)
2901 contains the shared library.
2902 If such a package is found, a runtime dependency is added
2903 from the package that depends on the shared library to the
2904 package that contains the library.</para>
2905
2906 <para>The automatically added runtime dependency also
2907 includes a version restriction.
2908 This version restriction specifies that at least the
2909 current version of the package that provides the shared
2910 library must be used, as if
2911 "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"
2912 had been added to <filename>RDEPENDS</filename>.
2913 This forces an upgrade of the package containing the shared
2914 library when installing the package that depends on the
2915 library, if needed.</para>
2916
2917 <para>If you want to avoid a package being registered as
2918 providing a particular shared library (e.g. because the library
2919 is for internal use only), then add the library to
2920 <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink>
2921 inside the package's recipe.
2922 </para></listitem>
2923 <listitem><para>
2924 <filename>pcdeps</filename>:
2925 During the <filename>do_package</filename> task of each
2926 recipe, all pkg-config modules
2927 (<filename>*.pc</filename> files) installed by the recipe
2928 are located.
2929 For each module, the package that contains the module is
2930 registered as providing the module.
2931 The resulting module-to-package mapping is saved globally in
2932 <filename>PKGDATA_DIR</filename> by the
2933 <filename>do_packagedata</filename> task.</para>
2934
2935 <para>Simultaneously, all pkg-config modules installed by
2936 the recipe are inspected to see what other pkg-config
2937 modules they depend on.
2938 A module is seen as depending on another module if it
2939 contains a "Requires:" line that specifies the other module.
2940 For each module dependency,
2941 <filename>PKGDATA_DIR</filename> is queried to see if some
2942 package contains the module.
2943 If such a package is found, a runtime dependency is added
2944 from the package that depends on the module to the package
2945 that contains the module.
2946 <note>
2947 The <filename>pcdeps</filename> mechanism most often
2948 infers dependencies between <filename>-dev</filename>
2949 packages.
2950 </note>
2951 </para></listitem>
2952 <listitem><para>
2953 <filename>depchains</filename>:
2954 If a package <filename>foo</filename> depends on a package
2955 <filename>bar</filename>, then <filename>foo-dev</filename>
2956 and <filename>foo-dbg</filename> are also made to depend on
2957 <filename>bar-dev</filename> and
2958 <filename>bar-dbg</filename>, respectively.
2959 Taking the <filename>-dev</filename> packages as an
2960 example, the <filename>bar-dev</filename> package might
2961 provide headers and shared library symlinks needed by
2962 <filename>foo-dev</filename>, which shows the need
2963 for a dependency between the packages.</para>
2964
2965 <para>The dependencies added by
2966 <filename>depchains</filename> are in the form of
2967 <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>.
2968 <note>
2969 By default, <filename>foo-dev</filename> also has an
2970 <filename>RDEPENDS</filename>-style dependency on
2971 <filename>foo</filename>, because the default value of
2972 <filename>RDEPENDS_${PN}-dev</filename> (set in
2973 <filename>bitbake.conf</filename>) includes
2974 "${PN}".
2975 </note></para>
2976
2977 <para>To ensure that the dependency chain is never broken,
2978 <filename>-dev</filename> and <filename>-dbg</filename>
2979 packages are always generated by default, even if the
2980 packages turn out to be empty.
2981 See the
2982 <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink>
2983 variable for more information.
2984 </para></listitem>
2985 </itemizedlist>
2986 </para>
2987
2988 <para>
2989 The <filename>do_package</filename> task depends on the
2990 <filename>do_packagedata</filename> task of each recipe in
2991 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>
2992 through use of a
2993 <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>
2994 declaration, which guarantees that the required
2995 shared-library/module-to-package mapping information will be available
2996 when needed as long as <filename>DEPENDS</filename> has been
2997 correctly set.
2998 </para>
2999 </section>
3000
3001 <section id='fakeroot-and-pseudo'>
3002 <title>Fakeroot and Pseudo</title>
3003
3004 <para>
3005 Some tasks are easier to implement when allowed to perform certain
3006 operations that are normally reserved for the root user (e.g.
3007 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>,
3008 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>,
3009 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>,
3010 and
3011 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>).
3012 For example, the <filename>do_install</filename> task benefits
3013 from being able to set the UID and GID of installed files to
3014 arbitrary values.
3015 </para>
3016
3017 <para>
3018 One approach to allowing tasks to perform root-only operations
3019 would be to require
3020 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
3021 to run as root.
3022 However, this method is cumbersome and has security issues.
3023 The approach that is actually used is to run tasks that benefit
3024 from root privileges in a "fake" root environment.
3025 Within this environment, the task and its child processes believe
3026 that they are running as the root user, and see an internally
3027 consistent view of the filesystem.
3028 As long as generating the final output (e.g. a package or an image)
3029 does not require root privileges, the fact that some earlier
3030 steps ran in a fake root environment does not cause problems.
3031 </para>
3032
3033 <para>
3034 The capability to run tasks in a fake root environment is known as
3035 "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>",
3036 which is derived from the BitBake keyword/variable
3037 flag that requests a fake root environment for a task.
3038 </para>
3039
3040 <para>
3041 In the
3042 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
3043 the program that implements fakeroot is known as Pseudo.
3044 Pseudo overrides system calls by using the environment variable
3045 <filename>LD_PRELOAD</filename>, which results in the illusion
3046 of running as root.
3047 To keep track of "fake" file ownership and permissions resulting
3048 from operations that require root permissions, Pseudo uses
3049 an SQLite 3 database.
3050 This database is stored in
3051 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename>
3052 for individual recipes.
3053 Storing the database in a file as opposed to in memory
3054 gives persistence between tasks and builds, which is not
3055 accomplished using fakeroot.
3056 <note><title>Caution</title>
3057 If you add your own task that manipulates the same files or
3058 directories as a fakeroot task, then that task also needs to
3059 run under fakeroot.
3060 Otherwise, the task cannot run root-only operations, and
3061 cannot see the fake file ownership and permissions set by the
3062 other task.
3063 You need to also add a dependency on
3064 <filename>virtual/fakeroot-native:do_populate_sysroot</filename>,
3065 giving the following:
3066 <literallayout class='monospaced'>
3067 fakeroot do_mytask () {
3068 ...
3069 }
3070 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
3071 </literallayout>
3072 </note>
3073 For more information, see the
3074 <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>
3075 variables in the BitBake User Manual.
3076 You can also reference the
3077 "<ulink url='http://www.ibm.com/developerworks/opensource/library/os-aapseudo1/index.html'>Pseudo</ulink>"
3078 and
3079 "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>"
3080 articles for background information on Pseudo.
3081 </para>
3082 </section>
3083
3084 <section id="wayland">
3085 <title>Wayland</title>
3086
3087 <para>
3088 <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink>
3089 is a computer display server protocol that
3090 provides a method for compositing window managers to communicate
3091 directly with applications and video hardware and expects them to
3092 communicate with input hardware using other libraries.
3093 Using Wayland with supporting targets can result in better control
3094 over graphics frame rendering than an application might otherwise
3095 achieve.
3096 </para>
3097
3098 <para>
3099 The Yocto Project provides the Wayland protocol libraries and the
3100 reference
3101 <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink>
3102 compositor as part of its release.
3103 This section describes what you need to do to implement Wayland and
3104 use the compositor when building an image for a supporting target.
3105 </para>
3106
3107 <section id="wayland-support">
3108 <title>Support</title>
3109
3110 <para>
3111 The Wayland protocol libraries and the reference Weston
3112 compositor ship as integrated packages in the
3113 <filename>meta</filename> layer of the
3114 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
3115 Specifically, you can find the recipes that build both Wayland
3116 and Weston at
3117 <filename>meta/recipes-graphics/wayland</filename>.
3118 </para>
3119
3120 <para>
3121 You can build both the Wayland and Weston packages for use only
3122 with targets that accept the
3123 <ulink url='https://en.wikipedia.org/wiki/Mesa_(computer_graphics)'>Mesa 3D and Direct Rendering Infrastructure</ulink>,
3124 which is also known as Mesa DRI.
3125 This implies that you cannot build and use the packages if your
3126 target uses, for example, the
3127 <trademark class='registered'>Intel</trademark> Embedded Media
3128 and Graphics Driver
3129 (<trademark class='registered'>Intel</trademark> EMGD) that
3130 overrides Mesa DRI.
3131 <note>
3132 Due to lack of EGL support, Weston 1.0.3 will not run
3133 directly on the emulated QEMU hardware.
3134 However, this version of Weston will run under X emulation
3135 without issues.
3136 </note>
3137 </para>
3138 </section>
3139
3140 <section id="enabling-wayland-in-an-image">
3141 <title>Enabling Wayland in an Image</title>
3142
3143 <para>
3144 To enable Wayland, you need to enable it to be built and enable
3145 it to be included in the image.
3146 </para>
3147
3148 <section id="enable-building">
3149 <title>Building</title>
3150
3151 <para>
3152 To cause Mesa to build the <filename>wayland-egl</filename>
3153 platform and Weston to build Wayland with Kernel Mode
3154 Setting
3155 (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>)
3156 support, include the "wayland" flag in the
3157 <ulink url="&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></ulink>
3158 statement in your <filename>local.conf</filename> file:
3159 <literallayout class='monospaced'>
3160 DISTRO_FEATURES_append = " wayland"
3161 </literallayout>
3162 <note>
3163 If X11 has been enabled elsewhere, Weston will build
3164 Wayland with X11 support
3165 </note>
3166 </para>
3167 </section>
3168
3169 <section id="enable-installation-in-an-image">
3170 <title>Installing</title>
3171
3172 <para>
3173 To install the Wayland feature into an image, you must
3174 include the following
3175 <ulink url='&YOCTO_DOCS_REF_URL;#var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></ulink>
3176 statement in your <filename>local.conf</filename> file:
3177 <literallayout class='monospaced'>
3178 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
3179 </literallayout>
3180 </para>
3181 </section>
3182 </section>
3183
3184 <section id="running-weston">
3185 <title>Running Weston</title>
3186
3187 <para>
3188 To run Weston inside X11, enabling it as described earlier and
3189 building a Sato image is sufficient.
3190 If you are running your image under Sato, a Weston Launcher
3191 appears in the "Utility" category.
3192 </para>
3193
3194 <para>
3195 Alternatively, you can run Weston through the command-line
3196 interpretor (CLI), which is better suited for development work.
3197 To run Weston under the CLI, you need to do the following after
3198 your image is built:
3199 <orderedlist>
3200 <listitem><para>
3201 Run these commands to export
3202 <filename>XDG_RUNTIME_DIR</filename>:
3203 <literallayout class='monospaced'>
3204 mkdir -p /tmp/$USER-weston
3205 chmod 0700 /tmp/$USER-weston
3206 export XDG_RUNTIME_DIR=/tmp/$USER-weston
3207 </literallayout>
3208 </para></listitem>
3209 <listitem><para>
3210 Launch Weston in the shell:
3211 <literallayout class='monospaced'>
3212 weston
3213 </literallayout></para></listitem>
3214 </orderedlist>
3215 </para>
3216 </section>
3217 </section>
3218
3219 <section id="overview-licenses">
3220 <title>Licenses</title>
3221
3222 <para>
3223 This section describes the mechanism by which the
3224 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
3225 tracks changes to licensing text.
3226 The section also describes how to enable commercially licensed
3227 recipes, which by default are disabled.
3228 </para>
3229
3230 <para>
3231 For information that can help you maintain compliance with
3232 various open source licensing during the lifecycle of the product,
3233 see the
3234 "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>"
3235 section in the Yocto Project Development Tasks Manual.
3236 </para>
3237
3238 <section id="usingpoky-configuring-LIC_FILES_CHKSUM">
3239 <title>Tracking License Changes</title>
3240
3241 <para>
3242 The license of an upstream project might change in the future.
3243 In order to prevent these changes going unnoticed, the
3244 <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink>
3245 variable tracks changes to the license text. The checksums are
3246 validated at the end of the configure step, and if the
3247 checksums do not match, the build will fail.
3248 </para>
3249
3250 <section id="usingpoky-specifying-LIC_FILES_CHKSUM">
3251 <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>
3252
3253 <para>
3254 The <filename>LIC_FILES_CHKSUM</filename>
3255 variable contains checksums of the license text in the
3256 source code for the recipe.
3257 Following is an example of how to specify
3258 <filename>LIC_FILES_CHKSUM</filename>:
3259 <literallayout class='monospaced'>
3260 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
3261 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
3262 file://licfile2.txt;endline=50;md5=zzzz \
3263 ..."
3264 </literallayout>
3265 <note><title>Notes</title>
3266 <itemizedlist>
3267 <listitem><para>
3268 When using "beginline" and "endline", realize
3269 that line numbering begins with one and not
3270 zero.
3271 Also, the included lines are inclusive (i.e.
3272 lines five through and including 29 in the
3273 previous example for
3274 <filename>licfile1.txt</filename>).
3275 </para></listitem>
3276 <listitem><para>
3277 When a license check fails, the selected license
3278 text is included as part of the QA message.
3279 Using this output, you can determine the exact
3280 start and finish for the needed license text.
3281 </para></listitem>
3282 </itemizedlist>
3283 </note>
3284 </para>
3285
3286 <para>
3287 The build system uses the
3288 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
3289 variable as the default directory when searching files
3290 listed in <filename>LIC_FILES_CHKSUM</filename>.
3291 The previous example employs the default directory.
3292 </para>
3293
3294 <para>
3295 Consider this next example:
3296 <literallayout class='monospaced'>
3297 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
3298 md5=bb14ed3c4cda583abc85401304b5cd4e"
3299 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
3300 </literallayout>
3301 </para>
3302
3303 <para>
3304 The first line locates a file in
3305 <filename>${S}/src/ls.c</filename> and isolates lines five
3306 through 16 as license text.
3307 The second line refers to a file in
3308 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
3309 </para>
3310
3311 <para>
3312 Note that <filename>LIC_FILES_CHKSUM</filename> variable is
3313 mandatory for all recipes, unless the
3314 <filename>LICENSE</filename> variable is set to "CLOSED".
3315 </para>
3316 </section>
3317
3318 <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
3319 <title>Explanation of Syntax</title>
3320
3321 <para>
3322 As mentioned in the previous section, the
3323 <filename>LIC_FILES_CHKSUM</filename> variable lists all
3324 the important files that contain the license text for the
3325 source code.
3326 It is possible to specify a checksum for an entire file,
3327 or a specific section of a file (specified by beginning and
3328 ending line numbers with the "beginline" and "endline"
3329 parameters, respectively).
3330 The latter is useful for source files with a license
3331 notice header, README documents, and so forth.
3332 If you do not use the "beginline" parameter, then it is
3333 assumed that the text begins on the first line of the file.
3334 Similarly, if you do not use the "endline" parameter,
3335 it is assumed that the license text ends with the last
3336 line of the file.
3337 </para>
3338
3339 <para>
3340 The "md5" parameter stores the md5 checksum of the license
3341 text.
3342 If the license text changes in any way as compared to
3343 this parameter then a mismatch occurs.
3344 This mismatch triggers a build failure and notifies
3345 the developer.
3346 Notification allows the developer to review and address
3347 the license text changes.
3348 Also note that if a mismatch occurs during the build,
3349 the correct md5 checksum is placed in the build log and
3350 can be easily copied to the recipe.
3351 </para>
3352
3353 <para>
3354 There is no limit to how many files you can specify using
3355 the <filename>LIC_FILES_CHKSUM</filename> variable.
3356 Generally, however, every project requires a few
3357 specifications for license tracking.
3358 Many projects have a "COPYING" file that stores the
3359 license information for all the source code files.
3360 This practice allows you to just track the "COPYING"
3361 file as long as it is kept up to date.
3362 <note><title>Tips</title>
3363 <itemizedlist>
3364 <listitem><para>
3365 If you specify an empty or invalid "md5"
3366 parameter,
3367 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
3368 returns an md5 mis-match
3369 error and displays the correct "md5" parameter
3370 value during the build.
3371 The correct parameter is also captured in
3372 the build log.
3373 </para></listitem>
3374 <listitem><para>
3375 If the whole file contains only license text,
3376 you do not need to use the "beginline" and
3377 "endline" parameters.
3378 </para></listitem>
3379 </itemizedlist>
3380 </note>
3381 </para>
3382 </section>
3383 </section>
3384
3385 <section id="enabling-commercially-licensed-recipes">
3386 <title>Enabling Commercially Licensed Recipes</title>
3387
3388 <para>
3389 By default, the OpenEmbedded build system disables
3390 components that have commercial or other special licensing
3391 requirements.
3392 Such requirements are defined on a
3393 recipe-by-recipe basis through the
3394 <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></ulink>
3395 variable definition in the affected recipe.
3396 For instance, the
3397 <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
3398 recipe contains the following statement:
3399 <literallayout class='monospaced'>
3400 LICENSE_FLAGS = "commercial"
3401 </literallayout>
3402 Here is a slightly more complicated example that contains both
3403 an explicit recipe name and version (after variable expansion):
3404 <literallayout class='monospaced'>
3405 LICENSE_FLAGS = "license_${PN}_${PV}"
3406 </literallayout>
3407 In order for a component restricted by a
3408 <filename>LICENSE_FLAGS</filename> definition to be enabled and
3409 included in an image, it needs to have a matching entry in the
3410 global
3411 <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></ulink>
3412 variable, which is a variable typically defined in your
3413 <filename>local.conf</filename> file.
3414 For example, to enable the
3415 <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
3416 package, you could add either the string
3417 "commercial_gst-plugins-ugly" or the more general string
3418 "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.
3419 See the
3420 "<link linkend='license-flag-matching'>License Flag Matching</link>"
3421 section for a full
3422 explanation of how <filename>LICENSE_FLAGS</filename> matching
3423 works.
3424 Here is the example:
3425 <literallayout class='monospaced'>
3426 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
3427 </literallayout>
3428 Likewise, to additionally enable the package built from the
3429 recipe containing
3430 <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>,
3431 and assuming that the actual recipe name was
3432 <filename>emgd_1.10.bb</filename>, the following string would
3433 enable that package as well as the original
3434 <filename>gst-plugins-ugly</filename> package:
3435 <literallayout class='monospaced'>
3436 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
3437 </literallayout>
3438 As a convenience, you do not need to specify the complete
3439 license string in the whitelist for every package.
3440 You can use an abbreviated form, which consists
3441 of just the first portion or portions of the license
3442 string before the initial underscore character or characters.
3443 A partial string will match any license that contains the
3444 given string as the first portion of its license.
3445 For example, the following whitelist string will also match
3446 both of the packages previously mentioned as well as any other
3447 packages that have licenses starting with "commercial" or
3448 "license".
3449 <literallayout class='monospaced'>
3450 LICENSE_FLAGS_WHITELIST = "commercial license"
3451 </literallayout>
3452 </para>
3453
3454 <section id="license-flag-matching">
3455 <title>License Flag Matching</title>
3456
3457 <para>
3458 License flag matching allows you to control what recipes
3459 the OpenEmbedded build system includes in the build.
3460 Fundamentally, the build system attempts to match
3461 <filename>LICENSE_FLAGS</filename> strings found in recipes
3462 against <filename>LICENSE_FLAGS_WHITELIST</filename>
3463 strings found in the whitelist.
3464 A match causes the build system to include a recipe in the
3465 build, while failure to find a match causes the build
3466 system to exclude a recipe.
3467 </para>
3468
3469 <para>
3470 In general, license flag matching is simple.
3471 However, understanding some concepts will help you
3472 correctly and effectively use matching.
3473 </para>
3474
3475 <para>
3476 Before a flag
3477 defined by a particular recipe is tested against the
3478 contents of the whitelist, the expanded string
3479 <filename>_${PN}</filename> is appended to the flag.
3480 This expansion makes each
3481 <filename>LICENSE_FLAGS</filename> value recipe-specific.
3482 After expansion, the string is then matched against the
3483 whitelist.
3484 Thus, specifying
3485 <filename>LICENSE_FLAGS = "commercial"</filename>
3486 in recipe "foo", for example, results in the string
3487 <filename>"commercial_foo"</filename>.
3488 And, to create a match, that string must appear in the
3489 whitelist.
3490 </para>
3491
3492 <para>
3493 Judicious use of the <filename>LICENSE_FLAGS</filename>
3494 strings and the contents of the
3495 <filename>LICENSE_FLAGS_WHITELIST</filename> variable
3496 allows you a lot of flexibility for including or excluding
3497 recipes based on licensing.
3498 For example, you can broaden the matching capabilities by
3499 using license flags string subsets in the whitelist.
3500 <note>
3501 When using a string subset, be sure to use the part of
3502 the expanded string that precedes the appended
3503 underscore character (e.g.
3504 <filename>usethispart_1.3</filename>,
3505 <filename>usethispart_1.4</filename>, and so forth).
3506 </note>
3507 For example, simply specifying the string "commercial" in
3508 the whitelist matches any expanded
3509 <filename>LICENSE_FLAGS</filename> definition that starts
3510 with the string "commercial" such as "commercial_foo" and
3511 "commercial_bar", which are the strings the build system
3512 automatically generates for hypothetical recipes named
3513 "foo" and "bar" assuming those recipes simply specify the
3514 following:
3515 <literallayout class='monospaced'>
3516 LICENSE_FLAGS = "commercial"
3517 </literallayout>
3518 Thus, you can choose to exhaustively
3519 enumerate each license flag in the whitelist and
3520 allow only specific recipes into the image, or
3521 you can use a string subset that causes a broader range of
3522 matches to allow a range of recipes into the image.
3523 </para>
3524
3525 <para>
3526 This scheme works even if the
3527 <filename>LICENSE_FLAGS</filename> string already
3528 has <filename>_${PN}</filename> appended.
3529 For example, the build system turns the license flag
3530 "commercial_1.2_foo" into "commercial_1.2_foo_foo" and
3531 would match both the general "commercial" and the specific
3532 "commercial_1.2_foo" strings found in the whitelist, as
3533 expected.
3534 </para>
3535
3536 <para>
3537 Here are some other scenarios:
3538 <itemizedlist>
3539 <listitem><para>
3540 You can specify a versioned string in the recipe
3541 such as "commercial_foo_1.2" in a "foo" recipe.
3542 The build system expands this string to
3543 "commercial_foo_1.2_foo".
3544 Combine this license flag with a whitelist that has
3545 the string "commercial" and you match the flag
3546 along with any other flag that starts with the
3547 string "commercial".
3548 </para></listitem>
3549 <listitem><para>
3550 Under the same circumstances, you can use
3551 "commercial_foo" in the whitelist and the build
3552 system not only matches "commercial_foo_1.2" but
3553 also matches any license flag with the string
3554 "commercial_foo", regardless of the version.
3555 </para></listitem>
3556 <listitem><para>
3557 You can be very specific and use both the
3558 package and version parts in the whitelist (e.g.
3559 "commercial_foo_1.2") to specifically match a
3560 versioned recipe.
3561 </para></listitem>
3562 </itemizedlist>
3563 </para>
3564 </section>
3565
3566 <section id="other-variables-related-to-commercial-licenses">
3567 <title>Other Variables Related to Commercial Licenses</title>
3568
3569 <para>
3570 Other helpful variables related to commercial
3571 license handling exist and are defined in the
3572 <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file:
3573 <literallayout class='monospaced'>
3574 COMMERCIAL_AUDIO_PLUGINS ?= ""
3575 COMMERCIAL_VIDEO_PLUGINS ?= ""
3576 </literallayout>
3577 If you want to enable these components, you can do so by
3578 making sure you have statements similar to the following
3579 in your <filename>local.conf</filename> configuration file:
3580 <literallayout class='monospaced'>
3581 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
3582 gst-plugins-ugly-mpegaudioparse"
3583 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
3584 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
3585 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
3586 </literallayout>
3587 Of course, you could also create a matching whitelist
3588 for those components using the more general "commercial"
3589 in the whitelist, but that would also enable all the
3590 other packages with <filename>LICENSE_FLAGS</filename>
3591 containing "commercial", which you may or may not want:
3592 <literallayout class='monospaced'>
3593 LICENSE_FLAGS_WHITELIST = "commercial"
3594 </literallayout>
3595 </para>
3596
3597 <para>
3598 Specifying audio and video plug-ins as part of the
3599 <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
3600 <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements
3601 (along with the enabling
3602 <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the
3603 plug-ins or components into built images, thus adding
3604 support for media formats or components.
3605 </para>
3606 </section>
3607 </section>
3608 </section>
3609</chapter>
3610<!--
3611vim: expandtab tw=80 ts=4
3612-->
diff --git a/documentation/overview-manual/overview-manual-customization.xsl b/documentation/overview-manual/overview-manual-customization.xsl
new file mode 100644
index 0000000000..22360e7bab
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-customization.xsl
@@ -0,0 +1,27 @@
1<?xml version='1.0'?>
2<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
3
4 <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
5
6<!--
7
8 <xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
9
10 <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/xhtml/docbook.xsl" />
11
12-->
13
14 <xsl:include href="../template/permalinks.xsl"/>
15 <xsl:include href="../template/section.title.xsl"/>
16 <xsl:include href="../template/component.title.xsl"/>
17 <xsl:include href="../template/division.title.xsl"/>
18 <xsl:include href="../template/formal.object.heading.xsl"/>
19
20 <xsl:param name="html.stylesheet" select="'overview-manual-style.css'" />
21 <xsl:param name="chapter.autolabel" select="1" />
22 <xsl:param name="appendix.autolabel" select="A" />
23 <xsl:param name="section.autolabel" select="1" />
24 <xsl:param name="section.label.includes.component.label" select="1" />
25 <xsl:param name="generate.id.attributes" select="1" />
26
27</xsl:stylesheet>
diff --git a/documentation/overview-manual/overview-manual-development-environment.xml b/documentation/overview-manual/overview-manual-development-environment.xml
new file mode 100644
index 0000000000..7bcf87ffd9
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-development-environment.xml
@@ -0,0 +1,972 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='overview-development-environment'>
6<title>The Yocto Project Development Environment</title>
7
8<para>
9 This chapter takes a look at the Yocto Project development
10 environment.
11 The chapter provides Yocto Project Development environment concepts that
12 help you understand how work is accomplished in an open source environment,
13 which is very different as compared to work accomplished in a closed,
14 proprietary environment.
15</para>
16
17<para>
18 Specifically, this chapter addresses open source philosophy, source
19 repositories, workflows, Git, and licensing.
20</para>
21
22<section id='open-source-philosophy'>
23 <title>Open Source Philosophy</title>
24
25 <para>
26 Open source philosophy is characterized by software development
27 directed by peer production and collaboration through an active
28 community of developers.
29 Contrast this to the more standard centralized development models
30 used by commercial software companies where a finite set of developers
31 produces a product for sale using a defined set of procedures that
32 ultimately result in an end product whose architecture and source
33 material are closed to the public.
34 </para>
35
36 <para>
37 Open source projects conceptually have differing concurrent agendas,
38 approaches, and production.
39 These facets of the development process can come from anyone in the
40 public (community) who has a stake in the software project.
41 The open source environment contains new copyright, licensing, domain,
42 and consumer issues that differ from the more traditional development
43 environment.
44 In an open source environment, the end product, source material,
45 and documentation are all available to the public at no cost.
46 </para>
47
48 <para>
49 A benchmark example of an open source project is the Linux kernel,
50 which was initially conceived and created by Finnish computer science
51 student Linus Torvalds in 1991.
52 Conversely, a good example of a non-open source project is the
53 <trademark class='registered'>Windows</trademark> family of operating
54 systems developed by
55 <trademark class='registered'>Microsoft</trademark> Corporation.
56 </para>
57
58 <para>
59 Wikipedia has a good historical description of the Open Source
60 Philosophy
61 <ulink url='http://en.wikipedia.org/wiki/Open_source'>here</ulink>.
62 You can also find helpful information on how to participate in the
63 Linux Community
64 <ulink url='http://ldn.linuxfoundation.org/book/how-participate-linux-community'>here</ulink>.
65 </para>
66</section>
67
68<section id='gs-the-development-host'>
69 <title>The Development Host</title>
70
71 <para>
72 A development host or build host is key to using the Yocto Project.
73 Because the goal of the Yocto Project is to develop images or
74 applications that run on embedded hardware, development of those
75 images and applications generally takes place on a system not
76 intended to run the software - the development host.
77 </para>
78
79 <para>
80 You need to set up a development host in order to use it with the
81 Yocto Project.
82 Most find that it is best to have a native Linux machine function as
83 the development host.
84 However, it is possible to use a system that does not run Linux
85 as its operating system as your development host.
86 When you have a Mac or Windows-based system, you can set it up
87 as the development host by using
88 <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/crops/about/'>CROPS</ulink>,
89 which leverages
90 <ulink url='https://www.docker.com/'>Docker Containers</ulink>.
91 Once you take the steps to set up a CROPS machine, you effectively
92 have access to a shell environment that is similar to what you see
93 when using a Linux-based development host.
94 For the steps needed to set up a system using CROPS, see the
95 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops'>Setting Up to Use CROss PlatformS (CROPS)</ulink>"
96 section in the Yocto Project Development Tasks Manual.
97 </para>
98
99 <para>
100 If your development host is going to be a system that runs a Linux
101 distribution, steps still exist that you must take to prepare the
102 system for use with the Yocto Project.
103 You need to be sure that the Linux distribution on the system is
104 one that supports the Yocto Project.
105 You also need to be sure that the correct set of host packages are
106 installed that allow development using the Yocto Project.
107 For the steps needed to set up a development host that runs Linux,
108 see the
109 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host'>Setting Up a Native Linux Host</ulink>"
110 section in the Yocto Project Development Tasks Manual.
111 </para>
112
113 <para>
114 Once your development host is set up to use the Yocto Project,
115 several methods exist for you to do work in the Yocto Project
116 environment:
117 <itemizedlist>
118 <listitem><para>
119 <emphasis>Command Lines, BitBake, and Shells:</emphasis>
120 Traditional development in the Yocto Project involves using
121 OpenEmbedded build system, which uses BitBake, in a
122 command-line environment from a shell on your development
123 host.
124 You can accomplish this from a host that is a native Linux
125 machine or from a host that has been set up with CROPS.
126 Either way, you create, modify, and build images and
127 applications all within a shell-based environment using
128 components and tools available through your Linux distribution
129 and the Yocto Project.</para>
130
131 <para>For a general flow of the build procedures, see the
132 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-building-an-image'>Building an Image</ulink>"
133 section in the Yocto Project Development Tasks Manual.
134 </para></listitem>
135 <listitem><para>
136 <emphasis>Board Support Package (BSP) Development:</emphasis>
137 Development of BSPs involves using the Yocto Project to
138 create and test layers that allow easy development of
139 images and applications targeted for specific hardware.
140 To development BSPs, you need to take some additional steps
141 beyond what was described in setting up a development host.
142 </para>
143
144 <para>The
145 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide'</ulink>
146 provides BSP-related development information.
147 For specifics on development host preparation, see the
148 "<ulink url='&YOCTO_DOCS_BSP_URL;#preparing-your-build-host-to-work-with-bsp-layers'>Preparing Your Build Host to Work With BSP Layers</ulink>"
149 section in the Yocto Project Board Support Package (BSP)
150 Developer's Guide.
151 </para></listitem>
152 <listitem><para>
153 <emphasis>Kernel Development:</emphasis>
154 If you are going to be developing kernels using the Yocto
155 Project you likely will be using <filename>devtool</filename>.
156 A workflow using <filename>devtool</filename> makes kernel
157 development quicker by reducing iteration cycle times.</para>
158
159 <para>The
160 <ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;'>Yocto Project Linux Kernel Development Manual</ulink>
161 provides kernel-related development information.
162 For specifics on development host preparation, see the
163 "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#preparing-the-build-host-to-work-on-the-kernel'>Preparing the Build Host to Work on the Kernel</ulink>"
164 section in the Yocto Project Linux Kernel Development Manual.
165 </para></listitem>
166 <listitem><para>
167 <emphasis>Using the <trademark class='trade'>Eclipse</trademark> IDE:</emphasis>
168 One of two Yocto Project development methods that involves an
169 interface that effectively puts the Yocto Project into the
170 background is the popular Eclipse IDE.
171 This method of development is advantageous if you are already
172 familiar with working within Eclipse.
173 Development is supported through a plugin that you install
174 onto your development host.</para>
175
176 <para>For steps that show you how to set up your development
177 host to use the Eclipse Yocto Project plugin, see the
178 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-eclipse-project'>Developing Applications Using <trademark class='trade'>Eclipse</trademark></ulink>"
179 Chapter in the Yocto Project Application Development and the
180 Extensible Software Development Kit (eSDK) manual.
181 </para></listitem>
182 <listitem><para>
183 <emphasis>Using the Toaster:</emphasis>
184 The other Yocto Project development method that involves an
185 interface that effectively puts the Yocto Project into the
186 background is Toaster.
187 Toaster provides an interface to the OpenEmbedded build system.
188 The interface enables you to configure and run your builds.
189 Information about builds is collected and stored in a database.
190 You can use Toaster to configure and start builds on multiple
191 remote build servers.</para>
192
193 <para>For steps that show you how to set up your development
194 host to use Toaster and on how to use Toaster in general,
195 see the
196 <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>.
197 </para></listitem>
198 </itemizedlist>
199 </para>
200</section>
201
202<section id='yocto-project-repositories'>
203 <title>Yocto Project Source Repositories</title>
204
205 <para>
206 The Yocto Project team maintains complete source repositories for all
207 Yocto Project files at
208 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'></ulink>.
209 This web-based source code browser is organized into categories by
210 function such as IDE Plugins, Matchbox, Poky, Yocto Linux Kernel, and
211 so forth.
212 From the interface, you can click on any particular item in the "Name"
213 column and see the URL at the bottom of the page that you need to clone
214 a Git repository for that particular item.
215 Having a local Git repository of the
216 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>,
217 which is usually named "poky", allows
218 you to make changes, contribute to the history, and ultimately enhance
219 the Yocto Project's tools, Board Support Packages, and so forth.
220 </para>
221
222 <para>
223 For any supported release of Yocto Project, you can also go to the
224 <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink> and
225 select the "Downloads" tab and get a released tarball of the
226 <filename>poky</filename> repository or any supported BSP tarballs.
227 Unpacking these tarballs gives you a snapshot of the released
228 files.
229 <note><title>Notes</title>
230 <itemizedlist>
231 <listitem><para>
232 The recommended method for setting up the Yocto Project
233 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
234 and the files for supported BSPs
235 (e.g., <filename>meta-intel</filename>) is to use
236 <link linkend='git'>Git</link> to create a local copy of
237 the upstream repositories.
238 </para></listitem>
239 <listitem><para>
240 Be sure to always work in matching branches for both
241 the selected BSP repository and the
242 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
243 (i.e. <filename>poky</filename>) repository.
244 For example, if you have checked out the "master" branch
245 of <filename>poky</filename> and you are going to use
246 <filename>meta-intel</filename>, be sure to checkout the
247 "master" branch of <filename>meta-intel</filename>.
248 </para></listitem>
249 </itemizedlist>
250 </note>
251 </para>
252
253 <para>
254 In summary, here is where you can get the project files needed for
255 development:
256 <itemizedlist>
257 <listitem><para id='source-repositories'>
258 <emphasis>
259 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'>Source Repositories:</ulink>
260 </emphasis>
261 This area contains IDE Plugins, Matchbox, Poky, Poky Support,
262 Tools, Yocto Linux Kernel, and Yocto Metadata Layers.
263 You can create local copies of Git repositories for each of
264 these areas.</para>
265
266 <para>
267 <imagedata fileref="figures/source-repos.png" align="center" width="6in" depth="4in" />
268 For steps on how to view and access these upstream Git
269 repositories, see the
270 "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-source-repositories'>Accessing Source Repositories</ulink>"
271 Section in the Yocto Project Development Tasks Manual.
272 </para></listitem>
273 <listitem><para><anchor id='index-downloads' />
274 <emphasis>
275 <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink>
276 </emphasis>
277 This is an index of releases such as
278 the <trademark class='trade'>Eclipse</trademark>
279 Yocto Plug-in, miscellaneous support, Poky, Pseudo, installers
280 for cross-development toolchains, and all released versions of
281 Yocto Project in the form of images or tarballs.
282 Downloading and extracting these files does not produce a local
283 copy of the Git repository but rather a snapshot of a
284 particular release or image.</para>
285
286 <para>
287 <imagedata fileref="figures/index-downloads.png" align="center" width="6in" depth="3.5in" />
288 For steps on how to view and access these files, see the
289 "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-index-of-releases'>Accessing Index of Releases</ulink>"
290 section in the Yocto Project Development Tasks Manual.
291 </para></listitem>
292 <listitem><para id='downloads-page'>
293 <emphasis>"Downloads" page for the
294 <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>:
295 </emphasis></para>
296
297 <para>The Yocto Project website includes a "DOWNLOADS" page
298 accessible through the "SOFTWARE" tab
299 that allows you to download any Yocto Project
300 release, tool, and Board Support Package (BSP) in tarball form.
301 The tarballs are similar to those found in the
302 <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink> area.</para>
303
304 <para>
305 <imagedata fileref="figures/yp-download.png" align="center" width="6in" depth="4in" />
306 For steps on how to use the "DOWNLOADS" page, see the
307 "<ulink url='&YOCTO_DOCS_DEV_URL;#using-the-downloads-page'>Using the Downloads Page</ulink>"
308 section in the Yocto Project Development Tasks Manual.
309 </para></listitem>
310 </itemizedlist>
311 </para>
312</section>
313
314<section id='gs-git-workflows-and-the-yocto-project'>
315 <title>Git Workflows and the Yocto Project</title>
316
317 <para>
318 Developing using the Yocto Project likely requires the use of
319 <link linkend='git'>Git</link>.
320 Git is a free, open source distributed version control system
321 used as part of many collaborative design environments.
322 This section provides workflow concepts using the Yocto Project and
323 Git.
324 In particular, the information covers basic practices that describe
325 roles and actions in a collaborative development environment.
326 <note>
327 If you are familiar with this type of development environment, you
328 might not want to read this section.
329 </note>
330 </para>
331
332 <para>
333 The Yocto Project files are maintained using Git in "branches"
334 whose Git histories track every change and whose structures
335 provide branches for all diverging functionality.
336 Although there is no need to use Git, many open source projects do so.
337 <para>
338
339 </para>
340 For the Yocto Project, a key individual called the "maintainer" is
341 responsible for the integrity of the "master" branch of a given Git
342 repository.
343 The "master" branch is the “upstream” repository from which final or
344 most recent builds of a project occur.
345 The maintainer is responsible for accepting changes from other
346 developers and for organizing the underlying branch structure to
347 reflect release strategies and so forth.
348 <note>
349 For information on finding out who is responsible for (maintains)
350 a particular area of code in the Yocto Project, see the
351 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
352 section of the Yocto Project Development Tasks Manual.
353 </note>
354 </para>
355
356 <para>
357 The Yocto Project <filename>poky</filename> Git repository also has an
358 upstream contribution Git repository named
359 <filename>poky-contrib</filename>.
360 You can see all the branches in this repository using the web interface
361 of the
362 <ulink url='&YOCTO_GIT_URL;'>Source Repositories</ulink> organized
363 within the "Poky Support" area.
364 These branches hold changes (commits) to the project that have been
365 submitted or committed by the Yocto Project development team and by
366 community members who contribute to the project.
367 The maintainer determines if the changes are qualified to be moved
368 from the "contrib" branches into the "master" branch of the Git
369 repository.
370 </para>
371
372 <para>
373 Developers (including contributing community members) create and
374 maintain cloned repositories of upstream branches.
375 The cloned repositories are local to their development platforms and
376 are used to develop changes.
377 When a developer is satisfied with a particular feature or change,
378 they "push" the change to the appropriate "contrib" repository.
379 </para>
380
381 <para>
382 Developers are responsible for keeping their local repository
383 up-to-date with whatever upstream branch they are working against.
384 They are also responsible for straightening out any conflicts that
385 might arise within files that are being worked on simultaneously by
386 more than one person.
387 All this work is done locally on the development host before
388 anything is pushed to a "contrib" area and examined at the maintainer’s
389 level.
390 </para>
391
392 <para>
393 A somewhat formal method exists by which developers commit changes
394 and push them into the "contrib" area and subsequently request that
395 the maintainer include them into an upstream branch.
396 This process is called “submitting a patch” or "submitting a change."
397 For information on submitting patches and changes, see the
398 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
399 section in the Yocto Project Development Tasks Manual.
400 </para>
401
402 <para>
403 To summarize the development workflow: a single point of entry
404 exists for changes into a "master" or development branch of the
405 Git repository, which is controlled by the project’s maintainer.
406 And, a set of developers exist who independently develop, test, and
407 submit changes to "contrib" areas for the maintainer to examine.
408 The maintainer then chooses which changes are going to become a
409 permanent part of the project.
410 </para>
411
412 <para>
413 <imagedata fileref="figures/git-workflow.png" width="6in" depth="3in" align="left" scalefit="1" />
414 </para>
415
416 <para>
417 While each development environment is unique, there are some best
418 practices or methods that help development run smoothly.
419 The following list describes some of these practices.
420 For more information about Git workflows, see the workflow topics in
421 the
422 <ulink url='http://book.git-scm.com'>Git Community Book</ulink>.
423 <itemizedlist>
424 <listitem><para>
425 <emphasis>Make Small Changes:</emphasis>
426 It is best to keep the changes you commit small as compared to
427 bundling many disparate changes into a single commit.
428 This practice not only keeps things manageable but also allows
429 the maintainer to more easily include or refuse changes.
430 </para></listitem>
431 <listitem><para>
432 <emphasis>Make Complete Changes:</emphasis>
433 It is also good practice to leave the repository in a
434 state that allows you to still successfully build your project.
435 In other words, do not commit half of a feature,
436 then add the other half as a separate, later commit.
437 Each commit should take you from one buildable project state
438 to another buildable state.
439 </para></listitem>
440 <listitem><para>
441 <emphasis>Use Branches Liberally:</emphasis>
442 It is very easy to create, use, and delete local branches in
443 your working Git repository on the development host.
444 You can name these branches anything you like.
445 It is helpful to give them names associated with the particular
446 feature or change on which you are working.
447 Once you are done with a feature or change and have merged it
448 into your local master branch, simply discard the temporary
449 branch.
450 </para></listitem>
451 <listitem><para>
452 <emphasis>Merge Changes:</emphasis>
453 The <filename>git merge</filename> command allows you to take
454 the changes from one branch and fold them into another branch.
455 This process is especially helpful when more than a single
456 developer might be working on different parts of the same
457 feature.
458 Merging changes also automatically identifies any collisions
459 or "conflicts" that might happen as a result of the same lines
460 of code being altered by two different developers.
461 </para></listitem>
462 <listitem><para>
463 <emphasis>Manage Branches:</emphasis>
464 Because branches are easy to use, you should use a system
465 where branches indicate varying levels of code readiness.
466 For example, you can have a "work" branch to develop in, a
467 "test" branch where the code or change is tested, a "stage"
468 branch where changes are ready to be committed, and so forth.
469 As your project develops, you can merge code across the
470 branches to reflect ever-increasing stable states of the
471 development.
472 </para></listitem>
473 <listitem><para>
474 <emphasis>Use Push and Pull:</emphasis>
475 The push-pull workflow is based on the concept of developers
476 "pushing" local commits to a remote repository, which is
477 usually a contribution repository.
478 This workflow is also based on developers "pulling" known
479 states of the project down into their local development
480 repositories.
481 The workflow easily allows you to pull changes submitted by
482 other developers from the upstream repository into your
483 work area ensuring that you have the most recent software
484 on which to develop.
485 The Yocto Project has two scripts named
486 <filename>create-pull-request</filename> and
487 <filename>send-pull-request</filename> that ship with the
488 release to facilitate this workflow.
489 You can find these scripts in the <filename>scripts</filename>
490 folder of the
491 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
492 For information on how to use these scripts, see the
493 "<ulink url='&YOCTO_DOCS_DEV_URL;#pushing-a-change-upstream'>Using Scripts to Push a Change Upstream and Request a Pull</ulink>"
494 section in the Yocto Project Development Tasks Manual.
495 </para></listitem>
496 <listitem><para>
497 <emphasis>Patch Workflow:</emphasis>
498 This workflow allows you to notify the maintainer through an
499 email that you have a change (or patch) you would like
500 considered for the "master" branch of the Git repository.
501 To send this type of change, you format the patch and then
502 send the email using the Git commands
503 <filename>git format-patch</filename> and
504 <filename>git send-email</filename>.
505 For information on how to use these scripts, see the
506 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
507 section in the Yocto Project Development Tasks Manual.
508 </para></listitem>
509 </itemizedlist>
510 </para>
511</section>
512
513<section id='git'>
514 <title>Git</title>
515
516 <para>
517 The Yocto Project makes extensive use of Git, which is a
518 free, open source distributed version control system.
519 Git supports distributed development, non-linear development,
520 and can handle large projects.
521 It is best that you have some fundamental understanding
522 of how Git tracks projects and how to work with Git if
523 you are going to use the Yocto Project for development.
524 This section provides a quick overview of how Git works and
525 provides you with a summary of some essential Git commands.
526 <note><title>Notes</title>
527 <itemizedlist>
528 <listitem><para>
529 For more information on Git, see
530 <ulink url='http://git-scm.com/documentation'></ulink>.
531 </para></listitem>
532 <listitem><para>
533 If you need to download Git, it is recommended that you add
534 Git to your system through your distribution's "software
535 store" (e.g. for Ubuntu, use the Ubuntu Software feature).
536 For the Git download page, see
537 <ulink url='http://git-scm.com/download'></ulink>.
538 </para></listitem>
539 <listitem><para>
540 For information beyond the introductory nature in this
541 section, see the
542 "<ulink url='&YOCTO_DOCS_DEV_URL;#working-with-yocto-project-source-files'>Working With Yocto Project Source Files</ulink>"
543 section in the Yocto Project Development Tasks Manual.
544 </para></listitem>
545 </itemizedlist>
546 </note>
547 </para>
548
549 <section id='repositories-tags-and-branches'>
550 <title>Repositories, Tags, and Branches</title>
551
552 <para>
553 As mentioned briefly in the previous section and also in the
554 "<link linkend='gs-git-workflows-and-the-yocto-project'>Git Workflows and the Yocto Project</link>"
555 section, the Yocto Project maintains source repositories at
556 <ulink url='&YOCTO_GIT_URL;/cgit.cgi'></ulink>.
557 If you look at this web-interface of the repositories, each item
558 is a separate Git repository.
559 </para>
560
561 <para>
562 Git repositories use branching techniques that track content
563 change (not files) within a project (e.g. a new feature or updated
564 documentation).
565 Creating a tree-like structure based on project divergence allows
566 for excellent historical information over the life of a project.
567 This methodology also allows for an environment from which you can
568 do lots of local experimentation on projects as you develop
569 changes or new features.
570 </para>
571
572 <para>
573 A Git repository represents all development efforts for a given
574 project.
575 For example, the Git repository <filename>poky</filename> contains
576 all changes and developments for that repository over the course
577 of its entire life.
578 That means that all changes that make up all releases are captured.
579 The repository maintains a complete history of changes.
580 </para>
581
582 <para>
583 You can create a local copy of any repository by "cloning" it
584 with the <filename>git clone</filename> command.
585 When you clone a Git repository, you end up with an identical
586 copy of the repository on your development system.
587 Once you have a local copy of a repository, you can take steps to
588 develop locally.
589 For examples on how to clone Git repositories, see the
590 "<ulink url='&YOCTO_DOCS_DEV_URL;#working-with-yocto-project-source-files'>Working With Yocto Project Source Files</ulink>"
591 section in the Yocto Project Development Tasks Manual.
592 </para>
593
594 <para>
595 It is important to understand that Git tracks content change and
596 not files.
597 Git uses "branches" to organize different development efforts.
598 For example, the <filename>poky</filename> repository has
599 several branches that include the current "&DISTRO_NAME_NO_CAP;"
600 branch, the "master" branch, and many branches for past
601 Yocto Project releases.
602 You can see all the branches by going to
603 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and
604 clicking on the
605 <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/heads'>[...]</ulink></filename>
606 link beneath the "Branch" heading.
607 </para>
608
609 <para>
610 Each of these branches represents a specific area of development.
611 The "master" branch represents the current or most recent
612 development.
613 All other branches represent offshoots of the "master" branch.
614 </para>
615
616 <para>
617 When you create a local copy of a Git repository, the copy has
618 the same set of branches as the original.
619 This means you can use Git to create a local working area
620 (also called a branch) that tracks a specific development branch
621 from the upstream source Git repository.
622 in other words, you can define your local Git environment to
623 work on any development branch in the repository.
624 To help illustrate, consider the following example Git commands:
625 <literallayout class='monospaced'>
626 $ cd ~
627 $ git clone git://git.yoctoproject.org/poky
628 $ cd poky
629 $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP;
630 </literallayout>
631 In the previous example after moving to the home directory, the
632 <filename>git clone</filename> command creates a
633 local copy of the upstream <filename>poky</filename> Git repository.
634 By default, Git checks out the "master" branch for your work.
635 After changing the working directory to the new local repository
636 (i.e. <filename>poky</filename>), the
637 <filename>git checkout</filename> command creates
638 and checks out a local branch named "&DISTRO_NAME_NO_CAP;", which
639 tracks the upstream "origin/&DISTRO_NAME_NO_CAP;" branch.
640 Changes you make while in this branch would ultimately affect
641 the upstream "&DISTRO_NAME_NO_CAP;" branch of the
642 <filename>poky</filename> repository.
643 </para>
644
645 <para>
646 It is important to understand that when you create and checkout a
647 local working branch based on a branch name,
648 your local environment matches the "tip" of that particular
649 development branch at the time you created your local branch,
650 which could be different from the files in the "master" branch
651 of the upstream repository.
652 In other words, creating and checking out a local branch based on
653 the "&DISTRO_NAME_NO_CAP;" branch name is not the same as
654 checking out the "master" branch in the repository.
655 Keep reading to see how you create a local snapshot of a Yocto
656 Project Release.
657 </para>
658
659 <para>
660 Git uses "tags" to mark specific changes in a repository branch
661 structure.
662 Typically, a tag is used to mark a special point such as the final
663 change (or commit) before a project is released.
664 You can see the tags used with the <filename>poky</filename> Git
665 repository by going to
666 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and
667 clicking on the
668 <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/tags'>[...]</ulink></filename>
669 link beneath the "Tag" heading.
670 </para>
671
672 <para>
673 Some key tags for the <filename>poky</filename> repository are
674 <filename>jethro-14.0.3</filename>,
675 <filename>morty-16.0.1</filename>,
676 <filename>pyro-17.0.0</filename>, and
677 <filename>&DISTRO_NAME_NO_CAP;-&POKYVERSION;</filename>.
678 These tags represent Yocto Project releases.
679 </para>
680
681 <para>
682 When you create a local copy of the Git repository, you also
683 have access to all the tags in the upstream repository.
684 Similar to branches, you can create and checkout a local working
685 Git branch based on a tag name.
686 When you do this, you get a snapshot of the Git repository that
687 reflects the state of the files when the change was made associated
688 with that tag.
689 The most common use is to checkout a working branch that matches
690 a specific Yocto Project release.
691 Here is an example:
692 <literallayout class='monospaced'>
693 $ cd ~
694 $ git clone git://git.yoctoproject.org/poky
695 $ cd poky
696 $ git fetch --all --tags --prune
697 $ git checkout tags/pyro-17.0.0 -b my-pyro-17.0.0
698 </literallayout>
699 In this example, the name of the top-level directory of your
700 local Yocto Project repository is <filename>poky</filename>.
701 After moving to the <filename>poky</filename> directory, the
702 <filename>git fetch</filename> command makes all the upstream
703 tags available locally in your repository.
704 Finally, the <filename>git checkout</filename> command
705 creates and checks out a branch named "my-pyro-17.0.0" that is
706 based on the upstream branch whose "HEAD" matches the
707 commit in the repository associated with the "pyro-17.0.0" tag.
708 The files in your repository now exactly match that particular
709 Yocto Project release as it is tagged in the upstream Git
710 repository.
711 It is important to understand that when you create and
712 checkout a local working branch based on a tag, your environment
713 matches a specific point in time and not the entire development
714 branch (i.e. from the "tip" of the branch backwards).
715 </para>
716 </section>
717
718 <section id='basic-commands'>
719 <title>Basic Commands</title>
720
721 <para>
722 Git has an extensive set of commands that lets you manage changes
723 and perform collaboration over the life of a project.
724 Conveniently though, you can manage with a small set of basic
725 operations and workflows once you understand the basic
726 philosophy behind Git.
727 You do not have to be an expert in Git to be functional.
728 A good place to look for instruction on a minimal set of Git
729 commands is
730 <ulink url='http://git-scm.com/documentation'>here</ulink>.
731 </para>
732
733 <para>
734 If you do not know much about Git, you should educate
735 yourself by visiting the links previously mentioned.
736 </para>
737
738 <para>
739 The following list of Git commands briefly describes some basic
740 Git operations as a way to get started.
741 As with any set of commands, this list (in most cases) simply shows
742 the base command and omits the many arguments it supports.
743 See the Git documentation for complete descriptions and strategies
744 on how to use these commands:
745 <itemizedlist>
746 <listitem><para>
747 <emphasis><filename>git init</filename>:</emphasis>
748 Initializes an empty Git repository.
749 You cannot use Git commands unless you have a
750 <filename>.git</filename> repository.
751 </para></listitem>
752 <listitem><para id='git-commands-clone'>
753 <emphasis><filename>git clone</filename>:</emphasis>
754 Creates a local clone of a Git repository that is on
755 equal footing with a fellow developer’s Git repository
756 or an upstream repository.
757 </para></listitem>
758 <listitem><para>
759 <emphasis><filename>git add</filename>:</emphasis>
760 Locally stages updated file contents to the index that
761 Git uses to track changes.
762 You must stage all files that have changed before you
763 can commit them.
764 </para></listitem>
765 <listitem><para>
766 <emphasis><filename>git commit</filename>:</emphasis>
767 Creates a local "commit" that documents the changes you
768 made.
769 Only changes that have been staged can be committed.
770 Commits are used for historical purposes, for determining
771 if a maintainer of a project will allow the change,
772 and for ultimately pushing the change from your local
773 Git repository into the project’s upstream repository.
774 </para></listitem>
775 <listitem><para>
776 <emphasis><filename>git status</filename>:</emphasis>
777 Reports any modified files that possibly need to be
778 staged and gives you a status of where you stand regarding
779 local commits as compared to the upstream repository.
780 </para></listitem>
781 <listitem><para>
782 <emphasis><filename>git checkout</filename> <replaceable>branch-name</replaceable>:</emphasis>
783 Changes your local working branch and in this form
784 assumes the local branch already exists.
785 This command is analogous to "cd".
786 </para></listitem>
787 <listitem><para>
788 <emphasis><filename>git checkout –b</filename> <replaceable>working-branch</replaceable> <replaceable>upstream-branch</replaceable>:</emphasis>
789 Creates and checks out a working branch on your local
790 machine.
791 The local branch tracks the upstream branch.
792 You can use your local branch to isolate your work.
793 It is a good idea to use local branches when adding
794 specific features or changes.
795 Using isolated branches facilitates easy removal of
796 changes if they do not work out.
797 </para></listitem>
798 <listitem><para><emphasis><filename>git branch</filename>:</emphasis>
799 Displays the existing local branches associated with your
800 local repository.
801 The branch that you have currently checked out is noted
802 with an asterisk character.
803 </para></listitem>
804 <listitem><para>
805 <emphasis><filename>git branch -D</filename> <replaceable>branch-name</replaceable>:</emphasis>
806 Deletes an existing local branch.
807 You need to be in a local branch other than the one you
808 are deleting in order to delete
809 <replaceable>branch-name</replaceable>.
810 </para></listitem>
811 <listitem><para>
812 <emphasis><filename>git pull --rebase</filename>:</emphasis>
813 Retrieves information from an upstream Git repository
814 and places it in your local Git repository.
815 You use this command to make sure you are synchronized with
816 the repository from which you are basing changes
817 (.e.g. the "master" branch).
818 The "--rebase" option ensures that any local commits you
819 have in your branch are preserved at the top of your
820 local branch.
821 </para></listitem>
822 <listitem><para>
823 <emphasis><filename>git push</filename> <replaceable>repo-name</replaceable> <replaceable>local-branch</replaceable><filename>:</filename><replaceable>upstream-branch</replaceable>:</emphasis>
824 Sends all your committed local changes to the upstream Git
825 repository that your local repository is tracking
826 (e.g. a contribution repository).
827 The maintainer of the project draws from these repositories
828 to merge changes (commits) into the appropriate branch
829 of project's upstream repository.
830 </para></listitem>
831 <listitem><para>
832 <emphasis><filename>git merge</filename>:</emphasis>
833 Combines or adds changes from one
834 local branch of your repository with another branch.
835 When you create a local Git repository, the default branch
836 is named "master".
837 A typical workflow is to create a temporary branch that is
838 based off "master" that you would use for isolated work.
839 You would make your changes in that isolated branch,
840 stage and commit them locally, switch to the "master"
841 branch, and then use the <filename>git merge</filename>
842 command to apply the changes from your isolated branch
843 into the currently checked out branch (e.g. "master").
844 After the merge is complete and if you are done with
845 working in that isolated branch, you can safely delete
846 the isolated branch.
847 </para></listitem>
848 <listitem><para>
849 <emphasis><filename>git cherry-pick</filename> <replaceable>commits</replaceable>:</emphasis>
850 Choose and apply specific commits from one branch
851 into another branch.
852 There are times when you might not be able to merge
853 all the changes in one branch with
854 another but need to pick out certain ones.
855 </para></listitem>
856 <listitem><para>
857 <emphasis><filename>gitk</filename>:</emphasis>
858 Provides a GUI view of the branches and changes in your
859 local Git repository.
860 This command is a good way to graphically see where things
861 have diverged in your local repository.
862 <note>
863 You need to install the <filename>gitk</filename>
864 package on your development system to use this
865 command.
866 </note>
867 </para></listitem>
868 <listitem><para>
869 <emphasis><filename>git log</filename>:</emphasis>
870 Reports a history of your commits to the repository.
871 This report lists all commits regardless of whether you
872 have pushed them upstream or not.
873 </para></listitem>
874 <listitem><para>
875 <emphasis><filename>git diff</filename>:</emphasis>
876 Displays line-by-line differences between a local
877 working file and the same file as understood by Git.
878 This command is useful to see what you have changed
879 in any given file.
880 </para></listitem>
881 </itemizedlist>
882 </para>
883 </section>
884</section>
885
886<section id='licensing'>
887 <title>Licensing</title>
888
889 <para>
890 Because open source projects are open to the public, they have
891 different licensing structures in place.
892 License evolution for both Open Source and Free Software has an
893 interesting history.
894 If you are interested in this history, you can find basic information
895 here:
896 <itemizedlist>
897 <listitem><para>
898 <ulink url='http://en.wikipedia.org/wiki/Open-source_license'>Open source license history</ulink>
899 </para></listitem>
900 <listitem><para>
901 <ulink url='http://en.wikipedia.org/wiki/Free_software_license'>Free software license history</ulink>
902 </para></listitem>
903 </itemizedlist>
904 </para>
905
906 <para>
907 In general, the Yocto Project is broadly licensed under the
908 Massachusetts Institute of Technology (MIT) License.
909 MIT licensing permits the reuse of software within proprietary
910 software as long as the license is distributed with that software.
911 MIT is also compatible with the GNU General Public License (GPL).
912 Patches to the Yocto Project follow the upstream licensing scheme.
913 You can find information on the MIT license
914 <ulink url='http://www.opensource.org/licenses/mit-license.php'>here</ulink>.
915 You can find information on the GNU GPL
916 <ulink url='http://www.opensource.org/licenses/LGPL-3.0'>here</ulink>.
917 </para>
918
919 <para>
920 When you build an image using the Yocto Project, the build process
921 uses a known list of licenses to ensure compliance.
922 You can find this list in the
923 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
924 at <filename>meta/files/common-licenses</filename>.
925 Once the build completes, the list of all licenses found and used
926 during that build are kept in the
927 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
928 at <filename>tmp/deploy/licenses</filename>.
929 </para>
930
931 <para>
932 If a module requires a license that is not in the base list, the
933 build process generates a warning during the build.
934 These tools make it easier for a developer to be certain of the
935 licenses with which their shipped products must comply.
936 However, even with these tools it is still up to the developer to
937 resolve potential licensing issues.
938 </para>
939
940 <para>
941 The base list of licenses used by the build process is a combination
942 of the Software Package Data Exchange (SPDX) list and the Open
943 Source Initiative (OSI) projects.
944 <ulink url='http://spdx.org'>SPDX Group</ulink> is a working group of
945 the Linux Foundation that maintains a specification for a standard
946 format for communicating the components, licenses, and copyrights
947 associated with a software package.
948 <ulink url='http://opensource.org'>OSI</ulink> is a corporation
949 dedicated to the Open Source Definition and the effort for reviewing
950 and approving licenses that conform to the Open Source Definition
951 (OSD).
952 </para>
953
954 <para>
955 You can find a list of the combined SPDX and OSI licenses that the
956 Yocto Project uses in the
957 <filename>meta/files/common-licenses</filename> directory in your
958 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
959 </para>
960
961 <para>
962 For information that can help you maintain compliance with various
963 open source licensing during the lifecycle of a product created using
964 the Yocto Project, see the
965 "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Product's Lifecycle</ulink>"
966 section in the Yocto Project Development Tasks Manual.
967 </para>
968</section>
969</chapter>
970<!--
971vim: expandtab tw=80 ts=4
972-->
diff --git a/documentation/overview-manual/overview-manual-eclipse-customization.xsl b/documentation/overview-manual/overview-manual-eclipse-customization.xsl
new file mode 100644
index 0000000000..aaf99ea1ba
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-eclipse-customization.xsl
@@ -0,0 +1,35 @@
1<?xml version='1.0'?>
2<xsl:stylesheet
3 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
4 xmlns="http://www.w3.org/1999/xhtml"
5 xmlns:fo="http://www.w3.org/1999/XSL/Format"
6 version="1.0">
7
8 <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/eclipse/eclipse3.xsl" />
9
10<!--
11
12 <xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/eclipse/eclipse3.xsl" />
13
14 <xsl:import
15 href="http://docbook.sourceforge.net/release/xsl/1.76.1/eclipse/eclipse3.xsl" />
16
17-->
18
19 <xsl:param name="chunker.output.indent" select="'yes'"/>
20 <xsl:param name="chunk.quietly" select="1"/>
21 <xsl:param name="chunk.first.sections" select="1"/>
22 <xsl:param name="chunk.section.depth" select="10"/>
23 <xsl:param name="use.id.as.filename" select="1"/>
24 <xsl:param name="ulink.target" select="'_self'" />
25 <xsl:param name="base.dir" select="'html/overview-manual/'"/>
26 <xsl:param name="html.stylesheet" select="'../book.css'"/>
27 <xsl:param name="eclipse.manifest" select="0"/>
28 <xsl:param name="create.plugin.xml" select="0"/>
29 <xsl:param name="suppress.navigation" select="1"/>
30 <xsl:param name="generate.index" select="0"/>
31 <xsl:param name="chapter.autolabel" select="1" />
32 <xsl:param name="appendix.autolabel" select="1" />
33 <xsl:param name="section.autolabel" select="1" />
34 <xsl:param name="section.label.includes.component.label" select="1" />
35</xsl:stylesheet>
diff --git a/documentation/overview-manual/overview-manual-intro.xml b/documentation/overview-manual/overview-manual-intro.xml
new file mode 100644
index 0000000000..2976f7c1b6
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-intro.xml
@@ -0,0 +1,111 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='overview-manual-intro'>
6
7<title>The Yocto Project Overview and Concepts Manual</title>
8 <section id='overview-manual-welcome'>
9 <title>Welcome</title>
10
11 <para>
12 Welcome to the Yocto Project Overview and Concepts Manual!
13 This manual introduces the Yocto Project by providing concepts,
14 software overviews, best-known-methods (BKMs), and any other
15 high-level introductory information suitable for a new Yocto
16 Project user.
17 </para>
18
19 <para>
20 The following list describes what you can get from this manual:
21 <itemizedlist>
22 <listitem><para>
23 <emphasis><link linkend='overview-yp'>Introducing the Yocto Project</link>:</emphasis>
24 This chapter provides an introduction to the Yocto
25 Project.
26 You will learn about features and challenges of the
27 Yocto Project, the layer model, components and tools,
28 development methods, the Poky reference distribution,
29 the OpenEmbedded build system workflow, and some basic
30 Yocto terms.
31 </para></listitem>
32 <listitem><para>
33 <emphasis><link linkend='overview-development-environment'>The Yocto Project Development Environment</link>:</emphasis>
34 This chapter helps you get started understanding the
35 Yocto Project development environment.
36 You will learn about open source, development hosts,
37 Yocto Project source repositories, workflows using Git
38 and the Yocto Project, a Git primer, and information
39 about licensing.
40 </para></listitem>
41 </itemizedlist>
42 </para>
43
44 <para>
45 This manual does not give you the following:
46 <itemizedlist>
47 <listitem><para>
48 <emphasis>Step-by-step Instructions for Development Tasks:</emphasis>
49 Instructional procedures reside in other manuals within
50 the Yocto Project documentation set.
51 For example, the
52 <ulink url='&YOCTO_DOCS_DEV_URL;'>Yocto Project Development Tasks Manual</ulink>
53 provides examples on how to perform various development
54 tasks.
55 As another example, the
56 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
57 manual contains detailed instructions on how to install an
58 SDK, which is used to develop applications for target
59 hardware.
60 </para></listitem>
61 <listitem><para>
62 <emphasis>Reference Material:</emphasis>
63 This type of material resides in an appropriate reference
64 manual.
65 For example, system variables are documented in the
66 <ulink url='&YOCTO_DOCS_REF_URL;'>Yocto Project Reference Manual</ulink>.
67 As another example, the
68 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>
69 contains reference information on BSPs.
70 </para></listitem>
71 <listitem><para>
72 <emphasis>Detailed Public Information Not Specific to the
73 Yocto Project:</emphasis>
74 For example, exhaustive information on how to use the
75 Source Control Manager Git is better covered with Internet
76 searches and official Git Documentation than through the
77 Yocto Project documentation.
78 </para></listitem>
79 </itemizedlist>
80 </para>
81 </section>
82
83 <section id='overview-manual-other-information'>
84 <title>Other Information</title>
85
86 <para>
87 Because this manual presents information for many different
88 topics, supplemental information is recommended for full
89 comprehension.
90 For additional introductory information on the Yocto Project, see
91 the <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>.
92 If you want to build an image with no knowledge of Yocto Project
93 as a way of quickly testing it out, see the
94 <ulink url='&YOCTO_DOCS_BRIEF_URL;'>Yocto Project Quick Build</ulink>
95 document.
96 </para>
97
98 <para>
99 For a comprehensive list of links and other documentation, see the
100 "<ulink url='&YOCTO_DOCS_REF_URL;#resources-links-and-related-documentation'>Links and Related Documentation</ulink>"
101 section in the Yocto Project Reference Manual.
102 For a paper showing how to set up and run a quick build using the
103 Yocto Project, see the
104 "<ulink url='&YOCTO_DOCS_BRIEF_URL;'>My First Yocto Project Build</ulink>"
105 paper.
106 </para>
107 </section>
108</chapter>
109<!--
110vim: expandtab tw=80 ts=4
111-->
diff --git a/documentation/overview-manual/overview-manual-style.css b/documentation/overview-manual/overview-manual-style.css
new file mode 100644
index 0000000000..97a364b125
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-style.css
@@ -0,0 +1,988 @@
1/*
2 Generic XHTML / DocBook XHTML CSS Stylesheet.
3
4 Browser wrangling and typographic design by
5 Oyvind Kolas / pippin@gimp.org
6
7 Customised for Poky by
8 Matthew Allum / mallum@o-hand.com
9
10 Thanks to:
11 Liam R. E. Quin
12 William Skaggs
13 Jakub Steiner
14
15 Structure
16 ---------
17
18 The stylesheet is divided into the following sections:
19
20 Positioning
21 Margins, paddings, width, font-size, clearing.
22 Decorations
23 Borders, style
24 Colors
25 Colors
26 Graphics
27 Graphical backgrounds
28 Nasty IE tweaks
29 Workarounds needed to make it work in internet explorer,
30 currently makes the stylesheet non validating, but up until
31 this point it is validating.
32 Mozilla extensions
33 Transparency for footer
34 Rounded corners on boxes
35
36*/
37
38
39 /*************** /
40 / Positioning /
41/ ***************/
42
43body {
44 font-family: Verdana, Sans, sans-serif;
45
46 min-width: 640px;
47 width: 80%;
48 margin: 0em auto;
49 padding: 2em 5em 5em 5em;
50 color: #333;
51}
52
53h1,h2,h3,h4,h5,h6,h7 {
54 font-family: Arial, Sans;
55 color: #00557D;
56 clear: both;
57}
58
59h1 {
60 font-size: 2em;
61 text-align: left;
62 padding: 0em 0em 0em 0em;
63 margin: 2em 0em 0em 0em;
64}
65
66h2.subtitle {
67 margin: 0.10em 0em 3.0em 0em;
68 padding: 0em 0em 0em 0em;
69 font-size: 1.8em;
70 padding-left: 20%;
71 font-weight: normal;
72 font-style: italic;
73}
74
75h2 {
76 margin: 2em 0em 0.66em 0em;
77 padding: 0.5em 0em 0em 0em;
78 font-size: 1.5em;
79 font-weight: bold;
80}
81
82h3.subtitle {
83 margin: 0em 0em 1em 0em;
84 padding: 0em 0em 0em 0em;
85 font-size: 142.14%;
86 text-align: right;
87}
88
89h3 {
90 margin: 1em 0em 0.5em 0em;
91 padding: 1em 0em 0em 0em;
92 font-size: 140%;
93 font-weight: bold;
94}
95
96h4 {
97 margin: 1em 0em 0.5em 0em;
98 padding: 1em 0em 0em 0em;
99 font-size: 120%;
100 font-weight: bold;
101}
102
103h5 {
104 margin: 1em 0em 0.5em 0em;
105 padding: 1em 0em 0em 0em;
106 font-size: 110%;
107 font-weight: bold;
108}
109
110h6 {
111 margin: 1em 0em 0em 0em;
112 padding: 1em 0em 0em 0em;
113 font-size: 110%;
114 font-weight: bold;
115}
116
117.authorgroup {
118 background-color: transparent;
119 background-repeat: no-repeat;
120 padding-top: 256px;
121 background-image: url("figures/overview-manual-title.png");
122 background-position: left top;
123 margin-top: -256px;
124 padding-right: 50px;
125 margin-left: 0px;
126 text-align: right;
127 width: 740px;
128}
129
130h3.author {
131 margin: 0em 0me 0em 0em;
132 padding: 0em 0em 0em 0em;
133 font-weight: normal;
134 font-size: 100%;
135 color: #333;
136 clear: both;
137}
138
139.author tt.email {
140 font-size: 66%;
141}
142
143.titlepage hr {
144 width: 0em;
145 clear: both;
146}
147
148.revhistory {
149 padding-top: 2em;
150 clear: both;
151}
152
153.toc,
154.list-of-tables,
155.list-of-examples,
156.list-of-figures {
157 padding: 1.33em 0em 2.5em 0em;
158 color: #00557D;
159}
160
161.toc p,
162.list-of-tables p,
163.list-of-figures p,
164.list-of-examples p {
165 padding: 0em 0em 0em 0em;
166 padding: 0em 0em 0.3em;
167 margin: 1.5em 0em 0em 0em;
168}
169
170.toc p b,
171.list-of-tables p b,
172.list-of-figures p b,
173.list-of-examples p b{
174 font-size: 100.0%;
175 font-weight: bold;
176}
177
178.toc dl,
179.list-of-tables dl,
180.list-of-figures dl,
181.list-of-examples dl {
182 margin: 0em 0em 0.5em 0em;
183 padding: 0em 0em 0em 0em;
184}
185
186.toc dt {
187 margin: 0em 0em 0em 0em;
188 padding: 0em 0em 0em 0em;
189}
190
191.toc dd {
192 margin: 0em 0em 0em 2.6em;
193 padding: 0em 0em 0em 0em;
194}
195
196div.glossary dl,
197div.variablelist dl {
198}
199
200.glossary dl dt,
201.variablelist dl dt,
202.variablelist dl dt span.term {
203 font-weight: normal;
204 width: 20em;
205 text-align: right;
206}
207
208.variablelist dl dt {
209 margin-top: 0.5em;
210}
211
212.glossary dl dd,
213.variablelist dl dd {
214 margin-top: -1em;
215 margin-left: 25.5em;
216}
217
218.glossary dd p,
219.variablelist dd p {
220 margin-top: 0em;
221 margin-bottom: 1em;
222}
223
224
225div.calloutlist table td {
226 padding: 0em 0em 0em 0em;
227 margin: 0em 0em 0em 0em;
228}
229
230div.calloutlist table td p {
231 margin-top: 0em;
232 margin-bottom: 1em;
233}
234
235div p.copyright {
236 text-align: left;
237}
238
239div.legalnotice p.legalnotice-title {
240 margin-bottom: 0em;
241}
242
243p {
244 line-height: 1.5em;
245 margin-top: 0em;
246
247}
248
249dl {
250 padding-top: 0em;
251}
252
253hr {
254 border: solid 1px;
255}
256
257
258.mediaobject,
259.mediaobjectco {
260 text-align: center;
261}
262
263img {
264 border: none;
265}
266
267ul {
268 padding: 0em 0em 0em 1.5em;
269}
270
271ul li {
272 padding: 0em 0em 0em 0em;
273}
274
275ul li p {
276 text-align: left;
277}
278
279table {
280 width :100%;
281}
282
283th {
284 padding: 0.25em;
285 text-align: left;
286 font-weight: normal;
287 vertical-align: top;
288}
289
290td {
291 padding: 0.25em;
292 vertical-align: top;
293}
294
295p a[id] {
296 margin: 0px;
297 padding: 0px;
298 display: inline;
299 background-image: none;
300}
301
302a {
303 text-decoration: underline;
304 color: #444;
305}
306
307pre {
308 overflow: auto;
309}
310
311a:hover {
312 text-decoration: underline;
313 /*font-weight: bold;*/
314}
315
316/* This style defines how the permalink character
317 appears by itself and when hovered over with
318 the mouse. */
319
320[alt='Permalink'] { color: #eee; }
321[alt='Permalink']:hover { color: black; }
322
323
324div.informalfigure,
325div.informalexample,
326div.informaltable,
327div.figure,
328div.table,
329div.example {
330 margin: 1em 0em;
331 padding: 1em;
332 page-break-inside: avoid;
333}
334
335
336div.informalfigure p.title b,
337div.informalexample p.title b,
338div.informaltable p.title b,
339div.figure p.title b,
340div.example p.title b,
341div.table p.title b{
342 padding-top: 0em;
343 margin-top: 0em;
344 font-size: 100%;
345 font-weight: normal;
346}
347
348.mediaobject .caption,
349.mediaobject .caption p {
350 text-align: center;
351 font-size: 80%;
352 padding-top: 0.5em;
353 padding-bottom: 0.5em;
354}
355
356.epigraph {
357 padding-left: 55%;
358 margin-bottom: 1em;
359}
360
361.epigraph p {
362 text-align: left;
363}
364
365.epigraph .quote {
366 font-style: italic;
367}
368.epigraph .attribution {
369 font-style: normal;
370 text-align: right;
371}
372
373span.application {
374 font-style: italic;
375}
376
377.programlisting {
378 font-family: monospace;
379 font-size: 80%;
380 white-space: pre;
381 margin: 1.33em 0em;
382 padding: 1.33em;
383}
384
385.tip,
386.warning,
387.caution,
388.note {
389 margin-top: 1em;
390 margin-bottom: 1em;
391
392}
393
394/* force full width of table within div */
395.tip table,
396.warning table,
397.caution table,
398.note table {
399 border: none;
400 width: 100%;
401}
402
403
404.tip table th,
405.warning table th,
406.caution table th,
407.note table th {
408 padding: 0.8em 0.0em 0.0em 0.0em;
409 margin : 0em 0em 0em 0em;
410}
411
412.tip p,
413.warning p,
414.caution p,
415.note p {
416 margin-top: 0.5em;
417 margin-bottom: 0.5em;
418 padding-right: 1em;
419 text-align: left;
420}
421
422.acronym {
423 text-transform: uppercase;
424}
425
426b.keycap,
427.keycap {
428 padding: 0.09em 0.3em;
429 margin: 0em;
430}
431
432.itemizedlist li {
433 clear: none;
434}
435
436.filename {
437 font-size: medium;
438 font-family: Courier, monospace;
439}
440
441
442div.navheader, div.heading{
443 position: absolute;
444 left: 0em;
445 top: 0em;
446 width: 100%;
447 background-color: #cdf;
448 width: 100%;
449}
450
451div.navfooter, div.footing{
452 position: fixed;
453 left: 0em;
454 bottom: 0em;
455 background-color: #eee;
456 width: 100%;
457}
458
459
460div.navheader td,
461div.navfooter td {
462 font-size: 66%;
463}
464
465div.navheader table th {
466 /*font-family: Georgia, Times, serif;*/
467 /*font-size: x-large;*/
468 font-size: 80%;
469}
470
471div.navheader table {
472 border-left: 0em;
473 border-right: 0em;
474 border-top: 0em;
475 width: 100%;
476}
477
478div.navfooter table {
479 border-left: 0em;
480 border-right: 0em;
481 border-bottom: 0em;
482 width: 100%;
483}
484
485div.navheader table td a,
486div.navfooter table td a {
487 color: #777;
488 text-decoration: none;
489}
490
491/* normal text in the footer */
492div.navfooter table td {
493 color: black;
494}
495
496div.navheader table td a:visited,
497div.navfooter table td a:visited {
498 color: #444;
499}
500
501
502/* links in header and footer */
503div.navheader table td a:hover,
504div.navfooter table td a:hover {
505 text-decoration: underline;
506 background-color: transparent;
507 color: #33a;
508}
509
510div.navheader hr,
511div.navfooter hr {
512 display: none;
513}
514
515
516.qandaset tr.question td p {
517 margin: 0em 0em 1em 0em;
518 padding: 0em 0em 0em 0em;
519}
520
521.qandaset tr.answer td p {
522 margin: 0em 0em 1em 0em;
523 padding: 0em 0em 0em 0em;
524}
525.answer td {
526 padding-bottom: 1.5em;
527}
528
529.emphasis {
530 font-weight: bold;
531}
532
533
534 /************* /
535 / decorations /
536/ *************/
537
538.titlepage {
539}
540
541.part .title {
542}
543
544.subtitle {
545 border: none;
546}
547
548/*
549h1 {
550 border: none;
551}
552
553h2 {
554 border-top: solid 0.2em;
555 border-bottom: solid 0.06em;
556}
557
558h3 {
559 border-top: 0em;
560 border-bottom: solid 0.06em;
561}
562
563h4 {
564 border: 0em;
565 border-bottom: solid 0.06em;
566}
567
568h5 {
569 border: 0em;
570}
571*/
572
573.programlisting {
574 border: solid 1px;
575}
576
577div.figure,
578div.table,
579div.informalfigure,
580div.informaltable,
581div.informalexample,
582div.example {
583 border: 1px solid;
584}
585
586
587
588.tip,
589.warning,
590.caution,
591.note {
592 border: 1px solid;
593}
594
595.tip table th,
596.warning table th,
597.caution table th,
598.note table th {
599 border-bottom: 1px solid;
600}
601
602.question td {
603 border-top: 1px solid black;
604}
605
606.answer {
607}
608
609
610b.keycap,
611.keycap {
612 border: 1px solid;
613}
614
615
616div.navheader, div.heading{
617 border-bottom: 1px solid;
618}
619
620
621div.navfooter, div.footing{
622 border-top: 1px solid;
623}
624
625 /********* /
626 / colors /
627/ *********/
628
629body {
630 color: #333;
631 background: white;
632}
633
634a {
635 background: transparent;
636}
637
638a:hover {
639 background-color: #dedede;
640}
641
642
643h1,
644h2,
645h3,
646h4,
647h5,
648h6,
649h7,
650h8 {
651 background-color: transparent;
652}
653
654hr {
655 border-color: #aaa;
656}
657
658
659.tip, .warning, .caution, .note {
660 border-color: #fff;
661}
662
663
664.tip table th,
665.warning table th,
666.caution table th,
667.note table th {
668 border-bottom-color: #fff;
669}
670
671
672.warning {
673 background-color: #f0f0f2;
674}
675
676.caution {
677 background-color: #f0f0f2;
678}
679
680.tip {
681 background-color: #f0f0f2;
682}
683
684.note {
685 background-color: #f0f0f2;
686}
687
688.glossary dl dt,
689.variablelist dl dt,
690.variablelist dl dt span.term {
691 color: #044;
692}
693
694div.figure,
695div.table,
696div.example,
697div.informalfigure,
698div.informaltable,
699div.informalexample {
700 border-color: #aaa;
701}
702
703pre.programlisting {
704 color: black;
705 background-color: #fff;
706 border-color: #aaa;
707 border-width: 2px;
708}
709
710.guimenu,
711.guilabel,
712.guimenuitem {
713 background-color: #eee;
714}
715
716
717b.keycap,
718.keycap {
719 background-color: #eee;
720 border-color: #999;
721}
722
723
724div.navheader {
725 border-color: black;
726}
727
728
729div.navfooter {
730 border-color: black;
731}
732
733.writernotes {
734 color: red;
735}
736
737
738 /*********** /
739 / graphics /
740/ ***********/
741
742/*
743body {
744 background-image: url("images/body_bg.jpg");
745 background-attachment: fixed;
746}
747
748.navheader,
749.note,
750.tip {
751 background-image: url("images/note_bg.jpg");
752 background-attachment: fixed;
753}
754
755.warning,
756.caution {
757 background-image: url("images/warning_bg.jpg");
758 background-attachment: fixed;
759}
760
761.figure,
762.informalfigure,
763.example,
764.informalexample,
765.table,
766.informaltable {
767 background-image: url("images/figure_bg.jpg");
768 background-attachment: fixed;
769}
770
771*/
772h1,
773h2,
774h3,
775h4,
776h5,
777h6,
778h7{
779}
780
781/*
782Example of how to stick an image as part of the title.
783
784div.article .titlepage .title
785{
786 background-image: url("figures/white-on-black.png");
787 background-position: center;
788 background-repeat: repeat-x;
789}
790*/
791
792div.preface .titlepage .title,
793div.colophon .title,
794div.chapter .titlepage .title,
795div.article .titlepage .title
796{
797}
798
799div.section div.section .titlepage .title,
800div.sect2 .titlepage .title {
801 background: none;
802}
803
804
805h1.title {
806 background-color: transparent;
807 background-repeat: no-repeat;
808 height: 256px;
809 text-indent: -9000px;
810 overflow:hidden;
811}
812
813h2.subtitle {
814 background-color: transparent;
815 text-indent: -9000px;
816 overflow:hidden;
817 width: 0px;
818 display: none;
819}
820
821 /*************************************** /
822 / pippin.gimp.org specific alterations /
823/ ***************************************/
824
825/*
826div.heading, div.navheader {
827 color: #777;
828 font-size: 80%;
829 padding: 0;
830 margin: 0;
831 text-align: left;
832 position: absolute;
833 top: 0px;
834 left: 0px;
835 width: 100%;
836 height: 50px;
837 background: url('/gfx/heading_bg.png') transparent;
838 background-repeat: repeat-x;
839 background-attachment: fixed;
840 border: none;
841}
842
843div.heading a {
844 color: #444;
845}
846
847div.footing, div.navfooter {
848 border: none;
849 color: #ddd;
850 font-size: 80%;
851 text-align:right;
852
853 width: 100%;
854 padding-top: 10px;
855 position: absolute;
856 bottom: 0px;
857 left: 0px;
858
859 background: url('/gfx/footing_bg.png') transparent;
860}
861*/
862
863
864
865 /****************** /
866 / nasty ie tweaks /
867/ ******************/
868
869/*
870div.heading, div.navheader {
871 width:expression(document.body.clientWidth + "px");
872}
873
874div.footing, div.navfooter {
875 width:expression(document.body.clientWidth + "px");
876 margin-left:expression("-5em");
877}
878body {
879 padding:expression("4em 5em 0em 5em");
880}
881*/
882
883 /**************************************** /
884 / mozilla vendor specific css extensions /
885/ ****************************************/
886/*
887div.navfooter, div.footing{
888 -moz-opacity: 0.8em;
889}
890
891div.figure,
892div.table,
893div.informalfigure,
894div.informaltable,
895div.informalexample,
896div.example,
897.tip,
898.warning,
899.caution,
900.note {
901 -moz-border-radius: 0.5em;
902}
903
904b.keycap,
905.keycap {
906 -moz-border-radius: 0.3em;
907}
908*/
909
910table tr td table tr td {
911 display: none;
912}
913
914
915hr {
916 display: none;
917}
918
919table {
920 border: 0em;
921}
922
923 .photo {
924 float: right;
925 margin-left: 1.5em;
926 margin-bottom: 1.5em;
927 margin-top: 0em;
928 max-width: 17em;
929 border: 1px solid gray;
930 padding: 3px;
931 background: white;
932}
933 .seperator {
934 padding-top: 2em;
935 clear: both;
936 }
937
938 #validators {
939 margin-top: 5em;
940 text-align: right;
941 color: #777;
942 }
943 @media print {
944 body {
945 font-size: 8pt;
946 }
947 .noprint {
948 display: none;
949 }
950 }
951
952
953.tip,
954.note {
955 background: #f0f0f2;
956 color: #333;
957 padding: 20px;
958 margin: 20px;
959}
960
961.tip h3,
962.note h3 {
963 padding: 0em;
964 margin: 0em;
965 font-size: 2em;
966 font-weight: bold;
967 color: #333;
968}
969
970.tip a,
971.note a {
972 color: #333;
973 text-decoration: underline;
974}
975
976.footnote {
977 font-size: small;
978 color: #333;
979}
980
981/* Changes the announcement text */
982.tip h3,
983.warning h3,
984.caution h3,
985.note h3 {
986 font-size:large;
987 color: #00557D;
988}
diff --git a/documentation/overview-manual/overview-manual-yp-intro.xml b/documentation/overview-manual/overview-manual-yp-intro.xml
new file mode 100644
index 0000000000..437635842a
--- /dev/null
+++ b/documentation/overview-manual/overview-manual-yp-intro.xml
@@ -0,0 +1,1335 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='overview-yp'>
6 <title>Introducing the Yocto Project</title>
7
8 <section id='what-is-the-yocto-project'>
9 <title>What is the Yocto Project?</title>
10
11 <para>
12 The Yocto Project is an open source collaboration project
13 that helps developers create custom Linux-based systems that are
14 designed for embedded products regardless of the product's hardware
15 architecture.
16 Yocto Project provides a flexible toolset and a development
17 environment that allows embedded device developers across the
18 world to collaborate through shared technologies, software stacks,
19 configurations, and best practices used to create these tailored
20 Linux images.
21 </para>
22
23 <para>
24 Thousands of developers worldwide have discovered that Yocto
25 Project provides advantages in both systems and applications
26 development, archival and management benefits, and customizations
27 used for speed, footprint, and memory utilization.
28 The project is a standard when it comes to delivering hardware
29 support and software stacks, allowing software configuration
30 and build interchange, and build and support customizations for
31 multiple hardware platforms and software stacks that can be
32 maintained and scaled.
33 </para>
34
35 <para id='yp-key-dev-elements'>
36 <imagedata fileref="figures/key-dev-elements.png" format="PNG" align='center' width="8in"/>
37 </para>
38
39 <para>
40 For further introductory information on the Yocto Project, you
41 might be interested in this
42 <ulink url='https://www.embedded.com/electronics-blogs/say-what-/4458600/Why-the-Yocto-Project-for-my-IoT-Project-'>article</ulink>
43 by Drew Moseley and in this short introductory
44 <ulink url='https://www.youtube.com/watch?v=utZpKM7i5Z4'>video</ulink>.
45 </para>
46
47 <para>
48 The remainder of this section overviews advantages and challenges
49 tied to the Yocto Project.
50 </para>
51
52 <section id='gs-features'>
53 <title>Features</title>
54
55 <para>
56 The following list describes features and advantages of the
57 Yocto Project:
58 <itemizedlist>
59 <listitem><para>
60 <emphasis>Widely Adopted Across the Industry:</emphasis>
61 Semiconductor, operating system, software, and
62 service vendors exist whose products and services
63 adopt and support the Yocto Project.
64 For a look at the companies involved with the Yocto
65 Project, see the membership, associate, and
66 participant pages on the Yocto Project home page.
67 </para></listitem>
68 <listitem><para>
69 <emphasis>Architecture Agnostic:</emphasis>
70 Yocto Project supports Intel, ARM, MIPS, AMD, PPC
71 and other architectures.
72 Most ODMs, OSVs, and chip vendors create and supply
73 BSPs that support their hardware.
74 If you have custom silicon, you can create a BSP
75 that supports that architecture.</para>
76
77 <para>Aside from lots of architecture support, the
78 Yocto Project fully supports a wide range of device
79 emulation through the Quick EMUlator (QEMU).
80 </para></listitem>
81 <listitem><para>
82 <emphasis>Images and Code Transfer Easily:</emphasis>
83 Yocto Project output can easily move between
84 architectures without moving to new development
85 environments.
86 Additionally, if you have used the Yocto Project to
87 create an image or application and you find yourself
88 not able to support it, commercial Linux vendors such
89 as Wind River, Mentor Graphics, Timesys, and ENEA could
90 take it and provide ongoing support.
91 These vendors have offerings that are built using
92 the Yocto Project.
93 </para></listitem>
94 <listitem><para>
95 <emphasis>Flexibility:</emphasis>
96 Corporations use the Yocto Project many different ways.
97 One example is to create an internal Linux distribution
98 as a code base the corporation can use across multiple
99 product groups.
100 Through customization and layering, a project group
101 can leverage the base Linux distribution to create
102 a distribution that works for their product needs.
103 </para></listitem>
104 <listitem><para>
105 <emphasis>Ideal for Constrained Embedded and IoT devices:</emphasis>
106 Unlike a full Linux distribution, you can use the
107 Yocto Project to create exactly what you need for
108 embedded devices.
109 You only add the feature support or packages that you
110 absolutely need for the device.
111 For devices that have display hardware, you can use
112 available system components such as X11, GTK+, Qt,
113 Clutter, and SDL (among others) to create a rich user
114 experience.
115 For devices that do not have a display or where you
116 want to use alternative UI frameworks, you can choose
117 to not install these components.
118 </para></listitem>
119 <listitem><para>
120 <emphasis>Comprehensive Toolchain Capabilities:</emphasis>
121 Toolchains for supported architectures satisfy most
122 use cases.
123 However, if your hardware supports features that are
124 not part of a standard toolchain, you can easily
125 customize that toolchain through specification of
126 platform-specific tuning parameters.
127 And, should you need to use a third-party toolchain,
128 mechanisms built into the Yocto Project allow for that.
129 </para></listitem>
130 <listitem><para>
131 <emphasis>Mechanism Rules Over Policy:</emphasis>
132 Focusing on mechanism rather than policy ensures that
133 you are free to set policies based on the needs of your
134 design instead of adopting decisions enforced by some
135 system software provider.
136 </para></listitem>
137 <listitem><para>
138 <emphasis>Uses a Layer Model:</emphasis>
139 The Yocto Project layer infrastructure groups related
140 functionality into separate bundles.
141 You can incrementally add these grouped functionalities
142 to your project as needed.
143 Using layers to isolate and group functionality
144 reduces project complexity and redundancy, allows you
145 to easily extend the system, make customizations,
146 and keep functionality organized.
147 </para></listitem>
148 <listitem><para>
149 <emphasis>Supports Partial Builds:</emphasis>
150 You can build and rebuild individual packages as
151 needed.
152 Yocto Project accomplishes this through its
153 shared-state cache (sstate) scheme.
154 Being able to build and debug components individually
155 eases project development.
156 </para></listitem>
157 <listitem><para>
158 <emphasis>Releases According to a Strict Schedule:</emphasis>
159 Major releases occur on a six-month cycle predictably
160 in October and April.
161 The most recent two releases support point releases
162 to address common vulnerabilities and exposures.
163 This predictability is crucial for projects based on
164 the Yocto Project and allows development teams to
165 plan activities.
166 </para></listitem>
167 <listitem><para>
168 <emphasis>Rich Ecosystem of Individuals and Organizations:</emphasis>
169 For open source projects, the value of community is
170 very important.
171 Support forums, expertise, and active developers who
172 continue to push the Yocto Project forward are readily
173 available.
174 </para></listitem>
175 <listitem><para>
176 <emphasis>Binary Reproducibility:</emphasis>
177 The Yocto Project allows you to be very specific about
178 dependencies and achieves very high percentages of
179 binary reproducibility (e.g. 99.8% for
180 <filename>core-image-minimal</filename>).
181 When distributions are not specific about which
182 packages are pulled in and in what order to support
183 dependencies, other build systems can arbitrarily
184 include packages.
185 </para></listitem>
186 <listitem><para>
187 <emphasis>License Manifest:</emphasis>
188 The Yocto Project provides a license manifest for
189 review by people who need to track the use of open
190 source licenses (e.g.legal teams).
191 </para></listitem>
192 </itemizedlist>
193 </para>
194 </section>
195
196 <section id='gs-challenges'>
197 <title>Challenges</title>
198
199 <para>
200 The following list presents challenges you might encounter
201 when developing using the Yocto Project:
202 <itemizedlist>
203 <listitem><para>
204 <emphasis>Steep Learning Curve:</emphasis>
205 The Yocto Project has a steep learning curve and has
206 many different ways to accomplish similar tasks.
207 It can be difficult to choose how to proceed when
208 varying methods exist by which to accomplish a given
209 task.
210 </para></listitem>
211 <listitem><para>
212 <emphasis>Understanding What Changes You Need to Make
213 For Your Design Requires Some Research:</emphasis>
214 Beyond the simple tutorial stage, understanding what
215 changes need to be made for your particular design
216 can require a significant amount of research and
217 investigation.
218 For information that helps you transition from
219 trying out the Yocto Project to using it for your
220 project, see the "What I wish I'd Known" and
221 "Transitioning to a Custom Environment for Systems
222 Development" documents on the Yocto Project website.
223 </para></listitem>
224 <listitem><para>
225 <emphasis>Project Workflow Could Be Confusing:</emphasis>
226 The Yocto Project workflow could be confusing if you
227 are used to traditional desktop and server software
228 development.
229 In a desktop development environment, mechanisms exist
230 to easily pull and install new packages, which are
231 typically pre-compiled binaries from servers accessible
232 over the Internet.
233 Using the Yocto Project, you must modify your
234 configuration and rebuild to add additional packages.
235 </para></listitem>
236 <listitem><para>
237 <emphasis>Working in a Cross-Build Environment Can
238 Feel Unfamiliar:</emphasis>
239 When developing code to run on a target, compilation,
240 execution, and testing done on the actual target
241 can be faster than running a BitBake build on a
242 development host and then deploying binaries to the
243 target for test.
244 While the Yocto Project does support development tools
245 on the target, the additional step of integrating your
246 changes back into the Yocto Project build environment
247 would be required.
248 Yocto Project supports an intermediate approach that
249 involves making changes on the development system
250 within the BitBake environment and then deploying only
251 the updated packages to the target.</para>
252
253 <para>The Yocto Project OpenEmbedded build system
254 produces packages in standard formats (i.e. RPM,
255 DEB, IPK, and TAR).
256 You can deploy these packages into the running system
257 on the target by using utilities on the target such
258 as <filename>rpm</filename> or
259 <filename>ipk</filename>.
260 </para></listitem>
261 <listitem><para>
262 <emphasis>Initial Build Times Can be Significant:</emphasis>
263 Long initial build times are unfortunately unavoidable
264 due to the large number of packages initially built
265 from scratch for a fully functioning Linux system.
266 Once that initial build is completed, however, the
267 shared-state (sstate) cache mechanism Yocto Project
268 uses keeps the system from rebuilding packages that
269 have not been "touched" since the last build.
270 The sstate mechanism significantly reduces times
271 for successive builds.
272 </para></listitem>
273 </itemizedlist>
274 </para>
275 </section>
276 </section>
277
278 <section id='the-yocto-project-layer-model'>
279 <title>The Yocto Project Layer Model</title>
280
281 <para>
282 The Yocto Project's "Layer Model" is a development model for
283 embedded and IoT Linux creation that distinguishes the
284 Yocto Project from other simple build systems.
285 The Layer Model simultaneously supports collaboration and
286 customization.
287 Layers are repositories that contain related sets of instructions
288 that tell the OpenEmbedded build system what to do.
289 You can collaborate, share, and reuse layers.
290 </para>
291
292 <para>
293 Layers can contain changes to previous instructions or settings
294 at any time.
295 This powerful override capability is what allows you to customize
296 previously supplied collaborative or community layers to suit your
297 product requirements.
298 </para>
299
300 <para>
301 You use different layers to logically separate information in your
302 build.
303 As an example, you could have BSP, GUI, distro configuration,
304 middleware, or application layers.
305 Putting your entire build into one layer limits and complicates
306 future customization and reuse.
307 Isolating information into layers, on the other hand, helps
308 simplify future customizations and reuse.
309 You might find it tempting to keep everything in one layer when
310 working on a single project.
311 However, the more modular your Metadata, the easier
312 it is to cope with future changes.
313 <note><title>Notes</title>
314 <itemizedlist>
315 <listitem><para>
316 Use Board Support Package (BSP) layers from silicon
317 vendors when possible.
318 </para></listitem>
319 <listitem><para>
320 Familiarize yourself with the
321 <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project curated layer index</ulink>
322 or the
323 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded layer index</ulink>.
324 The latter contains more layers but they are less
325 universally validated.
326 </para></listitem>
327 <listitem><para>
328 Layers support the inclusion of technologies, hardware
329 components, and software components.
330 The Yocto Project Compatible designation provides a
331 minimum level of standardization that contributes to a
332 strong ecosystem.
333 "YP Compatible" is applied to appropriate products and
334 software components such as BSPs, other OE-compatible
335 layers, and related open-source projects, allowing the
336 producer to use Yocto Project badges and branding
337 assets.
338 </para></listitem>
339 </itemizedlist>
340 </note>
341 </para>
342
343 <para>
344 To illustrate how layers are used to keep things modular, consider
345 machine customizations.
346 These types of customizations typically reside in a special layer,
347 rather than a general layer, called a BSP Layer.
348 Furthermore, the machine customizations should be isolated from
349 recipes and Metadata that support a new GUI environment,
350 for example.
351 This situation gives you a couple of layers: one for the machine
352 configurations, and one for the GUI environment.
353 It is important to understand, however, that the BSP layer can
354 still make machine-specific additions to recipes within the GUI
355 environment layer without polluting the GUI layer itself
356 with those machine-specific changes.
357 You can accomplish this through a recipe that is a BitBake append
358 (<filename>.bbappend</filename>) file, which is described later
359 in this section.
360 <note>
361 For general information on BSP layer structure, see the
362 <ulink url='&YOCTO_DOCS_BSP_URL;#bsp'>Board Support Packages (BSP) - Developer's Guide</ulink>.
363 </note>
364 </para>
365
366 <para>
367 The
368 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
369 contains both general layers and BSP layers right out of the box.
370 You can easily identify layers that ship with a Yocto Project
371 release in the Source Directory by their names.
372 Layers typically have names that begin with the string
373 <filename>meta-</filename>.
374 <note>
375 It is not a requirement that a layer name begin with the
376 prefix <filename>meta-</filename>, but it is a commonly
377 accepted standard in the Yocto Project community.
378 </note>
379 For example, if you were to examine the
380 <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/'>tree view</ulink>
381 of the <filename>poky</filename> repository, you will see several
382 layers: <filename>meta</filename>,
383 <filename>meta-skeleton</filename>,
384 <filename>meta-selftest</filename>,
385 <filename>meta-poky</filename>, and
386 <filename>meta-yocto-bsp</filename>.
387 Each of these repositories represents a distinct layer.
388 </para>
389
390 <para>
391 For procedures on how to create layers, see the
392 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
393 section in the Yocto Project Development Tasks Manual.
394 </para>
395 </section>
396
397 <section id='components-and-tools'>
398 <title>Components and Tools</title>
399
400 <para>
401 The Yocto Project employs a collection of components and
402 tools used by the project itself, by project developers,
403 and by those using the Yocto Project.
404 These components and tools are open source projects and
405 metadata that are separate from the reference distribution
406 (Poky) and the OpenEmbedded build system.
407 Most of the components and tools are downloaded separately.
408 </para>
409
410 <para>
411 This section provides brief overviews of the components and
412 tools associated with the Yocto Project.
413 </para>
414
415 <section id='gs-development-tools'>
416 <title>Development Tools</title>
417
418 <para>
419 The following list consists of tools that help you develop
420 images and applications using the Yocto Project:
421 <itemizedlist>
422 <listitem><para id='gs-crops-overview'>
423 <emphasis>CROPS:</emphasis>
424 <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/crops/about/'>CROPS</ulink>
425 is an open source, cross-platform development framework
426 that leverages
427 <ulink url='https://www.docker.com/'>Docker Containers</ulink>.
428 CROPS provides an easily managed, extensible environment
429 that allows you to build binaries for a variety of
430 architectures on Windows, Linux and Mac OS X hosts.
431 </para></listitem>
432 <listitem><para>
433 <emphasis><filename>devtool</filename>:</emphasis>
434 This command-line tool is available as part of the
435 extensible SDK (eSDK) and is its cornerstone.
436 You can use <filename>devtool</filename> to help build,
437 test, and package software within the eSDK.
438 You can use the tool to optionally integrate what you
439 build into an image built by the OpenEmbedded build
440 system.</para>
441
442 <para>The <filename>devtool</filename> command employs
443 a number of sub-commands that allow you to add, modify,
444 and upgrade recipes.
445 As with the OpenEmbedded build system, “recipes”
446 represent software packages within
447 <filename>devtool</filename>.
448 When you use <filename>devtool add</filename>, a recipe
449 is automatically created.
450 When you use <filename>devtool modify</filename>, the
451 specified existing recipe is used in order to determine
452 where to get the source code and how to patch it.
453 In both cases, an environment is set up so that when
454 you build the recipe a source tree that is under your
455 control is used in order to allow you to make changes
456 to the source as desired.
457 By default, both new recipes and the source go into
458 a “workspace” directory under the eSDK.
459 The <filename>devtool upgrade</filename> command
460 updates an existing recipe so that you can build it
461 for an updated set of source files.</para>
462
463 <para>You can read about the
464 <filename>devtool</filename> workflow in the Yocto
465 Project Application Development and Extensible
466 Software Development Kit (eSDK) Manual in the
467 "<ulink url='&YOCTO_DOCS_SDK_URL;#using-devtool-in-your-sdk-workflow'>Using <filename>devtool</filename> in Your SDK Workflow'</ulink>"
468 section.
469 </para></listitem>
470 <listitem><para>
471 <emphasis>Extensible Software Development Kit (eSDK):</emphasis>
472 The eSDK provides a cross-development toolchain and
473 libraries tailored to the contents of a specific image.
474 The eSDK makes it easy to add new applications and
475 libraries to an image, modify the source for an
476 existing component, test changes on the target
477 hardware, and integrate into the rest of the
478 OpenEmbedded build system.
479 The eSDK gives you a toolchain experience supplemented
480 with the powerful set of <filename>devtool</filename>
481 commands tailored for the Yocto Project environment.
482 </para>
483
484 <para>For information on the eSDK, see the
485 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and Extensible Software Development Kit (eSDK) Manual</ulink>.
486 </para></listitem>
487 <listitem><para>
488 <emphasis><trademark class='trade'>Eclipse</trademark> IDE Plug-in:</emphasis>
489 This plug-in enables you to use the popular Eclipse
490 Integrated Development Environment (IDE), which allows
491 for development using the Yocto Project all within the
492 Eclipse IDE.
493 You can work within Eclipse to cross-compile, deploy,
494 and execute your output into a QEMU emulation session
495 as well as onto actual target hardware.</para>
496
497 <para>The environment also supports performance
498 enhancing tools that allow you to perform remote
499 profiling, tracing, collection of power data,
500 collection of latency data, and collection of
501 performance data.</para>
502
503 <para>Once you enable the plug-in, standard Eclipse
504 functions automatically use the cross-toolchain
505 and target system libraries.
506 You can build applications using any of these
507 libraries.</para>
508
509 <para>For more information on the Eclipse plug-in,
510 see the
511 "<ulink url='&YOCTO_DOCS_SDK_URL;#adt-eclipse'>Working Within Eclipse</ulink>"
512 section in the Yocto Project Application Development
513 and the Extensible Software Development Kit (eSDK)
514 manual.
515 </para></listitem>
516 <listitem><para>
517 <emphasis>Toaster:</emphasis>
518 Toaster is a web interface to the Yocto Project
519 OpenEmbedded build system.
520 Toaster allows you to configure, run, and view
521 information about builds.
522 </para></listitem>
523 </itemizedlist>
524 </para>
525 </section>
526
527 <section id='gs-production-tools'>
528 <title>Production Tools</title>
529
530 <para>
531 The following list consists of tools that help production
532 related activities using the Yocto Project:
533 <itemizedlist>
534 <listitem><para>
535 <emphasis>Auto Upgrade Helper:</emphasis>
536 This utility when used in conjunction with the
537 OpenEmbedded build system (BitBake and OE-Core)
538 automatically generates upgrades for recipes that
539 are based on new versions of the recipes published
540 upstream.
541 </para></listitem>
542 <listitem><para>
543 <emphasis>Recipe Reporting System:</emphasis>
544 The Recipe Reporting System tracks recipe versions
545 available for Yocto Project.
546 The main purpose of the system is to help you
547 manage the recipes you maintain and to offer a dynamic
548 overview of the project.
549 The Recipe Reporting System is built on top
550 the of OpenEmbedded Metadata Index, which is a website
551 that indexes layers for the OpenEmbedded build system.
552 </para></listitem>
553 <listitem><para>
554 <emphasis>Patchwork:</emphasis>
555 <ulink url='http://jk.ozlabs.org/projects/patchwork/'>Patchwork</ulink>
556 is a fork of a project originally started by
557 <ulink url='http://ozlabs.org/'>OzLabs</ulink>.
558 The project is a web-based tracking system designed
559 to streamline the process of bringing contributions
560 into a project.
561 The Yocto Project uses Patchwork as an organizational
562 tool to handle patches, which number in the thousands
563 for every release.
564 </para></listitem>
565 <listitem><para>
566 <emphasis>AutoBuilder:</emphasis>
567 AutoBuilder is a project that automates build tests
568 and quality assurance (QA).
569 By using the public AutoBuilder, anyone can determine
570 the status of the current "master" branch of Poky.
571 </para>
572
573 <para>A goal of the Yocto Project is to lead the
574 open source industry with a project that automates
575 testing and QA procedures.
576 In doing so, the project encourages a development
577 community that publishes QA and test plans, publicly
578 demonstrates QA and test plans, and encourages
579 development of tools that automate and test and QA
580 procedures for the benefit of the development
581 community.</para>
582
583 <para>You can learn more about the AutoBuilder used
584 by the Yocto Project
585 <ulink url='&YOCTO_AB_URL;'>here</ulink>.
586 </para></listitem>
587 <listitem><para>
588 <emphasis>Cross-Prelink:</emphasis>
589 Prelinking is the process of pre-computing the load
590 addresses and link tables generated by the dynamic
591 linker as compared to doing this at runtime.
592 Doing this ahead of time results in performance
593 improvements when the application is launched and
594 reduced memory usage for libraries shared by many
595 applications.</para>
596
597 <para>Historically, cross-prelink is a variant of
598 prelink, which was conceived by
599 <ulink url='http://people.redhat.com/jakub/prelink.pdf'>Jakub Jel&iacute;nek</ulink>
600 a number of years ago.
601 Both prelink and cross-prelink are maintained in the
602 same repository albeit on separate branches.
603 By providing an emulated runtime dynamic linker
604 (i.e. <filename>glibc</filename>-derived
605 <filename>ld.so</filename> emulation), the
606 cross-prelink project extends the prelink software’s
607 ability to prelink a sysroot environment.
608 Additionally, the cross-prelink software enables the
609 ability to work in sysroot style environments.</para>
610
611 <para>The dynamic linker determines standard load
612 address calculations based on a variety of factors
613 such as mapping addresses, library usage, and library
614 function conflicts.
615 The prelink tool uses this information, from the
616 dynamic linker, to determine unique load addresses
617 for executable and linkable format (ELF) binaries
618 that are shared libraries and dynamically linked.
619 The prelink tool modifies these ELF binaries with the
620 pre-computed information.
621 The result is faster loading and often lower memory
622 consumption because more of the library code can
623 be re-used from shared Copy-On-Write (COW) pages.
624 </para>
625
626 <para>The original upstream prelink project only
627 supports running prelink on the end target device
628 due to the reliance on the target device’s dynamic
629 linker.
630 This restriction causes issues when developing a
631 cross-compiled system.
632 The cross-prelink adds a synthesized dynamic loader
633 that runs on the host, thus permitting cross-prelinking
634 without ever having to run on a read-write target
635 filesystem.
636 </para></listitem>
637 <listitem><para>
638 <emphasis>Pseudo:</emphasis>
639 Pseudo is the Yocto Project implementation of
640 <ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>,
641 which is used to run commands in an environment
642 that seemingly has root privileges.</para>
643
644 <para>During a build, it can be necessary to perform
645 operations that require system administrator
646 privileges.
647 For example, file ownership or permissions might need
648 definition.
649 Pseudo is a tool that you can either use directly or
650 through the environment variable
651 <filename>LD_PRELOAD</filename>.
652 Either method allows these operations to succeed as
653 if system administrator privileges exist even
654 when they do not.</para>
655
656 <para>You can read more about Pseudo in the
657 "<ulink url='&YOCTO_DOCS_GS_URL;#fakeroot-and-pseudo'>Fakeroot and Pseudo</ulink>"
658 section of the Yocto Project Overview and Concepts
659 Manual.
660 </para></listitem>
661 </itemizedlist>
662 </para>
663 </section>
664
665 <section id='gs-openembedded-build-system'>
666 <title>Open-Embedded Build System Components</title>
667
668 <para>
669 The following list consists of components associated with the
670 Open-Embedded build system:
671 <itemizedlist>
672 <listitem><para>
673 <emphasis>BitBake:</emphasis>
674 BitBake is a core component of the Yocto Project and is
675 used by the OpenEmbedded build system to build images.
676 While BitBake is key to the build system, BitBake
677 is maintained separately from the Yocto Project.</para>
678
679 <para>BitBake is a generic task execution engine that
680 allows shell and Python tasks to be run efficiently
681 and in parallel while working within complex inter-task
682 dependency constraints.
683 In short, BitBake is a build engine that works
684 through recipes written in a specific format in order
685 to perform sets of tasks.</para>
686
687 <para>You can learn more about BitBake in the
688 <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>.
689 </para></listitem>
690 <listitem><para>
691 <emphasis>OpenEmbedded-Core:</emphasis>
692 OpenEmbedded-Core (OE-Core) is a common layer of
693 metadata (i.e. recipes, classes, and associated files)
694 used by OpenEmbedded-derived systems, which includes
695 the Yocto Project.
696 The Yocto Project and the OpenEmbedded Project both
697 maintain the OpenEmbedded-Core.
698 You can find the OE-Core metadata in the Yocto
699 Project
700 <ulink url='&YOCTO_DOCS_GS_URL;#source-repositories'>Source Repositories</ulink>
701 <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta'>here</ulink>.
702 </para>
703
704 <para>Historically, the Yocto Project integrated the
705 OE-Core metadata throughout the Yocto Project
706 source repository reference system (Poky).
707 After Yocto Project Version 1.0, the Yocto Project
708 and OpenEmbedded agreed to work together and share a
709 common core set of metadata (OE-Core), which contained
710 much of the functionality previously found in Poky.
711 This collaboration achieved a long-standing
712 OpenEmbedded objective for having a more tightly
713 controlled and quality-assured core.
714 The results also fit well with the Yocto Project
715 objective of achieving a smaller number of fully
716 featured tools as compared to many different ones.
717 </para>
718
719 <para>Sharing a core set of metadata results in Poky
720 as an integration layer on top of OE-Core.
721 You can see that in this
722 <link linkend='yp-key-dev-elements'>figure</link>.
723 The Yocto Project combines various components such as
724 BitBake, OE-Core, script “glue”, and documentation
725 for its build system.
726 </para></listitem>
727 </itemizedlist>
728 </para>
729 </section>
730
731 <section id='gs-reference-distribution-poky'>
732 <title>Reference Distribution (Poky)</title>
733
734 <para>
735 Poky is the Yocto Project reference distribution.
736 It contains the OpenEmbedded build system (BitBake and OE-Core)
737 as well as a set of metadata to get you started building your
738 own distribution.
739 See the
740 <link linkend='what-is-the-yocto-project'>figure</link> in
741 "What is the Yocto Project?" section for an illustration
742 that shows Poky and its relationship with other parts of the
743 Yocto Project.</para>
744
745 <para>To use the Yocto Project tools and components, you
746 can download (<filename>clone</filename>) Poky and use it
747 to bootstrap your own distribution.
748 <note>
749 Poky does not contain binary files.
750 It is a working example of how to build your own custom
751 Linux distribution from source.
752 </note>
753 You can read more about Poky in the
754 "<link linkend='reference-embedded-distribution'>Reference Embedded Distribution (Poky)</link>"
755 section.
756 </para>
757 </section>
758
759 <section id='gs-packages-for-finished-targets'>
760 <title>Packages for Finished Targets</title>
761
762 <para>
763 The following lists components associated with packages
764 for finished targets:
765 <itemizedlist>
766 <listitem><para>
767 <emphasis>Matchbox:</emphasis>
768 Matchbox is an Open Source, base environment for the
769 X Window System running on non-desktop, embedded
770 platforms such as handhelds, set-top boxes, kiosks,
771 and anything else for which screen space, input
772 mechanisms, or system resources are limited.</para>
773
774 <para>Matchbox consists of a number of interchangeable
775 and optional applications that you can tailor to a
776 specific, non-desktop platform to enhance usability
777 in constrained environments.</para>
778
779 <para>You can find the Matchbox source in its
780 <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi'>repository</ulink>
781 listed in the Yocto Project
782 <ulink url='&YOCTO_DOCS_GS_URL;#source-repositories'>Source Repositories</ulink>.
783 </para></listitem>
784 <listitem><para>
785 <emphasis>Opkg</emphasis>
786 Open PacKaGe management (opkg) is a lightweight
787 package management system based on the itsy package
788 (ipkg) management system.
789 Opkg is written in C and resembles Advanced Package
790 Tool (APT) and Debian Package (dpkg) in operation.
791 </para>
792
793 <para>Opkg is intended for use on embedded Linux
794 devices and is used in this capacity in the
795 <ulink url='http://www.openembedded.org/wiki/Main_Page'>OpenEmbedded</ulink>
796 and
797 <ulink url='https://openwrt.org/'>OpenWrt</ulink>
798 projects, as well as the Yocto Project.
799 <note>
800 As best it can, opkg maintains backwards
801 compatibility with ipkg and conforms to a subset
802 of Debian’s policy manual regarding control files.
803 </note>
804 </para></listitem>
805 </itemizedlist>
806 </para>
807 </section>
808
809 <section id='gs-archived-components'>
810 <title>Archived Components</title>
811
812 <para>
813 The Build Appliance is a virtual machine image that enables
814 you to build and boot a custom embedded Linux image with
815 the Yocto Project using a non-Linux development system.
816 </para>
817
818 <para>
819 Historically, the Build Appliance was the second of three
820 methods by which you could use the Yocto Project on a system
821 that was not native to Linux.
822 <orderedlist>
823 <listitem><para>
824 <emphasis>Hob:</emphasis>
825 Hob, which is now deprecated and is no longer available
826 since the 2.1 release of the Yocto Project provided
827 a rudimentary, GUI-based interface to the Yocto
828 Project.
829 Toaster has fully replaced Hob.
830 </para></listitem>
831 <listitem><para>
832 <emphasis>Build Appliance:</emphasis>
833 Post Hob, the Build Appliance became available.
834 It was never recommended that you use the Build
835 Appliance as a day-to-day production development
836 environment with the Yocto Project.
837 Build Appliance was useful as a way to try out
838 development in the Yocto Project environment.
839 </para></listitem>
840 <listitem><para>
841 <emphasis>CROPS:</emphasis>
842 The final and best solution available now for
843 developing using the Yocto Project on a system
844 not native to Linux is with
845 <link linkend='gs-crops-overview'>CROPS</link>.
846 </para></listitem>
847 </orderedlist>
848 </para>
849 </section>
850 </section>
851
852 <section id='gs-development-methods'>
853 <title>Development Methods</title>
854
855 <para>
856 The Yocto Project development environment usually involves a
857 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>Build Host</ulink>
858 and target hardware.
859 You use the Build Host to build images and develop applications,
860 while you use the target hardware to test deployed software.
861 </para>
862
863 <para>
864 This section provides an introduction to the choices or
865 development methods you have when setting up your Build Host.
866 Depending on the your particular workflow preference and the
867 type of operating system your Build Host runs, several choices
868 exist that allow you to use the Yocto Project.
869 <note>
870 For additional detail about the Yocto Project development
871 environment, see the
872 "<link linkend='overview-development-environment'>The Yocto Project Development Environment</link>"
873 chapter.
874 </note>
875 <itemizedlist>
876 <listitem><para>
877 <emphasis>Native Linux Host:</emphasis>
878 By far the best option for a Build Host.
879 A system running Linux as its native operating system
880 allows you to develop software by directly using the
881 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
882 tool.
883 You can accomplish all aspects of development from a
884 familiar shell of a supported Linux distribution.</para>
885
886 <para>For information on how to set up a Build Host on
887 a system running Linux as its native operating system,
888 see the
889 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host'>Setting Up a Native Linux Host</ulink>"
890 section in the Yocto Project Development Tasks Manual.
891 </para></listitem>
892 <listitem><para>
893 <emphasis>CROss PlatformS (CROPS):</emphasis>
894 Typically, you use
895 <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/crops/about/'>CROPS</ulink>,
896 which leverages
897 <ulink url='https://www.docker.com/'>Docker Containers</ulink>,
898 to set up a Build Host that is not running Linux (e.g.
899 <trademark class='registered'>Microsoft</trademark>
900 <trademark class='trademark'>Windows</trademark>
901 or
902 <trademark class='registered'>macOS</trademark>).
903 <note>
904 You can, however, use CROPS on a Linux-based system.
905 </note>
906 CROPS is an open source, cross-platform development
907 framework that provides an easily managed, extensible
908 environment for building binaries targeted for a variety
909 of architectures on Windows, macOS, or Linux hosts.
910 Once the Build Host is set up using CROPS, you can prepare
911 a shell environment to mimic that of a shell being used
912 on a system natively running Linux.</para>
913
914 <para>For information on how to set up a Build Host with
915 CROPS, see the
916 "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops'>Setting Up to Use CROss PlatformS (CROPS)</ulink>"
917 section in the Yocto Project Development Tasks Manual.
918 </para></listitem>
919 <listitem><para>
920 <emphasis>Toaster:</emphasis>
921 Regardless of what your Build Host is running, you can
922 use Toaster to develop software using the Yocto Project.
923 Toaster is a web interface to the Yocto Project's
924 OpenEmbedded build system.
925 The interface enables you to configure and run your
926 builds.
927 Information about builds is collected and stored in a
928 database.
929 You can use Toaster to configure and start builds on
930 multiple remote build servers.</para>
931
932 <para>For information about and how to use Toaster,
933 see the
934 <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>.
935 </para></listitem>
936 <listitem><para>
937 <emphasis><trademark class='trade'>Eclipse</trademark> IDE:</emphasis>
938 If your Build Host supports and runs the popular
939 Eclipse IDE, you can install the Yocto Project Eclipse
940 plug-in and use the Yocto Project to develop software.
941 The plug-in integrates the Yocto Project functionality
942 into Eclipse development practices.</para>
943
944 <para>For information about how to install and use the
945 Yocto Project Eclipse plug-in, see the
946 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-eclipse-project'>Developing Applications Using Eclipse</ulink>"
947 section in the Yocto Project Application Development and
948 the Extensible Software Development Kit (eSDK) Manual.
949 </para></listitem>
950 </itemizedlist>
951 </para>
952 </section>
953
954 <section id='reference-embedded-distribution'>
955 <title>Reference Embedded Distribution (Poky)</title>
956
957 <para>
958 "Poky", which is pronounced <emphasis>Pock</emphasis>-ee, is the
959 name of the Yocto Project's reference distribution or Reference OS
960 Kit.
961 Poky contains the
962 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded Build System</ulink>
963 build system
964 (<ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> and
965 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>)
966 as well as a set of
967 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>metadata</ulink> to get
968 you started building your own distro.
969 In other words, Poky is a base specification of the functionality
970 needed for a typical embedded system as well as the components
971 from the Yocto Project that allow you to build a distribution into
972 a usable binary image.
973 </para>
974
975 <para>
976 Poky is a combined repository of BitBake, OpenEmbedded-Core
977 (which is found in <filename>meta</filename>),
978 <filename>meta-poky</filename>,
979 <filename>meta-yocto-bsp</filename>, and documentation provided
980 all together and known to work well together.
981 You can view these items that make up the Poky repository in the
982 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/'>Source Repositories</ulink>.
983 <note>
984 If you are interested in all the contents of the
985 <filename>poky</filename> Git repository, see the
986 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core'>Top-Level Core Components</ulink>"
987 section in the Yocto Project Reference Manual.
988 </note>
989 </para>
990
991 <para id='gs-poky-reference-distribution'>
992 The following figure illustrates what generally comprises Poky:
993 <imagedata fileref="figures/poky-reference-distribution.png" format="PNG" align='center' width="8in"/>
994 <itemizedlist>
995 <listitem><para>
996 BitBake is a task executor and scheduler that is the heart of
997 the OpenEmbedded build system.
998 </para></listitem>
999 <listitem><para>
1000 <filename>meta-poky</filename>, which is Poky-specific
1001 metadata.
1002 </para></listitem>
1003 <listitem><para>
1004 <filename>meta-yocto-bsp</filename>, which is Yocto
1005 Project-specific Board Support Packages (BSPs).
1006 </para></listitem>
1007 <listitem><para>
1008 OpenEmbedded-Core (OE-Core) metadata, which includes
1009 shared configurations, global variable definitions,
1010 shared classes, packaging, and recipes.
1011 Classes define the encapsulation and inheritance of build
1012 logic.
1013 Recipes are the logical units of software and images
1014 to be built.
1015 </para></listitem>
1016 <listitem><para>
1017 Documentation, which contains the Yocto Project source
1018 files used to make the set of user manuals.
1019 </para></listitem>
1020 </itemizedlist>
1021 <note>
1022 While Poky is a "complete" distribution specification and is
1023 tested and put through QA, you cannot use it as a product
1024 "out of the box" in its current form.
1025 </note>
1026 </para>
1027
1028 <para>
1029 To use the Yocto Project tools, you can use Git to clone (download)
1030 the Poky repository then use your local copy of the reference
1031 distribution to bootstrap your own distribution.
1032 <note>
1033 Poky does not contain binary files.
1034 It is a working example of how to build your own custom Linux distribution
1035 from source.
1036 </note>
1037 </para>
1038
1039 <para>
1040 Poky has a regular, well established, six-month release cycle
1041 under its own version.
1042 Major releases occur at the same time major releases (point
1043 releases) occur for the Yocto Project, which are typically in the
1044 Spring and Fall.
1045 For more information on the Yocto Project release schedule and
1046 cadence, see the
1047 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-release-process'>Yocto Project Releases and the Stable Release Process</ulink>"
1048 chapter in the Yocto Project Reference Manual.
1049 </para>
1050
1051 <para>
1052 Much has been said about Poky being a "default configuration."
1053 A default configuration provides a starting image footprint.
1054 You can use Poky out of the box to create an image ranging from a
1055 shell-accessible minimal image all the way up to a Linux
1056 Standard Base-compliant image that uses a GNOME Mobile and
1057 Embedded (GMAE) based reference user interface called Sato.
1058 </para>
1059
1060 <para>
1061 One of the most powerful properties of Poky is that every aspect
1062 of a build is controlled by the metadata.
1063 You can use metadata to augment these base image types by
1064 adding metadata layers that extend functionality.
1065 These layers can provide, for example, an additional software
1066 stack for an image type, add a board support package (BSP) for
1067 additional hardware, or even create a new image type.
1068 </para>
1069
1070 <para>
1071 Metadata is loosely grouped into configuration files or package
1072 recipes.
1073 A recipe is a collection of non-executable metadata used by
1074 BitBake to set variables or define additional build-time tasks.
1075 A recipe contains fields such as the recipe description, the recipe
1076 version, the license of the package and the upstream source
1077 repository.
1078 A recipe might also indicate that the build process uses autotools,
1079 make, distutils or any other build process, in which case the basic
1080 functionality can be defined by the classes it inherits from
1081 the OE-Core layer's class definitions in
1082 <filename>./meta/classes</filename>.
1083 Within a recipe you can also define additional tasks as well as
1084 task prerequisites.
1085 Recipe syntax through BitBake also supports both
1086 <filename>_prepend</filename> and <filename>_append</filename>
1087 operators as a method of extending task functionality.
1088 These operators inject code into the beginning or end of a task.
1089 For information on these BitBake operators, see the
1090 "<ulink url='&YOCTO_DOCS_BB_URL;#appending-and-prepending-override-style-syntax'>Appending and Prepending (Override Style Syntax)</ulink>"
1091 section in the BitBake User's Manual.
1092 </para>
1093 </section>
1094
1095 <section id='openembedded-build-system-workflow'>
1096 <title>The OpenEmbedded Build System Workflow</title>
1097
1098 <para>
1099 The OpenEmbedded build system uses a "workflow" to accomplish
1100 image and SDK generation.
1101 The following figure overviews that workflow:
1102 <imagedata fileref="figures/YP-flow-diagram.png"
1103 format="PNG" align='center' width="8in"/>
1104 Following is a brief summary of the "workflow":
1105 <orderedlist>
1106 <listitem><para>
1107 Developers specify architecture, policies, patches and
1108 configuration details.
1109 </para></listitem>
1110 <listitem><para>
1111 The build system fetches and downloads the source code
1112 from the specified location.
1113 The build system supports standard methods such as tarballs
1114 or source code repositories systems such as Git.
1115 </para></listitem>
1116 <listitem><para>
1117 Once downloaded, the build system extracts the sources
1118 into a local work area where patches are applied and
1119 common steps for configuring and compiling the software
1120 are run.
1121 </para></listitem>
1122 <listitem><para>
1123 The build system then installs the software into a
1124 temporary staging area where the binary package format you
1125 select (DEB, RPM, or IPK) is used to roll up the software.
1126 </para></listitem>
1127 <listitem><para>
1128 Different QA and sanity checks run throughout entire
1129 build process.
1130 </para></listitem>
1131 <listitem><para>
1132 After the binaries are created, the build system
1133 generates a binary package feed that is used to create
1134 the final root file image.
1135 </para></listitem>
1136 <listitem><para>
1137 The build system generates the file system image and a
1138 customized Extensible SDK (eSDSK) for application
1139 development in parallel.
1140 </para></listitem>
1141 </orderedlist>
1142 </para>
1143
1144 <para>
1145 For a very detailed look at this workflow, see the
1146 "<ulink url='&YOCTO_DOCS_GS_URL;#development-concepts'>Development Concepts</ulink>"
1147 section in the Yocto Project Overview and Concepts Manual.
1148 </para>
1149 </section>
1150
1151
1152 <section id='some-basic-terms'>
1153 <title>Some Basic Terms</title>
1154
1155 <para>
1156 It helps to understand some basic fundamental terms when
1157 learning the Yocto Project.
1158 Although a list of terms exists in the
1159 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-terms'>Yocto Project Terms</ulink>"
1160 section of the Yocto Project Reference Manual, this section
1161 provides the definitions of some terms helpful for getting started:
1162 <itemizedlist>
1163 <listitem><para>
1164 <emphasis>Configuration Files:</emphasis>
1165 Files that hold global definitions of variables,
1166 user-defined variables, and hardware configuration
1167 information.
1168 These files tell the OpenEmbedded build system what to
1169 build and what to put into the image to support a
1170 particular platform.
1171 </para></listitem>
1172 <listitem><para>
1173 <emphasis>Extensible Software Development Kit (eSDK):</emphasis>
1174 A custom SDK for application developers.
1175 This eSDK allows developers to incorporate their library
1176 and programming changes back into the image to make
1177 their code available to other application developers.
1178 For information on the eSDK, see the
1179 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and Extensible Software Development Kit (eSDK)</ulink>
1180 manual.
1181 </para></listitem>
1182 <listitem><para>
1183 <emphasis>Layer:</emphasis>
1184 A collection of related recipes.
1185 Layers allow you to consolidate related metadata to
1186 customize your build.
1187 Layers also isolate information used when building
1188 for multiple architectures.
1189 Layers are hierarchical in their ability to override
1190 previous specifications.
1191 You can include any number of available layers from the
1192 Yocto Project and customize the build by adding your
1193 layers after them.
1194 You can search the Layer Index for layers used within
1195 Yocto Project.</para>
1196
1197 <para>For more detailed information on layers, see the
1198 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
1199 section in the Yocto Project Development Tasks Manual.
1200 For a discussion specifically on BSP Layers, see the
1201 "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>"
1202 section in the Yocto Project Board Support Packages (BSP)
1203 Developer's Guide.
1204 </para></listitem>
1205 <listitem><para>
1206 <emphasis>Metadata:</emphasis>
1207 A key element of the Yocto Project is the Metadata that
1208 is used to construct a Linux distribution and is contained
1209 in the files that the
1210 <link linkend='gs-term-openembedded-build-system'>OpenEmbedded build system</link> parses
1211 when building an image.
1212 In general, Metadata includes recipes, configuration
1213 files, and other information that refers to the build
1214 instructions themselves, as well as the data used to
1215 control what things get built and the effects of the
1216 build.
1217 Metadata also includes commands and data used to
1218 indicate what versions of software are used, from
1219 where they are obtained, and changes or additions to the
1220 software itself (patches or auxiliary files) that
1221 are used to fix bugs or customize the software for use
1222 in a particular situation.
1223 OpenEmbedded-Core is an important set of validated
1224 metadata.
1225 </para></listitem>
1226 <listitem><para id='gs-term-openembedded-build-system'>
1227 <emphasis>OpenEmbedded Build System:</emphasis>
1228 The terms "BitBake" and "build system" are sometimes
1229 used for the OpenEmbedded Build System.</para>
1230
1231 <para>BitBake is a task scheduler and execution engine
1232 that parses instructions (i.e. recipes) and configuration
1233 data.
1234 After a parsing phase, BitBake creates a dependency tree
1235 to order the compilation, schedules the compilation of
1236 the included code, and finally executes the building
1237 of the specified custom Linux image (distribution).
1238 BitBake is similar to the <filename>make</filename>
1239 tool.</para>
1240
1241 <para>During a build process, the build system tracks
1242 dependencies and performs a native or cross-compilation
1243 of the package.
1244 As a first step in a cross-build setup, the framework
1245 attempts to create a cross-compiler toolchain
1246 (i.e. Extensible SDK) suited for the target platform.
1247 </para></listitem>
1248 <listitem><para>
1249 <emphasis>OpenEmbedded-Core (OE-Core):</emphasis>
1250 OE-Core is metadata comprised of foundation recipes,
1251 classes, and associated files that are meant to be
1252 common among many different OpenEmbedded-derived systems,
1253 including the Yocto Project.
1254 OE-Core is a curated subset of an original repository
1255 developed by the OpenEmbedded community that has been
1256 pared down into a smaller, core set of continuously
1257 validated recipes.
1258 The result is a tightly controlled and quality-assured
1259 core set of recipes.</para>
1260
1261 <para>You can see the Metadata in the
1262 <filename>meta</filename> directory of the Yocto Project
1263 <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi'>Source Repositories</ulink>.
1264 </para></listitem>
1265 <listitem><para>
1266 <emphasis>Packages:</emphasis>
1267 In the context of the Yocto Project, this term refers to a
1268 recipe's packaged output produced by BitBake (i.e. a
1269 "baked recipe").
1270 A package is generally the compiled binaries produced from the
1271 recipe's sources.
1272 You "bake" something by running it through BitBake.</para>
1273
1274 <para>It is worth noting that the term "package" can,
1275 in general, have subtle meanings.
1276 For example, the packages referred to in the
1277 "<ulink url='&YOCTO_DOCS_REF_URL;#required-packages-for-the-host-development-system'>Required Packages for the Host Development System</ulink>"
1278 section in the Yocto Project Reference Manual are compiled
1279 binaries that, when installed, add functionality to your
1280 Linux distribution.</para>
1281
1282 <para>Another point worth noting is that historically within
1283 the Yocto Project, recipes were referred to as packages - thus,
1284 the existence of several BitBake variables that are seemingly
1285 mis-named,
1286 (e.g. <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>,
1287 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>,
1288 and
1289 <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink>).
1290 </para></listitem>
1291 <listitem><para>
1292 <emphasis>Poky:</emphasis>
1293 Poky is a reference embedded distribution and a reference
1294 test configuration.
1295 Poky provides the following:
1296 <itemizedlist>
1297 <listitem><para>
1298 A base-level functional distro used to illustrate
1299 how to customize a distribution.
1300 </para></listitem>
1301 <listitem><para>
1302 A means by which to test the Yocto Project
1303 components (i.e. Poky is used to validate
1304 the Yocto Project).
1305 </para></listitem>
1306 <listitem><para>
1307 A vehicle through which you can download
1308 the Yocto Project.
1309 </para></listitem>
1310 </itemizedlist>
1311 Poky is not a product level distro.
1312 Rather, it is a good starting point for customization.
1313 <note>
1314 Poky is an integration layer on top of OE-Core.
1315 </note>
1316 </para></listitem>
1317 <listitem><para>
1318 <emphasis>Recipe:</emphasis>
1319 The most common form of metadata.
1320 A recipe contains a list of settings and tasks
1321 (i.e. instructions) for building packages that are then
1322 used to build the binary image.
1323 A recipe describes where you get source code and which
1324 patches to apply.
1325 Recipes describe dependencies for libraries or for other
1326 recipes as well as configuration and compilation options.
1327 Related recipes are consolidated into a layer.
1328 </para></listitem>
1329 </itemizedlist>
1330 </para>
1331 </section>
1332</chapter>
1333<!--
1334vim: expandtab tw=80 ts=4
1335-->
diff --git a/documentation/overview-manual/overview-manual.xml b/documentation/overview-manual/overview-manual.xml
new file mode 100644
index 0000000000..f04fbaf3bd
--- /dev/null
+++ b/documentation/overview-manual/overview-manual.xml
@@ -0,0 +1,108 @@
1<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<book id='overview-manual' lang='en'
6 xmlns:xi="http://www.w3.org/2003/XInclude"
7 xmlns="http://docbook.org/ns/docbook"
8 >
9 <bookinfo>
10
11 <mediaobject>
12 <imageobject>
13 <imagedata fileref='figures/overview-manual-title.png'
14 format='SVG'
15 align='left' scalefit='1' width='100%'/>
16 </imageobject>
17 </mediaobject>
18
19 <title>
20 Yocto Project Overview and Concepts Manual
21 </title>
22
23 <authorgroup>
24 <author>
25 <firstname>Scott</firstname> <surname>Rifenbark</surname>
26 <affiliation>
27 <orgname>Scotty's Documentation Services, INC</orgname>
28 </affiliation>
29 <email>srifenbark@gmail.com</email>
30 </author>
31 </authorgroup>
32
33 <revhistory>
34 <revision>
35 <revnumber>2.5</revnumber>
36 <date>April 2018</date>
37 <revremark>The initial document released with the Yocto Project 2.5 Release.</revremark>
38 </revision>
39 </revhistory>
40
41 <copyright>
42 <year>&COPYRIGHT_YEAR;</year>
43 <holder>Linux Foundation</holder>
44 </copyright>
45
46 <legalnotice>
47 <para>
48 Permission is granted to copy, distribute and/or modify this document under
49 the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">
50 Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by
51 Creative Commons.
52 </para>
53 <note><title>Manual Notes</title>
54 <itemizedlist>
55 <listitem><para>
56 This version of the
57 <emphasis>Yocto Project Overview and Concepts Manual</emphasis>
58 is for the &YOCTO_DOC_VERSION; release of the
59 Yocto Project.
60 To be sure you have the latest version of the manual
61 for this release, go to the
62 <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
63 and select the manual from that site.
64 Manuals from the site are more up-to-date than manuals
65 derived from the Yocto Project released TAR files.
66 </para></listitem>
67 <listitem><para>
68 If you located this manual through a web search, the
69 version of the manual might not be the one you want
70 (e.g. the search might have returned a manual much
71 older than the Yocto Project version with which you
72 are working).
73 You can see all Yocto Project major releases by
74 visiting the
75 <ulink url='&YOCTO_WIKI_URL;/wiki/Releases'>Releases</ulink>
76 page.
77 If you need a version of this manual for a different
78 Yocto Project release, visit the
79 <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
80 and select the manual set by using the
81 "ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARCHIVE"
82 pull-down menus.
83 </para></listitem>
84 <listitem><para>
85 To report any inaccuracies or problems with this
86 manual, send an email to the Yocto Project
87 discussion group at
88 <filename>yocto@yoctoproject.com</filename> or log into
89 the freenode <filename>#yocto</filename> channel.
90 </para></listitem>
91 </itemizedlist>
92 </note>
93 </legalnotice>
94
95 </bookinfo>
96
97 <xi:include href="overview-manual-intro.xml"/>
98
99 <xi:include href="overview-manual-yp-intro.xml"/>
100
101 <xi:include href="overview-manual-development-environment.xml"/>
102
103 <xi:include href="overview-manual-concepts.xml" />
104
105</book>
106<!--
107vim: expandtab tw=80 ts=4
108-->