summaryrefslogtreecommitdiffstats
path: root/documentation/poky-ref-manual
diff options
context:
space:
mode:
authorScott Rifenbark <scott.m.rifenbark@intel.com>2011-12-08 10:34:44 -0800
committerRichard Purdie <richard.purdie@linuxfoundation.org>2012-01-30 16:05:48 +0000
commit5dd34a717eb94f1934bf60842b1406349897c970 (patch)
treee21fe693b353d41d2a2333c9ff36ee39ea733b9b /documentation/poky-ref-manual
parente7cfb3b46935ccb8b2c3520fc478aeff61f22779 (diff)
downloadpoky-5dd34a717eb94f1934bf60842b1406349897c970.tar.gz
documentation/poky-ref-manual: New chapter introduced
Long-term strategy for the YP Reference Manual is that it contains reference material and not "how-to-information". A step in this direction is to isolate any discussions on components and other areas of YP that need talked about. So to start with, I have created a new chapter for now named "Technical Details" that so far has a discussion of some components and shared state. This is a step in the direction of making this manual a reference manual and not a "how to" manual. Changes included removing redundant material from the 'usingpoky.xml' chapter and also adding the new chapter 'technical-details' into the 'poky-ref-manual.xml' file used for the make. (From yocto-docs rev: a01477f787768230bc25da2d094326922be23dd4) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/poky-ref-manual')
-rw-r--r--documentation/poky-ref-manual/poky-ref-manual.xml2
-rw-r--r--documentation/poky-ref-manual/technical-details.xml229
-rw-r--r--documentation/poky-ref-manual/usingpoky.xml220
3 files changed, 232 insertions, 219 deletions
diff --git a/documentation/poky-ref-manual/poky-ref-manual.xml b/documentation/poky-ref-manual/poky-ref-manual.xml
index 37059096e3..e9f6b5eba4 100644
--- a/documentation/poky-ref-manual/poky-ref-manual.xml
+++ b/documentation/poky-ref-manual/poky-ref-manual.xml
@@ -92,6 +92,8 @@
92 92
93 <xi:include href="extendpoky.xml"/> 93 <xi:include href="extendpoky.xml"/>
94 94
95 <xi:include href="technical-details.xml"/>
96
95 <xi:include href="../bsp-guide/bsp.xml"/> 97 <xi:include href="../bsp-guide/bsp.xml"/>
96 98
97 <xi:include href="development.xml"/> 99 <xi:include href="development.xml"/>
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml
new file mode 100644
index 0000000000..8f83ecb3a1
--- /dev/null
+++ b/documentation/poky-ref-manual/technical-details.xml
@@ -0,0 +1,229 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
3<chapter id='technical-details'>
4<title>Technical Details</title>
5
6 <para>
7 This chapter provides technical details for various parts of the Yocto Project.
8 Currently, topics include Yocto Project components and shared state (sstate) cache.
9 </para>
10
11<section id='usingpoky-components'>
12 <title>Yocto Project Components</title>
13
14 <para>
15 The BitBake task executor together with various types of configuration files form the
16 Yocto Project core.
17 This section overviews the BitBake task executor and the
18 configuration files by describing what they are used for and how they interact.
19 </para>
20
21 <para>
22 BitBake handles the parsing and execution of the data files.
23 The data itself is of various types:
24 <itemizedlist>
25 <listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
26 pieces of software</para></listitem>
27 <listitem><para><emphasis>Class Data:</emphasis> An abstraction of common build
28 information (e.g. how to build a Linux kernel).</para></listitem>
29 <listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
30 policy decisions, etc.
31 Configuration data acts as the glue to bind everything together.</para></listitem>
32 </itemizedlist>
33 For more information on data, see the
34 <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#yocto-project-terms'>
35 Yocto Project Terms</ulink> section in
36 <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html'>
37 The Yocto Project Development Manual</ulink>.
38 </para>
39
40 <para>
41 BitBake knows how to combine multiple data sources together and refers to each data source
42 as a <link linkend='usingpoky-changes-layers'>'layer'</link>.
43 </para>
44
45 <para>
46 Following are some brief details on these core components.
47 For more detailed information on these components see the
48 <link linkend='ref-structure'>'Reference: Directory Structure'</link>
49 appendix.
50 </para>
51
52 <section id='usingpoky-components-bitbake'>
53 <title>BitBake</title>
54
55 <para>
56 BitBake is the tool at the heart of the Yocto Project and is responsible
57 for parsing the metadata, generating a list of tasks from it,
58 and then executing those tasks.
59 To see a list of the options BitBake supports, use the following help command:
60 <literallayout class='monospaced'>
61 $ bitbake --help
62 </literallayout>
63 </para>
64
65 <para>
66 The most common usage for BitBake is <filename>bitbake &lt;packagename&gt;</filename>, where
67 <filename>packagename</filename> is the name of the package you want to build
68 (referred to as the "target" in this manual).
69 The target often equates to the first part of a <filename>.bb</filename> filename.
70 So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
71 might type the following:
72 <literallayout class='monospaced'>
73 $ bitbake matchbox-desktop
74 </literallayout>
75 Several different versions of <filename>matchbox-desktop</filename> might exist.
76 BitBake chooses the one selected by the distribution configuration.
77 You can get more details about how BitBake chooses between different
78 target versions and providers in the
79 <link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
80 </para>
81
82 <para>
83 BitBake also tries to execute any dependent tasks first.
84 So for example, before building <filename>matchbox-desktop</filename>, BitBake
85 would build a cross compiler and <filename>eglibc</filename> if they had not already
86 been built.
87 <note>This release of the Yocto Project does not support the <filename>glibc</filename>
88 GNU version of the Unix standard C library. By default, the Yocto Project builds with
89 <filename>eglibc</filename>.</note>
90 </para>
91
92 <para>
93 A useful BitBake option to consider is the <filename>-k</filename> or
94 <filename>--continue</filename> option.
95 This option instructs BitBake to try and continue processing the job as much
96 as possible even after encountering an error.
97 When an error occurs, the target that
98 failed and those that depend on it cannot be remade.
99 However, when you use this option other dependencies can still be processed.
100 </para>
101 </section>
102
103 <section id='usingpoky-components-metadata'>
104 <title>Metadata (Recipes)</title>
105
106 <para>
107 The <filename>.bb</filename> files are usually referred to as "recipes."
108 In general, a recipe contains information about a single piece of software.
109 The information includes the location from which to download the source patches
110 (if any are needed), which special configuration options to apply,
111 how to compile the source files, and how to package the compiled output.
112 </para>
113
114 <para>
115 The term "package" can also be used to describe recipes.
116 However, since the same word is used for the packaged output from the Yocto
117 Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
118 this document avoids using the term "package" to refer to recipes.
119 </para>
120 </section>
121
122 <section id='usingpoky-components-classes'>
123 <title>Classes</title>
124
125 <para>
126 Class files (<filename>.bbclass</filename>) contain information that is useful to share
127 between metadata files.
128 An example is the Autotools class, which contains
129 common settings for any application that Autotools uses.
130 The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
131 about common classes and how to use them.
132 </para>
133 </section>
134
135 <section id='usingpoky-components-configuration'>
136 <title>Configuration</title>
137
138 <para>
139 The configuration files (<filename>.conf</filename>) define various configuration variables
140 that govern the Yocto Project build process.
141 These files fall into several areas that define machine configuration options,
142 distribution configuration options, compiler tuning options, general common configuration
143 options and user configuration options (<filename>local.conf</filename>, which is found
144 in the Yocto Project files build directory).
145 </para>
146 </section>
147</section>
148
149<section id="considering-shared-state-cache">
150 <title>Considering Shared State Cache</title>
151
152 <para>
153 By design, the Yocto Project builds everything from scratch unless it can determine that
154 a given task's inputs have not changed.
155 While building from scratch ensures that everything is current, it does also
156 mean that a lot of time could be spent rebuiding things that don't necessarily need built.
157 </para>
158
159 <para>
160 The Yocto Project build process uses a shared state caching scheme to avoid having to
161 rebuild software when it is not necessary.
162 Because the build time for a Yocto image can be significant, it is helpful to try and
163 determine what really needs built and what can be skipped given a particular project's
164 build process.
165 </para>
166
167 <para>
168 The scheme that the Yocto Project uses involves checksum generation and comparison for
169 a task's inputs.
170 The scheme also employs an area of memory called the shared state cache that is
171 pointed to by the <filename>SSTATE_DIR</filename> variable.
172 This area contains task output generated from a previous build.
173 If a given task's checksum matches the checksum of a previous build for the same
174 task, the build process uses the state of the cache rather than rerunning that
175 task.
176 </para>
177
178 <para>
179 The previous paragraph is a simplistic explanation of how the build process
180 uses checksums and shared state memory cache to avoide building tasks that
181 don't need built.
182 If you want a bit more explanation on the topic,
183 see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
184 State - What does it mean and why should I care?</ulink>" from the Yocto
185 Project discussion archives.
186 </para>
187
188 <para>
189 As with all schemes, this one has some drawbacks.
190 It is possible that you could make implicit changes that are not factored into the checksum
191 calculation, but do affect a task's output.
192 A good example is perhaps when a tool changes its output.
193 Let's say that the output of <filename>rpmdeps</filename> needed to change.
194 The result of the change should be that all the "package", "package_write_rpm",
195 and "package_deploy-rpm" sstate-cache items would become invalid.
196 But, because this is a change that is external to the code and therefore implicit,
197 the associated sstate-cache items do not become invalidated.
198 In this case, the build process would use the cache items rather than running the
199 task again.
200 Obviously, these types of implicit changes can cause problems.
201 </para>
202
203 <para>
204 To avoid these problems during the build, you need to understand the effects of any
205 change you make.
206 Note that any changes you make directly to a function automatically are factored into
207 the checksum calculation and thus, will invalidate the associated area of sstate cache.
208 You need to be aware of any implicit changes that are not obvious changes to the
209 code and could affect the output of a given task.
210 Once you are aware of such a change, you can take steps to invalidate the cache
211 and force the task to run.
212 The step to take is as simple as changing a function's comments in the source code.
213 For example, to invalidate package sstate files, change the comment statments
214 of <filename>do_package</filename> or one of the functions it calls.
215 The change is purely cosmetic, but it causes the checksum to be recalculated and
216 forces the task to be run again.
217 </para>
218
219 <note>
220 For an example of a commit that makes a cosmetic change to invalidate an sstate,
221 see this
222 <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
223 </note>
224</section>
225
226</chapter>
227<!--
228vim: expandtab tw=80 ts=4
229-->
diff --git a/documentation/poky-ref-manual/usingpoky.xml b/documentation/poky-ref-manual/usingpoky.xml
index 1e21c93778..e741e23d93 100644
--- a/documentation/poky-ref-manual/usingpoky.xml
+++ b/documentation/poky-ref-manual/usingpoky.xml
@@ -4,150 +4,9 @@
4<title>Using the Yocto Project</title> 4<title>Using the Yocto Project</title>
5 5
6 <para> 6 <para>
7 This section gives an overview of the components that make up the Yocto Project 7 This section describes the common usage for the Yocto Project.
8 followed by information about Yocto Project builds and dealing with any
9 problems that might arise.
10 </para> 8 </para>
11 9
12<section id='usingpoky-components'>
13 <title>Yocto Project Components</title>
14
15 <para>
16 The BitBake task executor together with various types of configuration files form the
17 Yocto Project core.
18 This section overviews the BitBake task executor and the
19 configuration files by describing what they are used for and they they interact.
20 </para>
21
22 <para>
23 BitBake handles the parsing and execution of the data files.
24 The data itself is of various types:
25 <itemizedlist>
26 <listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
27 pieces of software</para></listitem>
28 <listitem><para><emphasis>Class Data:</emphasis> An abstraction of common build
29 information (e.g. how to build a Linux kernel).</para></listitem>
30 <listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
31 policy decisions, etc.
32 Configuration data acts a the glue to bind everything together.</para></listitem>
33 </itemizedlist>
34 For more information on data, see the
35 <ulink url='http://www.yoctoproject.org/docs/1.1/dev-manual/dev-manual.html#yocto-project-terms'>
36 Yocto Project Terms</ulink> section in
37 <ulink url='http://www.yoctoproject.org/docs/1.1/dev-manual/dev-manual.html'>
38 The Yocto Project Development Manual</ulink>.
39 </para>
40
41 <para>
42 BitBake knows how to combine multiple data sources together and refers to each data source
43 as a <link linkend='usingpoky-changes-layers'>'layer'</link>.
44 </para>
45
46 <para>
47 Following are some brief details on these core components.
48 For more detailed information on these components see the
49 <link linkend='ref-structure'>'Reference: Directory Structure'</link>
50 appendix.
51 </para>
52
53 <section id='usingpoky-components-bitbake'>
54 <title>BitBake</title>
55
56 <para>
57 BitBake is the tool at the heart of the Yocto Project and is responsible
58 for parsing the metadata, generating a list of tasks from it,
59 and then executing those tasks.
60 To see a list of the options BitBake supports, use the following help command:
61 <literallayout class='monospaced'>
62 $ bitbake --help
63 </literallayout>
64 </para>
65
66 <para>
67 The most common usage for BitBake is <filename>bitbake &lt;packagename&gt;</filename>, where
68 <filename>packagename</filename> is the name of the package you want to build
69 (referred to as the "target" in this manual).
70 The target often equates to the first part of a <filename>.bb</filename> filename.
71 So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
72 might type the following:
73 <literallayout class='monospaced'>
74 $ bitbake matchbox-desktop
75 </literallayout>
76 Several different versions of <filename>matchbox-desktop</filename> might exist.
77 BitBake chooses the one selected by the distribution configuration.
78 You can get more details about how BitBake chooses between different
79 target versions and providers in the
80 <link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
81 </para>
82
83 <para>
84 BitBake also tries to execute any dependent tasks first.
85 So for example, before building <filename>matchbox-desktop</filename>, BitBake
86 would build a cross compiler and <filename>eglibc</filename> if they had not already
87 been built.
88 <note>This release of the Yocto Project does not support the <filename>glibc</filename>
89 GNU version of the Unix standard C library. By default, the Yocto Project builds with
90 <filename>eglibc</filename>.</note>
91 </para>
92
93 <para>
94 A useful BitBake option to consider is the <filename>-k</filename> or
95 <filename>--continue</filename> option.
96 This option instructs BitBake to try and continue processing the job as much
97 as possible even after encountering an error.
98 When an error occurs, the target that
99 failed and those that depend on it cannot be remade.
100 However, when you use this option other dependencies can still be processed.
101 </para>
102 </section>
103
104 <section id='usingpoky-components-metadata'>
105 <title>Metadata (Recipes)</title>
106
107 <para>
108 The <filename>.bb</filename> files are usually referred to as "recipes."
109 In general, a recipe contains information about a single piece of software.
110 The information includes the location from which to download the source patches
111 (if any are needed), which special configuration options to apply,
112 how to compile the source files, and how to package the compiled output.
113 </para>
114
115 <para>
116 The term "package" can also be used to describe recipes.
117 However, since the same word is used for the packaged output from the Yocto
118 Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
119 this document avoids using the term "package" to refer to recipes.
120 </para>
121 </section>
122
123 <section id='usingpoky-components-classes'>
124 <title>Classes</title>
125
126 <para>
127 Class files (<filename>.bbclass</filename>) contain information that is useful to share
128 between metadata files.
129 An example is the Autotools class, which contains
130 common settings for any application that Autotools uses.
131 The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
132 about common classes and how to use them.
133 </para>
134 </section>
135
136 <section id='usingpoky-components-configuration'>
137 <title>Configuration</title>
138
139 <para>
140 The configuration files (<filename>.conf</filename>) define various configuration variables
141 that govern the Yocto Project build process.
142 These files fall into several areas that define machine configuration options,
143 distribution configuration options, compiler tuning options, general common configuration
144 options and user configuration options (<filename>local.conf</filename>, which is found
145 in the Yocto Project files build directory).
146 </para>
147 </section>
148</section>
149
150
151<section id='usingpoky-build'> 10<section id='usingpoky-build'>
152 <title>Running a Build</title> 11 <title>Running a Build</title>
153 12
@@ -221,83 +80,6 @@
221 due to some components not being rebuilt. 80 due to some components not being rebuilt.
222 </para> 81 </para>
223 </section> 82 </section>
224
225 <section id="considering-shared-state-cache">
226 <title>Considering Shared State Cache</title>
227
228 <para>
229 By design, the Yocto Project builds everything from scratch unless it can determine that
230 a given task's inputs have not changed.
231 While building from scratch ensures that everything is current, it does also
232 mean that a lot of time could be spent rebuiding things that don't necessarily need built.
233 </para>
234
235 <para>
236 The Yocto Project build process uses a shared state caching scheme to avoid having to
237 rebuild software when it is not necessary.
238 Because the build time for a Yocto image can be significant, it is helpful to try and
239 determine what really needs built and what can be skipped given a particular project's
240 build process.
241 </para>
242
243 <para>
244 The scheme that the Yocto Project uses involves checksum generation and comparison for
245 a task's inputs.
246 The scheme also employs an area of memory called the shared state cache that is
247 pointed to by the <filename>SSTATE_DIR</filename> variable.
248 This area contains task output generated from a previous build.
249 If a given task's checksum matches the checksum of a previous build for the same
250 task, the build process uses the state of the cache rather than rerunning that
251 task.
252 </para>
253
254 <para>
255 The previous paragraph is a simplistic explanation of how the build process
256 uses checksums and shared state memory cache to avoide building tasks that
257 don't need built.
258 If you want a bit more explanation on the topic,
259 see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
260 State - What does it mean and why should I care?</ulink>" from the Yocto
261 Project discussion archives.
262 </para>
263
264 <para>
265 As with all schemes, this one has some drawbacks.
266 It is possible that you could make implicit changes that are not factored into the checksum
267 calculation, but do affect a task's output.
268 A good example is perhaps when a tool changes its output.
269 Let's say that the output of <filename>rpmdeps</filename> needed to change.
270 The result of the change should be that all the "package", "package_write_rpm",
271 and "package_deploy-rpm" sstate-cache items would become invalid.
272 But, because this is a change that is external to the code and therefore implicit,
273 the associated sstate-cache items do not become invalidated.
274 In this case, the build process would use the cache items rather than running the
275 task again.
276 Obviously, these types of implicit changes can cause problems.
277 </para>
278
279 <para>
280 To avoid these problems during the build, you need to understand the effects of any
281 change you make.
282 Note that any changes you make directly to a function automatically are factored into
283 the checksum calculation and thus, will invalidate the associated area of sstate cache.
284 You need to be aware of any implicit changes that are not obvious changes to the
285 code and could affect the output of a given task.
286 Once you are aware of such a change, you can take steps to invalidate the cache
287 and force the task to run.
288 The step to take is as simple as changing a function's comments in the source code.
289 For example, to invalidate package sstate files, change the comment statments
290 of <filename>do_package</filename> or one of the functions it calls.
291 The change is purely cosmetic, but it causes the checksum to be recalculated and
292 forces the task to be run again.
293 </para>
294
295 <note>
296 For an example of a commit that makes a cosmetic change to invalidate an sstate,
297 see this
298 <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
299 </note>
300 </section>
301</section> 83</section>
302 84
303<section id='usingpoky-install'> 85<section id='usingpoky-install'>