diff options
Diffstat (limited to 'documentation/ref-manual/technical-details.xml')
-rw-r--r-- | documentation/ref-manual/technical-details.xml | 1421 |
1 files changed, 1421 insertions, 0 deletions
diff --git a/documentation/ref-manual/technical-details.xml b/documentation/ref-manual/technical-details.xml new file mode 100644 index 0000000000..2df36521c2 --- /dev/null +++ b/documentation/ref-manual/technical-details.xml | |||
@@ -0,0 +1,1421 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | ||
3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | ||
4 | |||
5 | <chapter id='technical-details'> | ||
6 | <title>Technical Details</title> | ||
7 | |||
8 | <para> | ||
9 | This chapter provides technical details for various parts of the | ||
10 | Yocto Project. | ||
11 | Currently, topics include Yocto Project components, | ||
12 | cross-toolchain generation, shared state (sstate) cache, | ||
13 | x32, Wayland support, and Licenses. | ||
14 | </para> | ||
15 | |||
16 | <section id='usingpoky-components'> | ||
17 | <title>Yocto Project Components</title> | ||
18 | |||
19 | <para> | ||
20 | The | ||
21 | <ulink url='&YOCTO_DOCS_DEV_URL;#bitbake-term'>BitBake</ulink> | ||
22 | task executor together with various types of configuration files form | ||
23 | the OpenEmbedded Core. | ||
24 | This section overviews these components by describing their use and | ||
25 | how they interact. | ||
26 | </para> | ||
27 | |||
28 | <para> | ||
29 | BitBake handles the parsing and execution of the data files. | ||
30 | The data itself is of various types: | ||
31 | <itemizedlist> | ||
32 | <listitem><para><emphasis>Recipes:</emphasis> Provides details | ||
33 | about particular pieces of software. | ||
34 | </para></listitem> | ||
35 | <listitem><para><emphasis>Class Data:</emphasis> Abstracts | ||
36 | common build information (e.g. how to build a Linux kernel). | ||
37 | </para></listitem> | ||
38 | <listitem><para><emphasis>Configuration Data:</emphasis> Defines | ||
39 | machine-specific settings, policy decisions, and so forth. | ||
40 | Configuration data acts as the glue to bind everything | ||
41 | together. | ||
42 | </para></listitem> | ||
43 | </itemizedlist> | ||
44 | </para> | ||
45 | |||
46 | <para> | ||
47 | BitBake knows how to combine multiple data sources together and refers | ||
48 | to each data source as a layer. | ||
49 | For information on layers, see the | ||
50 | "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and | ||
51 | Creating Layers</ulink>" section of the Yocto Project Development Manual. | ||
52 | </para> | ||
53 | |||
54 | <para> | ||
55 | Following are some brief details on these core components. | ||
56 | For additional information on how these components interact during | ||
57 | a build, see the | ||
58 | "<link linkend='closer-look'>A Closer Look at the Yocto Project Development Environment</link>" | ||
59 | Chapter. | ||
60 | </para> | ||
61 | |||
62 | <section id='usingpoky-components-bitbake'> | ||
63 | <title>BitBake</title> | ||
64 | |||
65 | <para> | ||
66 | BitBake is the tool at the heart of the OpenEmbedded build system | ||
67 | and is responsible for parsing the | ||
68 | <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>, | ||
69 | generating a list of tasks from it, and then executing those tasks. | ||
70 | </para> | ||
71 | |||
72 | <para> | ||
73 | This section briefly introduces BitBake. | ||
74 | If you want more information on BitBake, see the | ||
75 | <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>. | ||
76 | </para> | ||
77 | |||
78 | <para> | ||
79 | To see a list of the options BitBake supports, use either of | ||
80 | the following commands: | ||
81 | <literallayout class='monospaced'> | ||
82 | $ bitbake -h | ||
83 | $ bitbake --help | ||
84 | </literallayout> | ||
85 | </para> | ||
86 | |||
87 | <para> | ||
88 | The most common usage for BitBake is <filename>bitbake <replaceable>packagename</replaceable></filename>, where | ||
89 | <filename>packagename</filename> is the name of the package you want to build | ||
90 | (referred to as the "target" in this manual). | ||
91 | The target often equates to the first part of a recipe's filename | ||
92 | (e.g. "foo" for a recipe named | ||
93 | <filename>foo_1.3.0-r0.bb</filename>). | ||
94 | So, to process the <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you | ||
95 | might type the following: | ||
96 | <literallayout class='monospaced'> | ||
97 | $ bitbake matchbox-desktop | ||
98 | </literallayout> | ||
99 | Several different versions of <filename>matchbox-desktop</filename> might exist. | ||
100 | BitBake chooses the one selected by the distribution configuration. | ||
101 | You can get more details about how BitBake chooses between different | ||
102 | target versions and providers in the | ||
103 | "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>" | ||
104 | section of the BitBake User Manual. | ||
105 | </para> | ||
106 | |||
107 | <para> | ||
108 | BitBake also tries to execute any dependent tasks first. | ||
109 | So for example, before building <filename>matchbox-desktop</filename>, BitBake | ||
110 | would build a cross compiler and <filename>glibc</filename> if they had not already | ||
111 | been built. | ||
112 | </para> | ||
113 | |||
114 | <para> | ||
115 | A useful BitBake option to consider is the <filename>-k</filename> or | ||
116 | <filename>--continue</filename> option. | ||
117 | This option instructs BitBake to try and continue processing the job | ||
118 | as long as possible even after encountering an error. | ||
119 | When an error occurs, the target that | ||
120 | failed and those that depend on it cannot be remade. | ||
121 | However, when you use this option other dependencies can still be | ||
122 | processed. | ||
123 | </para> | ||
124 | </section> | ||
125 | |||
126 | <section id='usingpoky-components-metadata'> | ||
127 | <title>Metadata (Recipes)</title> | ||
128 | |||
129 | <para> | ||
130 | Files that have the <filename>.bb</filename> suffix are "recipes" | ||
131 | files. | ||
132 | In general, a recipe contains information about a single piece of | ||
133 | software. | ||
134 | This information includes the location from which to download the | ||
135 | unaltered source, any source patches to be applied to that source | ||
136 | (if needed), which special configuration options to apply, | ||
137 | how to compile the source files, and how to package the compiled | ||
138 | output. | ||
139 | </para> | ||
140 | |||
141 | <para> | ||
142 | The term "package" is sometimes used to refer to recipes. However, | ||
143 | since the word "package" is used for the packaged output from the OpenEmbedded | ||
144 | build system (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), | ||
145 | this document avoids using the term "package" when referring to recipes. | ||
146 | </para> | ||
147 | </section> | ||
148 | |||
149 | <section id='usingpoky-components-classes'> | ||
150 | <title>Classes</title> | ||
151 | |||
152 | <para> | ||
153 | Class files (<filename>.bbclass</filename>) contain information that | ||
154 | is useful to share between | ||
155 | <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> files. | ||
156 | An example is the | ||
157 | <link linkend='ref-classes-autotools'><filename>autotools</filename></link> | ||
158 | class, which contains common settings for any application that | ||
159 | Autotools uses. | ||
160 | The "<link linkend='ref-classes'>Classes</link>" chapter provides | ||
161 | details about classes and how to use them. | ||
162 | </para> | ||
163 | </section> | ||
164 | |||
165 | <section id='usingpoky-components-configuration'> | ||
166 | <title>Configuration</title> | ||
167 | |||
168 | <para> | ||
169 | The configuration files (<filename>.conf</filename>) define various configuration variables | ||
170 | that govern the OpenEmbedded build process. | ||
171 | These files fall into several areas that define machine configuration options, | ||
172 | distribution configuration options, compiler tuning options, general common configuration | ||
173 | options, and user configuration options in <filename>local.conf</filename>, which is found | ||
174 | in the | ||
175 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>. | ||
176 | </para> | ||
177 | </section> | ||
178 | </section> | ||
179 | |||
180 | <section id="cross-development-toolchain-generation"> | ||
181 | <title>Cross-Development Toolchain Generation</title> | ||
182 | |||
183 | <para> | ||
184 | The Yocto Project does most of the work for you when it comes to | ||
185 | creating | ||
186 | <ulink url='&YOCTO_DOCS_DEV_URL;#cross-development-toolchain'>cross-development toolchains</ulink>. | ||
187 | This section provides some technical background on how | ||
188 | cross-development toolchains are created and used. | ||
189 | For more information on toolchains, you can also see the | ||
190 | <ulink url='&YOCTO_DOCS_ADT_URL;'>Yocto Project Application Developer's Guide</ulink>. | ||
191 | </para> | ||
192 | |||
193 | <para> | ||
194 | In the Yocto Project development environment, cross-development | ||
195 | toolchains are used to build the image and applications that run on the | ||
196 | target hardware. | ||
197 | With just a few commands, the OpenEmbedded build system creates | ||
198 | these necessary toolchains for you. | ||
199 | </para> | ||
200 | |||
201 | <para> | ||
202 | The following figure shows a high-level build environment regarding | ||
203 | toolchain construction and use. | ||
204 | </para> | ||
205 | |||
206 | <para> | ||
207 | <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" /> | ||
208 | </para> | ||
209 | |||
210 | <para> | ||
211 | Most of the work occurs on the Build Host. | ||
212 | This is the machine used to build images and generally work within the | ||
213 | the Yocto Project environment. | ||
214 | When you run BitBake to create an image, the OpenEmbedded build system | ||
215 | uses the host <filename>gcc</filename> compiler to bootstrap a | ||
216 | cross-compiler named <filename>gcc-cross</filename>. | ||
217 | The <filename>gcc-cross</filename> compiler is what BitBake uses to | ||
218 | compile source files when creating the target image. | ||
219 | You can think of <filename>gcc-cross</filename> simply as an | ||
220 | automatically generated cross-compiler that is used internally within | ||
221 | BitBake only. | ||
222 | </para> | ||
223 | |||
224 | <para> | ||
225 | The chain of events that occurs when <filename>gcc-cross</filename> is | ||
226 | bootstrapped is as follows: | ||
227 | <literallayout class='monospaced'> | ||
228 | gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime | ||
229 | </literallayout> | ||
230 | <itemizedlist> | ||
231 | <listitem><para><filename>gcc</filename>: | ||
232 | The build host's GNU Compiler Collection (GCC). | ||
233 | </para></listitem> | ||
234 | <listitem><para><filename>binutils-cross</filename>: | ||
235 | The bare minimum binary utilities needed in order to run | ||
236 | the <filename>gcc-cross-initial</filename> phase of the | ||
237 | bootstrap operation. | ||
238 | </para></listitem> | ||
239 | <listitem><para><filename>gcc-cross-initial</filename>: | ||
240 | An early stage of the bootstrap process for creating | ||
241 | the cross-compiler. | ||
242 | This stage builds enough of the <filename>gcc-cross</filename>, | ||
243 | the C library, and other pieces needed to finish building the | ||
244 | final cross-compiler in later stages. | ||
245 | This tool is a "native" package (i.e. it is designed to run on | ||
246 | the build host). | ||
247 | </para></listitem> | ||
248 | <listitem><para><filename>linux-libc-headers</filename>: | ||
249 | Headers needed for the cross-compiler. | ||
250 | </para></listitem> | ||
251 | <listitem><para><filename>glibc-initial</filename>: | ||
252 | An initial version of the Embedded GLIBC needed to bootstrap | ||
253 | <filename>glibc</filename>. | ||
254 | </para></listitem> | ||
255 | <listitem><para><filename>gcc-cross</filename>: | ||
256 | The final stage of the bootstrap process for the | ||
257 | cross-compiler. | ||
258 | This stage results in the actual cross-compiler that | ||
259 | BitBake uses when it builds an image for a targeted | ||
260 | device. | ||
261 | <note> | ||
262 | If you are replacing this cross compiler toolchain | ||
263 | with a custom version, you must replace | ||
264 | <filename>gcc-cross</filename>. | ||
265 | </note> | ||
266 | This tool is also a "native" package (i.e. it is | ||
267 | designed to run on the build host). | ||
268 | </para></listitem> | ||
269 | <listitem><para><filename>gcc-runtime</filename>: | ||
270 | Runtime libraries resulting from the toolchain bootstrapping | ||
271 | process. | ||
272 | This tool produces a binary that consists of the | ||
273 | runtime libraries need for the targeted device. | ||
274 | </para></listitem> | ||
275 | </itemizedlist> | ||
276 | </para> | ||
277 | |||
278 | <para> | ||
279 | You can use the OpenEmbedded build system to build an installer for | ||
280 | the relocatable SDK used to develop applications. | ||
281 | When you run the installer, it installs the toolchain, which contains | ||
282 | the development tools (e.g., the | ||
283 | <filename>gcc-cross-canadian</filename>), | ||
284 | <filename>binutils-cross-canadian</filename>, and other | ||
285 | <filename>nativesdk-*</filename> tools you need to cross-compile and | ||
286 | test your software. | ||
287 | The figure shows the commands you use to easily build out this | ||
288 | toolchain. | ||
289 | This cross-development toolchain is built to execute on the | ||
290 | <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>, | ||
291 | which might or might not be the same | ||
292 | machine as the Build Host. | ||
293 | <note> | ||
294 | If your target architecture is supported by the Yocto Project, | ||
295 | you can take advantage of pre-built images that ship with the | ||
296 | Yocto Project and already contain cross-development toolchain | ||
297 | installers. | ||
298 | </note> | ||
299 | </para> | ||
300 | |||
301 | <para> | ||
302 | Here is the bootstrap process for the relocatable toolchain: | ||
303 | <literallayout class='monospaced'> | ||
304 | gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> | ||
305 | glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian | ||
306 | </literallayout> | ||
307 | <itemizedlist> | ||
308 | <listitem><para><filename>gcc</filename>: | ||
309 | The build host's GNU Compiler Collection (GCC). | ||
310 | </para></listitem> | ||
311 | <listitem><para><filename>binutils-crosssdk</filename>: | ||
312 | The bare minimum binary utilities needed in order to run | ||
313 | the <filename>gcc-crosssdk-initial</filename> phase of the | ||
314 | bootstrap operation. | ||
315 | </para></listitem> | ||
316 | <listitem><para><filename>gcc-crosssdk-initial</filename>: | ||
317 | An early stage of the bootstrap process for creating | ||
318 | the cross-compiler. | ||
319 | This stage builds enough of the | ||
320 | <filename>gcc-crosssdk</filename> and supporting pieces so that | ||
321 | the final stage of the bootstrap process can produce the | ||
322 | finished cross-compiler. | ||
323 | This tool is a "native" binary that runs on the build host. | ||
324 | </para></listitem> | ||
325 | <listitem><para><filename>linux-libc-headers</filename>: | ||
326 | Headers needed for the cross-compiler. | ||
327 | </para></listitem> | ||
328 | <listitem><para><filename>glibc-initial</filename>: | ||
329 | An initial version of the Embedded GLIBC needed to bootstrap | ||
330 | <filename>nativesdk-glibc</filename>. | ||
331 | </para></listitem> | ||
332 | <listitem><para><filename>nativesdk-glibc</filename>: | ||
333 | The Embedded GLIBC needed to bootstrap the | ||
334 | <filename>gcc-crosssdk</filename>. | ||
335 | </para></listitem> | ||
336 | <listitem><para><filename>gcc-crosssdk</filename>: | ||
337 | The final stage of the bootstrap process for the | ||
338 | relocatable cross-compiler. | ||
339 | The <filename>gcc-crosssdk</filename> is a transitory compiler | ||
340 | and never leaves the build host. | ||
341 | Its purpose is to help in the bootstrap process to create the | ||
342 | eventual relocatable <filename>gcc-cross-canadian</filename> | ||
343 | compiler, which is relocatable. | ||
344 | This tool is also a "native" package (i.e. it is | ||
345 | designed to run on the build host). | ||
346 | </para></listitem> | ||
347 | <listitem><para><filename>gcc-cross-canadian</filename>: | ||
348 | The final relocatable cross-compiler. | ||
349 | When run on the | ||
350 | <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>, | ||
351 | this tool | ||
352 | produces executable code that runs on the target device. | ||
353 | Only one cross-canadian compiler is produced per architecture | ||
354 | since they can be targeted at different processor optimizations | ||
355 | using configurations passed to the compiler through the | ||
356 | compile commands. | ||
357 | This circumvents the need for multiple compilers and thus | ||
358 | reduces the size of the toolchains. | ||
359 | </para></listitem> | ||
360 | </itemizedlist> | ||
361 | </para> | ||
362 | |||
363 | <note> | ||
364 | For information on advantages gained when building a | ||
365 | cross-development toolchain installer, see the | ||
366 | "<ulink url='&YOCTO_DOCS_ADT_URL;#optionally-building-a-toolchain-installer'>Optionally Building a Toolchain Installer</ulink>" | ||
367 | section in the Yocto Project Application Developer's Guide. | ||
368 | </note> | ||
369 | </section> | ||
370 | |||
371 | <section id="shared-state-cache"> | ||
372 | <title>Shared State Cache</title> | ||
373 | |||
374 | <para> | ||
375 | By design, the OpenEmbedded build system builds everything from scratch unless | ||
376 | BitBake can determine that parts do not need to be rebuilt. | ||
377 | Fundamentally, building from scratch is attractive as it means all parts are | ||
378 | built fresh and there is no possibility of stale data causing problems. | ||
379 | When developers hit problems, they typically default back to building from scratch | ||
380 | so they know the state of things from the start. | ||
381 | </para> | ||
382 | |||
383 | <para> | ||
384 | Building an image from scratch is both an advantage and a disadvantage to the process. | ||
385 | As mentioned in the previous paragraph, building from scratch ensures that | ||
386 | everything is current and starts from a known state. | ||
387 | However, building from scratch also takes much longer as it generally means | ||
388 | rebuilding things that do not necessarily need to be rebuilt. | ||
389 | </para> | ||
390 | |||
391 | <para> | ||
392 | The Yocto Project implements shared state code that supports incremental builds. | ||
393 | The implementation of the shared state code answers the following questions that | ||
394 | were fundamental roadblocks within the OpenEmbedded incremental build support system: | ||
395 | <itemizedlist> | ||
396 | <listitem><para>What pieces of the system have changed and what pieces have | ||
397 | not changed?</para></listitem> | ||
398 | <listitem><para>How are changed pieces of software removed and replaced?</para></listitem> | ||
399 | <listitem><para>How are pre-built components that do not need to be rebuilt from scratch | ||
400 | used when they are available?</para></listitem> | ||
401 | </itemizedlist> | ||
402 | </para> | ||
403 | |||
404 | <para> | ||
405 | For the first question, the build system detects changes in the "inputs" to a given task by | ||
406 | creating a checksum (or signature) of the task's inputs. | ||
407 | If the checksum changes, the system assumes the inputs have changed and the task needs to be | ||
408 | rerun. | ||
409 | For the second question, the shared state (sstate) code tracks which tasks add which output | ||
410 | to the build process. | ||
411 | This means the output from a given task can be removed, upgraded or otherwise manipulated. | ||
412 | The third question is partly addressed by the solution for the second question | ||
413 | assuming the build system can fetch the sstate objects from remote locations and | ||
414 | install them if they are deemed to be valid. | ||
415 | </para> | ||
416 | |||
417 | <note> | ||
418 | The OpenEmbedded build system does not maintain | ||
419 | <link linkend='var-PR'><filename>PR</filename></link> information | ||
420 | as part of the shared state packages. | ||
421 | Consequently, considerations exist that affect maintaining shared | ||
422 | state feeds. | ||
423 | For information on how the OpenEmbedded build system | ||
424 | works with packages and can | ||
425 | track incrementing <filename>PR</filename> information, see the | ||
426 | "<ulink url='&YOCTO_DOCS_DEV_URL;#incrementing-a-package-revision-number'>Incrementing a Package Revision Number</ulink>" | ||
427 | section. | ||
428 | </note> | ||
429 | |||
430 | <para> | ||
431 | The rest of this section goes into detail about the overall incremental build | ||
432 | architecture, the checksums (signatures), shared state, and some tips and tricks. | ||
433 | </para> | ||
434 | |||
435 | <section id='overall-architecture'> | ||
436 | <title>Overall Architecture</title> | ||
437 | |||
438 | <para> | ||
439 | When determining what parts of the system need to be built, BitBake | ||
440 | works on a per-task basis rather than a per-recipe basis. | ||
441 | You might wonder why using a per-task basis is preferred over a per-recipe basis. | ||
442 | To help explain, consider having the IPK packaging backend enabled and then switching to DEB. | ||
443 | In this case, the | ||
444 | <link linkend='ref-tasks-install'><filename>do_install</filename></link> | ||
445 | and | ||
446 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> | ||
447 | task outputs are still valid. | ||
448 | However, with a per-recipe approach, the build would not include the | ||
449 | <filename>.deb</filename> files. | ||
450 | Consequently, you would have to invalidate the whole build and rerun it. | ||
451 | Rerunning everything is not the best solution. | ||
452 | Also, in this case, the core must be "taught" much about specific tasks. | ||
453 | This methodology does not scale well and does not allow users to easily add new tasks | ||
454 | in layers or as external recipes without touching the packaged-staging core. | ||
455 | </para> | ||
456 | </section> | ||
457 | |||
458 | <section id='checksums'> | ||
459 | <title>Checksums (Signatures)</title> | ||
460 | |||
461 | <para> | ||
462 | The shared state code uses a checksum, which is a unique signature of a task's | ||
463 | inputs, to determine if a task needs to be run again. | ||
464 | Because it is a change in a task's inputs that triggers a rerun, the process | ||
465 | needs to detect all the inputs to a given task. | ||
466 | For shell tasks, this turns out to be fairly easy because | ||
467 | the build process generates a "run" shell script for each task and | ||
468 | it is possible to create a checksum that gives you a good idea of when | ||
469 | the task's data changes. | ||
470 | </para> | ||
471 | |||
472 | <para> | ||
473 | To complicate the problem, there are things that should not be included in | ||
474 | the checksum. | ||
475 | First, there is the actual specific build path of a given task - | ||
476 | the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>. | ||
477 | It does not matter if the work directory changes because it should not | ||
478 | affect the output for target packages. | ||
479 | Also, the build process has the objective of making native or cross packages relocatable. | ||
480 | The checksum therefore needs to exclude <filename>WORKDIR</filename>. | ||
481 | The simplistic approach for excluding the work directory is to set | ||
482 | <filename>WORKDIR</filename> to some fixed value and create the checksum | ||
483 | for the "run" script. | ||
484 | </para> | ||
485 | |||
486 | <para> | ||
487 | Another problem results from the "run" scripts containing functions that | ||
488 | might or might not get called. | ||
489 | The incremental build solution contains code that figures out dependencies | ||
490 | between shell functions. | ||
491 | This code is used to prune the "run" scripts down to the minimum set, | ||
492 | thereby alleviating this problem and making the "run" scripts much more | ||
493 | readable as a bonus. | ||
494 | </para> | ||
495 | |||
496 | <para> | ||
497 | So far we have solutions for shell scripts. | ||
498 | What about Python tasks? | ||
499 | The same approach applies even though these tasks are more difficult. | ||
500 | The process needs to figure out what variables a Python function accesses | ||
501 | and what functions it calls. | ||
502 | Again, the incremental build solution contains code that first figures out | ||
503 | the variable and function dependencies, and then creates a checksum for the data | ||
504 | used as the input to the task. | ||
505 | </para> | ||
506 | |||
507 | <para> | ||
508 | Like the <filename>WORKDIR</filename> case, situations exist where dependencies | ||
509 | should be ignored. | ||
510 | For these cases, you can instruct the build process to ignore a dependency | ||
511 | by using a line like the following: | ||
512 | <literallayout class='monospaced'> | ||
513 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
514 | </literallayout> | ||
515 | This example ensures that the | ||
516 | <link linkend='var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></link> | ||
517 | variable does not | ||
518 | depend on the value of | ||
519 | <link linkend='var-MACHINE'><filename>MACHINE</filename></link>, | ||
520 | even if it does reference it. | ||
521 | </para> | ||
522 | |||
523 | <para> | ||
524 | Equally, there are cases where we need to add dependencies BitBake is not able to find. | ||
525 | You can accomplish this by using a line like the following: | ||
526 | <literallayout class='monospaced'> | ||
527 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
528 | </literallayout> | ||
529 | This example explicitly adds the <filename>MACHINE</filename> variable as a | ||
530 | dependency for <filename>PACKAGE_ARCHS</filename>. | ||
531 | </para> | ||
532 | |||
533 | <para> | ||
534 | Consider a case with in-line Python, for example, where BitBake is not | ||
535 | able to figure out dependencies. | ||
536 | When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake | ||
537 | produces output when it discovers something for which it cannot figure out | ||
538 | dependencies. | ||
539 | The Yocto Project team has currently not managed to cover those dependencies | ||
540 | in detail and is aware of the need to fix this situation. | ||
541 | </para> | ||
542 | |||
543 | <para> | ||
544 | Thus far, this section has limited discussion to the direct inputs into a task. | ||
545 | Information based on direct inputs is referred to as the "basehash" in the | ||
546 | code. | ||
547 | However, there is still the question of a task's indirect inputs - the | ||
548 | things that were already built and present in the | ||
549 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>. | ||
550 | The checksum (or signature) for a particular task needs to add the hashes | ||
551 | of all the tasks on which the particular task depends. | ||
552 | Choosing which dependencies to add is a policy decision. | ||
553 | However, the effect is to generate a master checksum that combines the basehash | ||
554 | and the hashes of the task's dependencies. | ||
555 | </para> | ||
556 | |||
557 | <para> | ||
558 | At the code level, there are a variety of ways both the basehash and the | ||
559 | dependent task hashes can be influenced. | ||
560 | Within the BitBake configuration file, we can give BitBake some extra information | ||
561 | to help it construct the basehash. | ||
562 | The following statement effectively results in a list of global variable | ||
563 | dependency excludes - variables never included in any checksum: | ||
564 | <literallayout class='monospaced'> | ||
565 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ | ||
566 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ | ||
567 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ | ||
568 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ | ||
569 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
570 | </literallayout> | ||
571 | The previous example excludes | ||
572 | <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link> | ||
573 | since that variable is actually constructed as a path within | ||
574 | <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on | ||
575 | the whitelist. | ||
576 | </para> | ||
577 | |||
578 | <para> | ||
579 | The rules for deciding which hashes of dependent tasks to include through | ||
580 | dependency chains are more complex and are generally accomplished with a | ||
581 | Python function. | ||
582 | The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples | ||
583 | of this and also illustrates how you can insert your own policy into the system | ||
584 | if so desired. | ||
585 | This file defines the two basic signature generators <filename>OE-Core</filename> | ||
586 | uses: "OEBasic" and "OEBasicHash". | ||
587 | By default, there is a dummy "noop" signature handler enabled in BitBake. | ||
588 | This means that behavior is unchanged from previous versions. | ||
589 | <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default | ||
590 | through this setting in the <filename>bitbake.conf</filename> file: | ||
591 | <literallayout class='monospaced'> | ||
592 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
593 | </literallayout> | ||
594 | The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the | ||
595 | "OEBasic" version but adds the task hash to the stamp files. | ||
596 | This results in any | ||
597 | <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> | ||
598 | change that changes the task hash, automatically | ||
599 | causing the task to be run again. | ||
600 | This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link> | ||
601 | values, and changes to Metadata automatically ripple across the build. | ||
602 | </para> | ||
603 | |||
604 | <para> | ||
605 | It is also worth noting that the end result of these signature generators is to | ||
606 | make some dependency and hash information available to the build. | ||
607 | This information includes: | ||
608 | <itemizedlist> | ||
609 | <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: | ||
610 | The base hashes for each task in the recipe. | ||
611 | </para></listitem> | ||
612 | <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: | ||
613 | The base hashes for each dependent task. | ||
614 | </para></listitem> | ||
615 | <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: | ||
616 | The task dependencies for each task. | ||
617 | </para></listitem> | ||
618 | <listitem><para><filename>BB_TASKHASH</filename>: | ||
619 | The hash of the currently running task. | ||
620 | </para></listitem> | ||
621 | </itemizedlist> | ||
622 | </para> | ||
623 | </section> | ||
624 | |||
625 | <section id='shared-state'> | ||
626 | <title>Shared State</title> | ||
627 | |||
628 | <para> | ||
629 | Checksums and dependencies, as discussed in the previous section, solve half the | ||
630 | problem of supporting a shared state. | ||
631 | The other part of the problem is being able to use checksum information during the build | ||
632 | and being able to reuse or rebuild specific components. | ||
633 | </para> | ||
634 | |||
635 | <para> | ||
636 | The | ||
637 | <link linkend='ref-classes-sstate'><filename>sstate</filename></link> | ||
638 | class is a relatively generic implementation of how to "capture" | ||
639 | a snapshot of a given task. | ||
640 | The idea is that the build process does not care about the source of a task's output. | ||
641 | Output could be freshly built or it could be downloaded and unpacked from | ||
642 | somewhere - the build process does not need to worry about its origin. | ||
643 | </para> | ||
644 | |||
645 | <para> | ||
646 | There are two types of output, one is just about creating a directory | ||
647 | in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>. | ||
648 | A good example is the output of either | ||
649 | <link linkend='ref-tasks-install'><filename>do_install</filename></link> | ||
650 | or | ||
651 | <link linkend='ref-tasks-package'><filename>do_package</filename></link>. | ||
652 | The other type of output occurs when a set of data is merged into a shared directory | ||
653 | tree such as the sysroot. | ||
654 | </para> | ||
655 | |||
656 | <para> | ||
657 | The Yocto Project team has tried to keep the details of the | ||
658 | implementation hidden in <filename>sstate</filename> class. | ||
659 | From a user's perspective, adding shared state wrapping to a task | ||
660 | is as simple as this | ||
661 | <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link> | ||
662 | example taken from the | ||
663 | <link linkend='ref-classes-deploy'><filename>deploy</filename></link> | ||
664 | class: | ||
665 | <literallayout class='monospaced'> | ||
666 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" | ||
667 | SSTATETASKS += "do_deploy" | ||
668 | do_deploy[sstate-name] = "deploy" | ||
669 | do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" | ||
670 | do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" | ||
671 | |||
672 | python do_deploy_setscene () { | ||
673 | sstate_setscene(d) | ||
674 | } | ||
675 | addtask do_deploy_setscene | ||
676 | do_deploy[dirs] = "${DEPLOYDIR} ${B}" | ||
677 | </literallayout> | ||
678 | In this example, we add some extra flags to the task, a name field ("deploy"), an | ||
679 | input directory where the task sends data, and the output | ||
680 | directory where the data from the task should eventually be copied. | ||
681 | We also add a <filename>_setscene</filename> variant of the task and add the task | ||
682 | name to the <filename>SSTATETASKS</filename> list. | ||
683 | </para> | ||
684 | |||
685 | <para> | ||
686 | If you have a directory whose contents you need to preserve, you can do this with | ||
687 | a line like the following: | ||
688 | <literallayout class='monospaced'> | ||
689 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | ||
690 | </literallayout> | ||
691 | This method, as well as the following example, also works for multiple directories. | ||
692 | <literallayout class='monospaced'> | ||
693 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" | ||
694 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" | ||
695 | do_package[sstate-lockfile] = "${PACKAGELOCK}" | ||
696 | </literallayout> | ||
697 | These methods also include the ability to take a lockfile when manipulating | ||
698 | shared state directory structures since some cases are sensitive to file | ||
699 | additions or removals. | ||
700 | </para> | ||
701 | |||
702 | <para> | ||
703 | Behind the scenes, the shared state code works by looking in | ||
704 | <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and | ||
705 | <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link> | ||
706 | for shared state files. | ||
707 | Here is an example: | ||
708 | <literallayout class='monospaced'> | ||
709 | SSTATE_MIRRORS ?= "\ | ||
710 | file://.* http://someserver.tld/share/sstate/PATH \n \ | ||
711 | file://.* file:///some/local/dir/sstate/PATH" | ||
712 | </literallayout> | ||
713 | <note> | ||
714 | The shared state directory (<filename>SSTATE_DIR</filename>) is | ||
715 | organized into two-character subdirectories, where the subdirectory | ||
716 | names are based on the first two characters of the hash. | ||
717 | If the shared state directory structure for a mirror has the | ||
718 | same structure as <filename>SSTATE_DIR</filename>, you must | ||
719 | specify "PATH" as part of the URI to enable the build system | ||
720 | to map to the appropriate subdirectory. | ||
721 | </note> | ||
722 | </para> | ||
723 | |||
724 | <para> | ||
725 | The shared state package validity can be detected just by looking at the | ||
726 | filename since the filename contains the task checksum (or signature) as | ||
727 | described earlier in this section. | ||
728 | If a valid shared state package is found, the build process downloads it | ||
729 | and uses it to accelerate the task. | ||
730 | </para> | ||
731 | |||
732 | <para> | ||
733 | The build processes use the <filename>*_setscene</filename> tasks | ||
734 | for the task acceleration phase. | ||
735 | BitBake goes through this phase before the main execution code and tries | ||
736 | to accelerate any tasks for which it can find shared state packages. | ||
737 | If a shared state package for a task is available, the shared state | ||
738 | package is used. | ||
739 | This means the task and any tasks on which it is dependent are not | ||
740 | executed. | ||
741 | </para> | ||
742 | |||
743 | <para> | ||
744 | As a real world example, the aim is when building an IPK-based image, | ||
745 | only the | ||
746 | <link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link> | ||
747 | tasks would have their | ||
748 | shared state packages fetched and extracted. | ||
749 | Since the sysroot is not used, it would never get extracted. | ||
750 | This is another reason why a task-based approach is preferred over a | ||
751 | recipe-based approach, which would have to install the output from every task. | ||
752 | </para> | ||
753 | </section> | ||
754 | |||
755 | <section id='tips-and-tricks'> | ||
756 | <title>Tips and Tricks</title> | ||
757 | |||
758 | <para> | ||
759 | The code in the build system that supports incremental builds is not | ||
760 | simple code. | ||
761 | This section presents some tips and tricks that help you work around | ||
762 | issues related to shared state code. | ||
763 | </para> | ||
764 | |||
765 | <section id='debugging'> | ||
766 | <title>Debugging</title> | ||
767 | |||
768 | <para> | ||
769 | When things go wrong, debugging needs to be straightforward. | ||
770 | Because of this, the Yocto Project includes strong debugging | ||
771 | tools: | ||
772 | <itemizedlist> | ||
773 | <listitem><para>Whenever a shared state package is written out, so is a | ||
774 | corresponding <filename>.siginfo</filename> file. | ||
775 | This practice results in a pickled Python database of all | ||
776 | the metadata that went into creating the hash for a given shared state | ||
777 | package.</para></listitem> | ||
778 | <listitem><para>If you run BitBake with the <filename>--dump-signatures</filename> | ||
779 | (or <filename>-S</filename>) option, BitBake dumps out | ||
780 | <filename>.siginfo</filename> files in | ||
781 | the stamp directory for every task it would have executed instead of | ||
782 | building the specified target package.</para></listitem> | ||
783 | <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that | ||
784 | can process <filename>.siginfo</filename> files. | ||
785 | If you specify one of these files, BitBake dumps out the dependency | ||
786 | information in the file. | ||
787 | If you specify two files, BitBake compares the two files and dumps out | ||
788 | the differences between the two. | ||
789 | This more easily helps answer the question of "What | ||
790 | changed between X and Y?"</para></listitem> | ||
791 | </itemizedlist> | ||
792 | </para> | ||
793 | </section> | ||
794 | |||
795 | <section id='invalidating-shared-state'> | ||
796 | <title>Invalidating Shared State</title> | ||
797 | |||
798 | <para> | ||
799 | The OpenEmbedded build system uses checksums and shared state | ||
800 | cache to avoid unnecessarily rebuilding tasks. | ||
801 | Collectively, this scheme is known as "shared state code." | ||
802 | </para> | ||
803 | |||
804 | <para> | ||
805 | As with all schemes, this one has some drawbacks. | ||
806 | It is possible that you could make implicit changes to your | ||
807 | code that the checksum calculations do not take into | ||
808 | account. | ||
809 | These implicit changes affect a task's output but do not trigger | ||
810 | the shared state code into rebuilding a recipe. | ||
811 | Consider an example during which a tool changes its output. | ||
812 | Assume that the output of <filename>rpmdeps</filename> changes. | ||
813 | The result of the change should be that all the | ||
814 | <filename>package</filename> and | ||
815 | <filename>package_write_rpm</filename> shared state cache | ||
816 | items become invalid. | ||
817 | However, because the change to the output is | ||
818 | external to the code and therefore implicit, | ||
819 | the associated shared state cache items do not become | ||
820 | invalidated. | ||
821 | In this case, the build process uses the cached items rather | ||
822 | than running the task again. | ||
823 | Obviously, these types of implicit changes can cause problems. | ||
824 | </para> | ||
825 | |||
826 | <para> | ||
827 | To avoid these problems during the build, you need to | ||
828 | understand the effects of any changes you make. | ||
829 | Realize that changes you make directly to a function | ||
830 | are automatically factored into the checksum calculation. | ||
831 | Thus, these explicit changes invalidate the associated area of | ||
832 | shared state cache. | ||
833 | However, you need to be aware of any implicit changes that | ||
834 | are not obvious changes to the code and could affect the output | ||
835 | of a given task. | ||
836 | </para> | ||
837 | |||
838 | <para> | ||
839 | When you identify an implicit change, you can easily take steps | ||
840 | to invalidate the cache and force the tasks to run. | ||
841 | The steps you can take are as simple as changing a function's | ||
842 | comments in the source code. | ||
843 | For example, to invalidate package shared state files, change | ||
844 | the comment statements of | ||
845 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> | ||
846 | or the comments of one of the functions it calls. | ||
847 | Even though the change is purely cosmetic, it causes the | ||
848 | checksum to be recalculated and forces the OpenEmbedded build | ||
849 | system to run the task again. | ||
850 | </para> | ||
851 | |||
852 | <note> | ||
853 | For an example of a commit that makes a cosmetic change to | ||
854 | invalidate shared state, see this | ||
855 | <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. | ||
856 | </note> | ||
857 | </section> | ||
858 | </section> | ||
859 | </section> | ||
860 | |||
861 | <section id='x32'> | ||
862 | <title>x32</title> | ||
863 | |||
864 | <para> | ||
865 | x32 is a processor-specific Application Binary Interface (psABI) for x86_64. | ||
866 | An ABI defines the calling conventions between functions in a processing environment. | ||
867 | The interface determines what registers are used and what the sizes are for various C data types. | ||
868 | </para> | ||
869 | |||
870 | <para> | ||
871 | Some processing environments prefer using 32-bit applications even when running | ||
872 | on Intel 64-bit platforms. | ||
873 | Consider the i386 psABI, which is a very old 32-bit ABI for Intel 64-bit platforms. | ||
874 | The i386 psABI does not provide efficient use and access of the Intel 64-bit processor resources, | ||
875 | leaving the system underutilized. | ||
876 | Now consider the x86_64 psABI. | ||
877 | This ABI is newer and uses 64-bits for data sizes and program pointers. | ||
878 | The extra bits increase the footprint size of the programs, libraries, | ||
879 | and also increases the memory and file system size requirements. | ||
880 | Executing under the x32 psABI enables user programs to utilize CPU and system resources | ||
881 | more efficiently while keeping the memory footprint of the applications low. | ||
882 | Extra bits are used for registers but not for addressing mechanisms. | ||
883 | </para> | ||
884 | |||
885 | <section id='support'> | ||
886 | <title>Support</title> | ||
887 | |||
888 | <para> | ||
889 | This Yocto Project release supports the final specifications of x32 | ||
890 | psABI. | ||
891 | Support for x32 psABI exists as follows: | ||
892 | <itemizedlist> | ||
893 | <listitem><para>You can create packages and images in x32 psABI format on x86_64 architecture targets. | ||
894 | </para></listitem> | ||
895 | <listitem><para>You can successfully build many recipes with the x32 toolchain.</para></listitem> | ||
896 | <listitem><para>You can create and boot <filename>core-image-minimal</filename> and | ||
897 | <filename>core-image-sato</filename> images.</para></listitem> | ||
898 | </itemizedlist> | ||
899 | </para> | ||
900 | </section> | ||
901 | |||
902 | <section id='completing-x32'> | ||
903 | <title>Completing x32</title> | ||
904 | |||
905 | <para> | ||
906 | Future Plans for the x32 psABI in the Yocto Project include the following: | ||
907 | <itemizedlist> | ||
908 | <listitem><para>Enhance and fix the few remaining recipes so they | ||
909 | work with and support x32 toolchains.</para></listitem> | ||
910 | <listitem><para>Enhance RPM Package Manager (RPM) support for x32 binaries.</para></listitem> | ||
911 | <listitem><para>Support larger images.</para></listitem> | ||
912 | </itemizedlist> | ||
913 | </para> | ||
914 | </section> | ||
915 | |||
916 | <section id='using-x32-right-now'> | ||
917 | <title>Using x32 Right Now</title> | ||
918 | |||
919 | <para> | ||
920 | Follow these steps to use the x32 spABI: | ||
921 | <itemizedlist> | ||
922 | <listitem><para>Enable the x32 psABI tuning file for <filename>x86_64</filename> | ||
923 | machines by editing the <filename>conf/local.conf</filename> like this: | ||
924 | <literallayout class='monospaced'> | ||
925 | MACHINE = "qemux86-64" | ||
926 | DEFAULTTUNE = "x86-64-x32" | ||
927 | baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \ | ||
928 | or 'INVALID'), True) or 'lib'}" | ||
929 | #MACHINE = "genericx86" | ||
930 | #DEFAULTTUNE = "core2-64-x32" | ||
931 | </literallayout></para></listitem> | ||
932 | <listitem><para>As usual, use BitBake to build an image that supports the x32 psABI. | ||
933 | Here is an example: | ||
934 | <literallayout class='monospaced'> | ||
935 | $ bitbake core-image-sato | ||
936 | </literallayout></para></listitem> | ||
937 | <listitem><para>As usual, run your image using QEMU: | ||
938 | <literallayout class='monospaced'> | ||
939 | $ runqemu qemux86-64 core-image-sato | ||
940 | </literallayout></para></listitem> | ||
941 | </itemizedlist> | ||
942 | </para> | ||
943 | </section> | ||
944 | </section> | ||
945 | |||
946 | <section id="wayland"> | ||
947 | <title>Wayland</title> | ||
948 | |||
949 | <para> | ||
950 | <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink> | ||
951 | is a computer display server protocol that | ||
952 | provides a method for compositing window managers to communicate | ||
953 | directly with applications and video hardware and expects them to | ||
954 | communicate with input hardware using other libraries. | ||
955 | Using Wayland with supporting targets can result in better control | ||
956 | over graphics frame rendering than an application might otherwise | ||
957 | achieve. | ||
958 | </para> | ||
959 | |||
960 | <para> | ||
961 | The Yocto Project provides the Wayland protocol libraries and the | ||
962 | reference | ||
963 | <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink> | ||
964 | compositor as part of its release. | ||
965 | This section describes what you need to do to implement Wayland and | ||
966 | use the compositor when building an image for a supporting target. | ||
967 | </para> | ||
968 | |||
969 | <section id="wayland-support"> | ||
970 | <title>Support</title> | ||
971 | |||
972 | <para> | ||
973 | The Wayland protocol libraries and the reference Weston compositor | ||
974 | ship as integrated packages in the <filename>meta</filename> layer | ||
975 | of the | ||
976 | <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>. | ||
977 | Specifically, you can find the recipes that build both Wayland | ||
978 | and Weston at <filename>meta/recipes-graphics/wayland</filename>. | ||
979 | </para> | ||
980 | |||
981 | <para> | ||
982 | You can build both the Wayland and Weston packages for use only | ||
983 | with targets that accept the | ||
984 | <ulink url='http://dri.freedesktop.org/wiki/'>Mesa 3D and Direct Rendering Infrastructure</ulink>, | ||
985 | which is also known as Mesa DRI. | ||
986 | This implies that you cannot build and use the packages if your | ||
987 | target uses, for example, the | ||
988 | <trademark class='registered'>Intel</trademark> Embedded Media and | ||
989 | Graphics Driver (<trademark class='registered'>Intel</trademark> | ||
990 | EMGD) that overrides Mesa DRI. | ||
991 | </para> | ||
992 | |||
993 | <note> | ||
994 | Due to lack of EGL support, Weston 1.0.3 will not run directly on | ||
995 | the emulated QEMU hardware. | ||
996 | However, this version of Weston will run under X emulation without | ||
997 | issues. | ||
998 | </note> | ||
999 | </section> | ||
1000 | |||
1001 | <section id="enabling-wayland-in-an-image"> | ||
1002 | <title>Enabling Wayland in an Image</title> | ||
1003 | |||
1004 | <para> | ||
1005 | To enable Wayland, you need to enable it to be built and enable | ||
1006 | it to be included in the image. | ||
1007 | </para> | ||
1008 | |||
1009 | <section id="enable-building"> | ||
1010 | <title>Building</title> | ||
1011 | |||
1012 | <para> | ||
1013 | To cause Mesa to build the <filename>wayland-egl</filename> | ||
1014 | platform and Weston to build Wayland with Kernel Mode | ||
1015 | Setting | ||
1016 | (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>) | ||
1017 | support, include the "wayland" flag in the | ||
1018 | <link linkend="var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></link> | ||
1019 | statement in your <filename>local.conf</filename> file: | ||
1020 | <literallayout class='monospaced'> | ||
1021 | DISTRO_FEATURES_append = " wayland" | ||
1022 | </literallayout> | ||
1023 | </para> | ||
1024 | |||
1025 | <note> | ||
1026 | If X11 has been enabled elsewhere, Weston will build Wayland | ||
1027 | with X11 support | ||
1028 | </note> | ||
1029 | </section> | ||
1030 | |||
1031 | <section id="enable-installation-in-an-image"> | ||
1032 | <title>Installing</title> | ||
1033 | |||
1034 | <para> | ||
1035 | To install the Wayland feature into an image, you must | ||
1036 | include the following | ||
1037 | <link linkend='var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></link> | ||
1038 | statement in your <filename>local.conf</filename> file: | ||
1039 | <literallayout class='monospaced'> | ||
1040 | CORE_IMAGE_EXTRA_INSTALL += "wayland weston" | ||
1041 | </literallayout> | ||
1042 | </para> | ||
1043 | </section> | ||
1044 | </section> | ||
1045 | |||
1046 | <section id="running-weston"> | ||
1047 | <title>Running Weston</title> | ||
1048 | |||
1049 | <para> | ||
1050 | To run Weston inside X11, enabling it as described earlier and | ||
1051 | building a Sato image is sufficient. | ||
1052 | If you are running your image under Sato, a Weston Launcher appears | ||
1053 | in the "Utility" category. | ||
1054 | </para> | ||
1055 | |||
1056 | <para> | ||
1057 | Alternatively, you can run Weston through the command-line | ||
1058 | interpretor (CLI), which is better suited for development work. | ||
1059 | To run Weston under the CLI, you need to do the following after | ||
1060 | your image is built: | ||
1061 | <orderedlist> | ||
1062 | <listitem><para>Run these commands to export | ||
1063 | <filename>XDG_RUNTIME_DIR</filename>: | ||
1064 | <literallayout class='monospaced'> | ||
1065 | mkdir -p /tmp/$USER-weston | ||
1066 | chmod 0700 /tmp/$USER-weston | ||
1067 | export XDG_RUNTIME_DIR=/tmp/$USER-weston | ||
1068 | </literallayout></para></listitem> | ||
1069 | <listitem><para>Launch Weston in the shell: | ||
1070 | <literallayout class='monospaced'> | ||
1071 | weston | ||
1072 | </literallayout></para></listitem> | ||
1073 | </orderedlist> | ||
1074 | </para> | ||
1075 | </section> | ||
1076 | </section> | ||
1077 | |||
1078 | <section id="licenses"> | ||
1079 | <title>Licenses</title> | ||
1080 | |||
1081 | <para> | ||
1082 | This section describes the mechanism by which the OpenEmbedded build system | ||
1083 | tracks changes to licensing text. | ||
1084 | The section also describes how to enable commercially licensed recipes, | ||
1085 | which by default are disabled. | ||
1086 | </para> | ||
1087 | |||
1088 | <para> | ||
1089 | For information that can help you maintain compliance with various open | ||
1090 | source licensing during the lifecycle of the product, see the | ||
1091 | "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>" section | ||
1092 | in the Yocto Project Development Manual. | ||
1093 | </para> | ||
1094 | |||
1095 | <section id="usingpoky-configuring-LIC_FILES_CHKSUM"> | ||
1096 | <title>Tracking License Changes</title> | ||
1097 | |||
1098 | <para> | ||
1099 | The license of an upstream project might change in the future. | ||
1100 | In order to prevent these changes going unnoticed, the | ||
1101 | <filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename> | ||
1102 | variable tracks changes to the license text. The checksums are validated at the end of the | ||
1103 | configure step, and if the checksums do not match, the build will fail. | ||
1104 | </para> | ||
1105 | |||
1106 | <section id="usingpoky-specifying-LIC_FILES_CHKSUM"> | ||
1107 | <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title> | ||
1108 | |||
1109 | <para> | ||
1110 | The <filename>LIC_FILES_CHKSUM</filename> | ||
1111 | variable contains checksums of the license text in the source code for the recipe. | ||
1112 | Following is an example of how to specify <filename>LIC_FILES_CHKSUM</filename>: | ||
1113 | <literallayout class='monospaced'> | ||
1114 | LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \ | ||
1115 | file://licfile1.txt;beginline=5;endline=29;md5=yyyy \ | ||
1116 | file://licfile2.txt;endline=50;md5=zzzz \ | ||
1117 | ..." | ||
1118 | </literallayout> | ||
1119 | </para> | ||
1120 | |||
1121 | <para> | ||
1122 | The build system uses the | ||
1123 | <filename><link linkend='var-S'>S</link></filename> variable as | ||
1124 | the default directory when searching files listed in | ||
1125 | <filename>LIC_FILES_CHKSUM</filename>. | ||
1126 | The previous example employs the default directory. | ||
1127 | </para> | ||
1128 | |||
1129 | <para> | ||
1130 | Consider this next example: | ||
1131 | <literallayout class='monospaced'> | ||
1132 | LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\ | ||
1133 | md5=bb14ed3c4cda583abc85401304b5cd4e" | ||
1134 | LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6" | ||
1135 | </literallayout> | ||
1136 | </para> | ||
1137 | |||
1138 | <para> | ||
1139 | The first line locates a file in | ||
1140 | <filename>${S}/src/ls.c</filename>. | ||
1141 | The second line refers to a file in | ||
1142 | <filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>. | ||
1143 | </para> | ||
1144 | <para> | ||
1145 | Note that <filename>LIC_FILES_CHKSUM</filename> variable is | ||
1146 | mandatory for all recipes, unless the | ||
1147 | <filename>LICENSE</filename> variable is set to "CLOSED". | ||
1148 | </para> | ||
1149 | </section> | ||
1150 | |||
1151 | <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax"> | ||
1152 | <title>Explanation of Syntax</title> | ||
1153 | <para> | ||
1154 | As mentioned in the previous section, the | ||
1155 | <filename>LIC_FILES_CHKSUM</filename> variable lists all the | ||
1156 | important files that contain the license text for the source code. | ||
1157 | It is possible to specify a checksum for an entire file, or a specific section of a | ||
1158 | file (specified by beginning and ending line numbers with the "beginline" and "endline" | ||
1159 | parameters, respectively). | ||
1160 | The latter is useful for source files with a license notice header, | ||
1161 | README documents, and so forth. | ||
1162 | If you do not use the "beginline" parameter, then it is assumed that the text begins on the | ||
1163 | first line of the file. | ||
1164 | Similarly, if you do not use the "endline" parameter, it is assumed that the license text | ||
1165 | ends with the last line of the file. | ||
1166 | </para> | ||
1167 | |||
1168 | <para> | ||
1169 | The "md5" parameter stores the md5 checksum of the license text. | ||
1170 | If the license text changes in any way as compared to this parameter | ||
1171 | then a mismatch occurs. | ||
1172 | This mismatch triggers a build failure and notifies the developer. | ||
1173 | Notification allows the developer to review and address the license text changes. | ||
1174 | Also note that if a mismatch occurs during the build, the correct md5 | ||
1175 | checksum is placed in the build log and can be easily copied to the recipe. | ||
1176 | </para> | ||
1177 | |||
1178 | <para> | ||
1179 | There is no limit to how many files you can specify using the | ||
1180 | <filename>LIC_FILES_CHKSUM</filename> variable. | ||
1181 | Generally, however, every project requires a few specifications for license tracking. | ||
1182 | Many projects have a "COPYING" file that stores the license information for all the source | ||
1183 | code files. | ||
1184 | This practice allows you to just track the "COPYING" file as long as it is kept up to date. | ||
1185 | </para> | ||
1186 | |||
1187 | <tip> | ||
1188 | If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match | ||
1189 | error and displays the correct "md5" parameter value during the build. | ||
1190 | The correct parameter is also captured in the build log. | ||
1191 | </tip> | ||
1192 | |||
1193 | <tip> | ||
1194 | If the whole file contains only license text, you do not need to use the "beginline" and | ||
1195 | "endline" parameters. | ||
1196 | </tip> | ||
1197 | </section> | ||
1198 | </section> | ||
1199 | |||
1200 | <section id="enabling-commercially-licensed-recipes"> | ||
1201 | <title>Enabling Commercially Licensed Recipes</title> | ||
1202 | |||
1203 | <para> | ||
1204 | By default, the OpenEmbedded build system disables | ||
1205 | components that have commercial or other special licensing | ||
1206 | requirements. | ||
1207 | Such requirements are defined on a | ||
1208 | recipe-by-recipe basis through the | ||
1209 | <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link> | ||
1210 | variable definition in the affected recipe. | ||
1211 | For instance, the | ||
1212 | <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> | ||
1213 | recipe contains the following statement: | ||
1214 | <literallayout class='monospaced'> | ||
1215 | LICENSE_FLAGS = "commercial" | ||
1216 | </literallayout> | ||
1217 | Here is a slightly more complicated example that contains both an | ||
1218 | explicit recipe name and version (after variable expansion): | ||
1219 | <literallayout class='monospaced'> | ||
1220 | LICENSE_FLAGS = "license_${PN}_${PV}" | ||
1221 | </literallayout> | ||
1222 | In order for a component restricted by a <filename>LICENSE_FLAGS</filename> | ||
1223 | definition to be enabled and included in an image, it | ||
1224 | needs to have a matching entry in the global | ||
1225 | <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link> | ||
1226 | variable, which is a variable | ||
1227 | typically defined in your <filename>local.conf</filename> file. | ||
1228 | For example, to enable | ||
1229 | the <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> | ||
1230 | package, you could add either the string | ||
1231 | "commercial_gst-plugins-ugly" or the more general string | ||
1232 | "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>. | ||
1233 | See the | ||
1234 | "<link linkend='license-flag-matching'>License Flag Matching</link>" section | ||
1235 | for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works. | ||
1236 | Here is the example: | ||
1237 | <literallayout class='monospaced'> | ||
1238 | LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly" | ||
1239 | </literallayout> | ||
1240 | Likewise, to additionally enable the package built from the recipe containing | ||
1241 | <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming | ||
1242 | that the actual recipe name was <filename>emgd_1.10.bb</filename>, | ||
1243 | the following string would enable that package as well as | ||
1244 | the original <filename>gst-plugins-ugly</filename> package: | ||
1245 | <literallayout class='monospaced'> | ||
1246 | LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10" | ||
1247 | </literallayout> | ||
1248 | As a convenience, you do not need to specify the complete license string | ||
1249 | in the whitelist for every package. | ||
1250 | You can use an abbreviated form, which consists | ||
1251 | of just the first portion or portions of the license string before | ||
1252 | the initial underscore character or characters. | ||
1253 | A partial string will match | ||
1254 | any license that contains the given string as the first | ||
1255 | portion of its license. | ||
1256 | For example, the following | ||
1257 | whitelist string will also match both of the packages | ||
1258 | previously mentioned as well as any other packages that have | ||
1259 | licenses starting with "commercial" or "license". | ||
1260 | <literallayout class='monospaced'> | ||
1261 | LICENSE_FLAGS_WHITELIST = "commercial license" | ||
1262 | </literallayout> | ||
1263 | </para> | ||
1264 | |||
1265 | <section id="license-flag-matching"> | ||
1266 | <title>License Flag Matching</title> | ||
1267 | |||
1268 | <para> | ||
1269 | License flag matching allows you to control what recipes the | ||
1270 | OpenEmbedded build system includes in the build. | ||
1271 | Fundamentally, the build system attempts to match | ||
1272 | <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link> | ||
1273 | strings found in recipes against | ||
1274 | <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link> | ||
1275 | strings found in the whitelist. | ||
1276 | A match causes the build system to include a recipe in the | ||
1277 | build, while failure to find a match causes the build system to | ||
1278 | exclude a recipe. | ||
1279 | </para> | ||
1280 | |||
1281 | <para> | ||
1282 | In general, license flag matching is simple. | ||
1283 | However, understanding some concepts will help you | ||
1284 | correctly and effectively use matching. | ||
1285 | </para> | ||
1286 | |||
1287 | <para> | ||
1288 | Before a flag | ||
1289 | defined by a particular recipe is tested against the | ||
1290 | contents of the whitelist, the expanded string | ||
1291 | <filename>_${PN}</filename> is appended to the flag. | ||
1292 | This expansion makes each <filename>LICENSE_FLAGS</filename> | ||
1293 | value recipe-specific. | ||
1294 | After expansion, the string is then matched against the | ||
1295 | whitelist. | ||
1296 | Thus, specifying | ||
1297 | <filename>LICENSE_FLAGS = "commercial"</filename> | ||
1298 | in recipe "foo", for example, results in the string | ||
1299 | <filename>"commercial_foo"</filename>. | ||
1300 | And, to create a match, that string must appear in the | ||
1301 | whitelist. | ||
1302 | </para> | ||
1303 | |||
1304 | <para> | ||
1305 | Judicious use of the <filename>LICENSE_FLAGS</filename> | ||
1306 | strings and the contents of the | ||
1307 | <filename>LICENSE_FLAGS_WHITELIST</filename> variable | ||
1308 | allows you a lot of flexibility for including or excluding | ||
1309 | recipes based on licensing. | ||
1310 | For example, you can broaden the matching capabilities by | ||
1311 | using license flags string subsets in the whitelist. | ||
1312 | <note>When using a string subset, be sure to use the part of | ||
1313 | the expanded string that precedes the appended underscore | ||
1314 | character (e.g. <filename>usethispart_1.3</filename>, | ||
1315 | <filename>usethispart_1.4</filename>, and so forth). | ||
1316 | </note> | ||
1317 | For example, simply specifying the string "commercial" in | ||
1318 | the whitelist matches any expanded | ||
1319 | <filename>LICENSE_FLAGS</filename> definition that starts with | ||
1320 | the string "commercial" such as "commercial_foo" and | ||
1321 | "commercial_bar", which are the strings the build system | ||
1322 | automatically generates for hypothetical recipes named | ||
1323 | "foo" and "bar" assuming those recipes simply specify the | ||
1324 | following: | ||
1325 | <literallayout class='monospaced'> | ||
1326 | LICENSE_FLAGS = "commercial" | ||
1327 | </literallayout> | ||
1328 | Thus, you can choose to exhaustively | ||
1329 | enumerate each license flag in the whitelist and | ||
1330 | allow only specific recipes into the image, or | ||
1331 | you can use a string subset that causes a broader range of | ||
1332 | matches to allow a range of recipes into the image. | ||
1333 | </para> | ||
1334 | |||
1335 | <para> | ||
1336 | This scheme works even if the | ||
1337 | <filename>LICENSE_FLAGS</filename> string already | ||
1338 | has <filename>_${PN}</filename> appended. | ||
1339 | For example, the build system turns the license flag | ||
1340 | "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would | ||
1341 | match both the general "commercial" and the specific | ||
1342 | "commercial_1.2_foo" strings found in the whitelist, as | ||
1343 | expected. | ||
1344 | </para> | ||
1345 | |||
1346 | <para> | ||
1347 | Here are some other scenarios: | ||
1348 | <itemizedlist> | ||
1349 | <listitem><para>You can specify a versioned string in the | ||
1350 | recipe such as "commercial_foo_1.2" in a "foo" recipe. | ||
1351 | The build system expands this string to | ||
1352 | "commercial_foo_1.2_foo". | ||
1353 | Combine this license flag with a whitelist that has | ||
1354 | the string "commercial" and you match the flag along | ||
1355 | with any other flag that starts with the string | ||
1356 | "commercial".</para></listitem> | ||
1357 | <listitem><para>Under the same circumstances, you can | ||
1358 | use "commercial_foo" in the whitelist and the | ||
1359 | build system not only matches "commercial_foo_1.2" but | ||
1360 | also matches any license flag with the string | ||
1361 | "commercial_foo", regardless of the version. | ||
1362 | </para></listitem> | ||
1363 | <listitem><para>You can be very specific and use both the | ||
1364 | package and version parts in the whitelist (e.g. | ||
1365 | "commercial_foo_1.2") to specifically match a | ||
1366 | versioned recipe.</para></listitem> | ||
1367 | </itemizedlist> | ||
1368 | </para> | ||
1369 | </section> | ||
1370 | |||
1371 | <section id="other-variables-related-to-commercial-licenses"> | ||
1372 | <title>Other Variables Related to Commercial Licenses</title> | ||
1373 | |||
1374 | <para> | ||
1375 | Other helpful variables related to commercial | ||
1376 | license handling exist and are defined in the | ||
1377 | <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file: | ||
1378 | <literallayout class='monospaced'> | ||
1379 | COMMERCIAL_AUDIO_PLUGINS ?= "" | ||
1380 | COMMERCIAL_VIDEO_PLUGINS ?= "" | ||
1381 | COMMERCIAL_QT = "" | ||
1382 | </literallayout> | ||
1383 | If you want to enable these components, you can do so by making sure you have | ||
1384 | statements similar to the following | ||
1385 | in your <filename>local.conf</filename> configuration file: | ||
1386 | <literallayout class='monospaced'> | ||
1387 | COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \ | ||
1388 | gst-plugins-ugly-mpegaudioparse" | ||
1389 | COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \ | ||
1390 | gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse" | ||
1391 | COMMERCIAL_QT ?= "qmmp" | ||
1392 | LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp" | ||
1393 | </literallayout> | ||
1394 | Of course, you could also create a matching whitelist | ||
1395 | for those components using the more general "commercial" | ||
1396 | in the whitelist, but that would also enable all the | ||
1397 | other packages with | ||
1398 | <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link> | ||
1399 | containing "commercial", which you may or may not want: | ||
1400 | <literallayout class='monospaced'> | ||
1401 | LICENSE_FLAGS_WHITELIST = "commercial" | ||
1402 | </literallayout> | ||
1403 | </para> | ||
1404 | |||
1405 | <para> | ||
1406 | Specifying audio and video plug-ins as part of the | ||
1407 | <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and | ||
1408 | <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements | ||
1409 | or commercial Qt components as part of | ||
1410 | the <filename>COMMERCIAL_QT</filename> statement (along | ||
1411 | with the enabling <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the | ||
1412 | plug-ins or components into built images, thus adding | ||
1413 | support for media formats or components. | ||
1414 | </para> | ||
1415 | </section> | ||
1416 | </section> | ||
1417 | </section> | ||
1418 | </chapter> | ||
1419 | <!-- | ||
1420 | vim: expandtab tw=80 ts=4 | ||
1421 | --> | ||