summaryrefslogtreecommitdiffstats
path: root/documentation/poky-ref-manual/technical-details.xml
diff options
context:
space:
mode:
authorScott Rifenbark <scott.m.rifenbark@intel.com>2012-12-11 12:24:29 -0600
committerRichard Purdie <richard.purdie@linuxfoundation.org>2013-01-07 14:43:26 +0000
commit8753c6b2888cbe64760800cba1e55e4ce53309d2 (patch)
tree7b026a799e528737fb5bfed2cc73f61788a00feb /documentation/poky-ref-manual/technical-details.xml
parentbb8e9d0599bdaef032741078820490385ea6b0c3 (diff)
downloadpoky-8753c6b2888cbe64760800cba1e55e4ce53309d2.tar.gz
Documentation: ref-manual - removing old poky-ref-manual files
Removed the old poky-ref-manuals from the new ref-manual structure. (From yocto-docs rev: b5db4ddea205875ed3acacb90f46efd557337e0d) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/poky-ref-manual/technical-details.xml')
-rw-r--r--documentation/poky-ref-manual/technical-details.xml1011
1 files changed, 0 insertions, 1011 deletions
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml
deleted file mode 100644
index b1d7c40799..0000000000
--- a/documentation/poky-ref-manual/technical-details.xml
+++ /dev/null
@@ -1,1011 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='technical-details'>
6<title>Technical Details</title>
7
8 <para>
9 This chapter provides technical details for various parts of the Yocto Project.
10 Currently, topics include Yocto Project components and shared state (sstate) cache.
11 </para>
12
13<section id='usingpoky-components'>
14 <title>Yocto Project Components</title>
15
16 <para>
17 The BitBake task executor together with various types of configuration files form the
18 OpenEmbedded Core.
19 This section overviews the BitBake task executor and the
20 configuration files by describing what they are used for and how they interact.
21 </para>
22
23 <para>
24 BitBake handles the parsing and execution of the data files.
25 The data itself is of various types:
26 <itemizedlist>
27 <listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
28 pieces of software</para></listitem>
29 <listitem><para><emphasis>Class Data:</emphasis> An abstraction of common build
30 information (e.g. how to build a Linux kernel).</para></listitem>
31 <listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
32 policy decisions, etc.
33 Configuration data acts as the glue to bind everything together.</para></listitem>
34 </itemizedlist>
35 For more information on data, see the
36 "<ulink url='&YOCTO_DOCS_DEV_URL;#yocto-project-terms'>Yocto Project Terms</ulink>"
37 section in the Yocto Project Development Manual.
38 </para>
39
40 <para>
41 BitBake knows how to combine multiple data sources together and refers to each data source
42 as a layer.
43 For information on layers, see the
44 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and
45 Creating Layers</ulink>" section of the Yocto Project Development Manual.
46 </para>
47
48 <para>
49 Following are some brief details on these core components.
50 For more detailed information on these components see the
51 "<link linkend='ref-structure'>Directory Structure</link>" chapter.
52 </para>
53
54 <section id='usingpoky-components-bitbake'>
55 <title>BitBake</title>
56
57 <para>
58 BitBake is the tool at the heart of the OpenEmbedded build system and is responsible
59 for parsing the metadata, generating a list of tasks from it,
60 and then executing those tasks.
61 To see a list of the options BitBake supports, use the following help command:
62 <literallayout class='monospaced'>
63 $ bitbake --help
64 </literallayout>
65 </para>
66
67 <para>
68 The most common usage for BitBake is <filename>bitbake &lt;packagename&gt;</filename>, where
69 <filename>packagename</filename> is the name of the package you want to build
70 (referred to as the "target" in this manual).
71 The target often equates to the first part of a <filename>.bb</filename> filename.
72 So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
73 might type the following:
74 <literallayout class='monospaced'>
75 $ bitbake matchbox-desktop
76 </literallayout>
77 Several different versions of <filename>matchbox-desktop</filename> might exist.
78 BitBake chooses the one selected by the distribution configuration.
79 You can get more details about how BitBake chooses between different
80 target versions and providers in the
81 "<link linkend='ref-bitbake-providers'>Preferences and Providers</link>" section.
82 </para>
83
84 <para>
85 BitBake also tries to execute any dependent tasks first.
86 So for example, before building <filename>matchbox-desktop</filename>, BitBake
87 would build a cross compiler and <filename>eglibc</filename> if they had not already
88 been built.
89 <note>This release of the Yocto Project does not support the <filename>glibc</filename>
90 GNU version of the Unix standard C library. By default, the OpenEmbedded build system
91 builds with <filename>eglibc</filename>.</note>
92 </para>
93
94 <para>
95 A useful BitBake option to consider is the <filename>-k</filename> or
96 <filename>--continue</filename> option.
97 This option instructs BitBake to try and continue processing the job as much
98 as possible even after encountering an error.
99 When an error occurs, the target that
100 failed and those that depend on it cannot be remade.
101 However, when you use this option other dependencies can still be processed.
102 </para>
103 </section>
104
105 <section id='usingpoky-components-metadata'>
106 <title>Metadata (Recipes)</title>
107
108 <para>
109 The <filename>.bb</filename> files are usually referred to as "recipes."
110 In general, a recipe contains information about a single piece of software.
111 The information includes the location from which to download the source patches
112 (if any are needed), which special configuration options to apply,
113 how to compile the source files, and how to package the compiled output.
114 </para>
115
116 <para>
117 The term "package" can also be used to describe recipes.
118 However, since the same word is used for the packaged output from the OpenEmbedded
119 build system (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
120 this document avoids using the term "package" when referring to recipes.
121 </para>
122 </section>
123
124 <section id='usingpoky-components-classes'>
125 <title>Classes</title>
126
127 <para>
128 Class files (<filename>.bbclass</filename>) contain information that is useful to share
129 between metadata files.
130 An example is the Autotools class, which contains
131 common settings for any application that Autotools uses.
132 The "<link linkend='ref-classes'>Classes</link>" chapter provides details
133 about common classes and how to use them.
134 </para>
135 </section>
136
137 <section id='usingpoky-components-configuration'>
138 <title>Configuration</title>
139
140 <para>
141 The configuration files (<filename>.conf</filename>) define various configuration variables
142 that govern the OpenEmbedded build process.
143 These files fall into several areas that define machine configuration options,
144 distribution configuration options, compiler tuning options, general common configuration
145 options and user configuration options (<filename>local.conf</filename>, which is found
146 in the <ulink url='build-directory'>Build Directory</ulink>).
147 </para>
148 </section>
149</section>
150
151<section id="shared-state-cache">
152 <title>Shared State Cache</title>
153
154 <para>
155 By design, the OpenEmbedded build system builds everything from scratch unless
156 BitBake can determine that parts don't need to be rebuilt.
157 Fundamentally, building from scratch is attractive as it means all parts are
158 built fresh and there is no possibility of stale data causing problems.
159 When developers hit problems, they typically default back to building from scratch
160 so they know the state of things from the start.
161 </para>
162
163 <para>
164 Building an image from scratch is both an advantage and a disadvantage to the process.
165 As mentioned in the previous paragraph, building from scratch ensures that
166 everything is current and starts from a known state.
167 However, building from scratch also takes much longer as it generally means
168 rebuilding things that don't necessarily need rebuilt.
169 </para>
170
171 <para>
172 The Yocto Project implements shared state code that supports incremental builds.
173 The implementation of the shared state code answers the following questions that
174 were fundamental roadblocks within the OpenEmbedded incremental build support system:
175 <itemizedlist>
176 <listitem>What pieces of the system have changed and what pieces have not changed?</listitem>
177 <listitem>How are changed pieces of software removed and replaced?</listitem>
178 <listitem>How are pre-built components that don't need to be rebuilt from scratch
179 used when they are available?</listitem>
180 </itemizedlist>
181 </para>
182
183 <para>
184 For the first question, the build system detects changes in the "inputs" to a given task by
185 creating a checksum (or signature) of the task's inputs.
186 If the checksum changes, the system assumes the inputs have changed and the task needs to be
187 rerun.
188 For the second question, the shared state (sstate) code tracks which tasks add which output
189 to the build process.
190 This means the output from a given task can be removed, upgraded or otherwise manipulated.
191 The third question is partly addressed by the solution for the second question
192 assuming the build system can fetch the sstate objects from remote locations and
193 install them if they are deemed to be valid.
194 </para>
195
196 <para>
197 The rest of this section goes into detail about the overall incremental build
198 architecture, the checksums (signatures), shared state, and some tips and tricks.
199 </para>
200
201 <section id='overall-architecture'>
202 <title>Overall Architecture</title>
203
204 <para>
205 When determining what parts of the system need to be built, BitBake
206 uses a per-task basis and does not use a per-recipe basis.
207 You might wonder why using a per-task basis is preferred over a per-recipe basis.
208 To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
209 In this case, <filename>do_install</filename> and <filename>do_package</filename>
210 output are still valid.
211 However, with a per-recipe approach, the build would not include the
212 <filename>.deb</filename> files.
213 Consequently, you would have to invalidate the whole build and rerun it.
214 Rerunning everything is not the best situation.
215 Also in this case, the core must be "taught" much about specific tasks.
216 This methodology does not scale well and does not allow users to easily add new tasks
217 in layers or as external recipes without touching the packaged-staging core.
218 </para>
219 </section>
220
221 <section id='checksums'>
222 <title>Checksums (Signatures)</title>
223
224 <para>
225 The shared state code uses a checksum, which is a unique signature of a task's
226 inputs, to determine if a task needs to be run again.
227 Because it is a change in a task's inputs that triggers a rerun, the process
228 needs to detect all the inputs to a given task.
229 For shell tasks, this turns out to be fairly easy because
230 the build process generates a "run" shell script for each task and
231 it is possible to create a checksum that gives you a good idea of when
232 the task's data changes.
233 </para>
234
235 <para>
236 To complicate the problem, there are things that should not be included in
237 the checksum.
238 First, there is the actual specific build path of a given task -
239 the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
240 It does not matter if the working directory changes because it should not
241 affect the output for target packages.
242 Also, the build process has the objective of making native/cross packages relocatable.
243 The checksum therefore needs to exclude <filename>WORKDIR</filename>.
244 The simplistic approach for excluding the working directory is to set
245 <filename>WORKDIR</filename> to some fixed value and create the checksum
246 for the "run" script.
247 </para>
248
249 <para>
250 Another problem results from the "run" scripts containing functions that
251 might or might not get called.
252 The incremental build solution contains code that figures out dependencies
253 between shell functions.
254 This code is used to prune the "run" scripts down to the minimum set,
255 thereby alleviating this problem and making the "run" scripts much more
256 readable as a bonus.
257 </para>
258
259 <para>
260 So far we have solutions for shell scripts.
261 What about python tasks?
262 The same approach applies even though these tasks are more difficult.
263 The process needs to figure out what variables a python function accesses
264 and what functions it calls.
265 Again, the incremental build solution contains code that first figures out
266 the variable and function dependencies, and then creates a checksum for the data
267 used as the input to the task.
268 </para>
269
270 <para>
271 Like the <filename>WORKDIR</filename> case, situations exist where dependencies
272 should be ignored.
273 For these cases, you can instruct the build process to ignore a dependency
274 by using a line like the following:
275 <literallayout class='monospaced'>
276 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
277 </literallayout>
278 This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
279 depend on the value of <filename>MACHINE</filename>, even if it does reference it.
280 </para>
281
282 <para>
283 Equally, there are cases where we need to add dependencies BitBake is not able to find.
284 You can accomplish this by using a line like the following:
285 <literallayout class='monospaced'>
286 PACKAGE_ARCHS[vardeps] = "MACHINE"
287 </literallayout>
288 This example explicitly adds the <filename>MACHINE</filename> variable as a
289 dependency for <filename>PACKAGE_ARCHS</filename>.
290 </para>
291
292 <para>
293 Consider a case with inline python, for example, where BitBake is not
294 able to figure out dependencies.
295 When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
296 produces output when it discovers something for which it cannot figure out
297 dependencies.
298 The Yocto Project team has currently not managed to cover those dependencies
299 in detail and is aware of the need to fix this situation.
300 </para>
301
302 <para>
303 Thus far, this section has limited discussion to the direct inputs into a task.
304 Information based on direct inputs is referred to as the "basehash" in the
305 code.
306 However, there is still the question of a task's indirect inputs - the
307 things that were already built and present in the Build Directory.
308 The checksum (or signature) for a particular task needs to add the hashes
309 of all the tasks on which the particular task depends.
310 Choosing which dependencies to add is a policy decision.
311 However, the effect is to generate a master checksum that combines the basehash
312 and the hashes of the task's dependencies.
313 </para>
314
315 <para>
316 At the code level, there are a variety of ways both the basehash and the
317 dependent task hashes can be influenced.
318 Within the BitBake configuration file, we can give BitBake some extra information
319 to help it construct the basehash.
320 The following statements effectively result in a list of global variable
321 dependency excludes - variables never included in any checksum:
322 <literallayout class='monospaced'>
323 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH"
324 BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS"
325 BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER"
326 BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET"
327 </literallayout>
328 The previous example actually excludes
329 <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
330 since it is actually constructed as a path within
331 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on
332 the whitelist.
333 </para>
334
335 <para>
336 The rules for deciding which hashes of dependent tasks to include through
337 dependency chains are more complex and are generally accomplished with a
338 python function.
339 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
340 of this and also illustrates how you can insert your own policy into the system
341 if so desired.
342 This file defines the two basic signature generators <filename>OE-Core</filename>
343 uses: "OEBasic" and "OEBasicHash".
344 By default, there is a dummy "noop" signature handler enabled in BitBake.
345 This means that behavior is unchanged from previous versions.
346 <filename>OE-Core</filename> uses the "OEBasic" signature handler by default
347 through this setting in the <filename>bitbake.conf</filename> file:
348 <literallayout class='monospaced'>
349 BB_SIGNATURE_HANDLER ?= "OEBasic"
350 </literallayout>
351 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
352 "OEBasic" version but adds the task hash to the stamp files.
353 This results in any metadata change that changes the task hash, automatically
354 causing the task to be run again.
355 This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>
356 values and changes to metadata automatically ripple across the build.
357 Currently, this behavior is not the default behavior for <filename>OE-Core</filename>
358 but is the default in <filename>poky</filename>.
359 </para>
360
361 <para>
362 It is also worth noting that the end result of these signature generators is to
363 make some dependency and hash information available to the build.
364 This information includes:
365 <literallayout class='monospaced'>
366 BB_BASEHASH_task-&lt;taskname&gt; - the base hashes for each task in the recipe
367 BB_BASEHASH_&lt;filename:taskname&gt; - the base hashes for each dependent task
368 BBHASHDEPS_&lt;filename:taskname&gt; - The task dependencies for each task
369 BB_TASKHASH - the hash of the currently running task
370 </literallayout>
371 </para>
372 </section>
373
374 <section id='shared-state'>
375 <title>Shared State</title>
376
377 <para>
378 Checksums and dependencies, as discussed in the previous section, solve half the
379 problem.
380 The other part of the problem is being able to use checksum information during the build
381 and being able to reuse or rebuild specific components.
382 </para>
383
384 <para>
385 The shared state class (<filename>sstate.bbclass</filename>)
386 is a relatively generic implementation of how to "capture" a snapshot of a given task.
387 The idea is that the build process does not care about the source of a task's output.
388 Output could be freshly built or it could be downloaded and unpacked from
389 somewhere - the build process doesn't need to worry about its source.
390 </para>
391
392 <para>
393 There are two types of output, one is just about creating a directory
394 in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
395 A good example is the output of either <filename>do_install</filename> or
396 <filename>do_package</filename>.
397 The other type of output occurs when a set of data is merged into a shared directory
398 tree such as the sysroot.
399 </para>
400
401 <para>
402 The Yocto Project team has tried to keep the details of the implementation hidden in
403 <filename>sstate.bbclass</filename>.
404 From a user's perspective, adding shared state wrapping to a task
405 is as simple as this <filename>do_deploy</filename> example taken from
406 <filename>do_deploy.bbclass</filename>:
407 <literallayout class='monospaced'>
408 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
409 SSTATETASKS += "do_deploy"
410 do_deploy[sstate-name] = "deploy"
411 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
412 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
413
414 python do_deploy_setscene () {
415 sstate_setscene(d)
416 }
417 addtask do_deploy_setscene
418 </literallayout>
419 In the example, we add some extra flags to the task, a name field ("deploy"), an
420 input directory where the task sends data, and the output
421 directory where the data from the task should eventually be copied.
422 We also add a <filename>_setscene</filename> variant of the task and add the task
423 name to the <filename>SSTATETASKS</filename> list.
424 </para>
425
426 <para>
427 If you have a directory whose contents you need to preserve, you can do this with
428 a line like the following:
429 <literallayout class='monospaced'>
430 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
431 </literallayout>
432 This method, as well as the following example, also works for multiple directories.
433 <literallayout class='monospaced'>
434 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
435 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
436 do_package[sstate-lockfile] = "${PACKAGELOCK}"
437 </literallayout>
438 These methods also include the ability to take a lockfile when manipulating
439 shared state directory structures since some cases are sensitive to file
440 additions or removals.
441 </para>
442
443 <para>
444 Behind the scenes, the shared state code works by looking in
445 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and
446 <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
447 for shared state files.
448 Here is an example:
449 <literallayout class='monospaced'>
450 SSTATE_MIRRORS ?= "\
451 file://.* http://someserver.tld/share/sstate/PATH \n \
452 file://.* file:///some/local/dir/sstate/PATH"
453 </literallayout>
454 <note>
455 The shared state directory (<filename>SSTATE_DIR</filename>) is
456 organized into two-character subdirectories, where the subdirectory
457 names are based on the first two characters of the hash.
458 If the shared state directory structure for a mirror has the
459 same structure as <filename>SSTATE_DIR</filename>, you must
460 specify "PATH" as part of the URI to enable the build system
461 to map to the appropriate subdirectory.
462 </note>
463 </para>
464
465 <para>
466 The shared state package validity can be detected just by looking at the
467 filename since the filename contains the task checksum (or signature) as
468 described earlier in this section.
469 If a valid shared state package is found, the build process downloads it
470 and uses it to accelerate the task.
471 </para>
472
473 <para>
474 The build processes uses the <filename>*_setscene</filename> tasks
475 for the task acceleration phase.
476 BitBake goes through this phase before the main execution code and tries
477 to accelerate any tasks for which it can find shared state packages.
478 If a shared state package for a task is available, the shared state
479 package is used.
480 This means the task and any tasks on which it is dependent are not
481 executed.
482 </para>
483
484 <para>
485 As a real world example, the aim is when building an IPK-based image,
486 only the <filename>do_package_write_ipk</filename> tasks would have their
487 shared state packages fetched and extracted.
488 Since the sysroot is not used, it would never get extracted.
489 This is another reason why a task-based approach is preferred over a
490 recipe-based approach, which would have to install the output from every task.
491 </para>
492 </section>
493
494 <section id='tips-and-tricks'>
495 <title>Tips and Tricks</title>
496
497 <para>
498 The code in the build system that supports incremental builds is not
499 simple code.
500 This section presents some tips and tricks that help you work around
501 issues related to shared state code.
502 </para>
503
504 <section id='debugging'>
505 <title>Debugging</title>
506
507 <para>
508 When things go wrong, debugging needs to be straightforward.
509 Because of this, the Yocto Project team included strong debugging
510 tools:
511 <itemizedlist>
512 <listitem><para>Whenever a shared state package is written out, so is a
513 corresponding <filename>.siginfo</filename> file.
514 This practice results in a pickled python database of all
515 the metadata that went into creating the hash for a given shared state
516 package.</para></listitem>
517 <listitem><para>If BitBake is run with the <filename>--dump-signatures</filename>
518 (or <filename>-S</filename>) option, BitBake dumps out
519 <filename>.siginfo</filename> files in
520 the stamp directory for every task it would have executed instead of
521 building the specified target package.</para></listitem>
522 <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that
523 can process these <filename>.siginfo</filename> files.
524 If one file is specified, it will dump out the dependency
525 information in the file.
526 If two files are specified, it will compare the two files and dump out
527 the differences between the two.
528 This allows the question of "What changed between X and Y?" to be
529 answered easily.</para></listitem>
530 </itemizedlist>
531 </para>
532 </section>
533
534 <section id='invalidating-shared-state'>
535 <title>Invalidating Shared State</title>
536
537 <para>
538 The shared state code uses checksums and shared state
539 cache to avoid unnecessarily rebuilding tasks.
540 As with all schemes, this one has some drawbacks.
541 It is possible that you could make implicit changes that are not factored
542 into the checksum calculation, but do affect a task's output.
543 A good example is perhaps when a tool changes its output.
544 Let's say that the output of <filename>rpmdeps</filename> needed to change.
545 The result of the change should be that all the "package", "package_write_rpm",
546 and "package_deploy-rpm" shared state cache items would become invalid.
547 But, because this is a change that is external to the code and therefore implicit,
548 the associated shared state cache items do not become invalidated.
549 In this case, the build process would use the cached items rather than running the
550 task again.
551 Obviously, these types of implicit changes can cause problems.
552 </para>
553
554 <para>
555 To avoid these problems during the build, you need to understand the effects of any
556 change you make.
557 Note that any changes you make directly to a function automatically are factored into
558 the checksum calculation and thus, will invalidate the associated area of sstate cache.
559 You need to be aware of any implicit changes that are not obvious changes to the
560 code and could affect the output of a given task.
561 Once you are aware of such a change, you can take steps to invalidate the cache
562 and force the task to run.
563 The step to take is as simple as changing a function's comments in the source code.
564 For example, to invalidate package shared state files, change the comment statements
565 of <filename>do_package</filename> or the comments of one of the functions it calls.
566 The change is purely cosmetic, but it causes the checksum to be recalculated and
567 forces the task to be run again.
568 </para>
569
570 <note>
571 For an example of a commit that makes a cosmetic change to invalidate
572 a shared state, see this
573 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
574 </note>
575 </section>
576 </section>
577</section>
578
579<section id='x32'>
580 <title>x32</title>
581
582 <para>
583 x32 is a new processor-specific Application Binary Interface (psABI) for x86_64.
584 An ABI defines the calling conventions between functions in a processing environment.
585 The interface determines what registers are used and what the sizes are for various C data types.
586 </para>
587
588 <para>
589 Some processing environments prefer using 32-bit applications even when running
590 on Intel 64-bit platforms.
591 Consider the i386 psABI, which is a very old 32-bit ABI for Intel 64-bit platforms.
592 The i386 psABI does not provide efficient use and access of the Intel 64-bit processor resources,
593 leaving the system underutilized.
594 Now consider the x86_64 psABI.
595 This ABI is newer and uses 64-bits for data sizes and program pointers.
596 The extra bits increase the footprint size of the programs, libraries,
597 and also increases the memory and file system size requirements.
598 Executing under the x32 psABI enables user programs to utilize CPU and system resources
599 more efficiently while keeping the memory footprint of the applications low.
600 Extra bits are used for registers but not for addressing mechanisms.
601 </para>
602
603 <section id='support'>
604 <title>Support</title>
605
606 <para>
607 While the x32 psABI specifications are not fully finalized, this Yocto Project
608 release supports current development specifications of x32 psABI.
609 As of this release of the Yocto Project, x32 psABI support exists as follows:
610 <itemizedlist>
611 <listitem><para>You can create packages and images in x32 psABI format on x86_64 architecture targets.
612 </para></listitem>
613 <listitem><para>You can use the x32 psABI support through the <filename>meta-x32</filename>
614 layer on top of the OE-core/Yocto layer.</para></listitem>
615 <listitem><para>The toolchain from the <filename>experimental/meta-x32</filename> layer
616 is used for building x32 psABI program binaries.</para></listitem>
617 <listitem><para>You can successfully build many recipes with the x32 toolchain.</para></listitem>
618 <listitem><para>You can create and boot <filename>core-image-minimal</filename> and
619 <filename>core-image-sato</filename> images.</para></listitem>
620 </itemizedlist>
621 </para>
622 </section>
623
624 <section id='future-development-and-limitations'>
625 <title>Future Development and Limitations</title>
626
627 <para>
628 As of this Yocto Project release, the x32 psABI kernel and library interfaces
629 specifications are not finalized.
630 </para>
631
632 <para>
633 Future Plans for the x32 psABI in the Yocto Project include the following:
634 <itemizedlist>
635 <listitem><para>Enhance and fix the few remaining recipes so they
636 work with and support x32 toolchains.</para></listitem>
637 <listitem><para>Enhance RPM Package Manager (RPM) support for x32 binaries.</para></listitem>
638 <listitem><para>Support larger images.</para></listitem>
639 <listitem><para>Integrate x32 recipes, toolchain, and kernel changes from
640 <filename>experimental/meta-x32</filename> into OE-core.</para></listitem>
641 </itemizedlist>
642 </para>
643 </section>
644
645 <section id='using-x32-right-now'>
646 <title>Using x32 Right Now</title>
647
648 <para>
649 Despite the fact the x32 psABI support is in development state for this release of the
650 Yocto Project, you can follow these steps to use the x32 spABI:
651 <itemizedlist>
652 <listitem><para>Add the <filename>experimental/meta-x32</filename> layer to your local
653 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
654 You can find the <filename>experimental/meta-x32</filename> source repository at
655 <ulink url='&YOCTO_GIT_URL;'></ulink>.</para></listitem>
656 <listitem><para>Edit your <filename>conf/bblayers.conf</filename> file so that it includes
657 the <filename>meta-x32</filename>.
658 Here is an example:
659 <literallayout class='monospaced'>
660 BBLAYERS ?= " \
661 /home/nitin/prj/poky.git/meta \
662 /home/nitin/prj/poky.git/meta-yocto \
663 /home/nitin/prj/poky.git/meta-yocto-bsp \
664 /home/nitin/prj/meta-x32.git \
665 "
666 BBLAYERS_NON_REMOVABLE ?= " \
667 /home/nitin/prj/poky.git/meta \
668 /home/nitin/prj/poky.git/meta-yocto \
669 "
670 </literallayout></para></listitem>
671 <listitem><para>Enable the x32 psABI tuning file for <filename>x86_64</filename>
672 machines by editing the <filename>conf/local.conf</filename> like this:
673 <literallayout class='monospaced'>
674 MACHINE = "qemux86-64"
675 DEFAULTTUNE = "x86-64-x32"
676 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \
677 or 'INVALID'), True) or 'lib'}"
678 #MACHINE = "atom-pc"
679 #DEFAULTTUNE = "core2-64-x32"
680 </literallayout></para></listitem>
681 <listitem><para>As usual, use BitBake to build an image that supports the x32 psABI.
682 Here is an example:
683 <literallayout class='monospaced'>
684 $ bitake core-image-sato
685 </literallayout></para></listitem>
686 <listitem><para>As usual, run your image using QEMU:
687 <literallayout class='monospaced'>
688 $ runqemu qemux86-64 core-image-sato
689 </literallayout></para></listitem>
690 </itemizedlist>
691 </para>
692 </section>
693</section>
694
695<section id="licenses">
696 <title>Licenses</title>
697
698 <para>
699 This section describes the mechanism by which the OpenEmbedded build system
700 tracks changes to licensing text.
701 The section also describes how to enable commercially licensed recipes,
702 which by default are disabled.
703 </para>
704
705 <para>
706 For information that can help you maintain compliance with various open
707 source licensing during the lifecycle of the product, see the
708 "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>" section
709 in the Yocto Project Development Manual.
710 </para>
711
712 <section id="usingpoky-configuring-LIC_FILES_CHKSUM">
713 <title>Tracking License Changes</title>
714
715 <para>
716 The license of an upstream project might change in the future.
717 In order to prevent these changes going unnoticed, the
718 <filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename>
719 variable tracks changes to the license text. The checksums are validated at the end of the
720 configure step, and if the checksums do not match, the build will fail.
721 </para>
722
723 <section id="usingpoky-specifying-LIC_FILES_CHKSUM">
724 <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>
725
726 <para>
727 The <filename>LIC_FILES_CHKSUM</filename>
728 variable contains checksums of the license text in the source code for the recipe.
729 Following is an example of how to specify <filename>LIC_FILES_CHKSUM</filename>:
730 <literallayout class='monospaced'>
731 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
732 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
733 file://licfile2.txt;endline=50;md5=zzzz \
734 ..."
735 </literallayout>
736 </para>
737
738 <para>
739 The build system uses the
740 <filename><link linkend='var-S'>S</link></filename> variable as the
741 default directory used when searching files listed in
742 <filename>LIC_FILES_CHKSUM</filename>.
743 The previous example employs the default directory.
744 </para>
745
746 <para>
747 You can also use relative paths as shown in the following example:
748 <literallayout class='monospaced'>
749 LIC_FILES_CHKSUM = "file://src/ls.c;startline=5;endline=16;\
750 md5=bb14ed3c4cda583abc85401304b5cd4e"
751 LIC_FILES_CHKSUM = "file://../license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
752 </literallayout>
753 </para>
754
755 <para>
756 In this example, the first line locates a file in
757 <filename>${S}/src/ls.c</filename>.
758 The second line refers to a file in
759 <filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>, which is the parent
760 of <filename><link linkend='var-S'>S</link></filename>.
761 </para>
762 <para>
763 Note that this variable is mandatory for all recipes, unless the
764 <filename>LICENSE</filename> variable is set to "CLOSED".
765 </para>
766 </section>
767
768 <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
769 <title>Explanation of Syntax</title>
770 <para>
771 As mentioned in the previous section, the
772 <filename>LIC_FILES_CHKSUM</filename> variable lists all the
773 important files that contain the license text for the source code.
774 It is possible to specify a checksum for an entire file, or a specific section of a
775 file (specified by beginning and ending line numbers with the "beginline" and "endline"
776 parameters, respectively).
777 The latter is useful for source files with a license notice header,
778 README documents, and so forth.
779 If you do not use the "beginline" parameter, then it is assumed that the text begins on the
780 first line of the file.
781 Similarly, if you do not use the "endline" parameter, it is assumed that the license text
782 ends with the last line of the file.
783 </para>
784
785 <para>
786 The "md5" parameter stores the md5 checksum of the license text.
787 If the license text changes in any way as compared to this parameter
788 then a mismatch occurs.
789 This mismatch triggers a build failure and notifies the developer.
790 Notification allows the developer to review and address the license text changes.
791 Also note that if a mismatch occurs during the build, the correct md5
792 checksum is placed in the build log and can be easily copied to the recipe.
793 </para>
794
795 <para>
796 There is no limit to how many files you can specify using the
797 <filename>LIC_FILES_CHKSUM</filename> variable.
798 Generally, however, every project requires a few specifications for license tracking.
799 Many projects have a "COPYING" file that stores the license information for all the source
800 code files.
801 This practice allows you to just track the "COPYING" file as long as it is kept up to date.
802 </para>
803
804 <tip>
805 If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match
806 error and displays the correct "md5" parameter value during the build.
807 The correct parameter is also captured in the build log.
808 </tip>
809
810 <tip>
811 If the whole file contains only license text, you do not need to use the "beginline" and
812 "endline" parameters.
813 </tip>
814 </section>
815 </section>
816
817 <section id="enabling-commercially-licensed-recipes">
818 <title>Enabling Commercially Licensed Recipes</title>
819
820 <para>
821 By default, the OpenEmbedded build system disables
822 components that have commercial or other special licensing
823 requirements.
824 Such requirements are defined on a
825 recipe-by-recipe basis through the <filename>LICENSE_FLAGS</filename> variable
826 definition in the affected recipe.
827 For instance, the
828 <filename>$HOME/poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
829 recipe contains the following statement:
830 <literallayout class='monospaced'>
831 LICENSE_FLAGS = "commercial"
832 </literallayout>
833 Here is a slightly more complicated example that contains both an
834 explicit recipe name and version (after variable expansion):
835 <literallayout class='monospaced'>
836 LICENSE_FLAGS = "license_${PN}_${PV}"
837 </literallayout>
838 In order for a component restricted by a <filename>LICENSE_FLAGS</filename>
839 definition to be enabled and included in an image, it
840 needs to have a matching entry in the global
841 <filename>LICENSE_FLAGS_WHITELIST</filename> variable, which is a variable
842 typically defined in your <filename>local.conf</filename> file.
843 For example, to enable
844 the <filename>$HOME/poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
845 package, you could add either the string
846 "commercial_gst-plugins-ugly" or the more general string
847 "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.
848 See the
849 "<link linkend='license-flag-matching'>License Flag Matching</link>" section
850 for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works.
851 Here is the example:
852 <literallayout class='monospaced'>
853 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
854 </literallayout>
855 Likewise, to additionally enable the package built from the recipe containing
856 <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming
857 that the actual recipe name was <filename>emgd_1.10.bb</filename>,
858 the following string would enable that package as well as
859 the original <filename>gst-plugins-ugly</filename> package:
860 <literallayout class='monospaced'>
861 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
862 </literallayout>
863 As a convenience, you do not need to specify the complete license string
864 in the whitelist for every package.
865 you can use an abbreviated form, which consists
866 of just the first portion or portions of the license string before
867 the initial underscore character or characters.
868 A partial string will match
869 any license that contains the given string as the first
870 portion of its license.
871 For example, the following
872 whitelist string will also match both of the packages
873 previously mentioned as well as any other packages that have
874 licenses starting with "commercial" or "license".
875 <literallayout class='monospaced'>
876 LICENSE_FLAGS_WHITELIST = "commercial license"
877 </literallayout>
878 </para>
879
880 <section id="license-flag-matching">
881 <title>License Flag Matching</title>
882
883 <para>
884 The definition of 'matching' in reference to a
885 recipe's <filename>LICENSE_FLAGS</filename> setting is simple.
886 However, some things exist that you should know about in order to
887 correctly and effectively use it.
888 </para>
889
890 <para>
891 Before a flag
892 defined by a particular recipe is tested against the
893 contents of the <filename>LICENSE_FLAGS_WHITELIST</filename> variable, the
894 string <filename>_${PN}</filename> (with
895 <link linkend='var-PN'><filename>PN</filename></link> expanded of course) is
896 appended to the flag, thus automatically making each
897 <filename>LICENSE_FLAGS</filename> value recipe-specific.
898 That string is
899 then matched against the whitelist.
900 So if you specify <filename>LICENSE_FLAGS = "commercial"</filename> in recipe
901 "foo" for example, the string <filename>"commercial_foo"</filename>
902 would normally be what is specified in the whitelist in order for it to
903 match.
904 </para>
905
906 <para>
907 You can broaden the match by
908 putting any "_"-separated beginning subset of a
909 <filename>LICENSE_FLAGS</filename> flag in the whitelist, which will also
910 match.
911 For example, simply specifying "commercial" in
912 the whitelist would match any expanded <filename>LICENSE_FLAGS</filename>
913 definition starting with "commercial" such as
914 "commercial_foo" and "commercial_bar", which are the
915 strings that would be automatically generated for
916 hypothetical "foo" and "bar" recipes assuming those
917 recipes had simply specified the following:
918 <literallayout class='monospaced'>
919 LICENSE_FLAGS = "commercial"
920 </literallayout>
921 </para>
922
923 <para>
924 Broadening the match allows for a range of specificity for the items
925 in the whitelist, from more general to perfectly
926 specific.
927 So you have the choice of exhaustively
928 enumerating each license flag in the whitelist to
929 allow only those specific recipes into the image, or
930 of using a more general string to pick up anything
931 matching just the first component or components of the specified
932 string.
933 </para>
934
935 <para>
936 This scheme works even if the flag already
937 has <filename>_${PN}</filename> appended - the extra <filename>_${PN}</filename> is
938 redundant, but does not affect the outcome.
939 For example, a license flag of "commercial_1.2_foo" would
940 turn into "commercial_1.2_foo_foo" and would match
941 both the general "commercial" and the specific
942 "commercial_1.2_foo", as expected.
943 The flag would also match
944 "commercial_1.2_foo_foo" and "commercial_1.2", which
945 does not make much sense regarding use in the whitelist.
946 </para>
947
948 <para>
949 For a versioned string, you could instead specify
950 "commercial_foo_1.2", which would turn into
951 "commercial_foo_1.2_foo".
952 And, as expected, this flag allows
953 you to pick up this package along with
954 anything else "commercial" when you specify "commercial"
955 in the whitelist.
956 Or, the flag allows you to pick up this package along with anything "commercial_foo"
957 regardless of version when you use "commercial_foo" in the whitelist.
958 Finally, you can be completely specific about the package and version and specify
959 "commercial_foo_1.2" package and version.
960 </para>
961 </section>
962
963 <section id="other-variables-related-to-commercial-licenses">
964 <title>Other Variables Related to Commercial Licenses</title>
965
966 <para>
967 Other helpful variables related to commercial
968 license handling exist and are defined in the
969 <filename>$HOME/poky/meta/conf/distro/include/default-distrovars.inc</filename> file:
970 <literallayout class='monospaced'>
971 COMMERCIAL_AUDIO_PLUGINS ?= ""
972 COMMERCIAL_VIDEO_PLUGINS ?= ""
973 COMMERCIAL_QT = ""
974 </literallayout>
975 If you want to enable these components, you can do so by making sure you have
976 the following statements in your <filename>local.conf</filename> configuration file:
977 <literallayout class='monospaced'>
978 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
979 gst-plugins-ugly-mpegaudioparse"
980 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
981 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
982 COMMERCIAL_QT ?= "qmmp"
983 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
984 </literallayout>
985 Of course, you could also create a matching whitelist
986 for those components using the more general "commercial"
987 in the whitelist, but that would also enable all the
988 other packages with <filename>LICENSE_FLAGS</filename> containing
989 "commercial", which you may or may not want:
990 <literallayout class='monospaced'>
991 LICENSE_FLAGS_WHITELIST = "commercial"
992 </literallayout>
993 </para>
994
995 <para>
996 Specifying audio and video plug-ins as part of the
997 <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
998 <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements
999 or commercial qt components as part of
1000 the <filename>COMMERCIAL_QT</filename> statement (along
1001 with the enabling <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the
1002 plug-ins or components into built images, thus adding
1003 support for media formats or components.
1004 </para>
1005 </section>
1006 </section>
1007</section>
1008</chapter>
1009<!--
1010vim: expandtab tw=80 ts=4
1011-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-17 11:16:58 +0000