summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
-rw-r--r--documentation/poky-ref-manual/technical-details.xml141
1 files changed, 68 insertions, 73 deletions
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml
index b279289643..f045b8f552 100644
--- a/documentation/poky-ref-manual/technical-details.xml
+++ b/documentation/poky-ref-manual/technical-details.xml
@@ -1,5 +1,7 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" 1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> 2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
3<chapter id='technical-details'> 5<chapter id='technical-details'>
4<title>Technical Details</title> 6<title>Technical Details</title>
5 7
@@ -31,10 +33,8 @@
31 Configuration data acts as the glue to bind everything together.</para></listitem> 33 Configuration data acts as the glue to bind everything together.</para></listitem>
32 </itemizedlist> 34 </itemizedlist>
33 For more information on data, see the 35 For more information on data, see the
34 <ulink url='http://www.yoctoproject.org/docs/1.1.1//dev-manual/dev-manual.html#yocto-project-terms'> 36 "<ulink url='&YOCTO_DOCS_DEV_URL;#yocto-project-terms'>Yocto Project Terms</ulink>"
35 Yocto Project Terms</ulink> section in 37 section in the Yocto Project Development Manual.
36 <ulink url='http://www.yoctoproject.org/docs/1.1.1//dev-manual/dev-manual.html'>
37 The Yocto Project Development Manual</ulink>.
38 </para> 38 </para>
39 39
40 <para> 40 <para>
@@ -45,8 +45,7 @@
45 <para> 45 <para>
46 Following are some brief details on these core components. 46 Following are some brief details on these core components.
47 For more detailed information on these components see the 47 For more detailed information on these components see the
48 <link linkend='ref-structure'>'Reference: Directory Structure'</link> 48 "<link linkend='ref-structure'>Reference: Directory Structure</link>" appendix.
49 appendix.
50 </para> 49 </para>
51 50
52 <section id='usingpoky-components-bitbake'> 51 <section id='usingpoky-components-bitbake'>
@@ -76,7 +75,7 @@
76 BitBake chooses the one selected by the distribution configuration. 75 BitBake chooses the one selected by the distribution configuration.
77 You can get more details about how BitBake chooses between different 76 You can get more details about how BitBake chooses between different
78 target versions and providers in the 77 target versions and providers in the
79 <link linkend='ref-bitbake-providers'>Preferences and Providers</link> section. 78 "<link linkend='ref-bitbake-providers'>Preferences and Providers</link>" section.
80 </para> 79 </para>
81 80
82 <para> 81 <para>
@@ -115,7 +114,7 @@
115 The term "package" can also be used to describe recipes. 114 The term "package" can also be used to describe recipes.
116 However, since the same word is used for the packaged output from the Yocto 115 However, since the same word is used for the packaged output from the Yocto
117 Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), 116 Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
118 this document avoids using the term "package" when refering to recipes. 117 this document avoids using the term "package" when referring to recipes.
119 </para> 118 </para>
120 </section> 119 </section>
121 120
@@ -127,7 +126,7 @@
127 between metadata files. 126 between metadata files.
128 An example is the Autotools class, which contains 127 An example is the Autotools class, which contains
129 common settings for any application that Autotools uses. 128 common settings for any application that Autotools uses.
130 The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details 129 The "<link linkend='ref-classes'>Reference: Classes</link>" appendix provides details
131 about common classes and how to use them. 130 about common classes and how to use them.
132 </para> 131 </para>
133 </section> 132 </section>
@@ -163,7 +162,7 @@
163 As mentioned in the previous paragraph, building from scratch ensures that 162 As mentioned in the previous paragraph, building from scratch ensures that
164 everything is current and starts from a known state. 163 everything is current and starts from a known state.
165 However, building from scratch also takes much longer as it generally means 164 However, building from scratch also takes much longer as it generally means
166 rebuiding things that don't necessarily need rebuilt. 165 rebuilding things that don't necessarily need rebuilt.
167 </para> 166 </para>
168 167
169 <para> 168 <para>
@@ -239,7 +238,7 @@
239 affect the output for target packages. 238 affect the output for target packages.
240 Also, the build process has the objective of making native/cross packages relocatable. 239 Also, the build process has the objective of making native/cross packages relocatable.
241 The checksum therefore needs to exclude <filename>WORKDIR</filename>. 240 The checksum therefore needs to exclude <filename>WORKDIR</filename>.
242 The simplistic approach for excluding the worknig directory is to set 241 The simplistic approach for excluding the working directory is to set
243 <filename>WORKDIR</filename> to some fixed value and create the checksum 242 <filename>WORKDIR</filename> to some fixed value and create the checksum
244 for the "run" script. 243 for the "run" script.
245 </para> 244 </para>
@@ -299,77 +298,73 @@
299 298
300 <para> 299 <para>
301 Thus far, this section has limited discussion to the direct inputs into a task. 300 Thus far, this section has limited discussion to the direct inputs into a task.
302 Information based on direct inputs is referred to as the "basehash" in the code. 301 Information based on direct inputs is referred to as the "basehash" in the
303 However, there is still the question of a task's indirect inputs, the things that 302 code.
304 were already built and present in the build directory. 303 However, there is still the question of a task's indirect inputs - the
305 The checksum (or signature) for a particular task needs to add the hashes of all the 304 things that were already built and present in the build directory.
306 tasks on which the particular task depends. 305 The checksum (or signature) for a particular task needs to add the hashes
307 Choosing which dependencies to add is a policy decision. 306 of all the tasks on which the particular task depends.
308 However, the effect is to generate a master checksum that combines the 307 Choosing which dependencies to add is a policy decision.
309 basehash and the hashes of the task's dependencies. 308 However, the effect is to generate a master checksum that combines the basehash
309 and the hashes of the task's dependencies.
310 </para> 310 </para>
311 311
312 <para> 312 <para>
313 While figuring out the dependencies and creating these checksums is good, 313 At the code level, there are a variety of ways both the basehash and the
314 what does the Yocto Project build system do with the checksum information? 314 dependent task hashes can be influenced.
315 The build system uses a signature handler that is responsible for 315 Within the BitBake configuration file, we can give BitBake some extra information
316 processing the checksum information. 316 to help it construct the basehash.
317 By default, there is a dummy "noop" signature handler enabled in BitBake. 317 The following statements effectively result in a list of global variable
318 This means that behaviour is unchanged from previous versions. 318 dependency excludes - variables never included in any checksum:
319 OECore uses the "basic" signature handler through this setting in the
320 <filename>bitbake.conf</filename> file:
321 <literallayout class='monospaced'>
322 BB_SIGNATURE_HANDLER ?= "basic"
323 </literallayout>
324 Also within the BitBake configuration file, we can give BitBake
325 some extra information to help it handle this information.
326 The following statements effectively result in a list of global
327 variable dependency excludes - variables never included in
328 any checksum:
329 <literallayout class='monospaced'> 319 <literallayout class='monospaced'>
330 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" 320 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH"
331 BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS" 321 BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS"
332 BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER" 322 BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER"
333 BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET" 323 BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET"
334 BB_HASHTASK_WHITELIST += "(.*-cross$|.*-native$|.*-cross-initial$| \
335 .*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
336 </literallayout> 324 </literallayout>
337 This example is actually where <filename>WORKDIR</filename> 325 The previous example actually excludes
338 is excluded since <filename>WORKDIR</filename> is constructed as a 326 <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
339 path within <filename>TMPDIR</filename>, which is on the whitelist. 327 since it is actually constructed as a path within
328 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on
329 the whitelist.
340 </para> 330 </para>
341 331
342 <para> 332 <para>
343 The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and 333 The rules for deciding which hashes of dependent tasks to include through
344 excludes certain kinds of tasks from the dependency chains. 334 dependency chains are more complex and are generally accomplished with a
345 The effect of the previous example is to isolate the native, target, 335 python function.
346 and cross-components. 336 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
347 So, for example, toolchain changes do not force a rebuild of the whole system. 337 of this and also illustrates how you can insert your own policy into the system
338 if so desired.
339 This file defines the two basic signature generators <filename>OE-Core</filename>
340 uses: "OEBasic" and "OEBasicHash".
341 By default, there is a dummy "noop" signature handler enabled in BitBake.
342 This means that behavior is unchanged from previous versions.
343 <filename>OE-Core</filename> uses the "OEBasic" signature handler by default
344 through this setting in the <filename>bitbake.conf</filename> file:
345 <literallayout class='monospaced'>
346 BB_SIGNATURE_HANDLER ?= "OEBasic"
347 </literallayout>
348 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
349 "OEBasic" version but adds the task hash to the stamp files.
350 This results in any metadata change that changes the task hash, automatically
351 causing the task to be run again.
352 This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>
353 values and changes to metadata automatically ripple across the build.
354 Currently, this behavior is not the default behavior for <filename>OE-Core</filename>
355 but is the default in <filename>poky</filename>.
348 </para> 356 </para>
349 357
350 <para> 358 <para>
351 The end result of the "basic" handler is to make some dependency and 359 It is also worth noting that the end result of these signature generators is to
352 hash information available to the build. 360 make some dependency and hash information available to the build.
353 This includes: 361 This information includes:
354 <literallayout class='monospaced'> 362 <literallayout class='monospaced'>
355 BB_BASEHASH_task-&lt;taskname&gt; - the base hashes for each task in the recipe 363 BB_BASEHASH_task-&lt;taskname&gt; - the base hashes for each task in the recipe
356 BB_BASEHASH_&lt;filename:taskname&gt; - the base hashes for each dependent task 364 BB_BASEHASH_&lt;filename:taskname&gt; - the base hashes for each dependent task
357 BBHASHDEPS_&lt;filename:taskname&gt; - The task dependencies for each task 365 BBHASHDEPS_&lt;filename:taskname&gt; - The task dependencies for each task
358 BB_TASKHASH - the hash of the currently running task 366 BB_TASKHASH - the hash of the currently running task
359 </literallayout> 367 </literallayout>
360 There is also a "basichash" <filename>BB_SIGNATURE_HANDLER</filename>,
361 which is the same as the basic version but adds the task hash to the stamp files.
362 This results in any metadata change that changes the task hash,
363 automatically causing the task to be run again.
364 This removes the need to bump <filename>PR</filename>
365 values and changes to metadata automatically ripple across the build.
366 Currently, this behavior is not the default behavior.
367 However, it is likely that the Yocto Project team will go forward with this
368 behavior in the future since all the functionality exists.
369 The reason for the delay is the potential impact to the distribution feed
370 creation as they need increasing <filename>PR</filename> fields
371 and the Yocto Project currently lacks a mechanism to automate incrementing
372 this field.
373 </para> 368 </para>
374 </section> 369 </section>
375 370
@@ -431,7 +426,7 @@
431 <literallayout class='monospaced'> 426 <literallayout class='monospaced'>
432 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" 427 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
433 </literallayout> 428 </literallayout>
434 This method, as well as the following example, also works for mutliple directories. 429 This method, as well as the following example, also works for multiple directories.
435 <literallayout class='monospaced'> 430 <literallayout class='monospaced'>
436 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" 431 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
437 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" 432 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
@@ -553,7 +548,7 @@
553 Once you are aware of such a change, you can take steps to invalidate the cache 548 Once you are aware of such a change, you can take steps to invalidate the cache
554 and force the task to run. 549 and force the task to run.
555 The step to take is as simple as changing a function's comments in the source code. 550 The step to take is as simple as changing a function's comments in the source code.
556 For example, to invalidate package shared state files, change the comment statments 551 For example, to invalidate package shared state files, change the comment statements
557 of <filename>do_package</filename> or the comments of one of the functions it calls. 552 of <filename>do_package</filename> or the comments of one of the functions it calls.
558 The change is purely cosmetic, but it causes the checksum to be recalculated and 553 The change is purely cosmetic, but it causes the checksum to be recalculated and
559 forces the task to be run again. 554 forces the task to be run again.
@@ -562,7 +557,7 @@
562 <note> 557 <note>
563 For an example of a commit that makes a cosmetic change to invalidate 558 For an example of a commit that makes a cosmetic change to invalidate
564 a shared state, see this 559 a shared state, see this
565 <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. 560 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
566 </note> 561 </note>
567 </section> 562 </section>
568 </section> 563 </section>