summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorScott Rifenbark <scott.m.rifenbark@intel.com>2011-12-15 08:22:30 -0800
committerRichard Purdie <richard.purdie@linuxfoundation.org>2011-12-16 16:58:41 +0000
commitb281fd127bac9ba77ab63a5c2b812ddd5d56df37 (patch)
tree39f6af0f880e046b82e5965af76f8c80ba2cdf8f
parentc7d27875880bd5b33efbec55947ac9e1c66b3d0e (diff)
downloadpoky-b281fd127bac9ba77ab63a5c2b812ddd5d56df37.tar.gz
documentation/poky-ref-manual/technical-details.xml: edits per Richard Purdie
Richard reviewed the new sections and had a couple comments. There was one technical error and he also wanted YP worked out when it was being used in the context of the code that was doing the work here. So I replaced with more generic terms or specifically called out BitBake as the responsible software. (From yocto-docs rev: 40343bf30b8492757f81bc0274d46931318f7120) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-rw-r--r--documentation/poky-ref-manual/technical-details.xml71
1 files changed, 33 insertions, 38 deletions
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml
index dfb5b726c0..9e40a66008 100644
--- a/documentation/poky-ref-manual/technical-details.xml
+++ b/documentation/poky-ref-manual/technical-details.xml
@@ -150,9 +150,9 @@
150 <title>Shared State Cache</title> 150 <title>Shared State Cache</title>
151 151
152 <para> 152 <para>
153 By design, the Yocto Project builds everything from scratch unless it can determine that 153 By design, the Yocto Project build system builds everything from scratch unless
154 parts don't need to be rebuilt. 154 BitBake can determine that parts don't need to be rebuilt.
155 Fundamentally, building from scratch is an attraction as it means all parts are 155 Fundamentally, building from scratch is attractive as it means all parts are
156 built fresh and there is no possibility of stale data causing problems. 156 built fresh and there is no possibility of stale data causing problems.
157 When developers hit problems, they typically default back to building from scratch 157 When developers hit problems, they typically default back to building from scratch
158 so they know the state of things from the start. 158 so they know the state of things from the start.
@@ -200,7 +200,7 @@
200 <title>Overall Architecture</title> 200 <title>Overall Architecture</title>
201 201
202 <para> 202 <para>
203 When determining what parts of the system need to be built, the Yocto Project 203 When determining what parts of the system need to be built, BitBake
204 uses a per-task basis and does not use a per-recipe basis. 204 uses a per-task basis and does not use a per-recipe basis.
205 You might wonder why using a per-task basis is preferred over a per-recipe basis. 205 You might wonder why using a per-task basis is preferred over a per-recipe basis.
206 To help explain, consider having the IPK packaging backend enabled and then switching to DEB. 206 To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
@@ -220,9 +220,9 @@
220 <title>Checksums (Signatures)</title> 220 <title>Checksums (Signatures)</title>
221 221
222 <para> 222 <para>
223 The Yocto Project uses a checksum, which is a unique signature of a task's 223 The shared state code uses a checksum, which is a unique signature of a task's
224 inputs, to determine if a task needs to be run again. 224 inputs, to determine if a task needs to be run again.
225 Because it is a change in a task's inputs that trigger a rerun, the process 225 Because it is a change in a task's inputs that triggers a rerun, the process
226 needs to detect all the inputs to a given task. 226 needs to detect all the inputs to a given task.
227 For shell tasks, this turns out to be fairly easy because 227 For shell tasks, this turns out to be fairly easy because
228 the build process generates a "run" shell script for each task and 228 the build process generates a "run" shell script for each task and
@@ -247,8 +247,8 @@
247 <para> 247 <para>
248 Another problem results from the "run" scripts containing functions that 248 Another problem results from the "run" scripts containing functions that
249 might or might not get called. 249 might or might not get called.
250 The Yocto Project contains code that figures out dependencies between shell 250 The incremental build solution contains code that figures out dependencies
251 functions. 251 between shell functions.
252 This code is used to prune the "run" scripts down to the minimum set, 252 This code is used to prune the "run" scripts down to the minimum set,
253 thereby alleviating this problem and making the "run" scripts much more 253 thereby alleviating this problem and making the "run" scripts much more
254 readable as a bonus. 254 readable as a bonus.
@@ -257,13 +257,12 @@
257 <para> 257 <para>
258 So far we have solutions for shell scripts. 258 So far we have solutions for shell scripts.
259 What about python tasks? 259 What about python tasks?
260 Handling these tasks are more difficult but the the same approach 260 The same approach applies even though these tasks are more difficult.
261 applies.
262 The process needs to figure out what variables a python function accesses 261 The process needs to figure out what variables a python function accesses
263 and what functions it calls. 262 and what functions it calls.
264 Again, the Yocto Project contains code that first figures out the variable and function 263 Again, the incremental build solution contains code that first figures out
265 dependencies, and then creates a checksum for the data used as the input to 264 the variable and function dependencies, and then creates a checksum for the data
266 the task. 265 used as the input to the task.
267 </para> 266 </para>
268 267
269 <para> 268 <para>
@@ -279,8 +278,7 @@
279 </para> 278 </para>
280 279
281 <para> 280 <para>
282 Equally, there are cases where we need to add in dependencies 281 Equally, there are cases where we need to add dependencies BitBake is not able to find.
283 BitBake is not able to find.
284 You can accomplish this by using a line like the following: 282 You can accomplish this by using a line like the following:
285 <literallayout class='monospaced'> 283 <literallayout class='monospaced'>
286 PACKAGE_ARCHS[vardeps] = "MACHINE" 284 PACKAGE_ARCHS[vardeps] = "MACHINE"
@@ -300,13 +298,12 @@
300 </para> 298 </para>
301 299
302 <para> 300 <para>
303 Thus far, this section has limited discussion to the direct inputs into a 301 Thus far, this section has limited discussion to the direct inputs into a task.
304 task.
305 Information based on direct inputs is referred to as the "basehash" in the code. 302 Information based on direct inputs is referred to as the "basehash" in the code.
306 However, there is still the question of a task's indirect inputs, the things that 303 However, there is still the question of a task's indirect inputs, the things that
307 were already built and present in the build directory. 304 were already built and present in the build directory.
308 The checksum (or signature) for a particular task needs to add the hashes of all the 305 The checksum (or signature) for a particular task needs to add the hashes of all the
309 tasks the particular task depends upon. 306 tasks on which the particular task depends.
310 Choosing which dependencies to add is a policy decision. 307 Choosing which dependencies to add is a policy decision.
311 However, the effect is to generate a master checksum that combines the 308 However, the effect is to generate a master checksum that combines the
312 basehash and the hashes of the task's dependencies. 309 basehash and the hashes of the task's dependencies.
@@ -327,7 +324,7 @@
327 Also within the BitBake configuration file, we can give BitBake 324 Also within the BitBake configuration file, we can give BitBake
328 some extra information to help it handle this information. 325 some extra information to help it handle this information.
329 The following statements effectively result in a list of global 326 The following statements effectively result in a list of global
330 list of variable dependency excludes - variables never included in 327 variable dependency excludes - variables never included in
331 any checksum: 328 any checksum:
332 <literallayout class='monospaced'> 329 <literallayout class='monospaced'>
333 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" 330 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH"
@@ -346,7 +343,7 @@
346 The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and 343 The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and
347 excludes certain kinds of tasks from the dependency chains. 344 excludes certain kinds of tasks from the dependency chains.
348 The effect of the previous example is to isolate the native, target, 345 The effect of the previous example is to isolate the native, target,
349 and cross components. 346 and cross-components.
350 So, for example, toolchain changes do not force a rebuild of the whole system. 347 So, for example, toolchain changes do not force a rebuild of the whole system.
351 </para> 348 </para>
352 349
@@ -380,7 +377,7 @@
380 <title>Shared State</title> 377 <title>Shared State</title>
381 378
382 <para> 379 <para>
383 Checksums and dependencies as discussed in the previous section solves half the 380 Checksums and dependencies, as discussed in the previous section, solve half the
384 problem. 381 problem.
385 The other part of the problem is being able to use checksum information during the build 382 The other part of the problem is being able to use checksum information during the build
386 and being able to reuse or rebuild specific components. 383 and being able to reuse or rebuild specific components.
@@ -388,10 +385,8 @@
388 385
389 <para> 386 <para>
390 The shared state class (<filename>sstate.bbclass</filename>) 387 The shared state class (<filename>sstate.bbclass</filename>)
391 is a relatively generic implementation of how to 388 is a relatively generic implementation of how to "capture" a snapshot of a given task.
392 "capture" a snapshot of a given task. 389 The idea is that the build process does not care about the source of a task's output.
393 The idea is that the build process does not care about the source of a
394 task's output.
395 Output could be freshly built or it could be downloaded and unpacked from 390 Output could be freshly built or it could be downloaded and unpacked from
396 somewhere - the build process doesn't need to worry about its source. 391 somewhere - the build process doesn't need to worry about its source.
397 </para> 392 </para>
@@ -431,8 +426,8 @@
431 </para> 426 </para>
432 427
433 <para> 428 <para>
434 If you have a directory whose contents you need to preserve, 429 If you have a directory whose contents you need to preserve, you can do this with
435 you can do this with a line like the following: 430 a line like the following:
436 <literallayout class='monospaced'> 431 <literallayout class='monospaced'>
437 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" 432 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
438 </literallayout> 433 </literallayout>
@@ -483,7 +478,7 @@
483 only the <filename>do_package_write_ipk</filename> tasks would have their 478 only the <filename>do_package_write_ipk</filename> tasks would have their
484 shared state packages fetched and extracted. 479 shared state packages fetched and extracted.
485 Since the sysroot is not used, it would never get extracted. 480 Since the sysroot is not used, it would never get extracted.
486 This is another reason to prefer the task-based approach over a 481 This is another reason why a task-based approach is preferred over a
487 recipe-based approach, which would have to install the output from every task. 482 recipe-based approach, which would have to install the output from every task.
488 </para> 483 </para>
489 </section> 484 </section>
@@ -515,13 +510,13 @@
515 (or <filename>-S</filename>) option, BitBake dumps out 510 (or <filename>-S</filename>) option, BitBake dumps out
516 <filename>.siginfo</filename> files in 511 <filename>.siginfo</filename> files in
517 the stamp directory for every task it would have executed instead of 512 the stamp directory for every task it would have executed instead of
518 building the target package specified.</para></listitem> 513 building the specified target package.</para></listitem>
519 <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that 514 <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that
520 can process these <filename>.siginfo</filename> files. 515 can process these <filename>.siginfo</filename> files.
521 If one file is specified, it will dump out the dependency 516 If one file is specified, it will dump out the dependency
522 information in the file. 517 information in the file.
523 If two files are specified, it will compare the 518 If two files are specified, it will compare the two files and dump out
524 two files and dump out the differences between the two. 519 the differences between the two.
525 This allows the question of "What changed between X and Y?" to be 520 This allows the question of "What changed between X and Y?" to be
526 answered easily.</para></listitem> 521 answered easily.</para></listitem>
527 </itemizedlist> 522 </itemizedlist>
@@ -532,18 +527,18 @@
532 <title>Invalidating Shared State</title> 527 <title>Invalidating Shared State</title>
533 528
534 <para> 529 <para>
535 The Yocto Project shared state code uses checksums and shared state memory 530 The shared state code uses checksums and shared state memory
536 cache to avoide building tasks that don't need built. 531 cache to avoid unnecessarily rebuilding tasks.
537 As with all schemes, this one has some drawbacks. 532 As with all schemes, this one has some drawbacks.
538 It is possible that you could make implicit changes that are not factored into the checksum 533 It is possible that you could make implicit changes that are not factored
539 calculation, but do affect a task's output. 534 into the checksum calculation, but do affect a task's output.
540 A good example is perhaps when a tool changes its output. 535 A good example is perhaps when a tool changes its output.
541 Let's say that the output of <filename>rpmdeps</filename> needed to change. 536 Let's say that the output of <filename>rpmdeps</filename> needed to change.
542 The result of the change should be that all the "package", "package_write_rpm", 537 The result of the change should be that all the "package", "package_write_rpm",
543 and "package_deploy-rpm" shared state cache items would become invalid. 538 and "package_deploy-rpm" shared state cache items would become invalid.
544 But, because this is a change that is external to the code and therefore implicit, 539 But, because this is a change that is external to the code and therefore implicit,
545 the associated shared state cache items do not become invalidated. 540 the associated shared state cache items do not become invalidated.
546 In this case, the build process would use the cache items rather than running the 541 In this case, the build process would use the cached items rather than running the
547 task again. 542 task again.
548 Obviously, these types of implicit changes can cause problems. 543 Obviously, these types of implicit changes can cause problems.
549 </para> 544 </para>
@@ -565,8 +560,8 @@
565 </para> 560 </para>
566 561
567 <note> 562 <note>
568 For an example of a commit that makes a cosmetic change to invalidate an sstate, 563 For an example of a commit that makes a cosmetic change to invalidate
569 see this 564 a shared state, see this
570 <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. 565 <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
571 </note> 566 </note>
572 </section> 567 </section>