diff options
| -rw-r--r-- | documentation/poky-ref-manual/technical-details.xml | 71 |
1 files changed, 33 insertions, 38 deletions
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml index dfb5b726c0..9e40a66008 100644 --- a/documentation/poky-ref-manual/technical-details.xml +++ b/documentation/poky-ref-manual/technical-details.xml | |||
| @@ -150,9 +150,9 @@ | |||
| 150 | <title>Shared State Cache</title> | 150 | <title>Shared State Cache</title> |
| 151 | 151 | ||
| 152 | <para> | 152 | <para> |
| 153 | By design, the Yocto Project builds everything from scratch unless it can determine that | 153 | By design, the Yocto Project build system builds everything from scratch unless |
| 154 | parts don't need to be rebuilt. | 154 | BitBake can determine that parts don't need to be rebuilt. |
| 155 | Fundamentally, building from scratch is an attraction as it means all parts are | 155 | Fundamentally, building from scratch is attractive as it means all parts are |
| 156 | built fresh and there is no possibility of stale data causing problems. | 156 | built fresh and there is no possibility of stale data causing problems. |
| 157 | When developers hit problems, they typically default back to building from scratch | 157 | When developers hit problems, they typically default back to building from scratch |
| 158 | so they know the state of things from the start. | 158 | so they know the state of things from the start. |
| @@ -200,7 +200,7 @@ | |||
| 200 | <title>Overall Architecture</title> | 200 | <title>Overall Architecture</title> |
| 201 | 201 | ||
| 202 | <para> | 202 | <para> |
| 203 | When determining what parts of the system need to be built, the Yocto Project | 203 | When determining what parts of the system need to be built, BitBake |
| 204 | uses a per-task basis and does not use a per-recipe basis. | 204 | uses a per-task basis and does not use a per-recipe basis. |
| 205 | You might wonder why using a per-task basis is preferred over a per-recipe basis. | 205 | You might wonder why using a per-task basis is preferred over a per-recipe basis. |
| 206 | To help explain, consider having the IPK packaging backend enabled and then switching to DEB. | 206 | To help explain, consider having the IPK packaging backend enabled and then switching to DEB. |
| @@ -220,9 +220,9 @@ | |||
| 220 | <title>Checksums (Signatures)</title> | 220 | <title>Checksums (Signatures)</title> |
| 221 | 221 | ||
| 222 | <para> | 222 | <para> |
| 223 | The Yocto Project uses a checksum, which is a unique signature of a task's | 223 | The shared state code uses a checksum, which is a unique signature of a task's |
| 224 | inputs, to determine if a task needs to be run again. | 224 | inputs, to determine if a task needs to be run again. |
| 225 | Because it is a change in a task's inputs that trigger a rerun, the process | 225 | Because it is a change in a task's inputs that triggers a rerun, the process |
| 226 | needs to detect all the inputs to a given task. | 226 | needs to detect all the inputs to a given task. |
| 227 | For shell tasks, this turns out to be fairly easy because | 227 | For shell tasks, this turns out to be fairly easy because |
| 228 | the build process generates a "run" shell script for each task and | 228 | the build process generates a "run" shell script for each task and |
| @@ -247,8 +247,8 @@ | |||
| 247 | <para> | 247 | <para> |
| 248 | Another problem results from the "run" scripts containing functions that | 248 | Another problem results from the "run" scripts containing functions that |
| 249 | might or might not get called. | 249 | might or might not get called. |
| 250 | The Yocto Project contains code that figures out dependencies between shell | 250 | The incremental build solution contains code that figures out dependencies |
| 251 | functions. | 251 | between shell functions. |
| 252 | This code is used to prune the "run" scripts down to the minimum set, | 252 | This code is used to prune the "run" scripts down to the minimum set, |
| 253 | thereby alleviating this problem and making the "run" scripts much more | 253 | thereby alleviating this problem and making the "run" scripts much more |
| 254 | readable as a bonus. | 254 | readable as a bonus. |
| @@ -257,13 +257,12 @@ | |||
| 257 | <para> | 257 | <para> |
| 258 | So far we have solutions for shell scripts. | 258 | So far we have solutions for shell scripts. |
| 259 | What about python tasks? | 259 | What about python tasks? |
| 260 | Handling these tasks are more difficult but the the same approach | 260 | The same approach applies even though these tasks are more difficult. |
| 261 | applies. | ||
| 262 | The process needs to figure out what variables a python function accesses | 261 | The process needs to figure out what variables a python function accesses |
| 263 | and what functions it calls. | 262 | and what functions it calls. |
| 264 | Again, the Yocto Project contains code that first figures out the variable and function | 263 | Again, the incremental build solution contains code that first figures out |
| 265 | dependencies, and then creates a checksum for the data used as the input to | 264 | the variable and function dependencies, and then creates a checksum for the data |
| 266 | the task. | 265 | used as the input to the task. |
| 267 | </para> | 266 | </para> |
| 268 | 267 | ||
| 269 | <para> | 268 | <para> |
| @@ -279,8 +278,7 @@ | |||
| 279 | </para> | 278 | </para> |
| 280 | 279 | ||
| 281 | <para> | 280 | <para> |
| 282 | Equally, there are cases where we need to add in dependencies | 281 | Equally, there are cases where we need to add dependencies BitBake is not able to find. |
| 283 | BitBake is not able to find. | ||
| 284 | You can accomplish this by using a line like the following: | 282 | You can accomplish this by using a line like the following: |
| 285 | <literallayout class='monospaced'> | 283 | <literallayout class='monospaced'> |
| 286 | PACKAGE_ARCHS[vardeps] = "MACHINE" | 284 | PACKAGE_ARCHS[vardeps] = "MACHINE" |
| @@ -300,13 +298,12 @@ | |||
| 300 | </para> | 298 | </para> |
| 301 | 299 | ||
| 302 | <para> | 300 | <para> |
| 303 | Thus far, this section has limited discussion to the direct inputs into a | 301 | Thus far, this section has limited discussion to the direct inputs into a task. |
| 304 | task. | ||
| 305 | Information based on direct inputs is referred to as the "basehash" in the code. | 302 | Information based on direct inputs is referred to as the "basehash" in the code. |
| 306 | However, there is still the question of a task's indirect inputs, the things that | 303 | However, there is still the question of a task's indirect inputs, the things that |
| 307 | were already built and present in the build directory. | 304 | were already built and present in the build directory. |
| 308 | The checksum (or signature) for a particular task needs to add the hashes of all the | 305 | The checksum (or signature) for a particular task needs to add the hashes of all the |
| 309 | tasks the particular task depends upon. | 306 | tasks on which the particular task depends. |
| 310 | Choosing which dependencies to add is a policy decision. | 307 | Choosing which dependencies to add is a policy decision. |
| 311 | However, the effect is to generate a master checksum that combines the | 308 | However, the effect is to generate a master checksum that combines the |
| 312 | basehash and the hashes of the task's dependencies. | 309 | basehash and the hashes of the task's dependencies. |
| @@ -327,7 +324,7 @@ | |||
| 327 | Also within the BitBake configuration file, we can give BitBake | 324 | Also within the BitBake configuration file, we can give BitBake |
| 328 | some extra information to help it handle this information. | 325 | some extra information to help it handle this information. |
| 329 | The following statements effectively result in a list of global | 326 | The following statements effectively result in a list of global |
| 330 | list of variable dependency excludes - variables never included in | 327 | variable dependency excludes - variables never included in |
| 331 | any checksum: | 328 | any checksum: |
| 332 | <literallayout class='monospaced'> | 329 | <literallayout class='monospaced'> |
| 333 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" | 330 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" |
| @@ -346,7 +343,7 @@ | |||
| 346 | The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and | 343 | The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and |
| 347 | excludes certain kinds of tasks from the dependency chains. | 344 | excludes certain kinds of tasks from the dependency chains. |
| 348 | The effect of the previous example is to isolate the native, target, | 345 | The effect of the previous example is to isolate the native, target, |
| 349 | and cross components. | 346 | and cross-components. |
| 350 | So, for example, toolchain changes do not force a rebuild of the whole system. | 347 | So, for example, toolchain changes do not force a rebuild of the whole system. |
| 351 | </para> | 348 | </para> |
| 352 | 349 | ||
| @@ -380,7 +377,7 @@ | |||
| 380 | <title>Shared State</title> | 377 | <title>Shared State</title> |
| 381 | 378 | ||
| 382 | <para> | 379 | <para> |
| 383 | Checksums and dependencies as discussed in the previous section solves half the | 380 | Checksums and dependencies, as discussed in the previous section, solve half the |
| 384 | problem. | 381 | problem. |
| 385 | The other part of the problem is being able to use checksum information during the build | 382 | The other part of the problem is being able to use checksum information during the build |
| 386 | and being able to reuse or rebuild specific components. | 383 | and being able to reuse or rebuild specific components. |
| @@ -388,10 +385,8 @@ | |||
| 388 | 385 | ||
| 389 | <para> | 386 | <para> |
| 390 | The shared state class (<filename>sstate.bbclass</filename>) | 387 | The shared state class (<filename>sstate.bbclass</filename>) |
| 391 | is a relatively generic implementation of how to | 388 | is a relatively generic implementation of how to "capture" a snapshot of a given task. |
| 392 | "capture" a snapshot of a given task. | 389 | The idea is that the build process does not care about the source of a task's output. |
| 393 | The idea is that the build process does not care about the source of a | ||
| 394 | task's output. | ||
| 395 | Output could be freshly built or it could be downloaded and unpacked from | 390 | Output could be freshly built or it could be downloaded and unpacked from |
| 396 | somewhere - the build process doesn't need to worry about its source. | 391 | somewhere - the build process doesn't need to worry about its source. |
| 397 | </para> | 392 | </para> |
| @@ -431,8 +426,8 @@ | |||
| 431 | </para> | 426 | </para> |
| 432 | 427 | ||
| 433 | <para> | 428 | <para> |
| 434 | If you have a directory whose contents you need to preserve, | 429 | If you have a directory whose contents you need to preserve, you can do this with |
| 435 | you can do this with a line like the following: | 430 | a line like the following: |
| 436 | <literallayout class='monospaced'> | 431 | <literallayout class='monospaced'> |
| 437 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | 432 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" |
| 438 | </literallayout> | 433 | </literallayout> |
| @@ -483,7 +478,7 @@ | |||
| 483 | only the <filename>do_package_write_ipk</filename> tasks would have their | 478 | only the <filename>do_package_write_ipk</filename> tasks would have their |
| 484 | shared state packages fetched and extracted. | 479 | shared state packages fetched and extracted. |
| 485 | Since the sysroot is not used, it would never get extracted. | 480 | Since the sysroot is not used, it would never get extracted. |
| 486 | This is another reason to prefer the task-based approach over a | 481 | This is another reason why a task-based approach is preferred over a |
| 487 | recipe-based approach, which would have to install the output from every task. | 482 | recipe-based approach, which would have to install the output from every task. |
| 488 | </para> | 483 | </para> |
| 489 | </section> | 484 | </section> |
| @@ -515,13 +510,13 @@ | |||
| 515 | (or <filename>-S</filename>) option, BitBake dumps out | 510 | (or <filename>-S</filename>) option, BitBake dumps out |
| 516 | <filename>.siginfo</filename> files in | 511 | <filename>.siginfo</filename> files in |
| 517 | the stamp directory for every task it would have executed instead of | 512 | the stamp directory for every task it would have executed instead of |
| 518 | building the target package specified.</para></listitem> | 513 | building the specified target package.</para></listitem> |
| 519 | <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that | 514 | <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that |
| 520 | can process these <filename>.siginfo</filename> files. | 515 | can process these <filename>.siginfo</filename> files. |
| 521 | If one file is specified, it will dump out the dependency | 516 | If one file is specified, it will dump out the dependency |
| 522 | information in the file. | 517 | information in the file. |
| 523 | If two files are specified, it will compare the | 518 | If two files are specified, it will compare the two files and dump out |
| 524 | two files and dump out the differences between the two. | 519 | the differences between the two. |
| 525 | This allows the question of "What changed between X and Y?" to be | 520 | This allows the question of "What changed between X and Y?" to be |
| 526 | answered easily.</para></listitem> | 521 | answered easily.</para></listitem> |
| 527 | </itemizedlist> | 522 | </itemizedlist> |
| @@ -532,18 +527,18 @@ | |||
| 532 | <title>Invalidating Shared State</title> | 527 | <title>Invalidating Shared State</title> |
| 533 | 528 | ||
| 534 | <para> | 529 | <para> |
| 535 | The Yocto Project shared state code uses checksums and shared state memory | 530 | The shared state code uses checksums and shared state memory |
| 536 | cache to avoide building tasks that don't need built. | 531 | cache to avoid unnecessarily rebuilding tasks. |
| 537 | As with all schemes, this one has some drawbacks. | 532 | As with all schemes, this one has some drawbacks. |
| 538 | It is possible that you could make implicit changes that are not factored into the checksum | 533 | It is possible that you could make implicit changes that are not factored |
| 539 | calculation, but do affect a task's output. | 534 | into the checksum calculation, but do affect a task's output. |
| 540 | A good example is perhaps when a tool changes its output. | 535 | A good example is perhaps when a tool changes its output. |
| 541 | Let's say that the output of <filename>rpmdeps</filename> needed to change. | 536 | Let's say that the output of <filename>rpmdeps</filename> needed to change. |
| 542 | The result of the change should be that all the "package", "package_write_rpm", | 537 | The result of the change should be that all the "package", "package_write_rpm", |
| 543 | and "package_deploy-rpm" shared state cache items would become invalid. | 538 | and "package_deploy-rpm" shared state cache items would become invalid. |
| 544 | But, because this is a change that is external to the code and therefore implicit, | 539 | But, because this is a change that is external to the code and therefore implicit, |
| 545 | the associated shared state cache items do not become invalidated. | 540 | the associated shared state cache items do not become invalidated. |
| 546 | In this case, the build process would use the cache items rather than running the | 541 | In this case, the build process would use the cached items rather than running the |
| 547 | task again. | 542 | task again. |
| 548 | Obviously, these types of implicit changes can cause problems. | 543 | Obviously, these types of implicit changes can cause problems. |
| 549 | </para> | 544 | </para> |
| @@ -565,8 +560,8 @@ | |||
| 565 | </para> | 560 | </para> |
| 566 | 561 | ||
| 567 | <note> | 562 | <note> |
| 568 | For an example of a commit that makes a cosmetic change to invalidate an sstate, | 563 | For an example of a commit that makes a cosmetic change to invalidate |
| 569 | see this | 564 | a shared state, see this |
| 570 | <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. | 565 | <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. |
| 571 | </note> | 566 | </note> |
| 572 | </section> | 567 | </section> |
