diff options
Diffstat (limited to 'documentation/poky-ref-manual/technical-details.xml')
-rw-r--r-- | documentation/poky-ref-manual/technical-details.xml | 71 |
1 files changed, 33 insertions, 38 deletions
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml index dfb5b726c0..9e40a66008 100644 --- a/documentation/poky-ref-manual/technical-details.xml +++ b/documentation/poky-ref-manual/technical-details.xml | |||
@@ -150,9 +150,9 @@ | |||
150 | <title>Shared State Cache</title> | 150 | <title>Shared State Cache</title> |
151 | 151 | ||
152 | <para> | 152 | <para> |
153 | By design, the Yocto Project builds everything from scratch unless it can determine that | 153 | By design, the Yocto Project build system builds everything from scratch unless |
154 | parts don't need to be rebuilt. | 154 | BitBake can determine that parts don't need to be rebuilt. |
155 | Fundamentally, building from scratch is an attraction as it means all parts are | 155 | Fundamentally, building from scratch is attractive as it means all parts are |
156 | built fresh and there is no possibility of stale data causing problems. | 156 | built fresh and there is no possibility of stale data causing problems. |
157 | When developers hit problems, they typically default back to building from scratch | 157 | When developers hit problems, they typically default back to building from scratch |
158 | so they know the state of things from the start. | 158 | so they know the state of things from the start. |
@@ -200,7 +200,7 @@ | |||
200 | <title>Overall Architecture</title> | 200 | <title>Overall Architecture</title> |
201 | 201 | ||
202 | <para> | 202 | <para> |
203 | When determining what parts of the system need to be built, the Yocto Project | 203 | When determining what parts of the system need to be built, BitBake |
204 | uses a per-task basis and does not use a per-recipe basis. | 204 | uses a per-task basis and does not use a per-recipe basis. |
205 | You might wonder why using a per-task basis is preferred over a per-recipe basis. | 205 | You might wonder why using a per-task basis is preferred over a per-recipe basis. |
206 | To help explain, consider having the IPK packaging backend enabled and then switching to DEB. | 206 | To help explain, consider having the IPK packaging backend enabled and then switching to DEB. |
@@ -220,9 +220,9 @@ | |||
220 | <title>Checksums (Signatures)</title> | 220 | <title>Checksums (Signatures)</title> |
221 | 221 | ||
222 | <para> | 222 | <para> |
223 | The Yocto Project uses a checksum, which is a unique signature of a task's | 223 | The shared state code uses a checksum, which is a unique signature of a task's |
224 | inputs, to determine if a task needs to be run again. | 224 | inputs, to determine if a task needs to be run again. |
225 | Because it is a change in a task's inputs that trigger a rerun, the process | 225 | Because it is a change in a task's inputs that triggers a rerun, the process |
226 | needs to detect all the inputs to a given task. | 226 | needs to detect all the inputs to a given task. |
227 | For shell tasks, this turns out to be fairly easy because | 227 | For shell tasks, this turns out to be fairly easy because |
228 | the build process generates a "run" shell script for each task and | 228 | the build process generates a "run" shell script for each task and |
@@ -247,8 +247,8 @@ | |||
247 | <para> | 247 | <para> |
248 | Another problem results from the "run" scripts containing functions that | 248 | Another problem results from the "run" scripts containing functions that |
249 | might or might not get called. | 249 | might or might not get called. |
250 | The Yocto Project contains code that figures out dependencies between shell | 250 | The incremental build solution contains code that figures out dependencies |
251 | functions. | 251 | between shell functions. |
252 | This code is used to prune the "run" scripts down to the minimum set, | 252 | This code is used to prune the "run" scripts down to the minimum set, |
253 | thereby alleviating this problem and making the "run" scripts much more | 253 | thereby alleviating this problem and making the "run" scripts much more |
254 | readable as a bonus. | 254 | readable as a bonus. |
@@ -257,13 +257,12 @@ | |||
257 | <para> | 257 | <para> |
258 | So far we have solutions for shell scripts. | 258 | So far we have solutions for shell scripts. |
259 | What about python tasks? | 259 | What about python tasks? |
260 | Handling these tasks are more difficult but the the same approach | 260 | The same approach applies even though these tasks are more difficult. |
261 | applies. | ||
262 | The process needs to figure out what variables a python function accesses | 261 | The process needs to figure out what variables a python function accesses |
263 | and what functions it calls. | 262 | and what functions it calls. |
264 | Again, the Yocto Project contains code that first figures out the variable and function | 263 | Again, the incremental build solution contains code that first figures out |
265 | dependencies, and then creates a checksum for the data used as the input to | 264 | the variable and function dependencies, and then creates a checksum for the data |
266 | the task. | 265 | used as the input to the task. |
267 | </para> | 266 | </para> |
268 | 267 | ||
269 | <para> | 268 | <para> |
@@ -279,8 +278,7 @@ | |||
279 | </para> | 278 | </para> |
280 | 279 | ||
281 | <para> | 280 | <para> |
282 | Equally, there are cases where we need to add in dependencies | 281 | Equally, there are cases where we need to add dependencies BitBake is not able to find. |
283 | BitBake is not able to find. | ||
284 | You can accomplish this by using a line like the following: | 282 | You can accomplish this by using a line like the following: |
285 | <literallayout class='monospaced'> | 283 | <literallayout class='monospaced'> |
286 | PACKAGE_ARCHS[vardeps] = "MACHINE" | 284 | PACKAGE_ARCHS[vardeps] = "MACHINE" |
@@ -300,13 +298,12 @@ | |||
300 | </para> | 298 | </para> |
301 | 299 | ||
302 | <para> | 300 | <para> |
303 | Thus far, this section has limited discussion to the direct inputs into a | 301 | Thus far, this section has limited discussion to the direct inputs into a task. |
304 | task. | ||
305 | Information based on direct inputs is referred to as the "basehash" in the code. | 302 | Information based on direct inputs is referred to as the "basehash" in the code. |
306 | However, there is still the question of a task's indirect inputs, the things that | 303 | However, there is still the question of a task's indirect inputs, the things that |
307 | were already built and present in the build directory. | 304 | were already built and present in the build directory. |
308 | The checksum (or signature) for a particular task needs to add the hashes of all the | 305 | The checksum (or signature) for a particular task needs to add the hashes of all the |
309 | tasks the particular task depends upon. | 306 | tasks on which the particular task depends. |
310 | Choosing which dependencies to add is a policy decision. | 307 | Choosing which dependencies to add is a policy decision. |
311 | However, the effect is to generate a master checksum that combines the | 308 | However, the effect is to generate a master checksum that combines the |
312 | basehash and the hashes of the task's dependencies. | 309 | basehash and the hashes of the task's dependencies. |
@@ -327,7 +324,7 @@ | |||
327 | Also within the BitBake configuration file, we can give BitBake | 324 | Also within the BitBake configuration file, we can give BitBake |
328 | some extra information to help it handle this information. | 325 | some extra information to help it handle this information. |
329 | The following statements effectively result in a list of global | 326 | The following statements effectively result in a list of global |
330 | list of variable dependency excludes - variables never included in | 327 | variable dependency excludes - variables never included in |
331 | any checksum: | 328 | any checksum: |
332 | <literallayout class='monospaced'> | 329 | <literallayout class='monospaced'> |
333 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" | 330 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" |
@@ -346,7 +343,7 @@ | |||
346 | The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and | 343 | The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and |
347 | excludes certain kinds of tasks from the dependency chains. | 344 | excludes certain kinds of tasks from the dependency chains. |
348 | The effect of the previous example is to isolate the native, target, | 345 | The effect of the previous example is to isolate the native, target, |
349 | and cross components. | 346 | and cross-components. |
350 | So, for example, toolchain changes do not force a rebuild of the whole system. | 347 | So, for example, toolchain changes do not force a rebuild of the whole system. |
351 | </para> | 348 | </para> |
352 | 349 | ||
@@ -380,7 +377,7 @@ | |||
380 | <title>Shared State</title> | 377 | <title>Shared State</title> |
381 | 378 | ||
382 | <para> | 379 | <para> |
383 | Checksums and dependencies as discussed in the previous section solves half the | 380 | Checksums and dependencies, as discussed in the previous section, solve half the |
384 | problem. | 381 | problem. |
385 | The other part of the problem is being able to use checksum information during the build | 382 | The other part of the problem is being able to use checksum information during the build |
386 | and being able to reuse or rebuild specific components. | 383 | and being able to reuse or rebuild specific components. |
@@ -388,10 +385,8 @@ | |||
388 | 385 | ||
389 | <para> | 386 | <para> |
390 | The shared state class (<filename>sstate.bbclass</filename>) | 387 | The shared state class (<filename>sstate.bbclass</filename>) |
391 | is a relatively generic implementation of how to | 388 | is a relatively generic implementation of how to "capture" a snapshot of a given task. |
392 | "capture" a snapshot of a given task. | 389 | The idea is that the build process does not care about the source of a task's output. |
393 | The idea is that the build process does not care about the source of a | ||
394 | task's output. | ||
395 | Output could be freshly built or it could be downloaded and unpacked from | 390 | Output could be freshly built or it could be downloaded and unpacked from |
396 | somewhere - the build process doesn't need to worry about its source. | 391 | somewhere - the build process doesn't need to worry about its source. |
397 | </para> | 392 | </para> |
@@ -431,8 +426,8 @@ | |||
431 | </para> | 426 | </para> |
432 | 427 | ||
433 | <para> | 428 | <para> |
434 | If you have a directory whose contents you need to preserve, | 429 | If you have a directory whose contents you need to preserve, you can do this with |
435 | you can do this with a line like the following: | 430 | a line like the following: |
436 | <literallayout class='monospaced'> | 431 | <literallayout class='monospaced'> |
437 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | 432 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" |
438 | </literallayout> | 433 | </literallayout> |
@@ -483,7 +478,7 @@ | |||
483 | only the <filename>do_package_write_ipk</filename> tasks would have their | 478 | only the <filename>do_package_write_ipk</filename> tasks would have their |
484 | shared state packages fetched and extracted. | 479 | shared state packages fetched and extracted. |
485 | Since the sysroot is not used, it would never get extracted. | 480 | Since the sysroot is not used, it would never get extracted. |
486 | This is another reason to prefer the task-based approach over a | 481 | This is another reason why a task-based approach is preferred over a |
487 | recipe-based approach, which would have to install the output from every task. | 482 | recipe-based approach, which would have to install the output from every task. |
488 | </para> | 483 | </para> |
489 | </section> | 484 | </section> |
@@ -515,13 +510,13 @@ | |||
515 | (or <filename>-S</filename>) option, BitBake dumps out | 510 | (or <filename>-S</filename>) option, BitBake dumps out |
516 | <filename>.siginfo</filename> files in | 511 | <filename>.siginfo</filename> files in |
517 | the stamp directory for every task it would have executed instead of | 512 | the stamp directory for every task it would have executed instead of |
518 | building the target package specified.</para></listitem> | 513 | building the specified target package.</para></listitem> |
519 | <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that | 514 | <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that |
520 | can process these <filename>.siginfo</filename> files. | 515 | can process these <filename>.siginfo</filename> files. |
521 | If one file is specified, it will dump out the dependency | 516 | If one file is specified, it will dump out the dependency |
522 | information in the file. | 517 | information in the file. |
523 | If two files are specified, it will compare the | 518 | If two files are specified, it will compare the two files and dump out |
524 | two files and dump out the differences between the two. | 519 | the differences between the two. |
525 | This allows the question of "What changed between X and Y?" to be | 520 | This allows the question of "What changed between X and Y?" to be |
526 | answered easily.</para></listitem> | 521 | answered easily.</para></listitem> |
527 | </itemizedlist> | 522 | </itemizedlist> |
@@ -532,18 +527,18 @@ | |||
532 | <title>Invalidating Shared State</title> | 527 | <title>Invalidating Shared State</title> |
533 | 528 | ||
534 | <para> | 529 | <para> |
535 | The Yocto Project shared state code uses checksums and shared state memory | 530 | The shared state code uses checksums and shared state memory |
536 | cache to avoide building tasks that don't need built. | 531 | cache to avoid unnecessarily rebuilding tasks. |
537 | As with all schemes, this one has some drawbacks. | 532 | As with all schemes, this one has some drawbacks. |
538 | It is possible that you could make implicit changes that are not factored into the checksum | 533 | It is possible that you could make implicit changes that are not factored |
539 | calculation, but do affect a task's output. | 534 | into the checksum calculation, but do affect a task's output. |
540 | A good example is perhaps when a tool changes its output. | 535 | A good example is perhaps when a tool changes its output. |
541 | Let's say that the output of <filename>rpmdeps</filename> needed to change. | 536 | Let's say that the output of <filename>rpmdeps</filename> needed to change. |
542 | The result of the change should be that all the "package", "package_write_rpm", | 537 | The result of the change should be that all the "package", "package_write_rpm", |
543 | and "package_deploy-rpm" shared state cache items would become invalid. | 538 | and "package_deploy-rpm" shared state cache items would become invalid. |
544 | But, because this is a change that is external to the code and therefore implicit, | 539 | But, because this is a change that is external to the code and therefore implicit, |
545 | the associated shared state cache items do not become invalidated. | 540 | the associated shared state cache items do not become invalidated. |
546 | In this case, the build process would use the cache items rather than running the | 541 | In this case, the build process would use the cached items rather than running the |
547 | task again. | 542 | task again. |
548 | Obviously, these types of implicit changes can cause problems. | 543 | Obviously, these types of implicit changes can cause problems. |
549 | </para> | 544 | </para> |
@@ -565,8 +560,8 @@ | |||
565 | </para> | 560 | </para> |
566 | 561 | ||
567 | <note> | 562 | <note> |
568 | For an example of a commit that makes a cosmetic change to invalidate an sstate, | 563 | For an example of a commit that makes a cosmetic change to invalidate |
569 | see this | 564 | a shared state, see this |
570 | <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. | 565 | <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. |
571 | </note> | 566 | </note> |
572 | </section> | 567 | </section> |