From 00f87f84165964f262f8bb97378cfbef4b0c325a Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Thu, 11 Jan 2018 10:01:23 -0800 Subject: overview-manual, ref-manual: Moved "Shared State Cache" to overview manual Fixes [YOCTO #12370] The section on shared state cache needed to be in the overview manual and not in the ref-manual. I moved it. Some links were affected, which I fixed. (From yocto-docs rev: 1c4e5207bdde19d4b48ef42b1de81390d8a02d64) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../dev-manual/dev-manual-common-tasks.xml | 6 +- .../overview-manual/overview-concepts.xml | 635 +++++++++++++++++++++ .../overview-development-environment.xml | 2 +- documentation/ref-manual/ref-bitbake.xml | 6 +- documentation/ref-manual/ref-classes.xml | 4 +- documentation/ref-manual/ref-tasks.xml | 14 +- documentation/ref-manual/ref-variables.xml | 4 +- documentation/ref-manual/technical-details.xml | 603 ------------------- documentation/ref-manual/usingpoky.xml | 2 +- 9 files changed, 656 insertions(+), 620 deletions(-) (limited to 'documentation') diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml index 78825c7f1e..8a0d6a3222 100644 --- a/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/documentation/dev-manual/dev-manual-common-tasks.xml @@ -7346,7 +7346,7 @@ Some notes from Cal: Because the OpenEmbedded build system uses - "signatures", + "signatures", which are unique to a given build, the build system knows when to rebuild packages. All the inputs into a given task are represented by a @@ -7450,8 +7450,8 @@ Some notes from Cal: For more information on shared state, see the - "Shared State Cache" - section in the Yocto Project Reference Manual. + "Shared State Cache" + section in the Yocto Project Overview Manual. diff --git a/documentation/overview-manual/overview-concepts.xml b/documentation/overview-manual/overview-concepts.xml index 0a45cd7256..2d704923fa 100644 --- a/documentation/overview-manual/overview-concepts.xml +++ b/documentation/overview-manual/overview-concepts.xml @@ -471,6 +471,641 @@ + + + +
+ Shared State Cache + + + By design, the OpenEmbedded build system builds everything from + scratch unless BitBake can determine that parts do not need to be + rebuilt. + Fundamentally, building from scratch is attractive as it means all + parts are built fresh and there is no possibility of stale data + causing problems. + When developers hit problems, they typically default back to + building from scratch so they know the state of things from the + start. + + + + Building an image from scratch is both an advantage and a + disadvantage to the process. + As mentioned in the previous paragraph, building from scratch + ensures that everything is current and starts from a known state. + However, building from scratch also takes much longer as it + generally means rebuilding things that do not necessarily need + to be rebuilt. + + + + The Yocto Project implements shared state code that supports + incremental builds. + The implementation of the shared state code answers the following + questions that were fundamental roadblocks within the OpenEmbedded + incremental build support system: + + + What pieces of the system have changed and what pieces have + not changed? + + + How are changed pieces of software removed and replaced? + + + How are pre-built components that do not need to be rebuilt + from scratch used when they are available? + + + + + + For the first question, the build system detects changes in the + "inputs" to a given task by creating a checksum (or signature) of + the task's inputs. + If the checksum changes, the system assumes the inputs have changed + and the task needs to be rerun. + For the second question, the shared state (sstate) code tracks + which tasks add which output to the build process. + This means the output from a given task can be removed, upgraded + or otherwise manipulated. + The third question is partly addressed by the solution for the + second question assuming the build system can fetch the sstate + objects from remote locations and install them if they are deemed + to be valid. + + The OpenEmbedded build system does not maintain + PR + information as part of the shared state packages. + Consequently, considerations exist that affect maintaining + shared state feeds. + For information on how the OpenEmbedded build system + works with packages and can track incrementing + PR information, see the + "Automatically Incrementing a Binary Package Revision Number" + section in the Yocto Project Development Tasks Manual. + + + + + The rest of this section goes into detail about the overall + incremental build architecture, the checksums (signatures), shared + state, and some tips and tricks. + + +
+ Overall Architecture + + + When determining what parts of the system need to be built, + BitBake works on a per-task basis rather than a per-recipe + basis. + You might wonder why using a per-task basis is preferred over + a per-recipe basis. + To help explain, consider having the IPK packaging backend + enabled and then switching to DEB. + In this case, the + do_install + and + do_package + task outputs are still valid. + However, with a per-recipe approach, the build would not + include the .deb files. + Consequently, you would have to invalidate the whole build and + rerun it. + Rerunning everything is not the best solution. + Also, in this case, the core must be "taught" much about + specific tasks. + This methodology does not scale well and does not allow users + to easily add new tasks in layers or as external recipes + without touching the packaged-staging core. + +
+ +
+ Checksums (Signatures) + + + The shared state code uses a checksum, which is a unique + signature of a task's inputs, to determine if a task needs to + be run again. + Because it is a change in a task's inputs that triggers a + rerun, the process needs to detect all the inputs to a given + task. + For shell tasks, this turns out to be fairly easy because + the build process generates a "run" shell script for each task + and it is possible to create a checksum that gives you a good + idea of when the task's data changes. + + + + To complicate the problem, there are things that should not be + included in the checksum. + First, there is the actual specific build path of a given + task - the + WORKDIR. + It does not matter if the work directory changes because it + should not affect the output for target packages. + Also, the build process has the objective of making native + or cross packages relocatable. + + Both native and cross packages run on the build host. + However, cross packages generate output for the target + architecture. + + The checksum therefore needs to exclude + WORKDIR. + The simplistic approach for excluding the work directory is to + set WORKDIR to some fixed value and + create the checksum for the "run" script. + + + + Another problem results from the "run" scripts containing + functions that might or might not get called. + The incremental build solution contains code that figures out + dependencies between shell functions. + This code is used to prune the "run" scripts down to the + minimum set, thereby alleviating this problem and making the + "run" scripts much more readable as a bonus. + + + + So far we have solutions for shell scripts. + What about Python tasks? + The same approach applies even though these tasks are more + difficult. + The process needs to figure out what variables a Python + function accesses and what functions it calls. + Again, the incremental build solution contains code that first + figures out the variable and function dependencies, and then + creates a checksum for the data used as the input to the task. + + + + Like the WORKDIR case, situations exist + where dependencies should be ignored. + For these cases, you can instruct the build process to + ignore a dependency by using a line like the following: + + PACKAGE_ARCHS[vardepsexclude] = "MACHINE" + + This example ensures that the + PACKAGE_ARCHS + variable does not depend on the value of + MACHINE, + even if it does reference it. + + + + Equally, there are cases where we need to add dependencies + BitBake is not able to find. + You can accomplish this by using a line like the following: + + PACKAGE_ARCHS[vardeps] = "MACHINE" + + This example explicitly adds the MACHINE + variable as a dependency for + PACKAGE_ARCHS. + + + + Consider a case with in-line Python, for example, where + BitBake is not able to figure out dependencies. + When running in debug mode (i.e. using + -DDD), BitBake produces output when it + discovers something for which it cannot figure out dependencies. + The Yocto Project team has currently not managed to cover + those dependencies in detail and is aware of the need to fix + this situation. + + + + Thus far, this section has limited discussion to the direct + inputs into a task. + Information based on direct inputs is referred to as the + "basehash" in the code. + However, there is still the question of a task's indirect + inputs - the things that were already built and present in the + Build Directory. + The checksum (or signature) for a particular task needs to add + the hashes of all the tasks on which the particular task + depends. + Choosing which dependencies to add is a policy decision. + However, the effect is to generate a master checksum that + combines the basehash and the hashes of the task's + dependencies. + + + + At the code level, there are a variety of ways both the + basehash and the dependent task hashes can be influenced. + Within the BitBake configuration file, we can give BitBake + some extra information to help it construct the basehash. + The following statement effectively results in a list of + global variable dependency excludes - variables never + included in any checksum: + + BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ + SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ + USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ + PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ + CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" + + The previous example excludes + WORKDIR + since that variable is actually constructed as a path within + TMPDIR, + which is on the whitelist. + + + + The rules for deciding which hashes of dependent tasks to + include through dependency chains are more complex and are + generally accomplished with a Python function. + The code in meta/lib/oe/sstatesig.py shows + two examples of this and also illustrates how you can insert + your own policy into the system if so desired. + This file defines the two basic signature generators + OE-Core + uses: "OEBasic" and "OEBasicHash". + By default, there is a dummy "noop" signature handler enabled + in BitBake. + This means that behavior is unchanged from previous versions. + OE-Core uses the "OEBasicHash" signature handler by default + through this setting in the bitbake.conf + file: + + BB_SIGNATURE_HANDLER ?= "OEBasicHash" + + The "OEBasicHash" BB_SIGNATURE_HANDLER + is the same as the "OEBasic" version but adds the task hash to + the stamp files. + This results in any + Metadata + change that changes the task hash, automatically + causing the task to be run again. + This removes the need to bump + PR + values, and changes to Metadata automatically ripple across + the build. + + + + It is also worth noting that the end result of these + signature generators is to make some dependency and hash + information available to the build. + This information includes: + + + BB_BASEHASH_task-taskname: + The base hashes for each task in the recipe. + + + BB_BASEHASH_filename:taskname: + The base hashes for each dependent task. + + + BBHASHDEPS_filename:taskname: + The task dependencies for each task. + + + BB_TASKHASH: + The hash of the currently running task. + + + +
+ +
+ Shared State + + + Checksums and dependencies, as discussed in the previous + section, solve half the problem of supporting a shared state. + The other part of the problem is being able to use checksum + information during the build and being able to reuse or rebuild + specific components. + + + + The + sstate + class is a relatively generic implementation of how to + "capture" a snapshot of a given task. + The idea is that the build process does not care about the + source of a task's output. + Output could be freshly built or it could be downloaded and + unpacked from somewhere - the build process does not need to + worry about its origin. + + + + There are two types of output, one is just about creating a + directory in + WORKDIR. + A good example is the output of either + do_install + or + do_package. + The other type of output occurs when a set of data is merged + into a shared directory tree such as the sysroot. + + + + The Yocto Project team has tried to keep the details of the + implementation hidden in sstate class. + From a user's perspective, adding shared state wrapping to a task + is as simple as this + do_deploy + example taken from the + deploy + class: + + DEPLOYDIR = "${WORKDIR}/deploy-${PN}" + SSTATETASKS += "do_deploy" + do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" + do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" + + python do_deploy_setscene () { + sstate_setscene(d) + } + addtask do_deploy_setscene + do_deploy[dirs] = "${DEPLOYDIR} ${B}" + + The following list explains the previous example: + + + Adding "do_deploy" to SSTATETASKS + adds some required sstate-related processing, which is + implemented in the + sstate + class, to before and after the + do_deploy + task. + + + The + do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" + declares that do_deploy places its + output in ${DEPLOYDIR} when run + normally (i.e. when not using the sstate cache). + This output becomes the input to the shared state cache. + + + The + do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" + line causes the contents of the shared state cache to be + copied to ${DEPLOY_DIR_IMAGE}. + + If do_deploy is not already in + the shared state cache or if its input checksum + (signature) has changed from when the output was + cached, the task will be run to populate the shared + state cache, after which the contents of the shared + state cache is copied to + ${DEPLOY_DIR_IMAGE}. + If do_deploy is in the shared + state cache and its signature indicates that the + cached output is still valid (i.e. if no + relevant task inputs have changed), then the + contents of the shared state cache will be copied + directly to + ${DEPLOY_DIR_IMAGE} by the + do_deploy_setscene task + instead, skipping the + do_deploy task. + + + + The following task definition is glue logic needed to + make the previous settings effective: + + python do_deploy_setscene () { + sstate_setscene(d) + } + addtask do_deploy_setscene + + sstate_setscene() takes the flags + above as input and accelerates the + do_deploy task through the + shared state cache if possible. + If the task was accelerated, + sstate_setscene() returns True. + Otherwise, it returns False, and the normal + do_deploy task runs. + For more information, see the + "setscene" + section in the BitBake User Manual. + + + The do_deploy[dirs] = "${DEPLOYDIR} ${B}" + line creates ${DEPLOYDIR} and + ${B} before the + do_deploy task runs, and also sets + the current working directory of + do_deploy to + ${B}. + For more information, see the + "Variable Flags" + section in the BitBake User Manual. + + In cases where + sstate-inputdirs and + sstate-outputdirs would be the + same, you can use + sstate-plaindirs. + For example, to preserve the + ${PKGD} and + ${PKGDEST} output from the + do_package + task, use the following: + + do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" + + + + + sstate-inputdirs and + sstate-outputdirs can also be used + with multiple directories. + For example, the following declares + PKGDESTWORK and + SHLIBWORK as shared state + input directories, which populates the shared state + cache, and PKGDATA_DIR and + SHLIBSDIR as the corresponding + shared state output directories: + + do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" + do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" + + + + These methods also include the ability to take a + lockfile when manipulating shared state directory + structures, for cases where file additions or removals + are sensitive: + + do_package[sstate-lockfile] = "${PACKAGELOCK}" + + + + + + + Behind the scenes, the shared state code works by looking in + SSTATE_DIR + and + SSTATE_MIRRORS + for shared state files. + Here is an example: + + SSTATE_MIRRORS ?= "\ + file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ + file://.* file:///some/local/dir/sstate/PATH" + + + The shared state directory + (SSTATE_DIR) is organized into + two-character subdirectories, where the subdirectory + names are based on the first two characters of the hash. + If the shared state directory structure for a mirror has the + same structure as SSTATE_DIR, you must + specify "PATH" as part of the URI to enable the build system + to map to the appropriate subdirectory. + + + + + The shared state package validity can be detected just by + looking at the filename since the filename contains the task + checksum (or signature) as described earlier in this section. + If a valid shared state package is found, the build process + downloads it and uses it to accelerate the task. + + + + The build processes use the *_setscene + tasks for the task acceleration phase. + BitBake goes through this phase before the main execution + code and tries to accelerate any tasks for which it can find + shared state packages. + If a shared state package for a task is available, the + shared state package is used. + This means the task and any tasks on which it is dependent + are not executed. + + + + As a real world example, the aim is when building an IPK-based + image, only the + do_package_write_ipk + tasks would have their shared state packages fetched and + extracted. + Since the sysroot is not used, it would never get extracted. + This is another reason why a task-based approach is preferred + over a recipe-based approach, which would have to install the + output from every task. + +
+ +
+ Tips and Tricks + + + The code in the build system that supports incremental builds + is not simple code. + This section presents some tips and tricks that help you work + around issues related to shared state code. + + +
+ Debugging + + + Seeing what metadata went into creating the input signature + of a shared state (sstate) task can be a useful debugging + aid. + This information is available in signature information + (siginfo) files in + SSTATE_DIR. + For information on how to view and interpret information in + siginfo files, see the + "Viewing Task Variable Dependencies" + section in the Yocto Project Reference Manual. + +
+ +
+ Invalidating Shared State + + + The OpenEmbedded build system uses checksums and shared + state cache to avoid unnecessarily rebuilding tasks. + Collectively, this scheme is known as "shared state code." + + + + As with all schemes, this one has some drawbacks. + It is possible that you could make implicit changes to your + code that the checksum calculations do not take into + account. + These implicit changes affect a task's output but do not + trigger the shared state code into rebuilding a recipe. + Consider an example during which a tool changes its output. + Assume that the output of rpmdeps + changes. + The result of the change should be that all the + package and + package_write_rpm shared state cache + items become invalid. + However, because the change to the output is + external to the code and therefore implicit, + the associated shared state cache items do not become + invalidated. + In this case, the build process uses the cached items + rather than running the task again. + Obviously, these types of implicit changes can cause + problems. + + + + To avoid these problems during the build, you need to + understand the effects of any changes you make. + Realize that changes you make directly to a function + are automatically factored into the checksum calculation. + Thus, these explicit changes invalidate the associated + area of shared state cache. + However, you need to be aware of any implicit changes that + are not obvious changes to the code and could affect + the output of a given task. + + + + When you identify an implicit change, you can easily + take steps to invalidate the cache and force the tasks + to run. + The steps you can take are as simple as changing a + function's comments in the source code. + For example, to invalidate package shared state files, + change the comment statements of + do_package + or the comments of one of the functions it calls. + Even though the change is purely cosmetic, it causes the + checksum to be recalculated and forces the OpenEmbedded + build system to run the task again. + + For an example of a commit that makes a cosmetic + change to invalidate shared state, see this + commit. + + +
+
+
+
x32 psABI diff --git a/documentation/overview-manual/overview-development-environment.xml b/documentation/overview-manual/overview-development-environment.xml index 62f3ccd438..e0d470b179 100644 --- a/documentation/overview-manual/overview-development-environment.xml +++ b/documentation/overview-manual/overview-development-environment.xml @@ -2490,7 +2490,7 @@ STAMP variable, and the end of the name consists of the task's name and current - input checksum. + input checksum. This naming scheme assumes that BB_SIGNATURE_HANDLER diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml index 17f4c54b9c..7d1dd48128 100644 --- a/documentation/ref-manual/ref-bitbake.xml +++ b/documentation/ref-manual/ref-bitbake.xml @@ -348,8 +348,10 @@ If doing so results in unnecessary rebuilds of tasks, you can whitelist the variable so that the shared state code ignores the dependency when it creates checksums. - For information on this process, see the BB_HASHBASE_WHITELIST - example in the "Checksums (Signatures)" section. + For information on this process, see the + BB_HASHBASE_WHITELIST example in the + "Checksums (Signatures)" + section in the Yocto Project Overview Manual.
diff --git a/documentation/ref-manual/ref-classes.xml b/documentation/ref-manual/ref-classes.xml index b11fbea63e..eabb60fe03 100644 --- a/documentation/ref-manual/ref-classes.xml +++ b/documentation/ref-manual/ref-classes.xml @@ -3182,8 +3182,8 @@ This check was removed for YP 2.3 release For more information on sstate, see the - "Shared State Cache" - section. + "Shared State Cache" + section in the Yocto Project Overview Manual. diff --git a/documentation/ref-manual/ref-tasks.xml b/documentation/ref-manual/ref-tasks.xml index 62a677a430..a90f02b7d8 100644 --- a/documentation/ref-manual/ref-tasks.xml +++ b/documentation/ref-manual/ref-tasks.xml @@ -628,8 +628,8 @@ Running this task does not remove the - sstate) cache - files. + sstate + cache files. Consequently, if no changes have been made and the recipe is rebuilt after cleaning, output files are simply restored from the sstate cache. @@ -645,8 +645,9 @@ Removes all output files, shared state - (sstate) cache, and - downloaded source files for a target (i.e. the contents of + (sstate) + cache, and downloaded source files for a target (i.e. the contents + of DL_DIR). Essentially, the do_cleanall task is identical to the @@ -675,13 +676,14 @@ Removes all output files and shared state - (sstate) + (sstate) cache for a target. Essentially, the do_cleansstate task is identical to the do_clean task with the added removal of shared state - (sstate) cache. + (sstate) + cache. diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index 48849b1fef..564bb38ac6 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -10654,11 +10654,11 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3" PR to know when to rebuild a recipe. The build system uses the task - input checksums + input checksums along with the stamp and - shared state cache + shared state cache mechanisms. The PR variable primarily becomes diff --git a/documentation/ref-manual/technical-details.xml b/documentation/ref-manual/technical-details.xml index 3d3def5a65..06a018b8a9 100644 --- a/documentation/ref-manual/technical-details.xml +++ b/documentation/ref-manual/technical-details.xml @@ -13,609 +13,6 @@ x32, Wayland support, and Licenses. -
- Shared State Cache - - - By design, the OpenEmbedded build system builds everything from scratch unless - BitBake can determine that parts do not need to be rebuilt. - Fundamentally, building from scratch is attractive as it means all parts are - built fresh and there is no possibility of stale data causing problems. - When developers hit problems, they typically default back to building from scratch - so they know the state of things from the start. - - - - Building an image from scratch is both an advantage and a disadvantage to the process. - As mentioned in the previous paragraph, building from scratch ensures that - everything is current and starts from a known state. - However, building from scratch also takes much longer as it generally means - rebuilding things that do not necessarily need to be rebuilt. - - - - The Yocto Project implements shared state code that supports incremental builds. - The implementation of the shared state code answers the following questions that - were fundamental roadblocks within the OpenEmbedded incremental build support system: - - What pieces of the system have changed and what pieces have - not changed? - How are changed pieces of software removed and replaced? - How are pre-built components that do not need to be rebuilt from scratch - used when they are available? - - - - - For the first question, the build system detects changes in the "inputs" to a given task by - creating a checksum (or signature) of the task's inputs. - If the checksum changes, the system assumes the inputs have changed and the task needs to be - rerun. - For the second question, the shared state (sstate) code tracks which tasks add which output - to the build process. - This means the output from a given task can be removed, upgraded or otherwise manipulated. - The third question is partly addressed by the solution for the second question - assuming the build system can fetch the sstate objects from remote locations and - install them if they are deemed to be valid. - - - - The OpenEmbedded build system does not maintain - PR information - as part of the shared state packages. - Consequently, considerations exist that affect maintaining shared - state feeds. - For information on how the OpenEmbedded build system - works with packages and can - track incrementing PR information, see the - "Automatically Incrementing a Binary Package Revision Number" - section in the Yocto Project Development Tasks Manual. - - - - The rest of this section goes into detail about the overall incremental build - architecture, the checksums (signatures), shared state, and some tips and tricks. - - -
- Overall Architecture - - - When determining what parts of the system need to be built, BitBake - works on a per-task basis rather than a per-recipe basis. - You might wonder why using a per-task basis is preferred over a per-recipe basis. - To help explain, consider having the IPK packaging backend enabled and then switching to DEB. - In this case, the - do_install - and - do_package - task outputs are still valid. - However, with a per-recipe approach, the build would not include the - .deb files. - Consequently, you would have to invalidate the whole build and rerun it. - Rerunning everything is not the best solution. - Also, in this case, the core must be "taught" much about specific tasks. - This methodology does not scale well and does not allow users to easily add new tasks - in layers or as external recipes without touching the packaged-staging core. - -
- -
- Checksums (Signatures) - - - The shared state code uses a checksum, which is a unique signature of a task's - inputs, to determine if a task needs to be run again. - Because it is a change in a task's inputs that triggers a rerun, the process - needs to detect all the inputs to a given task. - For shell tasks, this turns out to be fairly easy because - the build process generates a "run" shell script for each task and - it is possible to create a checksum that gives you a good idea of when - the task's data changes. - - - - To complicate the problem, there are things that should not be - included in the checksum. - First, there is the actual specific build path of a given task - - the WORKDIR. - It does not matter if the work directory changes because it should - not affect the output for target packages. - Also, the build process has the objective of making native - or cross packages relocatable. - - Both native and cross packages run on the build host. - However, cross packages generate output for the target - architecture. - - The checksum therefore needs to exclude - WORKDIR. - The simplistic approach for excluding the work directory is to set - WORKDIR to some fixed value and create the - checksum for the "run" script. - - - - Another problem results from the "run" scripts containing functions that - might or might not get called. - The incremental build solution contains code that figures out dependencies - between shell functions. - This code is used to prune the "run" scripts down to the minimum set, - thereby alleviating this problem and making the "run" scripts much more - readable as a bonus. - - - - So far we have solutions for shell scripts. - What about Python tasks? - The same approach applies even though these tasks are more difficult. - The process needs to figure out what variables a Python function accesses - and what functions it calls. - Again, the incremental build solution contains code that first figures out - the variable and function dependencies, and then creates a checksum for the data - used as the input to the task. - - - - Like the WORKDIR case, situations exist where dependencies - should be ignored. - For these cases, you can instruct the build process to ignore a dependency - by using a line like the following: - - PACKAGE_ARCHS[vardepsexclude] = "MACHINE" - - This example ensures that the - PACKAGE_ARCHS - variable does not - depend on the value of - MACHINE, - even if it does reference it. - - - - Equally, there are cases where we need to add dependencies BitBake is not able to find. - You can accomplish this by using a line like the following: - - PACKAGE_ARCHS[vardeps] = "MACHINE" - - This example explicitly adds the MACHINE variable as a - dependency for PACKAGE_ARCHS. - - - - Consider a case with in-line Python, for example, where BitBake is not - able to figure out dependencies. - When running in debug mode (i.e. using -DDD), BitBake - produces output when it discovers something for which it cannot figure out - dependencies. - The Yocto Project team has currently not managed to cover those dependencies - in detail and is aware of the need to fix this situation. - - - - Thus far, this section has limited discussion to the direct inputs into a task. - Information based on direct inputs is referred to as the "basehash" in the - code. - However, there is still the question of a task's indirect inputs - the - things that were already built and present in the - Build Directory. - The checksum (or signature) for a particular task needs to add the hashes - of all the tasks on which the particular task depends. - Choosing which dependencies to add is a policy decision. - However, the effect is to generate a master checksum that combines the basehash - and the hashes of the task's dependencies. - - - - At the code level, there are a variety of ways both the basehash and the - dependent task hashes can be influenced. - Within the BitBake configuration file, we can give BitBake some extra information - to help it construct the basehash. - The following statement effectively results in a list of global variable - dependency excludes - variables never included in any checksum: - - BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ - SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ - USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ - PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ - CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" - - The previous example excludes - WORKDIR - since that variable is actually constructed as a path within - TMPDIR, which is on - the whitelist. - - - - The rules for deciding which hashes of dependent tasks to include through - dependency chains are more complex and are generally accomplished with a - Python function. - The code in meta/lib/oe/sstatesig.py shows two examples - of this and also illustrates how you can insert your own policy into the system - if so desired. - This file defines the two basic signature generators - OE-Core uses: "OEBasic" and - "OEBasicHash". - By default, there is a dummy "noop" signature handler enabled in BitBake. - This means that behavior is unchanged from previous versions. - OE-Core uses the "OEBasicHash" signature handler by default - through this setting in the bitbake.conf file: - - BB_SIGNATURE_HANDLER ?= "OEBasicHash" - - The "OEBasicHash" BB_SIGNATURE_HANDLER is the same as the - "OEBasic" version but adds the task hash to the stamp files. - This results in any - Metadata - change that changes the task hash, automatically - causing the task to be run again. - This removes the need to bump PR - values, and changes to Metadata automatically ripple across the build. - - - - It is also worth noting that the end result of these signature generators is to - make some dependency and hash information available to the build. - This information includes: - - BB_BASEHASH_task-taskname: - The base hashes for each task in the recipe. - - BB_BASEHASH_filename:taskname: - The base hashes for each dependent task. - - BBHASHDEPS_filename:taskname: - The task dependencies for each task. - - BB_TASKHASH: - The hash of the currently running task. - - - -
- -
- Shared State - - - Checksums and dependencies, as discussed in the previous section, solve half the - problem of supporting a shared state. - The other part of the problem is being able to use checksum information during the build - and being able to reuse or rebuild specific components. - - - - The - sstate - class is a relatively generic implementation of how to "capture" - a snapshot of a given task. - The idea is that the build process does not care about the source of a task's output. - Output could be freshly built or it could be downloaded and unpacked from - somewhere - the build process does not need to worry about its origin. - - - - There are two types of output, one is just about creating a directory - in WORKDIR. - A good example is the output of either - do_install - or - do_package. - The other type of output occurs when a set of data is merged into a shared directory - tree such as the sysroot. - - - - The Yocto Project team has tried to keep the details of the - implementation hidden in sstate class. - From a user's perspective, adding shared state wrapping to a task - is as simple as this - do_deploy - example taken from the - deploy - class: - - DEPLOYDIR = "${WORKDIR}/deploy-${PN}" - SSTATETASKS += "do_deploy" - do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" - do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" - - python do_deploy_setscene () { - sstate_setscene(d) - } - addtask do_deploy_setscene - do_deploy[dirs] = "${DEPLOYDIR} ${B}" - - The following list explains the previous example: - - - Adding "do_deploy" to SSTATETASKS - adds some required sstate-related processing, which is - implemented in the - sstate - class, to before and after the - do_deploy - task. - - - The - do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" - declares that do_deploy places its - output in ${DEPLOYDIR} when run - normally (i.e. when not using the sstate cache). - This output becomes the input to the shared state cache. - - - The - do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" - line causes the contents of the shared state cache to be - copied to ${DEPLOY_DIR_IMAGE}. - - If do_deploy is not already in - the shared state cache or if its input checksum - (signature) has changed from when the output was - cached, the task will be run to populate the shared - state cache, after which the contents of the shared - state cache is copied to - ${DEPLOY_DIR_IMAGE}. - If do_deploy is in the shared - state cache and its signature indicates that the - cached output is still valid (i.e. if no - relevant task inputs have changed), then the contents - of the shared state cache will be copied directly to - ${DEPLOY_DIR_IMAGE} by the - do_deploy_setscene task instead, - skipping the do_deploy task. - - - - The following task definition is glue logic needed to make - the previous settings effective: - - python do_deploy_setscene () { - sstate_setscene(d) - } - addtask do_deploy_setscene - - sstate_setscene() takes the flags - above as input and accelerates the - do_deploy task through the - shared state cache if possible. - If the task was accelerated, - sstate_setscene() returns True. - Otherwise, it returns False, and the normal - do_deploy task runs. - For more information, see the - "setscene" - section in the BitBake User Manual. - - - The do_deploy[dirs] = "${DEPLOYDIR} ${B}" - line creates ${DEPLOYDIR} and - ${B} before the - do_deploy task runs, and also sets - the current working directory of - do_deploy to - ${B}. - For more information, see the - "Variable Flags" - section in the BitBake User Manual. - - In cases where - sstate-inputdirs and - sstate-outputdirs would be the - same, you can use - sstate-plaindirs. - For example, to preserve the - ${PKGD} and - ${PKGDEST} output from the - do_package - task, use the following: - - do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" - - - - - sstate-inputdirs and - sstate-outputdirs can also be used - with multiple directories. - For example, the following declares - PKGDESTWORK and - SHLIBWORK as shared state - input directories, which populates the shared state - cache, and PKGDATA_DIR and - SHLIBSDIR as the corresponding - shared state output directories: - - do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" - do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" - - - - These methods also include the ability to take a lockfile - when manipulating shared state directory structures, - for cases where file additions or removals are sensitive: - - do_package[sstate-lockfile] = "${PACKAGELOCK}" - - - - - - - - - Behind the scenes, the shared state code works by looking in - SSTATE_DIR and - SSTATE_MIRRORS - for shared state files. - Here is an example: - - SSTATE_MIRRORS ?= "\ - file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ - file://.* file:///some/local/dir/sstate/PATH" - - - The shared state directory (SSTATE_DIR) is - organized into two-character subdirectories, where the subdirectory - names are based on the first two characters of the hash. - If the shared state directory structure for a mirror has the - same structure as SSTATE_DIR, you must - specify "PATH" as part of the URI to enable the build system - to map to the appropriate subdirectory. - - - - - The shared state package validity can be detected just by looking at the - filename since the filename contains the task checksum (or signature) as - described earlier in this section. - If a valid shared state package is found, the build process downloads it - and uses it to accelerate the task. - - - - The build processes use the *_setscene tasks - for the task acceleration phase. - BitBake goes through this phase before the main execution code and tries - to accelerate any tasks for which it can find shared state packages. - If a shared state package for a task is available, the shared state - package is used. - This means the task and any tasks on which it is dependent are not - executed. - - - - As a real world example, the aim is when building an IPK-based image, - only the - do_package_write_ipk - tasks would have their - shared state packages fetched and extracted. - Since the sysroot is not used, it would never get extracted. - This is another reason why a task-based approach is preferred over a - recipe-based approach, which would have to install the output from every task. - -
- -
- Tips and Tricks - - - The code in the build system that supports incremental builds is not - simple code. - This section presents some tips and tricks that help you work around - issues related to shared state code. - - -
- Debugging - - - Seeing what metadata went into creating the input signature - of a shared state (sstate) task can be a useful debugging aid. - This information is available in signature information - (siginfo) files in - SSTATE_DIR. - For information on how to view and interpret information in - siginfo files, see the - "Viewing Task Variable Dependencies" - section. - -
- -
- Invalidating Shared State - - - The OpenEmbedded build system uses checksums and shared state - cache to avoid unnecessarily rebuilding tasks. - Collectively, this scheme is known as "shared state code." - - - - As with all schemes, this one has some drawbacks. - It is possible that you could make implicit changes to your - code that the checksum calculations do not take into - account. - These implicit changes affect a task's output but do not trigger - the shared state code into rebuilding a recipe. - Consider an example during which a tool changes its output. - Assume that the output of rpmdeps changes. - The result of the change should be that all the - package and - package_write_rpm shared state cache - items become invalid. - However, because the change to the output is - external to the code and therefore implicit, - the associated shared state cache items do not become - invalidated. - In this case, the build process uses the cached items rather - than running the task again. - Obviously, these types of implicit changes can cause problems. - - - - To avoid these problems during the build, you need to - understand the effects of any changes you make. - Realize that changes you make directly to a function - are automatically factored into the checksum calculation. - Thus, these explicit changes invalidate the associated area of - shared state cache. - However, you need to be aware of any implicit changes that - are not obvious changes to the code and could affect the output - of a given task. - - - - When you identify an implicit change, you can easily take steps - to invalidate the cache and force the tasks to run. - The steps you can take are as simple as changing a function's - comments in the source code. - For example, to invalidate package shared state files, change - the comment statements of - do_package - or the comments of one of the functions it calls. - Even though the change is purely cosmetic, it causes the - checksum to be recalculated and forces the OpenEmbedded build - system to run the task again. - - - - For an example of a commit that makes a cosmetic change to - invalidate shared state, see this - commit. - -
-
-
-
Automatically Added Runtime Dependencies diff --git a/documentation/ref-manual/usingpoky.xml b/documentation/ref-manual/usingpoky.xml index f1fd452a06..73b5df88dd 100644 --- a/documentation/ref-manual/usingpoky.xml +++ b/documentation/ref-manual/usingpoky.xml @@ -535,7 +535,7 @@ ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 For tasks that are accelerated through the shared state - (sstate) + (sstate) cache, an additional siginfo file is written into SSTATE_DIR -- cgit v1.2.3-54-g00ecf