overview-manual, ref-manual: Moved "Shared State Cache" to overview manual

Fixes [YOCTO #12370]

The section on shared state cache needed to be in the overview manual
and not in the ref-manual.  I moved it.  Some links were affected,
which I fixed.

(From yocto-docs rev: 1c4e5207bdde19d4b48ef42b1de81390d8a02d64)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

2 files changed, 636 insertions, 1 deletions

diff --git a/documentation/overview-manual/overview-concepts.xml b/documentation/overview-manual/overview-concepts.xml
index 0a45cd7256..2d704923fa 100644
--- a/documentation/overview-manual/overview-concepts.xml
+++ b/documentation/overview-manual/overview-concepts.xml
@@ -471,6 +471,641 @@
         </note>
     </section>
 
+
+
+
+    <section id="shared-state-cache">
+        <title>Shared State Cache</title>
+
+        <para>
+            By design, the OpenEmbedded build system builds everything from
+            scratch unless BitBake can determine that parts do not need to be
+            rebuilt.
+            Fundamentally, building from scratch is attractive as it means all
+            parts are built fresh and there is no possibility of stale data
+            causing problems.
+            When developers hit problems, they typically default back to
+            building from scratch so they know the state of things from the
+            start.
+        </para>
+
+        <para>
+            Building an image from scratch is both an advantage and a
+            disadvantage to the process.
+            As mentioned in the previous paragraph, building from scratch
+            ensures that everything is current and starts from a known state.
+            However, building from scratch also takes much longer as it
+            generally means rebuilding things that do not necessarily need
+            to be rebuilt.
+        </para>
+
+        <para>
+            The Yocto Project implements shared state code that supports
+            incremental builds.
+            The implementation of the shared state code answers the following
+            questions that were fundamental roadblocks within the OpenEmbedded
+            incremental build support system:
+            <itemizedlist>
+                <listitem><para>
+                    What pieces of the system have changed and what pieces have
+                    not changed?
+                    </para></listitem>
+                <listitem><para>
+                    How are changed pieces of software removed and replaced?
+                    </para></listitem>
+                <listitem><para>
+                    How are pre-built components that do not need to be rebuilt
+                    from scratch used when they are available?
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            For the first question, the build system detects changes in the
+            "inputs" to a given task by creating a checksum (or signature) of
+            the task's inputs.
+            If the checksum changes, the system assumes the inputs have changed
+            and the task needs to be rerun.
+            For the second question, the shared state (sstate) code tracks
+            which tasks add which output to the build process.
+            This means the output from a given task can be removed, upgraded
+            or otherwise manipulated.
+            The third question is partly addressed by the solution for the
+            second question assuming the build system can fetch the sstate
+            objects from remote locations and install them if they are deemed
+            to be valid.
+            <note>
+                The OpenEmbedded build system does not maintain
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
+                information as part of the shared state packages.
+                Consequently, considerations exist that affect maintaining
+                shared state feeds.
+                For information on how the OpenEmbedded build system
+                works with packages and can track incrementing
+                <filename>PR</filename> information, see the
+                "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
+                section in the Yocto Project Development Tasks Manual.
+            </note>
+        </para>
+
+        <para>
+            The rest of this section goes into detail about the overall
+            incremental build architecture, the checksums (signatures), shared
+            state, and some tips and tricks.
+        </para>
+
+        <section id='overall-architecture'>
+            <title>Overall Architecture</title>
+
+            <para>
+                When determining what parts of the system need to be built,
+                BitBake works on a per-task basis rather than a per-recipe
+                basis.
+                You might wonder why using a per-task basis is preferred over
+                a per-recipe basis.
+                To help explain, consider having the IPK packaging backend
+                enabled and then switching to DEB.
+                In this case, the
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
+                and
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
+                task outputs are still valid.
+                However, with a per-recipe approach, the build would not
+                include the <filename>.deb</filename> files.
+                Consequently, you would have to invalidate the whole build and
+                rerun it.
+                Rerunning everything is not the best solution.
+                Also, in this case, the core must be "taught" much about
+                specific tasks.
+                This methodology does not scale well and does not allow users
+                to easily add new tasks in layers or as external recipes
+                without touching the packaged-staging core.
+            </para>
+        </section>
+
+        <section id='overview-checksums'>
+            <title>Checksums (Signatures)</title>
+
+            <para>
+                The shared state code uses a checksum, which is a unique
+                signature of a task's inputs, to determine if a task needs to
+                be run again.
+                Because it is a change in a task's inputs that triggers a
+                rerun, the process needs to detect all the inputs to a given
+                task.
+                For shell tasks, this turns out to be fairly easy because
+                the build process generates a "run" shell script for each task
+                and it is possible to create a checksum that gives you a good
+                idea of when the task's data changes.
+            </para>
+
+            <para>
+                To complicate the problem, there are things that should not be
+                included in the checksum.
+                First, there is the actual specific build path of a given
+                task - the
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
+                It does not matter if the work directory changes because it
+                should not affect the output for target packages.
+                Also, the build process has the objective of making native
+                or cross packages relocatable.
+                <note>
+                    Both native and cross packages run on the build host.
+                    However, cross packages generate output for the target
+                    architecture.
+                </note>
+                The checksum therefore needs to exclude
+                <filename>WORKDIR</filename>.
+                The simplistic approach for excluding the work directory is to
+                set <filename>WORKDIR</filename> to some fixed value and
+                create the checksum for the "run" script.
+            </para>
+
+            <para>
+                Another problem results from the "run" scripts containing
+                functions that might or might not get called.
+                The incremental build solution contains code that figures out
+                dependencies between shell functions.
+                This code is used to prune the "run" scripts down to the
+                minimum set, thereby alleviating this problem and making the
+                "run" scripts much more readable as a bonus.
+            </para>
+
+            <para>
+                So far we have solutions for shell scripts.
+                What about Python tasks?
+                The same approach applies even though these tasks are more
+                difficult.
+                The process needs to figure out what variables a Python
+                function accesses and what functions it calls.
+                Again, the incremental build solution contains code that first
+                figures out the variable and function dependencies, and then
+                creates a checksum for the data used as the input to the task.
+            </para>
+
+            <para>
+                Like the <filename>WORKDIR</filename> case, situations exist
+                where dependencies should be ignored.
+                For these cases, you can instruct the build process to
+                ignore a dependency by using a line like the following:
+                <literallayout class='monospaced'>
+     PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
+                </literallayout>
+                This example ensures that the
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
+                variable does not depend on the value of
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
+                even if it does reference it.
+            </para>
+
+            <para>
+                Equally, there are cases where we need to add dependencies
+                BitBake is not able to find.
+                You can accomplish this by using a line like the following:
+                <literallayout class='monospaced'>
+      PACKAGE_ARCHS[vardeps] = "MACHINE"
+                </literallayout>
+                This example explicitly adds the <filename>MACHINE</filename>
+                variable as a dependency for
+                <filename>PACKAGE_ARCHS</filename>.
+            </para>
+
+            <para>
+                Consider a case with in-line Python, for example, where
+                BitBake is not able to figure out dependencies.
+                When running in debug mode (i.e. using
+                <filename>-DDD</filename>), BitBake produces output when it
+                discovers something for which it cannot figure out dependencies.
+                The Yocto Project team has currently not managed to cover
+                those dependencies in detail and is aware of the need to fix
+                this situation.
+            </para>
+
+            <para>
+                Thus far, this section has limited discussion to the direct
+                inputs into a task.
+                Information based on direct inputs is referred to as the
+                "basehash" in the code.
+                However, there is still the question of a task's indirect
+                inputs - the things that were already built and present in the
+                <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
+                The checksum (or signature) for a particular task needs to add
+                the hashes of all the tasks on which the particular task
+                depends.
+                Choosing which dependencies to add is a policy decision.
+                However, the effect is to generate a master checksum that
+                combines the basehash and the hashes of the task's
+                dependencies.
+            </para>
+
+            <para>
+                At the code level, there are a variety of ways both the
+                basehash and the dependent task hashes can be influenced.
+                Within the BitBake configuration file, we can give BitBake
+                some extra information to help it construct the basehash.
+                The following statement effectively results in a list of
+                global variable dependency excludes - variables never
+                included in any checksum:
+                <literallayout class='monospaced'>
+     BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
+         SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
+         USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
+         PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
+         CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
+                </literallayout>
+                The previous example excludes
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
+                since that variable is actually constructed as a path within
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
+                which is on the whitelist.
+            </para>
+
+            <para>
+                The rules for deciding which hashes of dependent tasks to
+                include through dependency chains are more complex and are
+                generally accomplished with a Python function.
+                The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
+                two examples of this and also illustrates how you can insert
+                your own policy into the system if so desired.
+                This file defines the two basic signature generators
+                <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
+                uses:  "OEBasic" and "OEBasicHash".
+                By default, there is a dummy "noop" signature handler enabled
+                in BitBake.
+                This means that behavior is unchanged from previous versions.
+                OE-Core uses the "OEBasicHash" signature handler by default
+                through this setting in the <filename>bitbake.conf</filename>
+                file:
+                <literallayout class='monospaced'>
+     BB_SIGNATURE_HANDLER ?= "OEBasicHash"
+                </literallayout>
+                The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
+                is the same as the "OEBasic" version but adds the task hash to
+                the stamp files.
+                This results in any
+                <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
+                change that changes the task hash, automatically
+                causing the task to be run again.
+                This removes the need to bump
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
+                values, and changes to Metadata automatically ripple across
+                the build.
+            </para>
+
+            <para>
+                It is also worth noting that the end result of these
+                signature generators is to make some dependency and hash
+                information available to the build.
+                This information includes:
+                <itemizedlist>
+                    <listitem><para>
+                        <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
+                        The base hashes for each task in the recipe.
+                        </para></listitem>
+                    <listitem><para>
+                        <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+                        The base hashes for each dependent task.
+                        </para></listitem>
+                    <listitem><para>
+                        <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+                        The task dependencies for each task.
+                        </para></listitem>
+                    <listitem><para>
+                        <filename>BB_TASKHASH</filename>:
+                        The hash of the currently running task.
+                        </para></listitem>
+                </itemizedlist>
+            </para>
+        </section>
+
+        <section id='shared-state'>
+            <title>Shared State</title>
+
+            <para>
+                Checksums and dependencies, as discussed in the previous
+                section, solve half the problem of supporting a shared state.
+                The other part of the problem is being able to use checksum
+                information during the build and being able to reuse or rebuild
+                specific components.
+            </para>
+
+            <para>
+                The
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
+                class is a relatively generic implementation of how to
+                "capture" a snapshot of a given task.
+                The idea is that the build process does not care about the
+                source of a task's output.
+                Output could be freshly built or it could be downloaded and
+                unpacked from somewhere - the build process does not need to
+                worry about its origin.
+            </para>
+
+            <para>
+                There are two types of output, one is just about creating a
+                directory in
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
+                A good example is the output of either
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
+                or
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
+                The other type of output occurs when a set of data is merged
+                into a shared directory tree such as the sysroot.
+            </para>
+
+            <para>
+                The Yocto Project team has tried to keep the details of the
+                implementation hidden in <filename>sstate</filename> class.
+                From a user's perspective, adding shared state wrapping to a task
+                is as simple as this
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
+                example taken from the
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
+                class:
+                <literallayout class='monospaced'>
+     DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+     SSTATETASKS += "do_deploy"
+     do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
+     do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+
+     python do_deploy_setscene () {
+         sstate_setscene(d)
+     }
+     addtask do_deploy_setscene
+     do_deploy[dirs] = "${DEPLOYDIR} ${B}"
+                </literallayout>
+                The following list explains the previous example:
+                <itemizedlist>
+                    <listitem><para>
+                        Adding "do_deploy" to <filename>SSTATETASKS</filename>
+                        adds some required sstate-related processing, which is
+                        implemented in the
+                        <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
+                        class, to before and after the
+                        <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
+                        task.
+                        </para></listitem>
+                    <listitem><para>
+                        The
+                        <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
+                        declares that <filename>do_deploy</filename> places its
+                        output in <filename>${DEPLOYDIR}</filename> when run
+                        normally (i.e. when not using the sstate cache).
+                        This output becomes the input to the shared state cache.
+                        </para></listitem>
+                    <listitem><para>
+                        The
+                        <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
+                        line causes the contents of the shared state cache to be
+                        copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
+                        <note>
+                            If <filename>do_deploy</filename> is not already in
+                            the shared state cache or if its input checksum
+                            (signature) has changed from when the output was
+                            cached, the task will be run to populate the shared
+                            state cache, after which the contents of the shared
+                            state cache is copied to
+                            <filename>${DEPLOY_DIR_IMAGE}</filename>.
+                            If <filename>do_deploy</filename> is in the shared
+                            state cache and its signature indicates that the
+                            cached output is still valid (i.e. if no
+                            relevant task inputs have changed), then the
+                            contents of the shared state cache will be copied
+                            directly to
+                            <filename>${DEPLOY_DIR_IMAGE}</filename> by the
+                            <filename>do_deploy_setscene</filename> task
+                            instead, skipping the
+                            <filename>do_deploy</filename> task.
+                        </note>
+                        </para></listitem>
+                    <listitem><para>
+                        The following task definition is glue logic needed to
+                        make the previous settings effective:
+                        <literallayout class='monospaced'>
+     python do_deploy_setscene () {
+         sstate_setscene(d)
+     }
+     addtask do_deploy_setscene
+                        </literallayout>
+                        <filename>sstate_setscene()</filename> takes the flags
+                        above as input and accelerates the
+                        <filename>do_deploy</filename> task through the
+                        shared state cache if possible.
+                        If the task was accelerated,
+                        <filename>sstate_setscene()</filename> returns True.
+                        Otherwise, it returns False, and the normal
+                        <filename>do_deploy</filename> task runs.
+                        For more information, see the
+                        "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
+                        section in the BitBake User Manual.
+                        </para></listitem>
+                    <listitem><para>
+                        The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
+                        line creates <filename>${DEPLOYDIR}</filename> and
+                        <filename>${B}</filename> before the
+                        <filename>do_deploy</filename> task runs, and also sets
+                        the current working directory of
+                        <filename>do_deploy</filename> to
+                        <filename>${B}</filename>.
+                        For more information, see the
+                        "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
+                        section in the BitBake User Manual.
+                        <note>
+                            In cases where
+                            <filename>sstate-inputdirs</filename> and
+                            <filename>sstate-outputdirs</filename> would be the
+                            same, you can use
+                            <filename>sstate-plaindirs</filename>.
+                            For example, to preserve the
+                            <filename>${PKGD}</filename> and
+                            <filename>${PKGDEST}</filename> output from the
+                            <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
+                            task, use the following:
+                            <literallayout class='monospaced'>
+     do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
+                            </literallayout>
+                        </note>
+                        </para></listitem>
+                    <listitem><para>
+                        <filename>sstate-inputdirs</filename> and
+                        <filename>sstate-outputdirs</filename> can also be used
+                        with multiple directories.
+                        For example, the following declares
+                        <filename>PKGDESTWORK</filename> and
+                        <filename>SHLIBWORK</filename> as shared state
+                        input directories, which populates the shared state
+                        cache, and <filename>PKGDATA_DIR</filename> and
+                        <filename>SHLIBSDIR</filename> as the corresponding
+                        shared state output directories:
+                        <literallayout class='monospaced'>
+     do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
+     do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
+                        </literallayout>
+                        </para></listitem>
+                    <listitem><para>
+                        These methods also include the ability to take a
+                        lockfile when manipulating shared state directory
+                        structures, for cases where file additions or removals
+                        are sensitive:
+                        <literallayout class='monospaced'>
+     do_package[sstate-lockfile] = "${PACKAGELOCK}"
+                        </literallayout>
+                        </para></listitem>
+                </itemizedlist>
+            </para>
+
+            <para>
+                Behind the scenes, the shared state code works by looking in
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
+                and
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
+                for shared state files.
+                Here is an example:
+                <literallayout class='monospaced'>
+     SSTATE_MIRRORS ?= "\
+     file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
+     file://.* file:///some/local/dir/sstate/PATH"
+                </literallayout>
+                <note>
+                    The shared state directory
+                    (<filename>SSTATE_DIR</filename>) is organized into
+                    two-character subdirectories, where the subdirectory
+                    names are based on the first two characters of the hash.
+                    If the shared state directory structure for a mirror has the
+                    same structure as <filename>SSTATE_DIR</filename>, you must
+                    specify "PATH" as part of the URI to enable the build system
+                    to map to the appropriate subdirectory.
+                </note>
+            </para>
+
+            <para>
+                The shared state package validity can be detected just by
+                looking at the filename since the filename contains the task
+                checksum (or signature) as described earlier in this section.
+                If a valid shared state package is found, the build process
+                downloads it and uses it to accelerate the task.
+            </para>
+
+            <para>
+                The build processes use the <filename>*_setscene</filename>
+                tasks for the task acceleration phase.
+                BitBake goes through this phase before the main execution
+                code and tries to accelerate any tasks for which it can find
+                shared state packages.
+                If a shared state package for a task is available, the
+                shared state package is used.
+                This means the task and any tasks on which it is dependent
+                are not executed.
+            </para>
+
+            <para>
+                As a real world example, the aim is when building an IPK-based
+                image, only the
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
+                tasks would have their shared state packages fetched and
+                extracted.
+                Since the sysroot is not used, it would never get extracted.
+                This is another reason why a task-based approach is preferred
+                over a recipe-based approach, which would have to install the
+                output from every task.
+            </para>
+        </section>
+
+        <section id='tips-and-tricks'>
+            <title>Tips and Tricks</title>
+
+            <para>
+                The code in the build system that supports incremental builds
+                is not simple code.
+                This section presents some tips and tricks that help you work
+                around issues related to shared state code.
+            </para>
+
+            <section id='overview-debugging'>
+                <title>Debugging</title>
+
+                <para>
+                    Seeing what metadata went into creating the input signature
+                    of a shared state (sstate) task can be a useful debugging
+                    aid.
+                    This information is available in signature information
+                    (<filename>siginfo</filename>) files in
+                    <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>.
+                    For information on how to view and interpret information in
+                    <filename>siginfo</filename> files, see the
+                    "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
+                    section in the Yocto Project Reference Manual.
+                </para>
+            </section>
+
+            <section id='invalidating-shared-state'>
+                <title>Invalidating Shared State</title>
+
+                <para>
+                    The OpenEmbedded build system uses checksums and shared
+                    state cache to avoid unnecessarily rebuilding tasks.
+                    Collectively, this scheme is known as "shared state code."
+                </para>
+
+                <para>
+                    As with all schemes, this one has some drawbacks.
+                    It is possible that you could make implicit changes to your
+                    code that the checksum calculations do not take into
+                    account.
+                    These implicit changes affect a task's output but do not
+                    trigger the shared state code into rebuilding a recipe.
+                    Consider an example during which a tool changes its output.
+                    Assume that the output of <filename>rpmdeps</filename>
+                    changes.
+                    The result of the change should be that all the
+                    <filename>package</filename> and
+                    <filename>package_write_rpm</filename> shared state cache
+                    items become invalid.
+                    However, because the change to the output is
+                    external to the code and therefore implicit,
+                    the associated shared state cache items do not become
+                    invalidated.
+                    In this case, the build process uses the cached items
+                    rather than running the task again.
+                    Obviously, these types of implicit changes can cause
+                    problems.
+                </para>
+
+                <para>
+                    To avoid these problems during the build, you need to
+                    understand the effects of any changes you make.
+                    Realize that changes you make directly to a function
+                    are automatically factored into the checksum calculation.
+                    Thus, these explicit changes invalidate the associated
+                    area of shared state cache.
+                    However, you need to be aware of any implicit changes that
+                    are not obvious changes to the code and could affect
+                    the output of a given task.
+                </para>
+
+                <para>
+                    When you identify an implicit change, you can easily
+                    take steps to invalidate the cache and force the tasks
+                    to run.
+                    The steps you can take are as simple as changing a
+                    function's comments in the source code.
+                    For example, to invalidate package shared state files,
+                    change the comment statements of
+                    <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
+                    or the comments of one of the functions it calls.
+                    Even though the change is purely cosmetic, it causes the
+                    checksum to be recalculated and forces the OpenEmbedded
+                    build system to run the task again.
+                    <note>
+                        For an example of a commit that makes a cosmetic
+                        change to invalidate shared state, see this
+                        <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
+                    </note>
+                </para>
+            </section>
+        </section>
+    </section>
+
     <section id='x32'>
         <title>x32 psABI</title>
 
diff --git a/documentation/overview-manual/overview-development-environment.xml b/documentation/overview-manual/overview-development-environment.xml
index 62f3ccd438..e0d470b179 100644
--- a/documentation/overview-manual/overview-development-environment.xml
+++ b/documentation/overview-manual/overview-development-environment.xml
@@ -2490,7 +2490,7 @@
                 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
                 variable, and the end of the name consists of the task's name
                 and current
-                <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
+                <link linkend='overview-checksums'>input checksum</link>.
                 <note>
                     This naming scheme assumes that
                     <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>