documentation/poky-ref-manual/usingpoky.xml: partial for YOCTO #1500

First draft of a re-write to the "Running a Build" section to try
and satisfy YOCTO #1500.  I segmented the section into three areas
rather than a single area.  This allowed me to create a sub-section
for the sstate stuff where it could be addressed on its own.  I sent
the draft out to Richard and Mark H. and got feedback from RP that
is going to cause further changes.  Thus, I am committing this partial
change.

(From yocto-docs rev: 4a99bcbda85ede0bd9490002c08461cc834b7518)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

1 files changed, 233 insertions, 47 deletions

diff --git a/documentation/poky-ref-manual/usingpoky.xml b/documentation/poky-ref-manual/usingpoky.xml
index d01cbaffa6..712dcc6fa9 100644
--- a/documentation/poky-ref-manual/usingpoky.xml
+++ b/documentation/poky-ref-manual/usingpoky.xml
@@ -152,67 +152,253 @@
     <title>Running a Build</title>
 
     <para>
-        You can find information on how to build an image using the Yocto Project in the 
+        You can find general information on how to build an image using the 
+        Yocto Project in the 
         <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html#building-image'>
         Building an Image</ulink> section of the 
         <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html'>
         Yocto Project Quick Start</ulink>.
-        This section provides a quick overview.
+        This section provides a summary of the build process and provides information
+        for less obvious aspects of the build process.
     </para>
 
-    <para>
-        The first thing you need to do is set up the Yocto Project build environment by sourcing
-        the environment setup script as follows:
-        <literallayout class='monospaced'>
+    <section id='build-overview'>
+        <title>Build Overview</title>
+
+        <para>
+            The first thing you need to do is set up the Yocto Project build environment by sourcing
+            the environment setup script as follows:
+            <literallayout class='monospaced'>
      $ source oe-init-build-env [build_dir]
-        </literallayout>
-    </para>
+            </literallayout>
+        </para>
 
-    <para>
-        The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
-        uses for the build.
-        If you do not specify a build directory it defaults to <filename>build</filename>
-        in your current working directory.
-        A common practice is to use a different build directory for different targets. 
-        For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
-        target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
-        See <link linkend="structure-core-script">oe-init-build-env</link>
-        for more information on this script.
-    </para>
+        <para>
+            The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
+            uses for the build.
+            If you do not specify a build directory it defaults to <filename>build</filename>
+            in your current working directory.
+            A common practice is to use a different build directory for different targets. 
+            For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
+            target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
+            See <link linkend="structure-core-script">oe-init-build-env</link>
+            for more information on this script.
+        </para>
 
-    <para>
-        Once the Yocto Project build environment is set up, you can build a target using:
-        <literallayout class='monospaced'>
+        <para>
+            Once the Yocto Project build environment is set up, you can build a target using:
+            <literallayout class='monospaced'>
      $ bitbake &lt;target&gt;
-        </literallayout>
-    </para>
+            </literallayout>
+        </para>
 
-    <para>
-        The <filename>target</filename> is the name of the recipe you want to build. 
-        Common targets are the images in <filename>meta/recipes-core/images</filename>,
-        <filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project
-        files.
-        Or, the target can be the name of a recipe for a specific piece of software such as 
-        <application>busybox</application>. 
-        For more details about the images Yocto Project supports, see the 
-        <link linkend="ref-images">'Reference: Images'</link> appendix.
-    </para>
+        <para>
+            The <filename>target</filename> is the name of the recipe you want to build. 
+            Common targets are the images in <filename>meta/recipes-core/images</filename>,
+            <filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project
+            files.
+            Or, the target can be the name of a recipe for a specific piece of software such as 
+            <application>busybox</application>. 
+            For more details about the images Yocto Project supports, see the 
+            <link linkend="ref-images">'Reference: Images'</link> appendix.
+        </para>
+
+        <note>
+            Building an image without GNU Public License Version 3 (GPLv3) components is 
+            only supported for minimal and base images.
+            See <link linkend='ref-images'>'Reference: Images'</link> for more information.
+        </note>
+    </section>
+
+    <section id='building-an-image-using-gpl-components'>
+        <title>Building an Image Using GPL Components</title>
+
+        <para>
+            When building an image using GPL components, you need to maintain your original 
+            settings and not switch back and forth applying different versions of the GNU
+            Public License.  
+            If you rebuild using different versions of GPL, dependency errors might occur
+            due to some components not being rebuilt.
+        </para>
+    </section>
 
-    <note>
-        Building an image without GNU Public License Version 3 (GPLv3) components is 
-        only supported for minimal and base images.
-        See <link linkend='ref-images'>'Reference: Images'</link> for more information.
-    </note>
-
-    <note>
-        When building an image using GPL components, you need to maintain your original 
-        settings and not switch back and forth applying different versions of the GNU
-        Public License.  
-        If you rebuild using different versions of GPL, dependency errors might occur
-        due to some components not being rebuilt.
-    </note>
+    <section id="considering-shared-state-cache">
+        <title>Considering Shared State Cache</title>
+
+        <para>
+            By design, the Yocto Project builds everything from scratch unless it can determine that
+            a given task's inputs have not changed.  
+            While building from scratch ensures that everything is current, it does also
+            mean that a lot of time could be spent rebuiding things that don't necessarily need built.
+        </para>
+
+        <para>
+            The Yocto Project build process uses a shared state caching scheme to avoid having to 
+            rebuild software when it is not necessary.  
+            Because the build time for a Yocto image can be significant, it is helpful to try and 
+            determine what really needs built and what can be skipped given a particular project's
+            build process.
+        </para>
+
+        <para>
+            The scheme that the Yocto Project uses involves checksum generation and comparison for
+            a task's inputs.
+            The scheme also employs an area of memory called the shared state cache that is 
+            pointed to by the <filename>SSTATE_DIR</filename> variable.
+            This area contains task output generated from a previous build.  
+            If a given task's checksum matches the checksum of a previous build for the same  
+            task, the build process uses the state of the cache rather than rerunning that 
+            task.
+        </para>
+
+        <para>
+            The previous paragraph is a simplistic explanation of how the build process 
+            uses checksums and shared state memory cache to avoide building tasks that 
+            don't need built.
+            If you want a bit more explanation on the topic, 
+            see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
+            State - What does it mean and why should I care?</ulink>" from the Yocto 
+            Project discussion archives.
+        </para>
+
+        <para>
+            As with all schemes, this one has some drawbacks.
+            It is possible that you could make implicit changes that are not factored into the checksum 
+            calculation, but do affect a task's output. 
+            A good example is perhaps when a tool changes its output.
+            Let's say that the output of <filename>rpmdeps</filename> needed to change.
+            The result of the change should be that all the "package", "package_write_rpm",
+            and "package_deploy-rpm" sstate-cache items would become invalid.
+            But, because this is a change that is external to the code and therefore implicit,
+            the associated sstate-cache items do not become invalidated.
+            In this case, the build process would use the cache items rather than running the 
+            task again. 
+            Obviously, these types of implicit changes can cause problems.
+        </para>
+
+        <para>
+            To avoid these problems during the build, you need to understand the effects of any
+            change you make.
+            Note that any changes you make directly to a function automatically are factored into 
+            the checksum calculation and thus, will invalidate the associated area of sstate cache.
+            You need to be aware of any implicit changes that are not obvious changes to the 
+            code and could affect the output of a given task. 
+            Once you are aware of such a change, you can take steps to invalidate the cache 
+            and force the task to run. 
+            The step to take is as simple as changing a function's comments in the source code. 
+            For example, to invalidate package sstate files, change the comment statments
+            of <filename>do_package</filename> or one of the functions it calls.
+            The change is purely cosmetic, but it causes the checksum to be recalculated and  
+            forces the task to be run again.
+        </para>
+
+        <note>
+            For an example of a commit that makes a cosmetic change to invalidate an sstate, 
+            see this
+            <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
+        </note>
+    </section>
+ 
+
+<!--
+
+        <section id="considering-shared-state-cache">
+            <title>Considering Shared State Cache</title>
+
+            <para>
+                What is shared state in general.
+                Benefits?
+                How we handle things 
+                (reference https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm),
+                which is RP's dissertation on how YP solved it.
+                We need to talk a bit about checksum generation for tasks and how the 
+                sstate code uses them to figure out what needs rebuilt and what can be re-loaded
+                from the sstate cache.  
+                Need to tell about cases where an implicit change can mess things up and under
+                normal situations the state in the sstate cache would be used but it shouldn't be.
+                This is the scenario described by bug 1500 - typical case.  
+                Then we talk about how we can invalidate parts of the cache on a per-class basis
+                if needed.  
+
+                there is a discussion at 
+                https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm
+                that talks about sstate and how the YP team attacked and solved the problem. 
+                This is probably a good place to get information from to broach the whole
+                sstate concept. 
+
+                YP, by default, builds from scratch.
+                This is good but it means spending a lot of time rebuilding things that don't 
+                necessarily need rebuilding.
+
+                The SSTATE_DIR variable points to the directory for the shared state cache that 
+                is used during a build.
+
+                A task's inputs have a checksum or signature associated with them.
+                If the checksum changes on an input as compared to a prior build, the task must be rerun.
+                The shared state (sstate) code keeps track of what output is generated by which tasks.
+                So if a task's inputs have not changed then the output associated with the task can 
+                be yanked from some place and re-used.  No re-build required for that particular task.
+
+                A "run" shell script is created for each task.
+                You can create a checksum for the task based on the inputs to the task.
+                When you have this checksum, the code will look at it and compare it to the previous
+                checksum to see if the task's inputs have changed.
+                If so, the task needs to be re-run.
+
+                Python tasks have python functions that access variables. 
+                Python functions will call other python functions as well.
+                The solution was to figure out the variable and function dependencies and create
+                a checksum value for the data coming into the python task.
+
+                Here is a conversation with Mark Hatle regarding bug 1500 (638 is related):
+
+(01:23:34 PM) scottrif: mark - you have a minute?
+(01:34:05 PM) Mark Hatle: sure..
+(01:34:11 PM) Mark Hatle: might be a bit slow to respond, but I'm here
+(01:34:45 PM) scottrif: Hi - I am looking at bug 1500 and trying to get a bit of better understanding.  Here is the link to the bug - http://bugzilla.pokylinux.org/show_bug.cgi?id=1500
+(01:35:25 PM) scottrif: It seems that the key for the user here is to when to "Know" when to put some comments into a function to invalidate certain areas of sstate.
+(01:35:49 PM) Mark Hatle: what the issue is, if you make changes to something that is not normally calculated in the checksums for sstate, then you can get package mismatches..
+(01:35:50 PM) scottrif: This trick of "knowing"... does it need to be explained?
+(01:36:25 PM) Mark Hatle: The developer will have to know they made such a change..  Then to deal with this, they can use a patch like what is referenced to cause specific steps to be invalidated and various steps re-run..
+(01:37:01 PM) scottrif: so my question is will the developer know when they make a change like this?
+(01:37:04 PM) Mark Hatle: In this case, we change part of the back-end packaging mechanisms.. which changed internal dependency generation.  The sstate code does not checksum the internal dependency generation, it assumes that is code that doesn't change behavior
+(01:37:24 PM) Mark Hatle: They should understand the ramifications of their changes — and thus know they need to do this.
+(01:37:46 PM) Mark Hatle: Examples of times you need to do this.  Back end packaging changes occur — i.e. you change the format of dependency generation..
+(01:38:38 PM) scottrif: do you have any other examples?
+(01:38:39 PM) Mark Hatle: when you change a recipe itself, source code.. it is -not- necessary to do this
+(01:38:49 PM) Mark Hatle: RP might be better at examples of when to do it..
+(01:39:11 PM) scottrif: right - If I change a recipe then every thing dependent further down the line gets regenerated right?
+(01:39:17 PM) Mark Hatle: This should never be necessary when a recipe changes.. it will only be necessary when some classes or back-end (packaging frameworks) change..
+(01:39:21 PM) Mark Hatle: ya
+(01:39:33 PM) Mark Hatle: Another way to think of this is implicit dependencies..
+(01:40:01 PM) Mark Hatle: I change RPM..  If you build something that produces an RPM package.. the assumption is the RPM package won't change, even if the RPM binary changes..
+(01:40:10 PM) Mark Hatle: If the format of the package changes.. you would need to do this
+(01:40:53 PM) Mark Hatle: RP can probably give you an idea of the various implicit dependencies, and which ones this type of change is needed for
+(01:41:26 PM) scottrif: okay.  I am struggling a bit with how to word it.  what I will do is write something up and send it out to you and RP for a look
+(01:41:47 PM) Mark Hatle: ya, I understand.. it's an odd set of situations that can cause this — but we definitely need to document it
+(01:42:01 PM) scottrif: I just want the information to help the user understand the conditions when they will want to invalidate parts of the sstate
+(01:42:18 PM) scottrif: I will likely use the RPM example as the case to illustrate it
+(01:42:26 PM) scottrif: as it seems pretty straight forward
+(01:42:28 PM) Mark Hatle: yup.  Key thing is it's only needed on implicit dependencies..  Normal case is back end packaging format changes..
+(01:42:31 PM) Mark Hatle: yup
+(01:42:47 PM) scottrif: ok - thanks Mark
+
+          Here is what RP wants to address 1500:
+
+If its desired to change the checksum of a given subset of tasks, maybe
+due to a change which isn't directly visible in the code itself (e.g. a
+tool changed its output) its possible to do this by changing a function
+comments since the sstate checksums include the body of functions. To
+invalidate package sstate files for example, do_package or one of the
+functions it calls can be changed, even if its just a cosmetic change to
+the commends.
+http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54
+is an example of a commit which does this.
+
+-->
 </section>
 
+
 <section id='usingpoky-install'>
     <title>Installing and Using the Result</title>