ref-manual, dev-manual: Updates to support recipe-specific sysroots

Changes covered several areas.  Version 2.3 of the YP now supports
recipe-specific sysroots and is not limited to machine-specific
as was prior releases.  Manual changes were as follows:

dev-manual: "Sharing Files Between Recipes" - Wordings were changed
to support the new functionality.

ref-manual: do_prepare_recipe_sysroot task added to the list of
tasks described in "Configuration and Compilation".

ref-manual: Extensive re-write of the "staging.bbclass" section.

ref-manual: Added a section to the build structure for
build/tmp/work/tunearch/recipename/version/.  This section breaks
down the recipe-specific subdirectories used to create (stage)
the sysroot.

ref-manual: New section (entry) for the do_prepare_recipe_sysroot
task in the task chapter.

ref-manual: Added a variable glossary description for the
SSTATE_SCAN_FILES variable.

In addition to these changes, I sprinkled in a liberal amount
of cross-referencing for the readers pleasure.

(From yocto-docs rev: 3a8ca96861f4c5d3badb91d0cdc5a3df513d4e59)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

6 files changed, 322 insertions, 29 deletions

diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml
index d2bd56dc13..e12fa32516 100644
--- a/documentation/dev-manual/dev-manual-common-tasks.xml
+++ b/documentation/dev-manual/dev-manual-common-tasks.xml
@@ -3193,10 +3193,11 @@
                 access to the library itself and its associated headers.
                 The way this access is accomplished is by populating sysroot
                 with files.
-                One sysroot exists per "machine" for which the image is
-                being built.
-                In practical terms, this means a sysroot exists for the target
-                machine, and a sysroot exists for the build host.
+                Each recipe has two sysroots in its work directory, one for
+                target files
+                (<filename>recipe-sysroot</filename>) and one for files that
+                are native to the build host
+                (<filename>recipe-sysroot-native</filename>).
                 <note>
                     You could find the term "staging" used within the Yocto
                     project regarding files populating sysroot (e.g. the
@@ -3214,12 +3215,6 @@
                 task within the
                 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink><filename>}</filename>
                 directory.
-            </para>
-
-            <para>
-                A subset of these files, as defined by the
-                the <ulink url='&YOCTO_DOCS_REF_URL;#var-SYSROOT_DIRS'><filename>SYSROOT_DIRS</filename></ulink>
-                variable, automatically populates the sysroot.
                 The reason for this limitation is that almost all files that
                 populate the sysroot are cataloged in manifests in order to
                 ensure the files can be removed later when a recipe is either
@@ -3228,6 +3223,13 @@
             </para>
 
             <para>
+                A subset of the files installed by the
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
+                task are used by the
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
+                task as defined by the the
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-SYSROOT_DIRS'><filename>SYSROOT_DIRS</filename></ulink>
+                variable to automatically populate the sysroot.
                 It is possible to modify the list of directories that populate
                 the sysroot.
                 The following example shows how you could add the
@@ -3239,13 +3241,11 @@
             </para>
 
             <para>
-                For information on variables you can use to help control how
-                files sysroot is populated, see the
-                <ulink url='&YOCTO_DOCS_REF_URL;#var-SYSROOT_DIRS'><filename>SYSROOT_DIRS</filename></ulink>,
-                <ulink url='&YOCTO_DOCS_REF_URL;#var-SYSROOT_DIRS_NATIVE'><filename>SYSROOT_DIRS_NATIVE</filename></ulink>,
-                and
-                <ulink url='&YOCTO_DOCS_REF_URL;#var-SYSROOT_DIRS_BLACKLIST'><filename>SYSROOT_DIRS_BLACKLIST</filename></ulink>
-                variables.
+                for a more complete description of the
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
+                task and its associated functions, see the
+                <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-staging'><filename>staging</filename></ulink>
+                class.
             </para>
         </section>
 
diff --git a/documentation/ref-manual/closer-look.xml b/documentation/ref-manual/closer-look.xml
index d7b47d83da..a6f1b23257 100644
--- a/documentation/ref-manual/closer-look.xml
+++ b/documentation/ref-manual/closer-look.xml
@@ -867,6 +867,19 @@
             <para>
                 This step in the build process consists of three tasks:
                 <itemizedlist>
+                    <listitem><para>
+                        <emphasis><link linkend='ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></link>:</emphasis>
+                        This task sets up the two sysroots in
+                        <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}</filename>
+                        (i.e. <filename>recipe-sysroot</filename> and
+                        <filename>recipe-sysroot-native</filename>) so that
+                        the sysroots contain the contents of the
+                        <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
+                        tasks of the recipes on which the recipe
+                        containing the tasks depends.
+                        A sysroot exists for both the target and for the native
+                        binaries, which run on the host system.
+                        </para></listitem>
                     <listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
                         This task configures the source by enabling and
                         disabling any build-time and configuration options for
diff --git a/documentation/ref-manual/ref-classes.xml b/documentation/ref-manual/ref-classes.xml
index 3d9fbf695d..8f25c521c6 100644
--- a/documentation/ref-manual/ref-classes.xml
+++ b/documentation/ref-manual/ref-classes.xml
@@ -3190,13 +3190,144 @@
     <title><filename>staging.bbclass</filename></title>
 
     <para>
-        The <filename>staging</filename> class provides the
-        <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
-        task, which stages files into the sysroot to make them available to
-        other recipes at build time.
-        The class is enabled by default because it is inherited by the
-        <link linkend='ref-classes-base'><filename>base</filename></link>
-        class.
+        The <filename>staging</filename> class installs files into individual
+        recipe work directories for sysroots.
+        The class contains the following key tasks:
+        <itemizedlist>
+            <listitem><para>
+                The
+                <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
+                task, which is responsible for handing the files that end up
+                in the recipe sysroots.
+                </para></listitem>
+            <listitem><para>
+                The
+                <link linkend='ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></link>
+                task (a "partner" task to the
+                <filename>populate_sysroot</filename> task), which installs
+                the files into the individual recipe work directories (i.e.
+                <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>).
+                </para></listitem>
+        </itemizedlist>
+    </para>
+
+    <para>
+        The code in the <filename>staging</filename> class is complex and
+        basically works in two stages:
+        <itemizedlist>
+            <listitem><para>
+                <emphasis>Stage One:</emphasis>
+                The first stage addresses recipes that have files they want
+                to share with other recipes that have dependencies on the
+                originating recipe.
+                Normally these dependencies are installed through the
+                <link linkend='ref-tasks-install'><filename>do_install</filename></link>
+                task into
+                <filename>${</filename><link linkend='var-D'><filename>D</filename></link><filename>}</filename>.
+                The <filename>do_populate_sysroot</filename> task copies
+                a subset of these files into
+                <filename>${SYSROOT_DESTDIR}</filename>.
+                This subset of files is controlled by the
+                <link linkend='var-SYSROOT_DIRS'><filename>SYSROOT_DIRS</filename></link>,
+                <link linkend='var-SYSROOT_DIRS_NATIVE'><filename>SYSROOT_DIRS_NATIVE</filename></link>,
+                and
+                <link linkend='var-SYSROOT_DIRS_BLACKLIST'><filename>SYSROOT_DIRS_BLACKLIST</filename></link>
+                variables.
+                <note>
+                    Additionally, a recipe can customize the files further by
+                declaring a processing function in the
+                    <link linkend='var-SYSROOT_PREPROCESS_FUNCS'><filename>SYSROOT_PREPROCESS_FUNCS</filename></link>
+                    variable.
+                </note>
+                </para>
+
+                <para>
+                A shared state (sstate) object is built from these files
+                and the files are placed into a subdirectory of
+                <filename>tmp/sysroot-components/</filename>.
+                The files are scanned for hardcoded paths to the original
+                installation location.
+                If the location is found in text files, the hardcoded
+                locations are replaced by tokens and a list of the files
+                needing such replacements is created.
+                These adjustments are referred to as "FIXMEs".
+                The list of files that are scanned for paths is controlled by
+                the
+                <link linkend='var-SSTATE_SCAN_FILES'><filename>SSTATE_SCAN_FILES</filename></link>
+                variable.
+                </para></listitem>
+            <listitem><para>
+                <emphasis>Stage Two:</emphasis>
+                The second stage addresses recipes that want to use something
+                from another recipe and declare a dependency on that recipe
+                through the
+                <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+                variable.
+                The recipe will have a
+                <link linkend='ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></link>
+                task and when
+                this task executes, it creates the
+                <filename>recipe-sysroot</filename> and
+                <filename>recipe-sysroot-native</filename> in the recipe
+                work directory (i.e.
+                <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
+                The OpenEmbedded build system creates hard links to copies of the
+                relevant files from <filename>sysroot-components</filename>
+                into the recipe work directory.
+                <note>
+                    If hard links are not possible, the build system uses
+                    actual copies.
+                </note>
+                The build system then addresses any "FIXMEs" to paths as
+                defined from the list created in the first stage.
+                </para>
+
+                <para>
+                Finally, any files in <filename>${bindir}</filename>
+                within the sysroot that have the prefix
+                "<filename>postinst-</filename>" are executed.
+                <note>
+                    Although these files are not recommended for general use,
+                    the files do allow some issues such as user creation
+                    and module indexes to be addressed.
+                </note>
+                </para>
+
+                <para>
+                Because recipes can have other dependencies outside of
+                <filename>DEPENDS</filename> (e.g.
+                <filename>do_unpack[depends] += "tar-native:do_populate_sysroot"</filename>),
+                the sysroot creation function
+                <filename>extend_recipe_sysroot</filename> is also added as
+                a pre-function for those tasks whose dependencies are not
+                through <filename>DEPENDS</filename> but operate similarly.
+                </para>
+
+                <para>
+                When installing dependencies into the sysroot, the code
+                traverses the dependency graph and processes dependencies
+                in exactly the same way as the dependencies would or would not
+                be when installed from sstate.
+                This processing means, for example, a native tool would have
+                its native dependencies added but a target library would not
+                have its dependencies traversed or installed.
+                The same sstate dependency code is used so that
+                builds should be identical regardless of whether sstate
+                was used or not.
+                For a closer look, see the
+                <filename>setscene_depvalid()</filename> function in the
+                <link linkend='ref-classes-sstate'><filename>sstate</filename></link>
+                class.
+                </para>
+
+                <para>
+                The build system is careful to maintain manifests of the files
+                it installs so that any given dependency can be installed as
+                needed.
+                The sstate hash of the installed item is also stored so that
+                if it changes, the build system can reinstall it.
+                </para></listitem>
+        </itemizedlist>
     </para>
 </section>
 
diff --git a/documentation/ref-manual/ref-structure.xml b/documentation/ref-manual/ref-structure.xml
index 541a47e556..b0c4bfeaa4 100644
--- a/documentation/ref-manual/ref-structure.xml
+++ b/documentation/ref-manual/ref-structure.xml
@@ -801,11 +801,24 @@
         <title><filename>build/tmp/sysroots/</filename></title>
 
         <para>
-            This directory contains shared header files and libraries as well as other shared
-            data.
-            Packages that need to share output with other packages do so within this directory.
-            The directory is subdivided by architecture so multiple builds can run within
-            the one Build Directory.
+            Previous versions of the OpenEmbedded build system used to
+            create a global shared sysroot per machine along with a native
+            sysroot.
+            Beginning with the &DISTRO; version of the Yocto Project,
+            sysroots exist in recipe-specific
+            <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
+            directories.
+            Thus, the <filename>build/tmp/sysroots/</filename> directory
+            is unused.
+            <note>
+                The <filename>build/tmp/sysroots/</filename> directory
+                can still be populated using the
+                <filename>bitbake build-sysroots</filename> command and can
+                be used for compatibility in some cases.
+                However, in general it is not recommended to populate
+                this directory.
+                Individual recipe-specific sysroots should be used.
+            </note>
         </para>
     </section>
 
@@ -894,6 +907,101 @@
         </para>
     </section>
 
+    <section id='structure-build-tmp-work-tunearch-recipename-version'>
+        <title><filename>build/tmp/work/<replaceable>tunearch</replaceable>/<replaceable>recipename</replaceable>/<replaceable>version</replaceable>/</filename></title>
+
+        <para>
+            The recipe work directory (<replaceable>recipe_work_directory</replaceable>).
+        </para>
+
+        <para>
+            As described earlier in the
+            "<link linkend='structure-build-tmp-sysroots'><filename>build/tmp/sysroots/</filename></link>"
+            section, beginning with the &DISTRO; release of the Yocto
+            Project, the OpenEmbedded build system builds each recipe in its
+            own work directory (i.e.
+            <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>).
+            The path to the work directory is constructed using the
+            architecture of the given build (e.g.
+            <link linkend='var-TUNE_PKGARCH'><filename>TUNE_PKGARCH</filename></link>,
+            <link linkend='var-MACHINE_ARCH'><filename>MACHINE_ARCH</filename></link>,
+            or "allarch"), the recipe name, and the version of the recipe (i.e.
+            <link linkend='var-PE'><filename>PE</filename></link><filename>:</filename><link linkend='var-PV'><filename>PV</filename></link><filename>-</filename><link linkend='var-PR'><filename>PR</filename></link>).
+        </para>
+
+        <para>
+            A number of key subdirectories exist within each recipe
+            work directory:
+            <itemizedlist>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/temp</filename>:
+                    Contains the log files of each task executed for this
+                    recipe, the "run" files for each executed task, which
+                    contain the code run, and a
+                    <filename>log.task_order</filename> file, which lists the
+                    order in which tasks were executed.
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/image</filename>:
+                    Contains the output of the
+                    <link linkend='ref-tasks-install'><filename>do_install</filename></link>
+                    task, which corresponds to the
+                    <filename>${</filename><link linkend='var-D'><filename>D</filename></link><filename>}</filename>
+                    variable in that task.
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/pseudo</filename>:
+                    Contains the pseudo database and log for any tasks executed
+                    under pseudo for the recipe.
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/sysroot-destdir</filename>:
+                    Contains the output of the
+                    <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
+                    task.
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/package</filename>:
+                    Contains the output of the
+                    <link linkend='ref-tasks-package'><filename>do_package</filename></link>
+                    task before the output is split into individual packages.
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/packages-split</filename>:
+                    Contains the output of the <filename>do_package</filename>
+                    task after the output has been split into individual
+                    packages.
+                    Subdirectories exist for each individual package created
+                    by the recipe.
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/recipe-sysroot</filename>:
+                    A directory populated with the target dependencies of the
+                    recipe.
+                    This directory looks like the target filesystem and
+                    contains libraries that the recipe might need to link
+                    against (e.g. the C library).
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/recipe-sysroot-native</filename>:
+                    A directory populated with the native dependencies of the
+                    recipe.
+                    This directory contains the tools the recipe needs to build
+                    (e.g. the compiler, Autoconf, libtool, and so forth).
+                   </para></listitem>
+                <listitem><para>
+                    <filename><replaceable>recipe_work_directory</replaceable>/build</filename>:
+                    This subdirectory applies only to recipes that support
+                    builds where the source is separate from the
+                    build artifacts.
+                    The OpenEmbedded build system uses this directory as a
+                    separate build directory (i.e.
+                    <filename>${</filename><link linkend='var-B'><filename>B</filename></link><filename>}</filename>).
+                   </para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
     <section id='structure-build-work-shared'>
         <title><filename>build/tmp/work-shared/</filename></title>
 
diff --git a/documentation/ref-manual/ref-tasks.xml b/documentation/ref-manual/ref-tasks.xml
index a3a41ebcc1..99e9f52b86 100644
--- a/documentation/ref-manual/ref-tasks.xml
+++ b/documentation/ref-manual/ref-tasks.xml
@@ -490,6 +490,18 @@
         </para>
     </section>
 
+    <section id='ref-tasks-prepare_recipe_sysroot'>
+        <title><filename>do_prepare_recipe_sysroot</filename></title>
+
+        <para>
+            Installs the files into the individual recipe work directories
+            (i.e. WORKDIR).
+            See the
+            "<link linkend='ref-classes-staging'><filename>staging</filename></link>"
+            class for more information.
+        </para>
+    </section>
+
     <section id='ref-tasks-rm_work'>
         <title><filename>do_rm_work</filename></title>
 
diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index 55f7a28c21..3699ae75bc 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -12776,6 +12776,35 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-SSTATE_SCAN_FILES'><glossterm>SSTATE_SCAN_FILES</glossterm>
+            <info>
+                SSTATE_SCAN_FILES[doc] = "Controls the list of files the OpenEmbedded build system scans for hardcoded installation paths."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    Controls the list of files the OpenEmbedded build system
+                    scans for hardcoded installation paths.
+                </para>
+
+                <para>
+                    During a build, the OpenEmbedded build system creates a
+                    shared state (sstate) object during the first stage of
+                    preparing the sysroots.
+                    During the build, the object is scanned for hardcoded paths
+                    for original installation locations.
+                    The list of files that are scanned for paths is controlled
+                    by the <filename>SSTATE_SCAN_FILES</filename> variable.
+                </para>
+
+                <para>
+                    For details on the process, see the
+                    <link linkend='ref-classes-staging'><filename>staging</filename></link>
+                    class.
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-STAGING_BASE_LIBDIR_NATIVE'><glossterm>STAGING_BASE_LIBDIR_NATIVE</glossterm>
             <info>
                 STAGING_BASE_LIBDIR_NATIVE[doc] = "Specifies the path to the /lib subdirectory of the sysroot directory for the build host."