From 3db33448592f737515289edef6ed25d407fd8039 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Thu, 13 Apr 2017 10:18:33 -0700 Subject: ref-manual, dev-manual: Updates to support recipe-specific sysroots Changes covered several areas. Version 2.3 of the YP now supports recipe-specific sysroots and is not limited to machine-specific as was prior releases. Manual changes were as follows: dev-manual: "Sharing Files Between Recipes" - Wordings were changed to support the new functionality. ref-manual: do_prepare_recipe_sysroot task added to the list of tasks described in "Configuration and Compilation". ref-manual: Extensive re-write of the "staging.bbclass" section. ref-manual: Added a section to the build structure for build/tmp/work/tunearch/recipename/version/. This section breaks down the recipe-specific subdirectories used to create (stage) the sysroot. ref-manual: New section (entry) for the do_prepare_recipe_sysroot task in the task chapter. ref-manual: Added a variable glossary description for the SSTATE_SCAN_FILES variable. In addition to these changes, I sprinkled in a liberal amount of cross-referencing for the readers pleasure. (From yocto-docs rev: 3a8ca96861f4c5d3badb91d0cdc5a3df513d4e59) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../dev-manual/dev-manual-common-tasks.xml | 34 ++--- documentation/ref-manual/closer-look.xml | 13 ++ documentation/ref-manual/ref-classes.xml | 145 ++++++++++++++++++++- documentation/ref-manual/ref-structure.xml | 118 ++++++++++++++++- documentation/ref-manual/ref-tasks.xml | 12 ++ documentation/ref-manual/ref-variables.xml | 29 +++++ 6 files changed, 322 insertions(+), 29 deletions(-) (limited to 'documentation') diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml index d2bd56dc13..e12fa32516 100644 --- a/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/documentation/dev-manual/dev-manual-common-tasks.xml @@ -3193,10 +3193,11 @@ access to the library itself and its associated headers. The way this access is accomplished is by populating sysroot with files. - One sysroot exists per "machine" for which the image is - being built. - In practical terms, this means a sysroot exists for the target - machine, and a sysroot exists for the build host. + Each recipe has two sysroots in its work directory, one for + target files + (recipe-sysroot) and one for files that + are native to the build host + (recipe-sysroot-native). You could find the term "staging" used within the Yocto project regarding files populating sysroot (e.g. the @@ -3214,12 +3215,6 @@ task within the ${D} directory. - - - - A subset of these files, as defined by the - the SYSROOT_DIRS - variable, automatically populates the sysroot. The reason for this limitation is that almost all files that populate the sysroot are cataloged in manifests in order to ensure the files can be removed later when a recipe is either @@ -3228,6 +3223,13 @@ + A subset of the files installed by the + do_install + task are used by the + do_populate_sysroot + task as defined by the the + SYSROOT_DIRS + variable to automatically populate the sysroot. It is possible to modify the list of directories that populate the sysroot. The following example shows how you could add the @@ -3239,13 +3241,11 @@ - For information on variables you can use to help control how - files sysroot is populated, see the - SYSROOT_DIRS, - SYSROOT_DIRS_NATIVE, - and - SYSROOT_DIRS_BLACKLIST - variables. + for a more complete description of the + do_populate_sysroot + task and its associated functions, see the + staging + class. diff --git a/documentation/ref-manual/closer-look.xml b/documentation/ref-manual/closer-look.xml index d7b47d83da..a6f1b23257 100644 --- a/documentation/ref-manual/closer-look.xml +++ b/documentation/ref-manual/closer-look.xml @@ -867,6 +867,19 @@ This step in the build process consists of three tasks: + + do_prepare_recipe_sysroot: + This task sets up the two sysroots in + ${WORKDIR} + (i.e. recipe-sysroot and + recipe-sysroot-native) so that + the sysroots contain the contents of the + do_populate_sysroot + tasks of the recipes on which the recipe + containing the tasks depends. + A sysroot exists for both the target and for the native + binaries, which run on the host system. + do_configure: This task configures the source by enabling and disabling any build-time and configuration options for diff --git a/documentation/ref-manual/ref-classes.xml b/documentation/ref-manual/ref-classes.xml index 3d9fbf695d..8f25c521c6 100644 --- a/documentation/ref-manual/ref-classes.xml +++ b/documentation/ref-manual/ref-classes.xml @@ -3190,13 +3190,144 @@ <filename>staging.bbclass</filename> - The staging class provides the - do_populate_sysroot - task, which stages files into the sysroot to make them available to - other recipes at build time. - The class is enabled by default because it is inherited by the - base - class. + The staging class installs files into individual + recipe work directories for sysroots. + The class contains the following key tasks: + + + The + do_populate_sysroot + task, which is responsible for handing the files that end up + in the recipe sysroots. + + + The + do_prepare_recipe_sysroot + task (a "partner" task to the + populate_sysroot task), which installs + the files into the individual recipe work directories (i.e. + WORKDIR). + + + + + + The code in the staging class is complex and + basically works in two stages: + + + Stage One: + The first stage addresses recipes that have files they want + to share with other recipes that have dependencies on the + originating recipe. + Normally these dependencies are installed through the + do_install + task into + ${D}. + The do_populate_sysroot task copies + a subset of these files into + ${SYSROOT_DESTDIR}. + This subset of files is controlled by the + SYSROOT_DIRS, + SYSROOT_DIRS_NATIVE, + and + SYSROOT_DIRS_BLACKLIST + variables. + + Additionally, a recipe can customize the files further by + declaring a processing function in the + SYSROOT_PREPROCESS_FUNCS + variable. + + + + + A shared state (sstate) object is built from these files + and the files are placed into a subdirectory of + tmp/sysroot-components/. + The files are scanned for hardcoded paths to the original + installation location. + If the location is found in text files, the hardcoded + locations are replaced by tokens and a list of the files + needing such replacements is created. + These adjustments are referred to as "FIXMEs". + The list of files that are scanned for paths is controlled by + the + SSTATE_SCAN_FILES + variable. + + + Stage Two: + The second stage addresses recipes that want to use something + from another recipe and declare a dependency on that recipe + through the + DEPENDS + variable. + The recipe will have a + do_prepare_recipe_sysroot + task and when + this task executes, it creates the + recipe-sysroot and + recipe-sysroot-native in the recipe + work directory (i.e. + WORKDIR. + The OpenEmbedded build system creates hard links to copies of the + relevant files from sysroot-components + into the recipe work directory. + + If hard links are not possible, the build system uses + actual copies. + + The build system then addresses any "FIXMEs" to paths as + defined from the list created in the first stage. + + + + Finally, any files in ${bindir} + within the sysroot that have the prefix + "postinst-" are executed. + + Although these files are not recommended for general use, + the files do allow some issues such as user creation + and module indexes to be addressed. + + + + + Because recipes can have other dependencies outside of + DEPENDS (e.g. + do_unpack[depends] += "tar-native:do_populate_sysroot"), + the sysroot creation function + extend_recipe_sysroot is also added as + a pre-function for those tasks whose dependencies are not + through DEPENDS but operate similarly. + + + + When installing dependencies into the sysroot, the code + traverses the dependency graph and processes dependencies + in exactly the same way as the dependencies would or would not + be when installed from sstate. + This processing means, for example, a native tool would have + its native dependencies added but a target library would not + have its dependencies traversed or installed. + The same sstate dependency code is used so that + builds should be identical regardless of whether sstate + was used or not. + For a closer look, see the + setscene_depvalid() function in the + sstate + class. + + + + The build system is careful to maintain manifests of the files + it installs so that any given dependency can be installed as + needed. + The sstate hash of the installed item is also stored so that + if it changes, the build system can reinstall it. + + diff --git a/documentation/ref-manual/ref-structure.xml b/documentation/ref-manual/ref-structure.xml index 541a47e556..b0c4bfeaa4 100644 --- a/documentation/ref-manual/ref-structure.xml +++ b/documentation/ref-manual/ref-structure.xml @@ -801,11 +801,24 @@ <filename>build/tmp/sysroots/</filename> - This directory contains shared header files and libraries as well as other shared - data. - Packages that need to share output with other packages do so within this directory. - The directory is subdivided by architecture so multiple builds can run within - the one Build Directory. + Previous versions of the OpenEmbedded build system used to + create a global shared sysroot per machine along with a native + sysroot. + Beginning with the &DISTRO; version of the Yocto Project, + sysroots exist in recipe-specific + WORKDIR + directories. + Thus, the build/tmp/sysroots/ directory + is unused. + + The build/tmp/sysroots/ directory + can still be populated using the + bitbake build-sysroots command and can + be used for compatibility in some cases. + However, in general it is not recommended to populate + this directory. + Individual recipe-specific sysroots should be used. + @@ -894,6 +907,101 @@ +
+ <filename>build/tmp/work/<replaceable>tunearch</replaceable>/<replaceable>recipename</replaceable>/<replaceable>version</replaceable>/</filename> + + + The recipe work directory (recipe_work_directory). + + + + As described earlier in the + "build/tmp/sysroots/" + section, beginning with the &DISTRO; release of the Yocto + Project, the OpenEmbedded build system builds each recipe in its + own work directory (i.e. + WORKDIR). + The path to the work directory is constructed using the + architecture of the given build (e.g. + TUNE_PKGARCH, + MACHINE_ARCH, + or "allarch"), the recipe name, and the version of the recipe (i.e. + PE:PV-PR). + + + + A number of key subdirectories exist within each recipe + work directory: + + + recipe_work_directory/temp: + Contains the log files of each task executed for this + recipe, the "run" files for each executed task, which + contain the code run, and a + log.task_order file, which lists the + order in which tasks were executed. + + + recipe_work_directory/image: + Contains the output of the + do_install + task, which corresponds to the + ${D} + variable in that task. + + + recipe_work_directory/pseudo: + Contains the pseudo database and log for any tasks executed + under pseudo for the recipe. + + + recipe_work_directory/sysroot-destdir: + Contains the output of the + do_populate_sysroot + task. + + + recipe_work_directory/package: + Contains the output of the + do_package + task before the output is split into individual packages. + + + recipe_work_directory/packages-split: + Contains the output of the do_package + task after the output has been split into individual + packages. + Subdirectories exist for each individual package created + by the recipe. + + + recipe_work_directory/recipe-sysroot: + A directory populated with the target dependencies of the + recipe. + This directory looks like the target filesystem and + contains libraries that the recipe might need to link + against (e.g. the C library). + + + recipe_work_directory/recipe-sysroot-native: + A directory populated with the native dependencies of the + recipe. + This directory contains the tools the recipe needs to build + (e.g. the compiler, Autoconf, libtool, and so forth). + + + recipe_work_directory/build: + This subdirectory applies only to recipes that support + builds where the source is separate from the + build artifacts. + The OpenEmbedded build system uses this directory as a + separate build directory (i.e. + ${B}). + + + +
+
<filename>build/tmp/work-shared/</filename> diff --git a/documentation/ref-manual/ref-tasks.xml b/documentation/ref-manual/ref-tasks.xml index a3a41ebcc1..99e9f52b86 100644 --- a/documentation/ref-manual/ref-tasks.xml +++ b/documentation/ref-manual/ref-tasks.xml @@ -490,6 +490,18 @@
+
+ <filename>do_prepare_recipe_sysroot</filename> + + + Installs the files into the individual recipe work directories + (i.e. WORKDIR). + See the + "staging" + class for more information. + +
+
<filename>do_rm_work</filename> diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index 55f7a28c21..3699ae75bc 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -12776,6 +12776,35 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3" + SSTATE_SCAN_FILES + + SSTATE_SCAN_FILES[doc] = "Controls the list of files the OpenEmbedded build system scans for hardcoded installation paths." + + + + + Controls the list of files the OpenEmbedded build system + scans for hardcoded installation paths. + + + + During a build, the OpenEmbedded build system creates a + shared state (sstate) object during the first stage of + preparing the sysroots. + During the build, the object is scanned for hardcoded paths + for original installation locations. + The list of files that are scanned for paths is controlled + by the SSTATE_SCAN_FILES variable. + + + + For details on the process, see the + staging + class. + + + + STAGING_BASE_LIBDIR_NATIVE STAGING_BASE_LIBDIR_NATIVE[doc] = "Specifies the path to the /lib subdirectory of the sysroot directory for the build host." -- cgit v1.2.3-54-g00ecf