From de6d45fefc3000ee8918d7c18448758d4216bae5 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Wed, 14 Jun 2017 09:50:55 -0700 Subject: documentation: Re-org for "closer-look" chapter Fixes [YOCTO #11630] The ref-manual needs expansion for the old "closer-look" chapter. This chapter previously held a detailed look at what happens when a user uses the YP to develop something. Now, the chapter needs to also contain YP development environment concepts (e.g. open- source philosophy, etc.), which are coming from the dev-manual. Because of this, I renamed the "closer-look.xml" chapter to be "ref-development-environment.xml". I also renamed the larger section that was formerly the entire chapter into its own section named "Development Concepts". Both these changes caused a few links to break. I fixed all the links from within the various manuals so they would find appropriate targets. I did some re-writing for introductory material to introduce the new chapter and the section on "Development Concepts". A new file ("ref-development-environment.xml") was added by basically renaming the "closer-look.xml" chapter. And, the tracking for "closer-look.xml" was deleted. (From yocto-docs rev: e37806474578b4f0ed137f64d68a39a17ab60644) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../dev-manual/dev-manual-common-tasks.xml | 4 +- documentation/ref-manual/closer-look.xml | 1627 ------------------- .../ref-manual/ref-development-environment.xml | 1635 ++++++++++++++++++++ documentation/ref-manual/ref-manual.xml | 2 +- documentation/ref-manual/technical-details.xml | 4 +- .../yocto-project-qs/yocto-project-qs.xml | 2 +- 6 files changed, 1641 insertions(+), 1633 deletions(-) delete mode 100644 documentation/ref-manual/closer-look.xml create mode 100644 documentation/ref-manual/ref-development-environment.xml diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml index 8f39d19044..0a9c1800e3 100644 --- a/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/documentation/dev-manual/dev-manual-common-tasks.xml @@ -2083,8 +2083,8 @@ - You can find more information about the build process in the - "A Closer Look at the Yocto Project Development Environment" + You can find more information about the build process in + "The Yocto Project Development Environment" chapter of the Yocto Project Reference Manual. diff --git a/documentation/ref-manual/closer-look.xml b/documentation/ref-manual/closer-look.xml deleted file mode 100644 index 5046465fba..0000000000 --- a/documentation/ref-manual/closer-look.xml +++ /dev/null @@ -1,1627 +0,0 @@ - %poky; ] > - - -A Closer Look at the Yocto Project Development Environment - - - This chapter takes a more detailed look at the Yocto Project - development environment. - The following diagram represents the development environment at a - high level. - The remainder of this chapter expands on the fundamental input, output, - process, and - Metadata) blocks - in the Yocto Project development environment. - - - - - - - - The generalized Yocto Project Development Environment consists of - several functional areas: - - User Configuration: - Metadata you can use to control the build process. - - Metadata Layers: - Various layers that provide software, machine, and - distro Metadata. - Source Files: - Upstream releases, local projects, and SCMs. - Build System: - Processes under the control of - BitBake. - This block expands on how BitBake fetches source, applies - patches, completes compilation, analyzes output for package - generation, creates and tests packages, generates images, and - generates cross-development tools. - Package Feeds: - Directories containing output packages (RPM, DEB or IPK), - which are subsequently used in the construction of an image or - SDK, produced by the build system. - These feeds can also be copied and shared using a web server or - other means to facilitate extending or updating existing - images on devices at runtime if runtime package management is - enabled. - Images: - Images produced by the development process. - - Application Development SDK: - Cross-development tools that are produced along with an image - or separately with BitBake. - - - -
- User Configuration - - - User configuration helps define the build. - Through user configuration, you can tell BitBake the - target architecture for which you are building the image, - where to store downloaded source, and other build properties. - - - - The following figure shows an expanded representation of the - "User Configuration" box of the - general Yocto Project Development Environment figure: - - - - - - - - BitBake needs some basic configuration files in order to complete - a build. - These files are *.conf files. - The minimally necessary ones reside as example files in the - Source Directory. - For simplicity, this section refers to the Source Directory as - the "Poky Directory." - - - - When you clone the poky Git repository or you - download and unpack a Yocto Project release, you can set up the - Source Directory to be named anything you want. - For this discussion, the cloned repository uses the default - name poky. - - The Poky repository is primarily an aggregation of existing - repositories. - It is not a canonical upstream source. - - - - - The meta-poky layer inside Poky contains - a conf directory that has example - configuration files. - These example files are used as a basis for creating actual - configuration files when you source the build environment - script - (i.e. - &OE_INIT_FILE; - or - oe-init-build-env-memres). - - - - Sourcing the build environment script creates a - Build Directory - if one does not already exist. - BitBake uses the Build Directory for all its work during builds. - The Build Directory has a conf directory that - contains default versions of your local.conf - and bblayers.conf configuration files. - These default configuration files are created only if versions - do not already exist in the Build Directory at the time you - source the build environment setup script. - - - - Because the Poky repository is fundamentally an aggregation of - existing repositories, some users might be familiar with running - the &OE_INIT_FILE; or - oe-init-build-env-memres script in the context - of separate OpenEmbedded-Core and BitBake repositories rather than a - single Poky repository. - This discussion assumes the script is executed from within a cloned - or unpacked version of Poky. - - - - Depending on where the script is sourced, different sub-scripts - are called to set up the Build Directory (Yocto or OpenEmbedded). - Specifically, the script - scripts/oe-setup-builddir inside the - poky directory sets up the Build Directory and seeds the directory - (if necessary) with configuration files appropriate for the - Yocto Project development environment. - - The scripts/oe-setup-builddir script - uses the $TEMPLATECONF variable to - determine which sample configuration files to locate. - - - - - The local.conf file provides many - basic variables that define a build environment. - Here is a list of a few. - To see the default configurations in a local.conf - file created by the build environment script, see the - local.conf.sample in the - meta-poky layer: - - Parallelism Options: - Controlled by the - BB_NUMBER_THREADS, - PARALLEL_MAKE, - and - BB_NUMBER_PARSE_THREADS - variables. - Target Machine Selection: - Controlled by the - MACHINE - variable. - Download Directory: - Controlled by the - DL_DIR - variable. - Shared State Directory: - Controlled by the - SSTATE_DIR - variable. - Build Output: - Controlled by the - TMPDIR - variable. - - - Configurations set in the conf/local.conf - file can also be set in the - conf/site.conf and - conf/auto.conf configuration files. - - - - - The bblayers.conf file tells BitBake what - layers you want considered during the build. - By default, the layers listed in this file include layers - minimally needed by the build system. - However, you must manually add any custom layers you have created. - You can find more information on working with the - bblayers.conf file in the - "Enabling Your Layer" - section in the Yocto Project Development Manual. - - - - The files site.conf and - auto.conf are not created by the environment - initialization script. - If you want the site.conf file, you need to - create that yourself. - The auto.conf file is typically created by - an autobuilder: - - site.conf: - You can use the conf/site.conf - configuration file to configure multiple build directories. - For example, suppose you had several build environments and - they shared some common features. - You can set these default build properties here. - A good example is perhaps the packaging format to use - through the - PACKAGE_CLASSES - variable. - One useful scenario for using the - conf/site.conf file is to extend your - BBPATH - variable to include the path to a - conf/site.conf. - Then, when BitBake looks for Metadata using - BBPATH, it finds the - conf/site.conf file and applies your - common configurations found in the file. - To override configurations in a particular build directory, - alter the similar configurations within that build - directory's conf/local.conf file. - - auto.conf: - The file is usually created and written to by - an autobuilder. - The settings put into the file are typically the same as - you would find in the conf/local.conf - or the conf/site.conf files. - - - - - - You can edit all configuration files to further define - any particular build environment. - This process is represented by the "User Configuration Edits" - box in the figure. - - - - When you launch your build with the - bitbake target - command, BitBake sorts out the configurations to ultimately - define your build environment. - It is important to understand that the OpenEmbedded build system - reads the configuration files in a specific order: - site.conf, auto.conf, - and local.conf. - And, the build system applies the normal assignment statement - rules. - Because the files are parsed in a specific order, variable - assignments for the same variable could be affected. - For example, if the auto.conf file and - the local.conf set - variable1 to different values, because - the build system parses local.conf after - auto.conf, - variable1 is assigned the value from - the local.conf file. - -
- -
- Metadata, Machine Configuration, and Policy Configuration - - - The previous section described the user configurations that - define BitBake's global behavior. - This section takes a closer look at the layers the build system - uses to further control the build. - These layers provide Metadata for the software, machine, and - policy. - - - - In general, three types of layer input exist: - - Policy Configuration: - Distribution Layers provide top-level or general - policies for the image or SDK being built. - For example, this layer would dictate whether BitBake - produces RPM or IPK packages. - Machine Configuration: - Board Support Package (BSP) layers provide machine - configurations. - This type of information is specific to a particular - target architecture. - Metadata: - Software layers contain user-supplied recipe files, - patches, and append files. - - - - - - The following figure shows an expanded representation of the - Metadata, Machine Configuration, and Policy Configuration input - (layers) boxes of the - general Yocto Project Development Environment figure: - - - - - - - - In general, all layers have a similar structure. - They all contain a licensing file - (e.g. COPYING) if the layer is to be - distributed, a README file as good practice - and especially if the layer is to be distributed, a - configuration directory, and recipe directories. - - - - The Yocto Project has many layers that can be used. - You can see a web-interface listing of them on the - Source Repositories - page. - The layers are shown at the bottom categorized under - "Yocto Metadata Layers." - These layers are fundamentally a subset of the - OpenEmbedded Metadata Index, - which lists all layers provided by the OpenEmbedded community. - - Layers exist in the Yocto Project Source Repositories that - cannot be found in the OpenEmbedded Metadata Index. - These layers are either deprecated or experimental in nature. - - - - - BitBake uses the conf/bblayers.conf file, - which is part of the user configuration, to find what layers it - should be using as part of the build. - - - - For more information on layers, see the - "Understanding and Creating Layers" - section in the Yocto Project Development Manual. - - -
- Distro Layer - - - The distribution layer provides policy configurations for your - distribution. - Best practices dictate that you isolate these types of - configurations into their own layer. - Settings you provide in - conf/distro/distro.conf override - similar - settings that BitBake finds in your - conf/local.conf file in the Build - Directory. - - - - The following list provides some explanation and references - for what you typically find in the distribution layer: - - classes: - Class files (.bbclass) hold - common functionality that can be shared among - recipes in the distribution. - When your recipes inherit a class, they take on the - settings and functions for that class. - You can read more about class files in the - "Classes" section. - - conf: - This area holds configuration files for the - layer (conf/layer.conf), - the distribution - (conf/distro/distro.conf), - and any distribution-wide include files. - - recipes-*: - Recipes and append files that affect common - functionality across the distribution. - This area could include recipes and append files - to add distribution-specific configuration, - initialization scripts, custom image recipes, - and so forth. - - -
- -
- BSP Layer - - - The BSP Layer provides machine configurations. - Everything in this layer is specific to the machine for which - you are building the image or the SDK. - A common structure or form is defined for BSP layers. - You can learn more about this structure in the - Yocto Project Board Support Package (BSP) Developer's Guide. - - In order for a BSP layer to be considered compliant with the - Yocto Project, it must meet some structural requirements. - - - - - The BSP Layer's configuration directory contains - configuration files for the machine - (conf/machine/machine.conf) and, - of course, the layer (conf/layer.conf). - - - - The remainder of the layer is dedicated to specific recipes - by function: recipes-bsp, - recipes-core, - recipes-graphics, and - recipes-kernel. - Metadata can exist for multiple formfactors, graphics - support systems, and so forth. - - While the figure shows several recipes-* - directories, not all these directories appear in all - BSP layers. - - -
- -
- Software Layer - - - The software layer provides the Metadata for additional - software packages used during the build. - This layer does not include Metadata that is specific to the - distribution or the machine, which are found in their - respective layers. - - - - This layer contains any new recipes that your project needs - in the form of recipe files. - -
-
- -
- Sources - - - In order for the OpenEmbedded build system to create an image or - any target, it must be able to access source files. - The - general Yocto Project Development Environment figure - represents source files using the "Upstream Project Releases", - "Local Projects", and "SCMs (optional)" boxes. - The figure represents mirrors, which also play a role in locating - source files, with the "Source Mirror(s)" box. - - - - The method by which source files are ultimately organized is - a function of the project. - For example, for released software, projects tend to use tarballs - or other archived files that can capture the state of a release - guaranteeing that it is statically represented. - On the other hand, for a project that is more dynamic or - experimental in nature, a project might keep source files in a - repository controlled by a Source Control Manager (SCM) such as - Git. - Pulling source from a repository allows you to control - the point in the repository (the revision) from which you want to - build software. - Finally, a combination of the two might exist, which would give the - consumer a choice when deciding where to get source files. - - - - BitBake uses the - SRC_URI - variable to point to source files regardless of their location. - Each recipe must have a SRC_URI variable - that points to the source. - - - - Another area that plays a significant role in where source files - come from is pointed to by the - DL_DIR - variable. - This area is a cache that can hold previously downloaded source. - You can also instruct the OpenEmbedded build system to create - tarballs from Git repositories, which is not the default behavior, - and store them in the DL_DIR by using the - BB_GENERATE_MIRROR_TARBALLS - variable. - - - - Judicious use of a DL_DIR directory can - save the build system a trip across the Internet when looking - for files. - A good method for using a download directory is to have - DL_DIR point to an area outside of your - Build Directory. - Doing so allows you to safely delete the Build Directory - if needed without fear of removing any downloaded source file. - - - - The remainder of this section provides a deeper look into the - source files and the mirrors. - Here is a more detailed look at the source file area of the - base figure: - - - -
- Upstream Project Releases - - - Upstream project releases exist anywhere in the form of an - archived file (e.g. tarball or zip file). - These files correspond to individual recipes. - For example, the figure uses specific releases each for - BusyBox, Qt, and Dbus. - An archive file can be for any released product that can be - built using a recipe. - -
- -
- Local Projects - - - Local projects are custom bits of software the user provides. - These bits reside somewhere local to a project - perhaps - a directory into which the user checks in items (e.g. - a local directory containing a development source tree - used by the group). - - - - The canonical method through which to include a local project - is to use the - externalsrc - class to include that local project. - You use either the local.conf or a - recipe's append file to override or set the - recipe to point to the local directory on your disk to pull - in the whole source tree. - - - - For information on how to use the - externalsrc class, see the - "externalsrc.bbclass" - section. - -
- -
- Source Control Managers (Optional) - - - Another place the build system can get source files from is - through an SCM such as Git or Subversion. - In this case, a repository is cloned or checked out. - The - do_fetch - task inside BitBake uses - the SRC_URI - variable and the argument's prefix to determine the correct - fetcher module. - - - - For information on how to have the OpenEmbedded build system - generate tarballs for Git repositories and place them in the - DL_DIR - directory, see the - BB_GENERATE_MIRROR_TARBALLS - variable. - - - - When fetching a repository, BitBake uses the - SRCREV - variable to determine the specific revision from which to - build. - -
- -
- Source Mirror(s) - - - Two kinds of mirrors exist: pre-mirrors and regular mirrors. - The PREMIRRORS - and - MIRRORS - variables point to these, respectively. - BitBake checks pre-mirrors before looking upstream for any - source files. - Pre-mirrors are appropriate when you have a shared directory - that is not a directory defined by the - DL_DIR - variable. - A Pre-mirror typically points to a shared directory that is - local to your organization. - - - - Regular mirrors can be any site across the Internet that is - used as an alternative location for source code should the - primary site not be functioning for some reason or another. - -
-
- -
- Package Feeds - - - When the OpenEmbedded build system generates an image or an SDK, - it gets the packages from a package feed area located in the - Build Directory. - The - general Yocto Project Development Environment figure - shows this package feeds area in the upper-right corner. - - - - This section looks a little closer into the package feeds area used - by the build system. - Here is a more detailed look at the area: - - - - - Package feeds are an intermediary step in the build process. - The OpenEmbedded build system provides classes to generate - different package types, and you specify which classes to enable - through the - PACKAGE_CLASSES - variable. - Before placing the packages into package feeds, - the build process validates them with generated output quality - assurance checks through the - insane - class. - - - - The package feed area resides in the Build Directory. - The directory the build system uses to temporarily store packages - is determined by a combination of variables and the particular - package manager in use. - See the "Package Feeds" box in the illustration and note the - information to the right of that area. - In particular, the following defines where package files are - kept: - - DEPLOY_DIR: - Defined as tmp/deploy in the Build - Directory. - - DEPLOY_DIR_*: - Depending on the package manager used, the package type - sub-folder. - Given RPM, IPK, or DEB packaging and tarball creation, the - DEPLOY_DIR_RPM, - DEPLOY_DIR_IPK, - DEPLOY_DIR_DEB, - or - DEPLOY_DIR_TAR, - variables are used, respectively. - - PACKAGE_ARCH: - Defines architecture-specific sub-folders. - For example, packages could exist for the i586 or qemux86 - architectures. - - - - - - BitBake uses the do_package_write_* tasks to - generate packages and place them into the package holding area (e.g. - do_package_write_ipk for IPK packages). - See the - "do_package_write_deb", - "do_package_write_ipk", - "do_package_write_rpm", - and - "do_package_write_tar" - sections for additional information. - As an example, consider a scenario where an IPK packaging manager - is being used and package architecture support for both i586 - and qemux86 exist. - Packages for the i586 architecture are placed in - build/tmp/deploy/ipk/i586, while packages for - the qemux86 architecture are placed in - build/tmp/deploy/ipk/qemux86. - -
- -
- BitBake - - - The OpenEmbedded build system uses - BitBake - to produce images. - You can see from the - general Yocto Project Development Environment figure, - the BitBake area consists of several functional areas. - This section takes a closer look at each of those areas. - - - - Separate documentation exists for the BitBake tool. - See the - BitBake User Manual - for reference material on BitBake. - - -
- Source Fetching - - - The first stages of building a recipe are to fetch and unpack - the source code: - - - - - The - do_fetch - and - do_unpack - tasks fetch the source files and unpack them into the work - directory. - - For every local file (e.g. file://) - that is part of a recipe's - SRC_URI - statement, the OpenEmbedded build system takes a checksum - of the file for the recipe and inserts the checksum into - the signature for the do_fetch. - If any local file has been modified, the - do_fetch task and all tasks that - depend on it are re-executed. - - By default, everything is accomplished in the - Build Directory, - which has a defined structure. - For additional general information on the Build Directory, - see the - "build/" - section. - - - - Unpacked source files are pointed to by the - S variable. - Each recipe has an area in the Build Directory where the - unpacked source code resides. - The name of that directory for any given recipe is defined from - several different variables. - You can see the variables that define these directories - by looking at the figure: - - TMPDIR - - The base directory where the OpenEmbedded build system - performs all its work during the build. - - PACKAGE_ARCH - - The architecture of the built package or packages. - - TARGET_OS - - The operating system of the target device. - - PN - - The name of the built package. - - PV - - The version of the recipe used to build the package. - - PR - - The revision of the recipe used to build the package. - - WORKDIR - - The location within TMPDIR where - a specific package is built. - - S - - Contains the unpacked source files for a given recipe. - - - -
- -
- Patching - - - Once source code is fetched and unpacked, BitBake locates - patch files and applies them to the source files: - - - - - The - do_patch - task processes recipes by - using the - SRC_URI - variable to locate applicable patch files, which by default - are *.patch or - *.diff files, or any file if - "apply=yes" is specified for the file in - SRC_URI. - - - - BitBake finds and applies multiple patches for a single recipe - in the order in which it finds the patches. - Patches are applied to the recipe's source files located in the - S directory. - - - - For more information on how the source directories are - created, see the - "Source Fetching" - section. - -
- -
- Configuration and Compilation - - - After source code is patched, BitBake executes tasks that - configure and compile the source code: - - - - - This step in the build process consists of three tasks: - - - do_prepare_recipe_sysroot: - This task sets up the two sysroots in - ${WORKDIR} - (i.e. recipe-sysroot and - recipe-sysroot-native) so that - the sysroots contain the contents of the - do_populate_sysroot - tasks of the recipes on which the recipe - containing the tasks depends. - A sysroot exists for both the target and for the native - binaries, which run on the host system. - - do_configure: - This task configures the source by enabling and - disabling any build-time and configuration options for - the software being built. - Configurations can come from the recipe itself as well - as from an inherited class. - Additionally, the software itself might configure itself - depending on the target for which it is being built. - - - The configurations handled by the - do_configure - task are specific - to source code configuration for the source code - being built by the recipe. - - If you are using the - autotools - class, - you can add additional configuration options by using - the EXTRA_OECONF - or - PACKAGECONFIG_CONFARGS - variables. - For information on how this variable works within - that class, see the - meta/classes/autotools.bbclass file. - - do_compile: - Once a configuration task has been satisfied, BitBake - compiles the source using the - do_compile - task. - Compilation occurs in the directory pointed to by the - B - variable. - Realize that the B directory is, by - default, the same as the - S - directory. - do_install: - Once compilation is done, BitBake executes the - do_install - task. - This task copies files from the B - directory and places them in a holding area pointed to - by the - D - variable. - - -
- -
- Package Splitting - - - After source code is configured and compiled, the - OpenEmbedded build system analyzes - the results and splits the output into packages: - - - - - The - do_package - and - do_packagedata - tasks combine to analyze - the files found in the - D directory - and split them into subsets based on available packages and - files. - The analyzing process involves the following as well as other - items: splitting out debugging symbols, - looking at shared library dependencies between packages, - and looking at package relationships. - The do_packagedata task creates package - metadata based on the analysis such that the - OpenEmbedded build system can generate the final packages. - Working, staged, and intermediate results of the analysis - and package splitting process use these areas: - - PKGD - - The destination directory for packages before they are - split. - - PKGDATA_DIR - - A shared, global-state directory that holds data - generated during the packaging process. - - PKGDESTWORK - - A temporary work area used by the - do_package task. - - PKGDEST - - The parent directory for packages after they have - been split. - - - The FILES - variable defines the files that go into each package in - PACKAGES. - If you want details on how this is accomplished, you can - look at the - package - class. - - - - Depending on the type of packages being created (RPM, DEB, or - IPK), the do_package_write_* task - creates the actual packages and places them in the - Package Feed area, which is - ${TMPDIR}/deploy. - You can see the - "Package Feeds" - section for more detail on that part of the build process. - - Support for creating feeds directly from the - deploy/* directories does not exist. - Creating such feeds usually requires some kind of feed - maintenance mechanism that would upload the new packages - into an official package feed (e.g. the - Ångström distribution). - This functionality is highly distribution-specific - and thus is not provided out of the box. - - -
- -
- Image Generation - - - Once packages are split and stored in the Package Feeds area, - the OpenEmbedded build system uses BitBake to generate the - root filesystem image: - - - - - The image generation process consists of several stages and - depends on several tasks and variables. - The - do_rootfs - task creates the root filesystem (file and directory structure) - for an image. - This task uses several key variables to help create the list - of packages to actually install: - - IMAGE_INSTALL: - Lists out the base set of packages to install from - the Package Feeds area. - PACKAGE_EXCLUDE: - Specifies packages that should not be installed. - - IMAGE_FEATURES: - Specifies features to include in the image. - Most of these features map to additional packages for - installation. - PACKAGE_CLASSES: - Specifies the package backend to use and consequently - helps determine where to locate packages within the - Package Feeds area. - IMAGE_LINGUAS: - Determines the language(s) for which additional - language support packages are installed. - - PACKAGE_INSTALL: - The final list of packages passed to the package manager - for installation into the image. - - - - - - With - IMAGE_ROOTFS - pointing to the location of the filesystem under construction and - the PACKAGE_INSTALL variable providing the - final list of packages to install, the root file system is - created. - - - - Package installation is under control of the package manager - (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of whether or - not package management is enabled for the target. - At the end of the process, if package management is not - enabled for the target, the package manager's data files - are deleted from the root filesystem. - As part of the final stage of package installation, postinstall - scripts that are part of the packages are run. - Any scripts that fail to run - on the build host are run on the target when the target system - is first booted. - If you are using a - read-only root filesystem, - all the post installation scripts must succeed during the - package installation phase since the root filesystem is - read-only. - - - - The final stages of the do_rootfs task - handle post processing. - Post processing includes creation of a manifest file and - optimizations. - - - - The manifest file (.manifest) resides - in the same directory as the root filesystem image. - This file lists out, line-by-line, the installed packages. - The manifest file is useful for the - testimage - class, for example, to determine whether or not to run - specific tests. - See the - IMAGE_MANIFEST - variable for additional information. - - - - Optimizing processes run across the image include - mklibs, prelink, - and any other post-processing commands as defined by the - ROOTFS_POSTPROCESS_COMMAND - variable. - The mklibs process optimizes the size - of the libraries, while the - prelink process optimizes the dynamic - linking of shared libraries to reduce start up time of - executables. - - - - After the root filesystem is built, processing begins on - the image through the do_image task. - The build system runs any pre-processing commands as defined - by the - IMAGE_PREPROCESS_COMMAND - variable. - This variable specifies a list of functions to call before - the OpenEmbedded build system creates the final image output - files. - - - - The do_image task dynamically creates - other do_image_* tasks as needed, which - include compressing the root filesystem image to reduce the - overall size of the image. - The process turns everything into an image file or a set of - image files. - The formats used for the root filesystem depend on the - IMAGE_FSTYPES - variable. - - - - The final task involved in image creation is the - do_image_complete task. - This task completes the image by applying any image - post processing as defined through the - IMAGE_POSTPROCESS_COMMAND - variable. - The variable specifies a list of functions to call once the - OpenEmbedded build system has created the final image output - files. - - - - The entire image generation process is run under Pseudo. - Running under Pseudo ensures that the files in the root - filesystem have correct ownership. - -
- -
- SDK Generation - - - The OpenEmbedded build system uses BitBake to generate the - Software Development Kit (SDK) installer script for both the - standard and extensible SDKs: - - - - - For more information on the cross-development toolchain - generation, see the - "Cross-Development Toolchain Generation" - section. - For information on advantages gained when building a - cross-development toolchain using the - do_populate_sdk - task, see the - "Building an SDK Installer" - section in the Yocto Project Software Development Kit (SDK) - Developer's Guide. - - - - Like image generation, the SDK script process consists of - several stages and depends on many variables. - The do_populate_sdk and - do_populate_sdk_ext tasks use these - key variables to help create the list of packages to actually - install. - For information on the variables listed in the figure, see the - "Application Development SDK" - section. - - - - The do_populate_sdk task helps create - the standard SDK and handles two parts: a target part and a - host part. - The target part is the part built for the target hardware and - includes libraries and headers. - The host part is the part of the SDK that runs on the - SDKMACHINE. - - - - The do_populate_sdk_ext task helps create - the extensible SDK and handles host and target parts - differently than its counter part does for the standard SDK. - For the extensible SDK, the task encapsulates the build system, - which includes everything needed (host and target) for the SDK. - - - - Regardless of the type of SDK being constructed, the - tasks perform some cleanup after which a cross-development - environment setup script and any needed configuration files - are created. - The final output is the Cross-development - toolchain installation script (.sh file), - which includes the environment setup script. - -
- -
- Stamp Files and the Rerunning of Tasks - - - For each task that completes successfully, BitBake writes a - stamp file into the - STAMPS_DIR - directory. - The beginning of the stamp file's filename is determined by the - STAMP - variable, and the end of the name consists of the task's name - and current - input checksum. - - This naming scheme assumes that - BB_SIGNATURE_HANDLER - is "OEBasicHash", which is almost always the case in - current OpenEmbedded. - - To determine if a task needs to be rerun, BitBake checks if a - stamp file with a matching input checksum exists for the task. - If such a stamp file exists, the task's output is assumed to - exist and still be valid. - If the file does not exist, the task is rerun. - - The stamp mechanism is more general than the shared - state (sstate) cache mechanism described in the - "Setscene Tasks and Shared State" - section. - BitBake avoids rerunning any task that has a valid - stamp file, not just tasks that can be accelerated through - the sstate cache. - However, you should realize that stamp files only - serve as a marker that some work has been done and that - these files do not record task output. - The actual task output would usually be somewhere in - TMPDIR - (e.g. in some recipe's - WORKDIR.) - What the sstate cache mechanism adds is a way to cache task - output that can then be shared between build machines. - - - Since STAMPS_DIR is usually a subdirectory - of TMPDIR, removing - TMPDIR will also remove - STAMPS_DIR, which means tasks will - properly be rerun to repopulate TMPDIR. - - - - If you want some task to always be considered "out of date", - you can mark it with the - nostamp - varflag. - If some other task depends on such a task, then that task will - also always be considered out of date, which might not be what - you want. - - - - For details on how to view information about a task's - signature, see the - "Viewing Task Variable Dependencies" - section. - -
- -
- Setscene Tasks and Shared State - - - The description of tasks so far assumes that BitBake needs to - build everything and there are no prebuilt objects available. - BitBake does support skipping tasks if prebuilt objects are - available. - These objects are usually made available in the form of a - shared state (sstate) cache. - - For information on variables affecting sstate, see the - SSTATE_DIR - and - SSTATE_MIRRORS - variables. - - - - - The idea of a setscene task (i.e - do_taskname_setscene) - is a version of the task where - instead of building something, BitBake can skip to the end - result and simply place a set of files into specific locations - as needed. - In some cases, it makes sense to have a setscene task variant - (e.g. generating package files in the - do_package_write_* task). - In other cases, it does not make sense, (e.g. a - do_patch - task or - do_unpack - task) since the work involved would be equal to or greater than - the underlying task. - - - - In the OpenEmbedded build system, the common tasks that have - setscene variants are do_package, - do_package_write_*, - do_deploy, - do_packagedata, - and - do_populate_sysroot. - Notice that these are most of the tasks whose output is an - end result. - - - - The OpenEmbedded build system has knowledge of the relationship - between these tasks and other tasks that precede them. - For example, if BitBake runs - do_populate_sysroot_setscene for - something, there is little point in running any of the - do_fetch, do_unpack, - do_patch, - do_configure, - do_compile, and - do_install tasks. - However, if do_package needs to be run, - BitBake would need to run those other tasks. - - - - It becomes more complicated if everything can come from an - sstate cache because some objects are simply not required at - all. - For example, you do not need a compiler or native tools, such - as quilt, if there is nothing to compile or patch. - If the do_package_write_* packages are - available from sstate, BitBake does not need the - do_package task data. - - - - To handle all these complexities, BitBake runs in two phases. - The first is the "setscene" stage. - During this stage, BitBake first checks the sstate cache for - any targets it is planning to build. - BitBake does a fast check to see if the object exists rather - than a complete download. - If nothing exists, the second phase, which is the setscene - stage, completes and the main build proceeds. - - - - If objects are found in the sstate cache, the OpenEmbedded - build system works backwards from the end targets specified - by the user. - For example, if an image is being built, the OpenEmbedded build - system first looks for the packages needed for that image and - the tools needed to construct an image. - If those are available, the compiler is not needed. - Thus, the compiler is not even downloaded. - If something was found to be unavailable, or the download or - setscene task fails, the OpenEmbedded build system then tries - to install dependencies, such as the compiler, from the cache. - - - - The availability of objects in the sstate cache is handled by - the function specified by the - BB_HASHCHECK_FUNCTION - variable and returns a list of the objects that are available. - The function specified by the - BB_SETSCENE_DEPVALID - variable is the function that determines whether a given - dependency needs to be followed, and whether for any given - relationship the function needs to be passed. - The function returns a True or False value. - -
-
- -
- Images - - - The images produced by the OpenEmbedded build system - are compressed forms of the - root filesystem that are ready to boot on a target device. - You can see from the - general Yocto Project Development Environment figure - that BitBake output, in part, consists of images. - This section is going to look more closely at this output: - - - - - For a list of example images that the Yocto Project provides, - see the - "Images" chapter. - - - - Images are written out to the - Build Directory - inside the tmp/deploy/images/machine/ - folder as shown in the figure. - This folder contains any files expected to be loaded on the - target device. - The - DEPLOY_DIR - variable points to the deploy directory, - while the - DEPLOY_DIR_IMAGE - variable points to the appropriate directory containing images for - the current configuration. - - kernel-image: - A kernel binary file. - The KERNEL_IMAGETYPE - variable setting determines the naming scheme for the - kernel image file. - Depending on that variable, the file could begin with - a variety of naming strings. - The deploy/images/machine - directory can contain multiple image files for the - machine. - root-filesystem-image: - Root filesystems for the target device (e.g. - *.ext3 or *.bz2 - files). - The IMAGE_FSTYPES - variable setting determines the root filesystem image - type. - The deploy/images/machine - directory can contain multiple root filesystems for the - machine. - kernel-modules: - Tarballs that contain all the modules built for the kernel. - Kernel module tarballs exist for legacy purposes and - can be suppressed by setting the - MODULE_TARBALL_DEPLOY - variable to "0". - The deploy/images/machine - directory can contain multiple kernel module tarballs - for the machine. - bootloaders: - Bootloaders supporting the image, if applicable to the - target machine. - The deploy/images/machine - directory can contain multiple bootloaders for the - machine. - symlinks: - The deploy/images/machine - folder contains - a symbolic link that points to the most recently built file - for each machine. - These links might be useful for external scripts that - need to obtain the latest version of each file. - - - -
- -
- Application Development SDK - - - In the - general Yocto Project Development Environment figure, - the output labeled "Application Development SDK" represents an - SDK. - The SDK generation process differs depending on whether you build - a standard SDK - (e.g. bitbake -c populate_sdk imagename) - or an extensible SDK - (e.g. bitbake -c populate_sdk_ext imagename). - This section is going to take a closer look at this output: - - - - - The specific form of this output is a self-extracting - SDK installer (*.sh) that, when run, - installs the SDK, which consists of a cross-development - toolchain, a set of libraries and headers, and an SDK - environment setup script. - Running this installer essentially sets up your - cross-development environment. - You can think of the cross-toolchain as the "host" - part because it runs on the SDK machine. - You can think of the libraries and headers as the "target" - part because they are built for the target hardware. - The environment setup script is added so that you can initialize - the environment before using the tools. - - - - - The Yocto Project supports several methods by which you can - set up this cross-development environment. - These methods include downloading pre-built SDK installers - or building and installing your own SDK installer. - - - - For background information on cross-development toolchains - in the Yocto Project development environment, see the - "Cross-Development Toolchain Generation" - section. - For information on setting up a cross-development - environment, see the - Yocto Project Software Development Kit (SDK) Developer's Guide. - - - - - Once built, the SDK installers are written out to the - deploy/sdk folder inside the - Build Directory - as shown in the figure at the beginning of this section. - Depending on the type of SDK, several variables exist that help - configure these files. - The following list shows the variables associated with a standard - SDK: - - DEPLOY_DIR: - Points to the deploy - directory. - SDKMACHINE: - Specifies the architecture of the machine - on which the cross-development tools are run to - create packages for the target hardware. - - SDKIMAGE_FEATURES: - Lists the features to include in the "target" part - of the SDK. - - TOOLCHAIN_HOST_TASK: - Lists packages that make up the host - part of the SDK (i.e. the part that runs on - the SDKMACHINE). - When you use - bitbake -c populate_sdk imagename - to create the SDK, a set of default packages - apply. - This variable allows you to add more packages. - - TOOLCHAIN_TARGET_TASK: - Lists packages that make up the target part - of the SDK (i.e. the part built for the - target hardware). - - SDKPATH: - Defines the default SDK installation path offered by the - installation script. - - - This next list, shows the variables associated with an extensible - SDK: - - DEPLOY_DIR: - Points to the deploy directory. - - SDK_EXT_TYPE: - Controls whether or not shared state artifacts are copied - into the extensible SDK. - By default, all required shared state artifacts are copied - into the SDK. - - SDK_INCLUDE_PKGDATA: - Specifies whether or not packagedata will be included in - the extensible SDK for all recipes in the "world" target. - - SDK_INCLUDE_TOOLCHAIN: - Specifies whether or not the toolchain will be included - when building the extensible SDK. - - SDK_LOCAL_CONF_WHITELIST: - A list of variables allowed through from the build system - configuration into the extensible SDK configuration. - - SDK_LOCAL_CONF_BLACKLIST: - A list of variables not allowed through from the build - system configuration into the extensible SDK configuration. - - SDK_INHERIT_BLACKLIST: - A list of classes to remove from the - INHERIT - value globally within the extensible SDK configuration. - - - -
- - - diff --git a/documentation/ref-manual/ref-development-environment.xml b/documentation/ref-manual/ref-development-environment.xml new file mode 100644 index 0000000000..a30cefc391 --- /dev/null +++ b/documentation/ref-manual/ref-development-environment.xml @@ -0,0 +1,1635 @@ + %poky; ] > + + +The Yocto Project Development Environment + + + This chapter takes a look at the Yocto Project development + environment and also provides a detailed look at what goes on during + development in that environment. + + +
+ Development Concepts + + + This section takes a more detailed look inside the development + process. + The following diagram represents development at a high level. + The remainder of this chapter expands on the fundamental input, output, + process, and + Metadata) blocks + that make up development in the Yocto Project environment. + + + + + + + + In general, development consists of several functional areas: + + User Configuration: + Metadata you can use to control the build process. + + Metadata Layers: + Various layers that provide software, machine, and + distro Metadata. + Source Files: + Upstream releases, local projects, and SCMs. + Build System: + Processes under the control of + BitBake. + This block expands on how BitBake fetches source, applies + patches, completes compilation, analyzes output for package + generation, creates and tests packages, generates images, and + generates cross-development tools. + Package Feeds: + Directories containing output packages (RPM, DEB or IPK), + which are subsequently used in the construction of an image or + SDK, produced by the build system. + These feeds can also be copied and shared using a web server or + other means to facilitate extending or updating existing + images on devices at runtime if runtime package management is + enabled. + Images: + Images produced by the development process. + + Application Development SDK: + Cross-development tools that are produced along with an image + or separately with BitBake. + + + +
+ User Configuration + + + User configuration helps define the build. + Through user configuration, you can tell BitBake the + target architecture for which you are building the image, + where to store downloaded source, and other build properties. + + + + The following figure shows an expanded representation of the + "User Configuration" box of the + general Yocto Project Development Environment figure: + + + + + + + + BitBake needs some basic configuration files in order to complete + a build. + These files are *.conf files. + The minimally necessary ones reside as example files in the + Source Directory. + For simplicity, this section refers to the Source Directory as + the "Poky Directory." + + + + When you clone the poky Git repository or you + download and unpack a Yocto Project release, you can set up the + Source Directory to be named anything you want. + For this discussion, the cloned repository uses the default + name poky. + + The Poky repository is primarily an aggregation of existing + repositories. + It is not a canonical upstream source. + + + + + The meta-poky layer inside Poky contains + a conf directory that has example + configuration files. + These example files are used as a basis for creating actual + configuration files when you source the build environment + script + (i.e. + &OE_INIT_FILE; + or + oe-init-build-env-memres). + + + + Sourcing the build environment script creates a + Build Directory + if one does not already exist. + BitBake uses the Build Directory for all its work during builds. + The Build Directory has a conf directory that + contains default versions of your local.conf + and bblayers.conf configuration files. + These default configuration files are created only if versions + do not already exist in the Build Directory at the time you + source the build environment setup script. + + + + Because the Poky repository is fundamentally an aggregation of + existing repositories, some users might be familiar with running + the &OE_INIT_FILE; or + oe-init-build-env-memres script in the context + of separate OpenEmbedded-Core and BitBake repositories rather than a + single Poky repository. + This discussion assumes the script is executed from within a cloned + or unpacked version of Poky. + + + + Depending on where the script is sourced, different sub-scripts + are called to set up the Build Directory (Yocto or OpenEmbedded). + Specifically, the script + scripts/oe-setup-builddir inside the + poky directory sets up the Build Directory and seeds the directory + (if necessary) with configuration files appropriate for the + Yocto Project development environment. + + The scripts/oe-setup-builddir script + uses the $TEMPLATECONF variable to + determine which sample configuration files to locate. + + + + + The local.conf file provides many + basic variables that define a build environment. + Here is a list of a few. + To see the default configurations in a local.conf + file created by the build environment script, see the + local.conf.sample in the + meta-poky layer: + + Parallelism Options: + Controlled by the + BB_NUMBER_THREADS, + PARALLEL_MAKE, + and + BB_NUMBER_PARSE_THREADS + variables. + Target Machine Selection: + Controlled by the + MACHINE + variable. + Download Directory: + Controlled by the + DL_DIR + variable. + Shared State Directory: + Controlled by the + SSTATE_DIR + variable. + Build Output: + Controlled by the + TMPDIR + variable. + + + Configurations set in the conf/local.conf + file can also be set in the + conf/site.conf and + conf/auto.conf configuration files. + + + + + The bblayers.conf file tells BitBake what + layers you want considered during the build. + By default, the layers listed in this file include layers + minimally needed by the build system. + However, you must manually add any custom layers you have created. + You can find more information on working with the + bblayers.conf file in the + "Enabling Your Layer" + section in the Yocto Project Development Manual. + + + + The files site.conf and + auto.conf are not created by the environment + initialization script. + If you want the site.conf file, you need to + create that yourself. + The auto.conf file is typically created by + an autobuilder: + + site.conf: + You can use the conf/site.conf + configuration file to configure multiple build directories. + For example, suppose you had several build environments and + they shared some common features. + You can set these default build properties here. + A good example is perhaps the packaging format to use + through the + PACKAGE_CLASSES + variable. + One useful scenario for using the + conf/site.conf file is to extend your + BBPATH + variable to include the path to a + conf/site.conf. + Then, when BitBake looks for Metadata using + BBPATH, it finds the + conf/site.conf file and applies your + common configurations found in the file. + To override configurations in a particular build directory, + alter the similar configurations within that build + directory's conf/local.conf file. + + auto.conf: + The file is usually created and written to by + an autobuilder. + The settings put into the file are typically the same as + you would find in the conf/local.conf + or the conf/site.conf files. + + + + + + You can edit all configuration files to further define + any particular build environment. + This process is represented by the "User Configuration Edits" + box in the figure. + + + + When you launch your build with the + bitbake target + command, BitBake sorts out the configurations to ultimately + define your build environment. + It is important to understand that the OpenEmbedded build system + reads the configuration files in a specific order: + site.conf, auto.conf, + and local.conf. + And, the build system applies the normal assignment statement + rules. + Because the files are parsed in a specific order, variable + assignments for the same variable could be affected. + For example, if the auto.conf file and + the local.conf set + variable1 to different values, because + the build system parses local.conf after + auto.conf, + variable1 is assigned the value from + the local.conf file. + +
+ +
+ Metadata, Machine Configuration, and Policy Configuration + + + The previous section described the user configurations that + define BitBake's global behavior. + This section takes a closer look at the layers the build system + uses to further control the build. + These layers provide Metadata for the software, machine, and + policy. + + + + In general, three types of layer input exist: + + Policy Configuration: + Distribution Layers provide top-level or general + policies for the image or SDK being built. + For example, this layer would dictate whether BitBake + produces RPM or IPK packages. + Machine Configuration: + Board Support Package (BSP) layers provide machine + configurations. + This type of information is specific to a particular + target architecture. + Metadata: + Software layers contain user-supplied recipe files, + patches, and append files. + + + + + + The following figure shows an expanded representation of the + Metadata, Machine Configuration, and Policy Configuration input + (layers) boxes of the + general Yocto Project Development Environment figure: + + + + + + + + In general, all layers have a similar structure. + They all contain a licensing file + (e.g. COPYING) if the layer is to be + distributed, a README file as good practice + and especially if the layer is to be distributed, a + configuration directory, and recipe directories. + + + + The Yocto Project has many layers that can be used. + You can see a web-interface listing of them on the + Source Repositories + page. + The layers are shown at the bottom categorized under + "Yocto Metadata Layers." + These layers are fundamentally a subset of the + OpenEmbedded Metadata Index, + which lists all layers provided by the OpenEmbedded community. + + Layers exist in the Yocto Project Source Repositories that + cannot be found in the OpenEmbedded Metadata Index. + These layers are either deprecated or experimental in nature. + + + + + BitBake uses the conf/bblayers.conf file, + which is part of the user configuration, to find what layers it + should be using as part of the build. + + + + For more information on layers, see the + "Understanding and Creating Layers" + section in the Yocto Project Development Manual. + + +
+ Distro Layer + + + The distribution layer provides policy configurations for your + distribution. + Best practices dictate that you isolate these types of + configurations into their own layer. + Settings you provide in + conf/distro/distro.conf override + similar + settings that BitBake finds in your + conf/local.conf file in the Build + Directory. + + + + The following list provides some explanation and references + for what you typically find in the distribution layer: + + classes: + Class files (.bbclass) hold + common functionality that can be shared among + recipes in the distribution. + When your recipes inherit a class, they take on the + settings and functions for that class. + You can read more about class files in the + "Classes" section. + + conf: + This area holds configuration files for the + layer (conf/layer.conf), + the distribution + (conf/distro/distro.conf), + and any distribution-wide include files. + + recipes-*: + Recipes and append files that affect common + functionality across the distribution. + This area could include recipes and append files + to add distribution-specific configuration, + initialization scripts, custom image recipes, + and so forth. + + +
+ +
+ BSP Layer + + + The BSP Layer provides machine configurations. + Everything in this layer is specific to the machine for which + you are building the image or the SDK. + A common structure or form is defined for BSP layers. + You can learn more about this structure in the + Yocto Project Board Support Package (BSP) Developer's Guide. + + In order for a BSP layer to be considered compliant with the + Yocto Project, it must meet some structural requirements. + + + + + The BSP Layer's configuration directory contains + configuration files for the machine + (conf/machine/machine.conf) and, + of course, the layer (conf/layer.conf). + + + + The remainder of the layer is dedicated to specific recipes + by function: recipes-bsp, + recipes-core, + recipes-graphics, and + recipes-kernel. + Metadata can exist for multiple formfactors, graphics + support systems, and so forth. + + While the figure shows several recipes-* + directories, not all these directories appear in all + BSP layers. + + +
+ +
+ Software Layer + + + The software layer provides the Metadata for additional + software packages used during the build. + This layer does not include Metadata that is specific to the + distribution or the machine, which are found in their + respective layers. + + + + This layer contains any new recipes that your project needs + in the form of recipe files. + +
+
+ +
+ Sources + + + In order for the OpenEmbedded build system to create an image or + any target, it must be able to access source files. + The + general Yocto Project Development Environment figure + represents source files using the "Upstream Project Releases", + "Local Projects", and "SCMs (optional)" boxes. + The figure represents mirrors, which also play a role in locating + source files, with the "Source Mirror(s)" box. + + + + The method by which source files are ultimately organized is + a function of the project. + For example, for released software, projects tend to use tarballs + or other archived files that can capture the state of a release + guaranteeing that it is statically represented. + On the other hand, for a project that is more dynamic or + experimental in nature, a project might keep source files in a + repository controlled by a Source Control Manager (SCM) such as + Git. + Pulling source from a repository allows you to control + the point in the repository (the revision) from which you want to + build software. + Finally, a combination of the two might exist, which would give the + consumer a choice when deciding where to get source files. + + + + BitBake uses the + SRC_URI + variable to point to source files regardless of their location. + Each recipe must have a SRC_URI variable + that points to the source. + + + + Another area that plays a significant role in where source files + come from is pointed to by the + DL_DIR + variable. + This area is a cache that can hold previously downloaded source. + You can also instruct the OpenEmbedded build system to create + tarballs from Git repositories, which is not the default behavior, + and store them in the DL_DIR by using the + BB_GENERATE_MIRROR_TARBALLS + variable. + + + + Judicious use of a DL_DIR directory can + save the build system a trip across the Internet when looking + for files. + A good method for using a download directory is to have + DL_DIR point to an area outside of your + Build Directory. + Doing so allows you to safely delete the Build Directory + if needed without fear of removing any downloaded source file. + + + + The remainder of this section provides a deeper look into the + source files and the mirrors. + Here is a more detailed look at the source file area of the + base figure: + + + +
+ Upstream Project Releases + + + Upstream project releases exist anywhere in the form of an + archived file (e.g. tarball or zip file). + These files correspond to individual recipes. + For example, the figure uses specific releases each for + BusyBox, Qt, and Dbus. + An archive file can be for any released product that can be + built using a recipe. + +
+ +
+ Local Projects + + + Local projects are custom bits of software the user provides. + These bits reside somewhere local to a project - perhaps + a directory into which the user checks in items (e.g. + a local directory containing a development source tree + used by the group). + + + + The canonical method through which to include a local project + is to use the + externalsrc + class to include that local project. + You use either the local.conf or a + recipe's append file to override or set the + recipe to point to the local directory on your disk to pull + in the whole source tree. + + + + For information on how to use the + externalsrc class, see the + "externalsrc.bbclass" + section. + +
+ +
+ Source Control Managers (Optional) + + + Another place the build system can get source files from is + through an SCM such as Git or Subversion. + In this case, a repository is cloned or checked out. + The + do_fetch + task inside BitBake uses + the SRC_URI + variable and the argument's prefix to determine the correct + fetcher module. + + + + For information on how to have the OpenEmbedded build system + generate tarballs for Git repositories and place them in the + DL_DIR + directory, see the + BB_GENERATE_MIRROR_TARBALLS + variable. + + + + When fetching a repository, BitBake uses the + SRCREV + variable to determine the specific revision from which to + build. + +
+ +
+ Source Mirror(s) + + + Two kinds of mirrors exist: pre-mirrors and regular mirrors. + The PREMIRRORS + and + MIRRORS + variables point to these, respectively. + BitBake checks pre-mirrors before looking upstream for any + source files. + Pre-mirrors are appropriate when you have a shared directory + that is not a directory defined by the + DL_DIR + variable. + A Pre-mirror typically points to a shared directory that is + local to your organization. + + + + Regular mirrors can be any site across the Internet that is + used as an alternative location for source code should the + primary site not be functioning for some reason or another. + +
+
+ +
+ Package Feeds + + + When the OpenEmbedded build system generates an image or an SDK, + it gets the packages from a package feed area located in the + Build Directory. + The + general Yocto Project Development Environment figure + shows this package feeds area in the upper-right corner. + + + + This section looks a little closer into the package feeds area used + by the build system. + Here is a more detailed look at the area: + + + + + Package feeds are an intermediary step in the build process. + The OpenEmbedded build system provides classes to generate + different package types, and you specify which classes to enable + through the + PACKAGE_CLASSES + variable. + Before placing the packages into package feeds, + the build process validates them with generated output quality + assurance checks through the + insane + class. + + + + The package feed area resides in the Build Directory. + The directory the build system uses to temporarily store packages + is determined by a combination of variables and the particular + package manager in use. + See the "Package Feeds" box in the illustration and note the + information to the right of that area. + In particular, the following defines where package files are + kept: + + DEPLOY_DIR: + Defined as tmp/deploy in the Build + Directory. + + DEPLOY_DIR_*: + Depending on the package manager used, the package type + sub-folder. + Given RPM, IPK, or DEB packaging and tarball creation, the + DEPLOY_DIR_RPM, + DEPLOY_DIR_IPK, + DEPLOY_DIR_DEB, + or + DEPLOY_DIR_TAR, + variables are used, respectively. + + PACKAGE_ARCH: + Defines architecture-specific sub-folders. + For example, packages could exist for the i586 or qemux86 + architectures. + + + + + + BitBake uses the do_package_write_* tasks to + generate packages and place them into the package holding area (e.g. + do_package_write_ipk for IPK packages). + See the + "do_package_write_deb", + "do_package_write_ipk", + "do_package_write_rpm", + and + "do_package_write_tar" + sections for additional information. + As an example, consider a scenario where an IPK packaging manager + is being used and package architecture support for both i586 + and qemux86 exist. + Packages for the i586 architecture are placed in + build/tmp/deploy/ipk/i586, while packages for + the qemux86 architecture are placed in + build/tmp/deploy/ipk/qemux86. + +
+ +
+ BitBake + + + The OpenEmbedded build system uses + BitBake + to produce images. + You can see from the + general Yocto Project Development Environment figure, + the BitBake area consists of several functional areas. + This section takes a closer look at each of those areas. + + + + Separate documentation exists for the BitBake tool. + See the + BitBake User Manual + for reference material on BitBake. + + +
+ Source Fetching + + + The first stages of building a recipe are to fetch and unpack + the source code: + + + + + The + do_fetch + and + do_unpack + tasks fetch the source files and unpack them into the work + directory. + + For every local file (e.g. file://) + that is part of a recipe's + SRC_URI + statement, the OpenEmbedded build system takes a checksum + of the file for the recipe and inserts the checksum into + the signature for the do_fetch. + If any local file has been modified, the + do_fetch task and all tasks that + depend on it are re-executed. + + By default, everything is accomplished in the + Build Directory, + which has a defined structure. + For additional general information on the Build Directory, + see the + "build/" + section. + + + + Unpacked source files are pointed to by the + S variable. + Each recipe has an area in the Build Directory where the + unpacked source code resides. + The name of that directory for any given recipe is defined from + several different variables. + You can see the variables that define these directories + by looking at the figure: + + TMPDIR - + The base directory where the OpenEmbedded build system + performs all its work during the build. + + PACKAGE_ARCH - + The architecture of the built package or packages. + + TARGET_OS - + The operating system of the target device. + + PN - + The name of the built package. + + PV - + The version of the recipe used to build the package. + + PR - + The revision of the recipe used to build the package. + + WORKDIR - + The location within TMPDIR where + a specific package is built. + + S - + Contains the unpacked source files for a given recipe. + + + +
+ +
+ Patching + + + Once source code is fetched and unpacked, BitBake locates + patch files and applies them to the source files: + + + + + The + do_patch + task processes recipes by + using the + SRC_URI + variable to locate applicable patch files, which by default + are *.patch or + *.diff files, or any file if + "apply=yes" is specified for the file in + SRC_URI. + + + + BitBake finds and applies multiple patches for a single recipe + in the order in which it finds the patches. + Patches are applied to the recipe's source files located in the + S directory. + + + + For more information on how the source directories are + created, see the + "Source Fetching" + section. + +
+ +
+ Configuration and Compilation + + + After source code is patched, BitBake executes tasks that + configure and compile the source code: + + + + + This step in the build process consists of three tasks: + + + do_prepare_recipe_sysroot: + This task sets up the two sysroots in + ${WORKDIR} + (i.e. recipe-sysroot and + recipe-sysroot-native) so that + the sysroots contain the contents of the + do_populate_sysroot + tasks of the recipes on which the recipe + containing the tasks depends. + A sysroot exists for both the target and for the native + binaries, which run on the host system. + + do_configure: + This task configures the source by enabling and + disabling any build-time and configuration options for + the software being built. + Configurations can come from the recipe itself as well + as from an inherited class. + Additionally, the software itself might configure itself + depending on the target for which it is being built. + + + The configurations handled by the + do_configure + task are specific + to source code configuration for the source code + being built by the recipe. + + If you are using the + autotools + class, + you can add additional configuration options by using + the EXTRA_OECONF + or + PACKAGECONFIG_CONFARGS + variables. + For information on how this variable works within + that class, see the + meta/classes/autotools.bbclass file. + + do_compile: + Once a configuration task has been satisfied, BitBake + compiles the source using the + do_compile + task. + Compilation occurs in the directory pointed to by the + B + variable. + Realize that the B directory is, by + default, the same as the + S + directory. + do_install: + Once compilation is done, BitBake executes the + do_install + task. + This task copies files from the B + directory and places them in a holding area pointed to + by the + D + variable. + + +
+ +
+ Package Splitting + + + After source code is configured and compiled, the + OpenEmbedded build system analyzes + the results and splits the output into packages: + + + + + The + do_package + and + do_packagedata + tasks combine to analyze + the files found in the + D directory + and split them into subsets based on available packages and + files. + The analyzing process involves the following as well as other + items: splitting out debugging symbols, + looking at shared library dependencies between packages, + and looking at package relationships. + The do_packagedata task creates package + metadata based on the analysis such that the + OpenEmbedded build system can generate the final packages. + Working, staged, and intermediate results of the analysis + and package splitting process use these areas: + + PKGD - + The destination directory for packages before they are + split. + + PKGDATA_DIR - + A shared, global-state directory that holds data + generated during the packaging process. + + PKGDESTWORK - + A temporary work area used by the + do_package task. + + PKGDEST - + The parent directory for packages after they have + been split. + + + The FILES + variable defines the files that go into each package in + PACKAGES. + If you want details on how this is accomplished, you can + look at the + package + class. + + + + Depending on the type of packages being created (RPM, DEB, or + IPK), the do_package_write_* task + creates the actual packages and places them in the + Package Feed area, which is + ${TMPDIR}/deploy. + You can see the + "Package Feeds" + section for more detail on that part of the build process. + + Support for creating feeds directly from the + deploy/* directories does not exist. + Creating such feeds usually requires some kind of feed + maintenance mechanism that would upload the new packages + into an official package feed (e.g. the + Ångström distribution). + This functionality is highly distribution-specific + and thus is not provided out of the box. + + +
+ +
+ Image Generation + + + Once packages are split and stored in the Package Feeds area, + the OpenEmbedded build system uses BitBake to generate the + root filesystem image: + + + + + The image generation process consists of several stages and + depends on several tasks and variables. + The + do_rootfs + task creates the root filesystem (file and directory structure) + for an image. + This task uses several key variables to help create the list + of packages to actually install: + + IMAGE_INSTALL: + Lists out the base set of packages to install from + the Package Feeds area. + PACKAGE_EXCLUDE: + Specifies packages that should not be installed. + + IMAGE_FEATURES: + Specifies features to include in the image. + Most of these features map to additional packages for + installation. + PACKAGE_CLASSES: + Specifies the package backend to use and consequently + helps determine where to locate packages within the + Package Feeds area. + IMAGE_LINGUAS: + Determines the language(s) for which additional + language support packages are installed. + + PACKAGE_INSTALL: + The final list of packages passed to the package manager + for installation into the image. + + + + + + With + IMAGE_ROOTFS + pointing to the location of the filesystem under construction and + the PACKAGE_INSTALL variable providing the + final list of packages to install, the root file system is + created. + + + + Package installation is under control of the package manager + (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of whether or + not package management is enabled for the target. + At the end of the process, if package management is not + enabled for the target, the package manager's data files + are deleted from the root filesystem. + As part of the final stage of package installation, postinstall + scripts that are part of the packages are run. + Any scripts that fail to run + on the build host are run on the target when the target system + is first booted. + If you are using a + read-only root filesystem, + all the post installation scripts must succeed during the + package installation phase since the root filesystem is + read-only. + + + + The final stages of the do_rootfs task + handle post processing. + Post processing includes creation of a manifest file and + optimizations. + + + + The manifest file (.manifest) resides + in the same directory as the root filesystem image. + This file lists out, line-by-line, the installed packages. + The manifest file is useful for the + testimage + class, for example, to determine whether or not to run + specific tests. + See the + IMAGE_MANIFEST + variable for additional information. + + + + Optimizing processes run across the image include + mklibs, prelink, + and any other post-processing commands as defined by the + ROOTFS_POSTPROCESS_COMMAND + variable. + The mklibs process optimizes the size + of the libraries, while the + prelink process optimizes the dynamic + linking of shared libraries to reduce start up time of + executables. + + + + After the root filesystem is built, processing begins on + the image through the do_image task. + The build system runs any pre-processing commands as defined + by the + IMAGE_PREPROCESS_COMMAND + variable. + This variable specifies a list of functions to call before + the OpenEmbedded build system creates the final image output + files. + + + + The do_image task dynamically creates + other do_image_* tasks as needed, which + include compressing the root filesystem image to reduce the + overall size of the image. + The process turns everything into an image file or a set of + image files. + The formats used for the root filesystem depend on the + IMAGE_FSTYPES + variable. + + + + The final task involved in image creation is the + do_image_complete task. + This task completes the image by applying any image + post processing as defined through the + IMAGE_POSTPROCESS_COMMAND + variable. + The variable specifies a list of functions to call once the + OpenEmbedded build system has created the final image output + files. + + + + The entire image generation process is run under Pseudo. + Running under Pseudo ensures that the files in the root + filesystem have correct ownership. + +
+ +
+ SDK Generation + + + The OpenEmbedded build system uses BitBake to generate the + Software Development Kit (SDK) installer script for both the + standard and extensible SDKs: + + + + + For more information on the cross-development toolchain + generation, see the + "Cross-Development Toolchain Generation" + section. + For information on advantages gained when building a + cross-development toolchain using the + do_populate_sdk + task, see the + "Building an SDK Installer" + section in the Yocto Project Software Development Kit (SDK) + Developer's Guide. + + + + Like image generation, the SDK script process consists of + several stages and depends on many variables. + The do_populate_sdk and + do_populate_sdk_ext tasks use these + key variables to help create the list of packages to actually + install. + For information on the variables listed in the figure, see the + "Application Development SDK" + section. + + + + The do_populate_sdk task helps create + the standard SDK and handles two parts: a target part and a + host part. + The target part is the part built for the target hardware and + includes libraries and headers. + The host part is the part of the SDK that runs on the + SDKMACHINE. + + + + The do_populate_sdk_ext task helps create + the extensible SDK and handles host and target parts + differently than its counter part does for the standard SDK. + For the extensible SDK, the task encapsulates the build system, + which includes everything needed (host and target) for the SDK. + + + + Regardless of the type of SDK being constructed, the + tasks perform some cleanup after which a cross-development + environment setup script and any needed configuration files + are created. + The final output is the Cross-development + toolchain installation script (.sh file), + which includes the environment setup script. + +
+ +
+ Stamp Files and the Rerunning of Tasks + + + For each task that completes successfully, BitBake writes a + stamp file into the + STAMPS_DIR + directory. + The beginning of the stamp file's filename is determined by the + STAMP + variable, and the end of the name consists of the task's name + and current + input checksum. + + This naming scheme assumes that + BB_SIGNATURE_HANDLER + is "OEBasicHash", which is almost always the case in + current OpenEmbedded. + + To determine if a task needs to be rerun, BitBake checks if a + stamp file with a matching input checksum exists for the task. + If such a stamp file exists, the task's output is assumed to + exist and still be valid. + If the file does not exist, the task is rerun. + + The stamp mechanism is more general than the shared + state (sstate) cache mechanism described in the + "Setscene Tasks and Shared State" + section. + BitBake avoids rerunning any task that has a valid + stamp file, not just tasks that can be accelerated through + the sstate cache. + However, you should realize that stamp files only + serve as a marker that some work has been done and that + these files do not record task output. + The actual task output would usually be somewhere in + TMPDIR + (e.g. in some recipe's + WORKDIR.) + What the sstate cache mechanism adds is a way to cache task + output that can then be shared between build machines. + + + Since STAMPS_DIR is usually a subdirectory + of TMPDIR, removing + TMPDIR will also remove + STAMPS_DIR, which means tasks will + properly be rerun to repopulate TMPDIR. + + + + If you want some task to always be considered "out of date", + you can mark it with the + nostamp + varflag. + If some other task depends on such a task, then that task will + also always be considered out of date, which might not be what + you want. + + + + For details on how to view information about a task's + signature, see the + "Viewing Task Variable Dependencies" + section. + +
+ +
+ Setscene Tasks and Shared State + + + The description of tasks so far assumes that BitBake needs to + build everything and there are no prebuilt objects available. + BitBake does support skipping tasks if prebuilt objects are + available. + These objects are usually made available in the form of a + shared state (sstate) cache. + + For information on variables affecting sstate, see the + SSTATE_DIR + and + SSTATE_MIRRORS + variables. + + + + + The idea of a setscene task (i.e + do_taskname_setscene) + is a version of the task where + instead of building something, BitBake can skip to the end + result and simply place a set of files into specific locations + as needed. + In some cases, it makes sense to have a setscene task variant + (e.g. generating package files in the + do_package_write_* task). + In other cases, it does not make sense, (e.g. a + do_patch + task or + do_unpack + task) since the work involved would be equal to or greater than + the underlying task. + + + + In the OpenEmbedded build system, the common tasks that have + setscene variants are do_package, + do_package_write_*, + do_deploy, + do_packagedata, + and + do_populate_sysroot. + Notice that these are most of the tasks whose output is an + end result. + + + + The OpenEmbedded build system has knowledge of the relationship + between these tasks and other tasks that precede them. + For example, if BitBake runs + do_populate_sysroot_setscene for + something, there is little point in running any of the + do_fetch, do_unpack, + do_patch, + do_configure, + do_compile, and + do_install tasks. + However, if do_package needs to be run, + BitBake would need to run those other tasks. + + + + It becomes more complicated if everything can come from an + sstate cache because some objects are simply not required at + all. + For example, you do not need a compiler or native tools, such + as quilt, if there is nothing to compile or patch. + If the do_package_write_* packages are + available from sstate, BitBake does not need the + do_package task data. + + + + To handle all these complexities, BitBake runs in two phases. + The first is the "setscene" stage. + During this stage, BitBake first checks the sstate cache for + any targets it is planning to build. + BitBake does a fast check to see if the object exists rather + than a complete download. + If nothing exists, the second phase, which is the setscene + stage, completes and the main build proceeds. + + + + If objects are found in the sstate cache, the OpenEmbedded + build system works backwards from the end targets specified + by the user. + For example, if an image is being built, the OpenEmbedded build + system first looks for the packages needed for that image and + the tools needed to construct an image. + If those are available, the compiler is not needed. + Thus, the compiler is not even downloaded. + If something was found to be unavailable, or the download or + setscene task fails, the OpenEmbedded build system then tries + to install dependencies, such as the compiler, from the cache. + + + + The availability of objects in the sstate cache is handled by + the function specified by the + BB_HASHCHECK_FUNCTION + variable and returns a list of the objects that are available. + The function specified by the + BB_SETSCENE_DEPVALID + variable is the function that determines whether a given + dependency needs to be followed, and whether for any given + relationship the function needs to be passed. + The function returns a True or False value. + +
+
+ +
+ Images + + + The images produced by the OpenEmbedded build system + are compressed forms of the + root filesystem that are ready to boot on a target device. + You can see from the + general Yocto Project Development Environment figure + that BitBake output, in part, consists of images. + This section is going to look more closely at this output: + + + + + For a list of example images that the Yocto Project provides, + see the + "Images" chapter. + + + + Images are written out to the + Build Directory + inside the tmp/deploy/images/machine/ + folder as shown in the figure. + This folder contains any files expected to be loaded on the + target device. + The + DEPLOY_DIR + variable points to the deploy directory, + while the + DEPLOY_DIR_IMAGE + variable points to the appropriate directory containing images for + the current configuration. + + kernel-image: + A kernel binary file. + The KERNEL_IMAGETYPE + variable setting determines the naming scheme for the + kernel image file. + Depending on that variable, the file could begin with + a variety of naming strings. + The deploy/images/machine + directory can contain multiple image files for the + machine. + root-filesystem-image: + Root filesystems for the target device (e.g. + *.ext3 or *.bz2 + files). + The IMAGE_FSTYPES + variable setting determines the root filesystem image + type. + The deploy/images/machine + directory can contain multiple root filesystems for the + machine. + kernel-modules: + Tarballs that contain all the modules built for the kernel. + Kernel module tarballs exist for legacy purposes and + can be suppressed by setting the + MODULE_TARBALL_DEPLOY + variable to "0". + The deploy/images/machine + directory can contain multiple kernel module tarballs + for the machine. + bootloaders: + Bootloaders supporting the image, if applicable to the + target machine. + The deploy/images/machine + directory can contain multiple bootloaders for the + machine. + symlinks: + The deploy/images/machine + folder contains + a symbolic link that points to the most recently built file + for each machine. + These links might be useful for external scripts that + need to obtain the latest version of each file. + + + +
+ +
+ Application Development SDK + + + In the + general Yocto Project Development Environment figure, + the output labeled "Application Development SDK" represents an + SDK. + The SDK generation process differs depending on whether you build + a standard SDK + (e.g. bitbake -c populate_sdk imagename) + or an extensible SDK + (e.g. bitbake -c populate_sdk_ext imagename). + This section is going to take a closer look at this output: + + + + + The specific form of this output is a self-extracting + SDK installer (*.sh) that, when run, + installs the SDK, which consists of a cross-development + toolchain, a set of libraries and headers, and an SDK + environment setup script. + Running this installer essentially sets up your + cross-development environment. + You can think of the cross-toolchain as the "host" + part because it runs on the SDK machine. + You can think of the libraries and headers as the "target" + part because they are built for the target hardware. + The environment setup script is added so that you can initialize + the environment before using the tools. + + + + + The Yocto Project supports several methods by which you can + set up this cross-development environment. + These methods include downloading pre-built SDK installers + or building and installing your own SDK installer. + + + + For background information on cross-development toolchains + in the Yocto Project development environment, see the + "Cross-Development Toolchain Generation" + section. + For information on setting up a cross-development + environment, see the + Yocto Project Software Development Kit (SDK) Developer's Guide. + + + + + Once built, the SDK installers are written out to the + deploy/sdk folder inside the + Build Directory + as shown in the figure at the beginning of this section. + Depending on the type of SDK, several variables exist that help + configure these files. + The following list shows the variables associated with a standard + SDK: + + DEPLOY_DIR: + Points to the deploy + directory. + SDKMACHINE: + Specifies the architecture of the machine + on which the cross-development tools are run to + create packages for the target hardware. + + SDKIMAGE_FEATURES: + Lists the features to include in the "target" part + of the SDK. + + TOOLCHAIN_HOST_TASK: + Lists packages that make up the host + part of the SDK (i.e. the part that runs on + the SDKMACHINE). + When you use + bitbake -c populate_sdk imagename + to create the SDK, a set of default packages + apply. + This variable allows you to add more packages. + + TOOLCHAIN_TARGET_TASK: + Lists packages that make up the target part + of the SDK (i.e. the part built for the + target hardware). + + SDKPATH: + Defines the default SDK installation path offered by the + installation script. + + + This next list, shows the variables associated with an extensible + SDK: + + DEPLOY_DIR: + Points to the deploy directory. + + SDK_EXT_TYPE: + Controls whether or not shared state artifacts are copied + into the extensible SDK. + By default, all required shared state artifacts are copied + into the SDK. + + SDK_INCLUDE_PKGDATA: + Specifies whether or not packagedata will be included in + the extensible SDK for all recipes in the "world" target. + + SDK_INCLUDE_TOOLCHAIN: + Specifies whether or not the toolchain will be included + when building the extensible SDK. + + SDK_LOCAL_CONF_WHITELIST: + A list of variables allowed through from the build system + configuration into the extensible SDK configuration. + + SDK_LOCAL_CONF_BLACKLIST: + A list of variables not allowed through from the build + system configuration into the extensible SDK configuration. + + SDK_INHERIT_BLACKLIST: + A list of classes to remove from the + INHERIT + value globally within the extensible SDK configuration. + + + +
+
+ + + diff --git a/documentation/ref-manual/ref-manual.xml b/documentation/ref-manual/ref-manual.xml index 16900adfb8..8096871c1c 100644 --- a/documentation/ref-manual/ref-manual.xml +++ b/documentation/ref-manual/ref-manual.xml @@ -163,7 +163,7 @@ - + diff --git a/documentation/ref-manual/technical-details.xml b/documentation/ref-manual/technical-details.xml index 768f70186b..7fc2e184e6 100644 --- a/documentation/ref-manual/technical-details.xml +++ b/documentation/ref-manual/technical-details.xml @@ -55,8 +55,8 @@ Following are some brief details on these core components. For additional information on how these components interact during a build, see the - "A Closer Look at the Yocto Project Development Environment" - Chapter. + "Development Concepts" + section.
diff --git a/documentation/yocto-project-qs/yocto-project-qs.xml b/documentation/yocto-project-qs/yocto-project-qs.xml index 1d9bd9f2e8..527fcd86cc 100644 --- a/documentation/yocto-project-qs/yocto-project-qs.xml +++ b/documentation/yocto-project-qs/yocto-project-qs.xml @@ -948,7 +948,7 @@ You can get build details, a - closer look + closer look at how the pieces of the Yocto Project development environment work together, information on various technical details, -- cgit v1.2.3-54-g00ecf