From cc265bf535f5dd77bf9818bebd24930c999cff85 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Mon, 9 Sep 2013 14:34:57 -0700 Subject: ref-manual: Created new "Closer Look" chapter Fixes [YOCTO #2808] I extracted the section that takes a closer look at the YP development process into its own chapter. Feedback during the review indicated that this information should not be buried as it was in a section but rather pulled higher out for visibility. So, The changes create a new chapter three that is dedicated to this topic. (From yocto-docs rev: 32c66976b6b84787d14d6174dab843862a0d184b) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- documentation/ref-manual/closer-look.xml | 1050 ++++++++++++++++++++++++++++++ 1 file changed, 1050 insertions(+) create mode 100644 documentation/ref-manual/closer-look.xml (limited to 'documentation/ref-manual/closer-look.xml') diff --git a/documentation/ref-manual/closer-look.xml b/documentation/ref-manual/closer-look.xml new file mode 100644 index 0000000000..8c3ceb7210 --- /dev/null +++ b/documentation/ref-manual/closer-look.xml @@ -0,0 +1,1050 @@ + %poky; ] > + + +A Closer Look at the Yocto Project Development Environment + + + This chapter takes a more detailed look at the Yocto Project + development environment. + The following diagram represents the development environment at a + high level. + The remainder of this chapter expands on the fundamental input, output, + process, and + Metadata) blocks + in the Yocto Project development environment. + + + + + + + + The generalized Yocto Project Development Environment consists of + several functional areas: + + User Configuration: + Metadata you can use to control the build process. + + Metadata Layers: + Various layers that provide software, machine, and + distro Metadata. + Source Files: + Upstream releases, local projects, and SCMs. + Build System: + Processes under the control of BitBake. + This block expands on how BitBake fetches source, applies + patches, completes compilation, analyzes output for package + generation, creates and tests packages, generates images, and + generates cross-development tools. + Package Feeds: + Directories containing output packages (rpm, deb or ipk), + which are subsequently used in the construction of an image or + SDK, produced by the build system. + These feeds can also be copied and shared using a web server or + other means to facilitate extending or updating existing + images on devices at runtime if runtime package management is + enabled. + Images: + Images produced by the development process. + Where do they go? + Can you mess with them (i.e. freely delete them or move them?). + + Application Development SDK: + Cross-development tools that are produced along with an image + or separately with BitBake. + + + +
+ User Configuration + + + User configuration helps define the build. + Through user configuration, you can tell BitBake the + target architecture for which you are building the image, + where to store downloaded source, and other build properties. + + + + The following figure shows an expanded representation of the + "User Configuration" box of the + general Yocto Project Development Environment figure: + + + + + + + + BitBake needs some basic configuration files in order to complete + a build. + These files are *.conf files. + The minimally necessary ones reside as example files in the + Source Directory. + For simplicity, this section refers to the Source Directory as + the "Poky Directory." + + + + When you clone the poky Git repository or you + download and unpack a Yocto Project release, you can set up the + Source Directory to be named anything you want. + For this discussion, the cloned repository uses the default + name poky. + + The Poky repository is primarily an aggregation of existing + repositories. + It is not a canonical upstream source. + + + + + The meta-yocto layer inside Poky contains + a conf directory that has example + configuration files. + These example files are used as a basis for creating actual + configuration files when you source the build environment + script oe-init-build-env. + + + + Sourcing the build environment script creates a + Build Directory + if one does not already exist. + BitBake uses the Build Directory for all its work during builds. + The Build Directory has a conf directory that + contains default versions of your local.conf + and bblayers.conf configuration files. + These default configuration files are created only if versions + do not already exist in the Build Directory at the time you + source the oe-init-build-env script. + + + + Because the Poky repository is fundamentally an aggregation of + existing repositories, some users might be familiar with running + the oe-init-build-env script in the context of + separate OpenEmbedded-Core and BitBake repositories rather than a + single Poky repository. + This discussion assumes the script is executed from within a cloned + or unpacked version of Poky. + + + + Depending on where the script is sourced, different sub-scripts + are called to set up the Build Directory (Yocto or OpenEmbedded). + Specifically, the script + scripts/oe-setup-builddir inside the + poky directory sets up the Build Directory and seeds the directory + (if necessary) with configuration files appropriate for the + Yocto Project development environment. + + The scripts/oe-setup-builddir script + uses the $TEMPLATECONF variable to + determine which sample configuration files to locate. + + + + + The local.conf file provides many + basic variables that define a build environment. + Here is a list of a few. + To see the default configurations in a local.conf + file created by the build environment script, see the + local.conf.sample in the + meta-yocto layer: + + Parallelism Options: + Controlled by the + BB_NUMBER_THREADS + and + PARALLEL_MAKE + variables. + Target Machine Selection: + Controlled by the + MACHINE + variable. + Download Directory: + Controlled by the + DL_DIR + variable. + Shared State Directory: + Controlled by the + SSTATE_DIR + variable. + Build Output: + Controlled by the + TMPDIR + variable. + + + Configurations set in the conf/local.conf + file can also be set in the + conf/site.conf and + conf/auto.conf configuration files. + + + + + The bblayers.conf file tells BitBake what + layers you want considered during the build. + By default, the layers listed in this file include layers + minimally needed by the build system. + However, you must manually add any custom layers you have created. + You can find more information on working with the + bblayers.conf file in the + "Enabling Your Layer" + section in the Yocto Project Development Manual. + + + + The files site.conf and + auto.conf are not created by the environment + initialization script. + If you want these configuration files, you must create them + yourself: + + site.conf: + You can use the conf/site.conf + configuration file to configure multiple build directories. + For example, suppose you had several build environments and + they shared some common features. + You can set these default build properties here. + A good example is perhaps the level of parallelism you want + to use through the + BB_NUMBER_THREADS + and + PARALLEL_MAKE + variables. + One useful scenario for using the + conf/site.conf file is to extend your + BBPATH + variable to include the path to a + conf/site.conf. + Then, when BitBake looks for Metadata using + BBPATH, it finds the + conf/site.conf file and applies your + common configurations found in the file. + To override configurations in a particular build directory, + alter the similar configurations within that build + directory's conf/local.conf file. + + auto.conf: + This file is not hand-created. + Rather, the file is usually created and written to by + an autobuilder. + The settings put into the file are typically the same as + you would find in the conf/local.conf + or the conf/site.conf files. + + + + + + You can edit all configuration files to further define + any particular build environment. + This process is represented by the "User Configuration Edits" + box in the figure. + + + + When you launch your build with the + bitbake <target> command, BitBake + sorts out the configurations to ultimately define your build + environment. + +
+ +
+ Metadata, Machine Configuration, and Policy Configuration + + + The previous section described the user configurations that + define the BitBake's global behavior. + This section takes a closer look at the layers the build system + uses to further control the build. + These layers provide Metadata for the software, machine, and + policy. + + + + In general, three types of layer input exist: + + Policy Configuration: + Distribution Layers provide top-level or general + policies for the image or SDK being built. + For example, this layer would dictate whether BitBake + produces RPM or IPK packages. + Machine Configuration: + Board Support Package (BSP) layers provide machine + configurations. + This type of information is specific to a particular + target architecture. + Metadata: + Software layers contain user-supplied recipe files, + patches, and append files. + + + + + + The following figure shows an expanded representation of the + Metadata, Machine Configuration, and Policy Configuration input + (layers) boxes of the + general Yocto Project Development Environment figure: + + + + + + + + In general, all layers have a similar structure. + They all contain a licensing file + (e.g. COPYING) if the layer is to be + distributed, a README file as good practice + and especially if the layer is to be distributed, a + configuration directory, and recipe directories. + + + + The Yocto Project has many layers that can be used. + You can see a web-interface listing of them on the + Source Repositories + page. + The layers are shown at the bottom categorized under + "Yocto Metadata Layers." + These layers are fundamentally a subset of the + OpenEmbedded Metadata Index, + which lists all layers provided by the OpenEmbedded community. + + Layers exist in the Yocto Project Source Repositories that + cannot be found in the OpenEmbedded Metadata Index. + These layers are either deprecated or experimental in nature. + + + + + BitBake uses the conf/bblayers.conf file, + which is part of the user configuration, to find what layers it + should be using as part of the build. + + + + For more information on layers, see the + "Understanding and Creating Layers" + section in the Yocto Project Development Manual. + + +
+ Distro Layer + + + The distribution layer provides policy configurations for your + distribution. + Best practices dictate that you isolate these types of + configurations into their own layer. + Settings you provide in + conf/<distro>.conf override similar + settings that BitBake finds in your + conf/local.conf file in the Build + Directory. + + + + The following list provides some explanation and references + for what you typically find in the distribution layer: + + classes: + Class files (.bbclass) holds + common functionality that can be shared among + recipes in the distribution. + When your recipes inherit a class, they take on the + settings and functions for that class. + You can read more about class files in the + "Classes" section. + + conf: + This area holds configuration files for the + layer (conf/layer.conf), + the distribution + (conf/distro/<distro>.conf), + and any distribution-wide include files. + + recipes-*: + Recipes and append files that affect common + functionality across the distribution. + This area could include recipes and append files to + to add distribution-specific configuration, + initialization scripts, custom image recipes, + and so forth. + + +
+ +
+ BSP Layer + + + The BSP Layer provides machine configurations. + Everything in this layer is specific to the machine for which + you are building the image or the SDK. + A common structure or form is defined for BSP layers. + You can learn more about this structure in the + Yocto Project Board Support Package (BSP) Developer's Guide. + + In order for a BSP layer to be considered compliant with the + Yocto Project, it must meet some structural requirements. + + + + + The BSP Layer's configuration directory contains + configuration files for the machine + (conf/machine/<machine>.conf) and, + of course, the layer (conf/layer.conf). + + + + The remainder of the layer is dedicated to specific recipes + by function: recipes-bsp, + recipes-core, + recipes-graphics, and + recipes-kernel. + Metadata can exist for multiple formfactors, graphics + support systems, and so forth. + + While the figure shows several recipe-* + directories, not all these directories appear in all + BSP layers. + + +
+ +
+ Software Layer + + + The software layer provides the Metadata for additional + software packages used during the build. + This layer does not include Metadata that is specific to the + distribution or the machine, which are found in their + respective layers. + + + + This layer contains any new recipes that your project needs + in the form of recipe files. + +
+
+ +
+ Sources + + + In order for the OpenEmbedded build system to create an image or + any target, it must be able to access source files. + The + general Yocto Project Development Environment figure + represents source files using the "Upstream Project Releases", + "Local Projects", and "SCMs (optional)" boxes. + The figure represents mirrors, which also play a role in locating + source files, with the "Source Mirror(s)" box. + + + + The method by which source files are ultimately organized is + a function of the project. + For example, for released software, projects tend to use tarballs + or other archived files that can capture the state of a release + guaranteeing that it is statically represented. + On the other hand, for a project that is more dynamic or + experimental in nature, a project might keep source files in a + repository controlled by a Source Control Manager (SCM) such as + Git. + Pulling source from a repository allows you to control + the point in the repository (the revision) from which you want to + build software. + Finally, a combination of the two might exist, which would give the + consumer a choice when deciding where to get source files. + + + + BitBake uses the + SRC_URI + variable to point to source files regardless of their location. + Each recipe must have a SRC_URI variable + that points to the source. + + + + Another area that plays a significant role in where source files + comes from is pointed to by the + DL_DIR + variable. + This area is a cache that can hold previously downloaded source. + Judicious use of a DL_DIR directory can + save the build system a trip across the Internet when looking + for files. + A good method for using a download directory is to have + DL_DIR point to an area outside of your + Build Directory. + Doing so allows you to safely delete the Build Directory + if needed without fear of removing any downloaded source file. + + + + The remainder of this section provides a deeper look into the + source files and the mirrors. + Here is a more detailed look at the source file area of the + base figure: + + + +
+ Upstream Project Releases + + + Upstream project releases exist anywhere in the form of an + archived file (e.g. tarball or zip file). + These files correspond to individual recipes. + For example, the figure uses specific releases each for + BusyBox, Qt, and Dbus. + An archive file can be for any released product that can be + built using a recipe. + +
+ +
+ Local Projects + + + Local projects are custom bits of software the user provides. + These bits reside somewhere local to a project - perhaps + a directory into which the user checks in items (e.g. + a local directory containing a development source tree + used by the group). + + + + The canonical method through which to include a local project + is to use the + externalsrc.bbclass + class to include local project. + You use either the local.conf or a + recipe's append file to override or set the + recipe to point to the local directory on your disk to pull + in the whole source tree. + + + + For information on how to use the + externalsrc.bbclass, see the + "externalsrc.bbclass" + section. + +
+ +
+ Source Control Managers (Optional) + + + Another place the build system can get source files from is + through an SCM such as Git or Subversion. + In this case, a repository is cloned or checked out. + The do_fetch task inside BitBake uses + the SRC_URI + variable and the argument's prefix to determine the correct + fetcher module. + + + + When fetching a repository, BitBake uses the + SRCREV + variable to determine the specific revision from which to + build. + +
+ +
+ Source Mirror(s) + + + Two kinds of mirrors exist: pre-mirrors and regular mirrors. + The PREMIRRORS + and + MIRRORS + variables point to these, respectively. + BitBake checks pre-mirrors before looking upstream for any + source files. + Pre-mirrors are appropriate when you have a shared directory + that is not a directory defined by the + DL_DIR + variable. + A Pre-mirror typically points to a shared directory that is + local to your organization. + + + + Regular mirrors can be any site across the Internet that is + used as an alternative location for source code should the + primary site not be functioning for some reason or another. + +
+
+ +
+ BitBake + + + The OpenEmbedded build system uses BitBake to produce images. + You can see from the + general Yocto Project Development Environment figure, + the BitBake area consists of several functional areas. + This section takes a closer look at each of those areas. + + +
+ Source Fetching + + + The first stages of building a recipe are to fetch and unpack + the source code: + + + + + The do_fetch and + do_unpack tasks fetch the source files + and unpack them into a working directory. + By default, everything is accomplished in the + Build Directory, + which has a defined structure. + For additional general information on the Build Directory, + see the + "build/" + section. + + + + Unpacked source source files are pointed to by the + S variable. + Each recipe has an area in the Build Directory where the + unpacked source code resides. + The name of directory for any given recipe is defined from + several different variables. + You can see the variables that define these directories + by looking at the figure: + + TMPDIR + + PACKAGE_ARCH + + TARGET_OS + + PN + + PV + + PR + + WORKDIR + + S + + + + + + Briefly, the S directory contains the + unpacked source files for a recipe. + The WORKDIR directory is where all the + building goes on for a given recipe. + +
+ +
+ Patching + + + Once source code is fetched and unpacked, BitBake locates + patch files and applies them to the source files: + + + + + The do_patch task processes recipes by + using the + SRC_URI + variable to locate applicable patch files, which by default + are *.patch or + *.diff files, or any file if + "apply=yes" is specified for the file in + SRC_URI. + + + + BitBake finds and applies multiple patches for a single recipe + in the order in which it finds the patches. + Patches are applied to the recipe's source files located in the + S directory. + + + + For more information on how the source directories are + created, see the + "Source Fetching" + section. + +
+ +
+ Configuration and Compilation + + + After source code is patched, BitBake executes tasks that + configure and compile the source code: + + + + + This step in the build process consists of three tasks: + + do_configure: + This task configures the source by enabling and + disabling any build-time and configuration options for + the software being built. + Configurations can come from the recipe itself as well + as from an inherited class. + Additionally, the software itself might configure itself + depending on the target for which it is being built. + + + The configurations handled by the + do_configure task are specific + to source code configuration for the source code + being built by the recipe. + + If you are using + autotools.bbclass, + you can add additional configuration options by using + the EXTRA_OECONF + variable. + For information on how this variable works within + that class, see the + meta/classes/autotools.bbclass. + + do_compile: + Once a configuration task has been satisfied, BitBake + compiles the source using the + do_compile task. + Compilation occurs in the directory pointed to by the + B + variable. + Realize that the B directory, by + default, is the same as the + S + directory. + do_install: + Once compilation is done, BitBake executes the + do_install task. + This task copies files from the B + directory and places them in a holding area pointed to + by the + D + variable. + + +
+ +
+ Package Splitting + + + After source code is configured and compiled, the + OpenEmbedded build system analyzes + the results and splits the output into packages: + + + + + The do_package and + do_packagedata tasks combine to analyze + the files found in the + D directory + and split them into subsets based on available packages and + files. + The analyzing process involves the following as well as other + items: splitting out debugging symbols, + looking at shared library dependencies between packages, + and looking at package relationships. + The do_packagedata task creates package + metadata based on the analysis such that the + OpenEmbedded build system can generate the final packages. + Working, staged, and intermediate results of the analysis + and package splitting process use these areas: + + PKGD + + PKGDATA_DIR + + PKGDESTWORK + + PKGDEST + + + The FILES + variable defines the files that go into each package in + PACKAGES. + If you want details on how this is accomplished, you can + look at + package.bbclass. + + + + Depending on the type of packages being created (RPM, DEB, or + IPK), the do_package_write_* task + creates the actual packages and places them in the + Package Feed area, which is + ${TMPDIR}/deploy. + You can see the + "Package Feeds" + section for more detail on that part of the build process. + + Support for creating feeds directly from the + deploy/* directories does not exist. + Creating such feeds usually requires some kind of feed + maintenance mechanism that would upload the new packages + into an official package feed (e.g. the + Ångström distribution). + This functionality is highly distribution-specific + and thus is not provided out of the box. + + +
+
+ +
+ Package Feeds + + + When the OpenEmbedded build system generates an image or an SDK, + it gets the packages from a package feed area located in the + Build Directory. + The + general Yocto Project Development Environment figure + shows this package feeds area in the upper-right corner. + + + + This section looks a little closer into the package feeds area used + by the build system. + Here is a more detailed look at the area: + + + + + Package feeds are an intermediary step in the build process. + BitBake generates packages whose type is defined by the + PACKAGE_CLASSES + variable. + Before placing the packages into package feeds, + the build process validates them with generated output quality + assurance checks through the + insane.bbclass + class. + + + + The package feed area resides in + tmp/deploy of the Build Directory. + Folders are created that correspond to the package type + (IPK, DEB, or RPM) created. + Further organization is derived through the value of the + PACKAGE_ARCH + variable for each package. + For example, packages can exist for the i586 or qemux86 + architectures. + The package files themselves reside within the appropriate + architecture folder. + + + + BitBake uses the do_package_write_* task to + place generated packages into the package holding area (e.g. + do_package_write_ipk for IPK packages). + +
+ +
+ Images + + + The images produced by the OpenEmbedded build system + are compressed forms of the + root filesystems that are ready to boot on a target device. + You can see from the + general Yocto Project Development Environment figure + that BitBake output in part consists of images. + This section is going to look more closely at this output: + + + + + For a list of example images that the Yocto Project provides, + the + "Images" chapter. + + + + Images are written out to the + Build Directory + inside the deploy/images folder as shown + in the figure. + This folder contains any files expected to be loaded on the + target device. + The + DEPLOY_DIR + variable points to the deploy directory. + + <kernel-image>: + A kernel binary file. + The KERNEL_IMAGETYPE + variable setting determines the naming scheme for the + kernel image file. + Depending on that variable, the file could begin with + a variety of naming strings. + The deploy/images directory can + contain multiple image files. + <root-filesystem-image>: + Root filesystems for the target device (e.g. + *.ext3 or *.bz2 + files). + The IMAGE_FSTYPES + variable setting determines the root filesystem image + type. + The deploy/images directory can + contain multiple root filesystems. + <kernel-modules>: + Tarballs that contain all the modules built for the kernel. + Kernel module tarballs exist for legacy purposes and + can be suppressed by setting the + MODULE_TARBALL_DEPLOY + variable to "0". + The deploy/images directory can + contain multiple kernel module tarballs. + + <bootloaders>: + Bootloaders supporting the image, if applicable to the + target machine. + The deploy/images directory can + contain multiple bootloaders. + + <symlinks>: + The deploy/images folder contains + a symbolic link that points to the most recently built file + for each machine. + These links might be useful for external scripts that + need to obtain the latest version of each file. + + + +
+ +
+ Application Development SDK + + + In the + general Yocto Project Development Environment figure, + the output labeled "Application Development SDK" represents an + SDK. + This section is going to take a closer look at this output: + + + + + The specific form of this output is a self-extracting + SDK installer (*.sh) that, when run, + installs the SDK, which consists of a cross-development + toolchain, a set of libraries and headers, and an SDK + environment setup script. + Running this installer essentially sets up your + cross-development environment. + You can think of the cross-toolchain as the "host" + part because it runs on the SDK machine. + You can think of the libraries and headers as the "target" + part because they are built for the target hardware. + The setup script is added so that you can initialize the + environment before using the tools. + + + + + The Yocto Project supports several methods by which you can + set up this cross-development environment. + These methods include downloading pre-built SDK installers, + building and installing your own SDK installer, or running + an Application Development Toolkit (ADT) installer to + install not just cross-development toolchains + but also additional tools to help in this type of + development. + + + + For background information on cross-development toolchains + in the Yocto Project development environment, see the + "Cross-Development Toolchain Generation" + section. + For information on setting up a cross-development + environment, see the + "Installing the ADT and Toolchains" + section in the Yocto Project Application Developer's Guide. + + + + + Once built, the SDK installers are written out to the + deploy/sdk folder inside the + Build Directory + as shown in the figure at the beginning of this section. + Several variables exist that help configure these files: + + DEPLOY_DIR: + Points to the deploy + directory. + SDKMACHINE: + Specifies the architecture of the machine + on which the cross-development tools are run to + create packages for the target hardware. + + SDKIMAGE_FEATURES: + Lists the features to include in the "target" part + of the SDK. + + TOOLCHAIN_HOST_TASK: + Lists packages that make up the host + part of the SDK (i.e. the part that runs on + the SDKMACHINE). + When you use + bitbake -c populate_sdk <imagename> + to create the SDK, a set of default packages + apply. + This variable allows you to add more packages. + + TOOLCHAIN_TARGET_TASK: + Lists packages that make up the target part + of the SDK (i.e. the part built for the + target hardware). + + + +
+ + + -- cgit v1.2.3-54-g00ecf