From 38010236158712295e19be068d79db7e9df46517 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Tue, 18 Feb 2014 14:46:13 -0600 Subject: bitbake: user-manual-execution.xml: Review edits applied. Significant review changes implemented from Richard Purdie. * Eliminated the lifted text from the "Closer Look" chapter of the YP ref-manual. * Added the "BitBake" chapter in from the YP ref-manual. * Significant re-writes and suggestions throughout the entire chapter. (Bitbake rev: 4a740385a4720e040af3d329d9c7cc77b03e7f46) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- bitbake/doc/user-manual/user-manual-execution.xml | 726 +++++++++++----------- 1 file changed, 364 insertions(+), 362 deletions(-) (limited to 'bitbake/doc') diff --git a/bitbake/doc/user-manual/user-manual-execution.xml b/bitbake/doc/user-manual/user-manual-execution.xml index aa345de698..e9f19be6de 100644 --- a/bitbake/doc/user-manual/user-manual-execution.xml +++ b/bitbake/doc/user-manual/user-manual-execution.xml @@ -5,8 +5,9 @@ Execution - The primary purpose for running BitBake is to produce an - image, which can be a kernel or a software development kit (SDK). + The primary purpose for running BitBake is to produce some kind + of output such as an image, a kernel, or a software development + kit. Of course, you can execute the bitbake command with options that cause it to execute single tasks, compile single recipe files, capture or clear data, or simply @@ -38,24 +39,56 @@ layers BitBake needs to recognize, all necessary layer.conf files (one from each layer), and bitbake.conf. + The data itself is of various types: + + Recipes: + Details about particular pieces of software. + + Class Data: + An abstraction of common build information + (e.g. how to build a Linux kernel). + + Configuration Data: + Machine-specific settings, policy decisions, + and so forth. + Configuration data acts as the glue to bind everything + together. + - The bitbake.conf file resides in the - conf directory, which must be listed in - BBPATH. - This configuratoin file lists and includes other configuration - files from the conf directory below the - directories listed in BBPATH. - In general, the most important of these included - configuration files from a user's perspective - is local.conf, which contains the user's - customized settings for the build environment. - - - - Other notable configuration files are the distribution configuration - file and the machine configuration file. + The layer.conf files are used to + construct key variables such as + BBPATH + and + BBFILES. + BBPATH is used to search for + configuration and class files under + conf/ and class/ + directories, respectively. + BBFILES is used to find recipe files + (.bb and .bbappend). + If there is no bblayers.conf file, + it is assumed the user has set the BBPATH + and BBFILES directly in the environment. + + + + Next, the bitbake.conf file is searched + using the BBPATH variable that was + just constructed. + The bitbake.conf file usually indicates + all the other key include files to parse. + The usual convention is to have machine, distro, site, and local + configurations. + This means a user provides their own customizations + through a local.conf file. + + + + As mentioned in the previous paragraph, two of the other notable + configuration files are the distro and machine configuration + files. These configuration files are normally identified by variables unique to the build systems using BitBake. For example, the Yocto Project uses the @@ -87,6 +120,9 @@ This file is expected to contain a BBLAYERS variable that is a space delimited list of 'layer' directories. + Recall that if BitBake cannot find a bblayers.conf + file then it is assumed the user has set the BBPATH + and BBFILES directly in the environment. @@ -194,460 +230,426 @@ -
- Locating Recipes - - - The BBFILES variable is how BitBake - locates files. - This variable is a space-separated list of files - that are available, supports wildcards, and is set shortly - after the parsing phase of BitBake's execution. - +
+ Locating and Parsing Recipes + During the configuration phase, BitBake will have + set + BBFILES. + BitBake now uses it to construct a list of recipes to parse, + along with any append files (.bbappend) + to apply. + BBFILES is a space-separated list of + available files and supports wildcards. + An example would be: BBFILES = "/path/to/bbfiles/*.bb" - With regard to dependencies, it expects the - .bb to define a - DEPENDS variable, which contains a - space separated list of “package names”, which themselves - are the PN variable. The - PN variable is, in general, - set to a component of the .bb - filename by default. + BitBake parses each recipe and append file located + with BBFILES and stores the values of + various variables into the datastore. + For each file, a fresh copy of the base configuration is + made, then the recipe is parsed line by line. + Any inherit statements cause BitBake to find and + then parse class files (.bbclass) + using + BBPATH + as the search path. + Finally, BitBake parses in order any append files found in + BBFILES. -
- -
- Parsing Recipes - After classes are included, the variable - BBFILES is set, usually in - local.conf, and defines the list of - places to search for recipe and append files. - Adding extra content to BBFILES is best - achieved through the use of BitBake layers. + One common convention is to use the recipe filename to define + pieces of metadata. + For example, in bitbake.conf the recipe + name and version set + PN and + PV: + + PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}" + PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or 'defaultpkgname'}" + + In this example, a recipe called "something_1.2.3.bb" sets + PN to "something" and + PV to "1.2.3". - BitBake parses each recipe and append file located with - BBFILES and stores the values of various - variables into the datastore. - In summary, for each recipe and append file pairing, the configuration - plus the base class of variables are set, followed by the data in the - recipe file itself, followed by any inherit commands - that the recipe file might contain. + By the time parsing is complete for a recipe, BitBake + has a list of tasks that the recipe defines and a set of + data consisting of keys and values. - Part of parsing a recipe is making sure that all the recipes - that the recipe being parsed depends on are understood. - These other recipes could be located in other layers or in - a specific layer version. - These two variables in a recipe can help with with these - cases: - - - LAYERDEPENDS - - - LAYERVERSION - - + BitBake does not need all this information. + It only needs a small subset of the information to make + decisions about the recipe. + Consequently, BitBake caches the values in which it is + interested. - Because parsing recipe and append files is a time consuming - process, a cache, referred to as the "setscene" - is kept to speed up subsequent parsing. - The setscene is invalid if the timestamps of a recipe changes, - any of the include files change, configuration files change, - or class files on which the recipe file depends change. + Subsequent BitBake commands then parse the base + configuration and compute a checksum of that data. + If that checksum matches what is in the cache, the + recipe and class files have not changed. + In this case, BitBake reloads the cached information + about the recipe instead of reparsing it from scratch.
-
- Executing Tasks +
+ Preferences and Providers - Tasks can either be a shell task or a Python task. - For shell tasks, BitBake writes a shell script to - ${WORKDIR}/temp/run.do_taskname.pid - and then executes the script. - The generated shell script contains all the exported variables, - and the shell functions with all variables expanded. - Output from the shell script goes to the file - ${WORKDIR}/temp/log.do_taskname.pid. - Looking at the expanded shell functions in the run file and - the output in the log files is a useful debugging technique. + Assuming BitBake has been instructed to execute a target and + that all the recipe files have been parsed, BitBake starts to + build the target and look for providers of that target. + Once a provider is selected, BitBake resolves all the dependencies for + the target. + As an example, suppose the target is + core-image-sato. + In this case, it would lead to + packagegroup-core-x11-sato, + which in turn leads to recipes like matchbox-terminal, + pcmanfm and gthumb. + These recipes in turn depend on eglibc and the toolchain. - For Python tasks, BitBake executes the task internally and logs - information to the controlling terminal. - Future versions of BitBake will write the functions to files - similar to the way shell tasks are handled. - Logging will be handled in a way similar to shell tasks as well. + Sometimes a target might have multiple providers. + A common example is "virtual/kernel", which is provided by each kernel package. + Each machine often selects the best kernel provider by using a line similar to the + following in the machine configuration file: + + PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" + + - Variables specific to scheduling functionality exist: - - - BB_SCHEDULER - - - BB_SCHEDULERS - - + The default + PREFERRED_PROVIDER + is the provider with the same name as the target. -
-
- Source Fetching + + Understanding how providers are chosen is made complicated by the fact + that multiple versions might exist. + BitBake defaults to the highest version of a provider. + Version comparisons are made using the same method as Debian. + You can use the + PREFERRED_VERSION + variable to specify a particular version (usually in the distro configuration). + You can influence the order by using the + DEFAULT_PREFERENCE + variable. + By default, files have a preference of "0". + Setting the DEFAULT_PREFERENCE to "-1" makes the + package unlikely to be used unless it is explicitly referenced. + Setting the DEFAULT_PREFERENCE to "1" makes it likely the package is used. + PREFERRED_VERSION overrides any DEFAULT_PREFERENCE setting. + DEFAULT_PREFERENCE is often used to mark newer and more experimental package + versions until they have undergone sufficient testing to be considered stable. + - The first stages of building a recipe are to fetch and unpack - the source code: - + In summary, BitBake has created a list of providers, which is prioritized, for each target. +
+ +
+ Dependencies - The do_fetch and - do_unpack tasks fetch the source files - and unpack them into the work directory. - By default, everything is accomplished in the - build directory, - which has a defined structure. + Each target BitBake builds consists of multiple tasks such as + fetch, unpack, + patch, configure, + and compile. + For best performance on multi-core systems, BitBake considers each + task as an independent + entity with its own set of dependencies. - Unpacked source files are pointed to by a variable. - For example, in the Yocto Project and OpenEmbedded build systems, - the S variable points to these source files. - Each recipe has an area in the Build Directory where the - unpacked source code resides. - The name of that directory for any given recipe is defined from - several different variables. - You can see the variables that define these directories - by looking at the figure that shows the structure and variables - used in the Yocto Project: - - TMPDIR - - PACKAGE_ARCH - - TARGET_OS - - PN - - PV - - PR - - WORKDIR - - S - - + Dependencies are defined through several variables. + You can find information about variables BitBake uses in + the Variables Glossary + near the end of this manual. + At a basic level, it is sufficient to know that BitBake uses the + DEPENDS and + RDEPENDS variables when + calculating dependencies. - Briefly, the S directory contains the - unpacked source files for a recipe. - The WORKDIR directory is where all the - building goes on for a given recipe. + For more information on how BitBake handles dependencies, see the + "Dependencies" section.
-
- Patching +
+ The Task List - Once source code is fetched and unpacked, BitBake locates - patch files and applies them to the source files: - + Based on the generated list of providers and the dependency information, + BitBake can now calculate exactly what tasks it needs to run and in what + order it needs to run them. + The build now starts with BitBake forking off threads up to the limit set in the + BB_NUMBER_THREADS + variable. + BitBake continues to fork threads as long as there are tasks ready to run, + those tasks have all their dependencies met, and the thread threshold has not been + exceeded. - The do_patch task processes recipes by - using the - SRC_URI - variable to locate applicable patch files, which by default - are *.patch or - *.diff files, or any file if - "apply=yes" is specified for the file in - SRC_URI. + It is worth noting that you can greatly speed up the build time by properly setting + the BB_NUMBER_THREADS variable. - BitBake finds and applies multiple patches for a single recipe - in the order in which it finds the patches. - Patches are applied to the recipe's source files located in the - S directory. + As each task completes, a timestamp is written to the directory specified by the + STAMP variable. + On subsequent runs, BitBake looks in the build directory within + tmp/stampsand does not rerun + tasks that are already completed unless a timestamp is found to be invalid. + Currently, invalid timestamps are only considered on a per + recipe file basis. + So, for example, if the configure stamp has a timestamp greater than the + compile timestamp for a given target, then the compile task would rerun. + Running the compile task again, however, has no effect on other providers + that depend on that target. + This behavior could change or become configurable in future versions of BitBake. + + Some tasks are marked as "nostamp" tasks. + No timestamp file is created when these tasks are run. + Consequently, "nostamp" tasks are always rerun. + + - For more information on how the source directories are - created, see the - "Source Fetching" - section. + For more information on tasks, see the + "Tasks" section.
-
- Configuration and Compilation - - - After source code is patched, BitBake executes tasks that - configure and compile the source code: - - +
+ Executing Tasks - This step in the build process consists of three tasks: - - do_configure: - This task configures the source by enabling and - disabling any build-time and configuration options for - the software being built. - Configurations can come from the recipe itself as well - as from an inherited class. - Additionally, the software itself might configure itself - depending on the target for which it is being built. - - - The configurations handled by the - do_configure task are specific - to source code configuration for the source code - being built by the recipe. - - If you are using the Autotools class - (autotools.bbclass), - you can add additional configuration options by using - the EXTRA_OECONF - variable. - For information on how this variable works within - that class, see the - meta/classes/autotools.bbclass file. - - do_compile: - Once a configuration task has been satisfied, BitBake - compiles the source using the - do_compile task. - Compilation occurs in the directory pointed to by the - B - variable. - Realize that the B directory is, by - default, the same as the - S - directory. - do_install: - Once compilation is done, BitBake executes the - do_install task. - This task copies files from the B - directory and places them in a holding area pointed to - by the D variable. - + Tasks can either be a shell task or a Python task. + For shell tasks, BitBake writes a shell script to + ${WORKDIR}/temp/run.do_taskname.pid + and then executes the script. + The generated shell script contains all the exported variables, + and the shell functions with all variables expanded. + Output from the shell script goes to the file + ${WORKDIR}/temp/log.do_taskname.pid. + Looking at the expanded shell functions in the run file and + the output in the log files is a useful debugging technique. -
- -
- Package Splitting - After source code is configured and compiled, the - OpenEmbedded build system analyzes - the results and splits the output into packages: - + For Python tasks, BitBake executes the task internally and logs + information to the controlling terminal. + Future versions of BitBake will write the functions to files + similar to the way shell tasks are handled. + Logging will be handled in a way similar to shell tasks as well. - The do_package and - do_packagedata tasks combine to analyze - the files found in the D directory - and split them into subsets based on available packages and - files. - The analyzing process involves the following as well as other - items: splitting out debugging symbols, - looking at shared library dependencies between packages, - and looking at package relationships. - The do_packagedata task creates package - metadata based on the analysis such that the - OpenEmbedded build system can generate the final packages. - Working, staged, and intermediate results of the analysis - and package splitting process use these areas: + Variables specific to scheduling functionality exist: - PKGD - - PKGDATA_DIR - - PKGDESTWORK + + BB_SCHEDULER - PKGDEST + + BB_SCHEDULERS - The FILES - variable defines the files that go into each package in - PACKAGES. - If you want details on how this is accomplished in the Yocto Project - for example, you can look at the package.bbclass - file in a Yocto tree. - - - - Depending on the type of packages being created (RPM, DEB, or - IPK), the do_package_write_* task - creates the actual packages and places them in the - Package Feed area, which is - ${TMPDIR}/deploy. - - Support for creating feeds directly from the - deploy/* directories does not exist. - Creating such feeds usually requires some kind of feed - maintenance mechanism that would upload the new packages - into an official package feed (e.g. the - Ångström distribution). - This functionality is highly distribution-specific - and thus is not provided out of the box. -
-
- Image Generation +
+ Setscene - Once packages are split and stored in the Package Feeds area, - the OpenEmbedded build system uses BitBake to generate the - root filesystem image: - + This section needs to get the concept of the setscene across. + The reader needs to know what it is and what it is used for during + the build process. - The image generation process consists of several stages and - depends on many variables. - The do_rootfs task uses these key variables - to help create the list of packages to actually install: - - IMAGE_INSTALL: - Lists out the base set of packages to install from - the Package Feeds area. - PACKAGE_EXCLUDE: - Specifies packages that should not be installed. - - IMAGE_FEATURES: - Specifies features to include in the image. - Most of these features map to additional packages for - installation. - PACKAGE_CLASSES: - Specifies the package backend to use and consequently - helps determine where to locate packages within the - Package Feeds area. - IMAGE_LINGUAS: - Determines the language(s) for which additional - language support packages are installed. - - + You can find more information on setscene metadata in the + "Task Checksums and Setscene" + section. +
+ +
+ Checksums (Signatures) - Package installation is under control of the package manager - (e.g. smart/rpm, opkg, or apt/dpkg) regardless of whether or - not package management is enabled for the target. - At the end of the process, if package management is not - enabled for the target, the package manager's data files - are deleted from the root filesystem. + A checksum is a unique signature of a task's inputs. + The setscene code uses a checksum to determine if a task needs + to be run. + Because it is a change in a task's inputs that triggers running + the task, the process needs to detect all the inputs to a given task. + For shell tasks, this turns out to be fairly easy because + BitBake generates a "run" shell script for each task and + it is possible to create a checksum that gives you a good idea of when + the task's data changes. - During image generation, the build system attempts to run - all post-installation scripts. - Any that fail to run on the build host are run on the - target when the target system is first booted. - If you are using a - read-only root filesystem, - all the post installation scripts must succeed during the - package installation phase since the root filesystem cannot be - written into. + To complicate the problem, some things should not be included in + the checksum. + First, there is the actual specific build path of a given task - + the working directory. + It does not matter if the working directory changes because it should not + affect the output for target packages. + The simplistic approach for excluding the working directory is to set + it to some fixed value and create the checksum for the "run" script. - During Optimization, optimizing processes are run across - the image. - These processes include mklibs and - prelink. - The mklibs process optimizes the size - of the libraries. - A prelink process optimizes the dynamic - linking of shared libraries to reduce start up time of - executables. + Another problem results from the "run" scripts containing functions that + might or might not get called. + The incremental build solution contains code that figures out dependencies + between shell functions. + This code is used to prune the "run" scripts down to the minimum set, + thereby alleviating this problem and making the "run" scripts much more + readable as a bonus. - Part of the image generation process includes compressing the - root filesystem image. - Compression is accomplished through several optimization - routines designed to reduce the overall size of the image. + So far we have solutions for shell scripts. + What about Python tasks? + The same approach applies even though these tasks are more difficult. + The process needs to figure out what variables a Python function accesses + and what functions it calls. + Again, the incremental build solution contains code that first figures out + the variable and function dependencies, and then creates a checksum for the data + used as the input to the task. - After the root filesystem has been constructed, the image - generation process turns everything into an image file or - a set of image files. - The formats used for the root filesystem depend on the - IMAGE_FSTYPES variable. + Like the working directory case, situations exist where dependencies + should be ignored. + For these cases, you can instruct the build process to ignore a dependency + by using a line like the following: + + PACKAGE_ARCHS[vardepsexclude] = "MACHINE" + + This example ensures that the PACKAGE_ARCHS variable does not + depend on the value of MACHINE, even if it does reference it. - - The entire image generation process is run under Pseudo. - Running under Pseudo ensures that the files in the root - filesystem have correct ownership. - -
- -
- SDK Generation + + Equally, there are cases where we need to add dependencies BitBake + is not able to find. + You can accomplish this by using a line like the following: + + PACKAGE_ARCHS[vardeps] = "MACHINE" + + This example explicitly adds the MACHINE variable as a + dependency for PACKAGE_ARCHS. + - The OpenEmbedded build system uses BitBake to generate the - Software Development Kit (SDK) installer script: - + Consider a case with in-line Python, for example, where BitBake is not + able to figure out dependencies. + When running in debug mode (i.e. using -DDD), BitBake + produces output when it discovers something for which it cannot figure out + dependencies. - Like image generation, the SDK script process consists of - several stages and depends on many variables. - The do_populate_sdk task uses these - key variables to help create the list of packages to actually - install. + Thus far, this section has limited discussion to the direct inputs into a task. + Information based on direct inputs is referred to as the "basehash" in the + code. + However, there is still the question of a task's indirect inputs - the + things that were already built and present in the build directory. + The checksum (or signature) for a particular task needs to add the hashes + of all the tasks on which the particular task depends. + Choosing which dependencies to add is a policy decision. + However, the effect is to generate a master checksum that combines the basehash + and the hashes of the task's dependencies. - The do_populate_sdk task handles two - parts: a target part and a host part. - The target part is the part built for the target hardware and - includes libraries and headers. - The host part is the part of the SDK that runs on the - SDKMACHINE. + At the code level, there are a variety of ways both the basehash and the + dependent task hashes can be influenced. + Within the BitBake configuration file, we can give BitBake some extra information + to help it construct the basehash. + The following statement effectively results in a list of global variable + dependency excludes - variables never included in any checksum. + This example uses variables from OpenEmbedded to help illustrate + the concept: + + BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ + SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ + USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ + PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ + CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" + + The previous example excludes the work directory, which is part of + TMPDIR. + + + + The rules for deciding which hashes of dependent tasks to include through + dependency chains are more complex and are generally accomplished with a + Python function. + The code in meta/lib/oe/sstatesig.py shows two examples + of this and also illustrates how you can insert your own policy into the system + if so desired. + This file defines the two basic signature generators OpenEmbedded Core + uses: "OEBasic" and "OEBasicHash". + By default, there is a dummy "noop" signature handler enabled in BitBake. + This means that behavior is unchanged from previous versions. + OE-Core uses the "OEBasicHash" signature handler by default + through this setting in the bitbake.conf file: + + BB_SIGNATURE_HANDLER ?= "OEBasicHash" + + The "OEBasicHash" BB_SIGNATURE_HANDLER is the same as the + "OEBasic" version but adds the task hash to the stamp files. + This results in any metadata change that changes the task hash, automatically + causing the task to be run again. + This removes the need to bump + PR + values, and changes to metadata automatically ripple across the build. - Once both parts are constructed, the - do_populate_sdk task performs some cleanup - on both parts. - After the cleanup, the task creates a cross-development - environment setup script and any configuration files that - might be needed. + It is also worth noting that the end result of these signature generators is to + make some dependency and hash information available to the build. + This information includes: + + BB_BASEHASH_task-<taskname>: + The base hashes for each task in the recipe. + + BB_BASEHASH_<filename:taskname>: + The base hashes for each dependent task. + + BBHASHDEPS_<filename:taskname>: + The task dependencies for each task. + + BB_TASKHASH: + The hash of the currently running task. + + - The final output of the task is the Cross-development - toolchain installation script (.sh file), - which includes the environment setup script. + You can find more information on checksum metadata in the + "Task Checksums and Setscene" + section.
-- cgit v1.2.3-54-g00ecf