From b4b3bf56f60fab51f0c2c66ad34e39208d360d2e Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Tue, 11 Feb 2014 13:45:54 -0600 Subject: bitbake: user-manual: Added new chapter to cover BB execution. This commit added need files, cleaned up bogus links, and adjusted the build file, which is user-manual.xml. The new chapter file is "user-manual-execution.xml". The file has base text from the bitbake section from the "Closer Look" chapter of the YP ref-manual. This means that I had to initially include some figures from that book for now. I am not including them as part of this commit though as they will likely get renamed. I made sure that all links were removed that were specific to the YP ref-manual. Another change was in the user-manual.xml file to include the new chapter as part of the build. Another block of text for the new chapter was the "Running a Task" section from the user-manual-metadata.xml file. Consequently, that section was deleted from the "Metafile" chapter. Another block of text for the new chapter was the "Parsing and Execution" section from the user-manual-metadata.xml file. Consequently, that section was deleted from the "Metafile chapter. (Bitbake rev: f3b21d1fb711f9625d2ac92d4f4fe0f269242bd7) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- bitbake/doc/user-manual/user-manual-execution.xml | 690 ++++++++++++++++++++++ bitbake/doc/user-manual/user-manual-metadata.xml | 333 ----------- bitbake/doc/user-manual/user-manual.xml | 2 + 3 files changed, 692 insertions(+), 333 deletions(-) create mode 100644 bitbake/doc/user-manual/user-manual-execution.xml (limited to 'bitbake') diff --git a/bitbake/doc/user-manual/user-manual-execution.xml b/bitbake/doc/user-manual/user-manual-execution.xml new file mode 100644 index 0000000000..365392dbff --- /dev/null +++ b/bitbake/doc/user-manual/user-manual-execution.xml @@ -0,0 +1,690 @@ + + + + Execution + + + Tasks can either be a shell task or a Python task. + For shell tasks, BitBake writes a shell script to + ${WORKDIR}/temp/run.do_taskname.pid + and then executes the script. + The generated shell script contains all the exported variables, + and the shell functions with all variables expanded. + Output from the shell script goes to the file + ${WORKDIR}/temp/log.do_taskname.pid. + Looking at the expanded shell functions in the run file and + the output in the log files is a useful debugging technique. + + + + For Python tasks, BitBake executes the task internally and logs + information to the controlling terminal. + Future versions of BitBake will write the functions to files + similar to the way shell tasks are handled. + Logging will be handled in a way similar to shell tasks as well. + + + + Once all the tasks have been completed BitBake exits. + + +
+ Parsing and Execution + +
+ Parsing Overview + + + BitBake parses configuration files, classes, recipes, and append + files. + + + + The first thing BitBake does is look for the + bitbake.conf file. + This file resides in the conf + directory, which must be listed in + BBPATH. + + + + The bitbake.conf file lists other configuration + files to include from the conf directory below the + directories listed in BBPATH. + In general, the most important of these + configuration files from a user's perspective + is local.conf, which contains the user's + customized settings for the build environment. + + + + Other notable configuration files are the distribution configuration + file and the machine configuration file. + These configuration files are normally identified by + variables unique to the build systems using BitBake. + For example, the Yocto Project uses the + DISTRO and MACHINE + variables, respectively. + + + + After parsing of the configuration files, some standard classes are + included. + The base.bbclass file + is always included. + Other classes that are specified in the configuration using the + INHERIT + variable are also included. + BitBake searches for class files in a "classes" subdirectory under + the paths in BBPATH in the same way as + configuration files. + + + + After classes are included, the variable + BBFILES is set, usually in + local.conf, and defines the list of + places to search for recipe and append files. + Adding extra content to BBFILES is best + achieved through the use of BitBake layers. + + + + BitBake parses each recipe and append file located with + BBFILES and stores the values of various + variables into the datastore. + In summary, for each recipe and append file pairing, the configuration + plus the base class of variables are set, followed by the data in the + recipe file itself, followed by any inherit commands + that the recipe file might contain. + + + + Because parsing recipe and append files is a time consuming + process, a cache, referred to as the "setscene" + is kept to speed up subsequent parsing. + The setscene is invalid if the timestamps of a recipe changes, + any of the include files change, configuration files change, + or class files on which the recipe file depends change. + +
+ +
+ Configuration files + + + Prior to parsing configuration files, Bitbake looks + at certain variables, including: + + BB_ENV_WHITELIST + BB_PRESERVE_ENV + BB_ENV_EXTRAWHITE + BB_ORIGENV + PREFERRED_VERSION + PREFERRED_PROVIDERS + + + + + The first kind of metadata in BitBake is configuration metadata. + This metadata is global, and therefore affects all packages and + tasks that are executed. + + + + BitBake will first search the current working directory for an + optional conf/bblayers.conf configuration file. + This file is expected to contain a + BBLAYERS + variable that is a space delimited list of 'layer' directories. + For each directory in this list, a conf/layer.conf + file will be searched for and parsed with the + LAYERDIR + variable being set to the directory where the layer was found. + The idea is these files will setup + BBPATH + and other variables correctly for a given build directory automatically + for the user. + + + + BitBake will then expect to find conf/bitbake.conf + file somewhere in the user specified BBPATH. + That configuration file generally has include directives to pull + in any other metadata (generally files specific to architecture, + machine, local and so on). + + + + Only variable definitions and include directives are allowed + in .conf files. + The following variables include: + + + BITBAKE_UI + + + BBDEBUG + + + MULTI_PROVIDER_WHITELIST + + + BB_NUMBER_PARSE_THREADS + + + BBPKGS + + + BB_DEFAULT_TASK + + + TOPDIR + + + BB_VERBOSE_LOGS + + + BB_NICE_LEVEL + + + BBFILE_COLLECTIONS + + + ASSUME_PROVIDED + + + BB_DANGLINGAPPENDS_WARNONLY + + + BBINCLUDED + + + BBFILE_PRIORITY + + + BUILDNAME + + + BBMASK + + + + +
+ Layers + + + Layers allow you to isolate different types of + customizations from each other. + While you might find it tempting to keep everything in one layer + when working on a single project, the more modular you organize + your metadata, the easier it is to cope with future changes. + + + + To illustrate how you can use layers to keep things modular, + consider machine customizations. + These types of customizations typically reside in a special layer, + rather than a general layer, called a Board Specific Package (BSP) Layer. + Furthermore, the machine customizations should be isolated from + recipes and metadata that support a new GUI environment, for + example. + This situation gives you a couple of layers: one for the machine + configurations and one for the GUI environment. + It is important to understand, however, that the BSP layer can still + make machine-specific additions to recipes within + the GUI environment layer without polluting the GUI layer itself + with those machine-specific changes. + You can accomplish this through a recipe that is a BitBake append + (.bbappend) file. + + + + There are certain variables specific to layers: + + + LAYERDEPENDS + + + LAYERVERSION + + + +
+ +
+ Schedulers + + + Variables specific to scheduling functionality exist: + + + BB_SCHEDULER + + + BB_SCHEDULERS + + + +
+
+ +
+ Classes + + + BitBake's rudimentary inheritance mechanism is accomplished using + classes. + As briefly mentioned in the metadata introduction, BitBake + parses a class when an inherit directive is encountered, and it + is located in the classes directory + relative to the directories in + BBPATH. + +
+ +
+ Recipe (<filename>.bb</filename>) Files + + + Recipe files, which are files that have the + .bb file extension, are logical units of + tasks for execution. + Normally, that logical unit is a package that needs to be + built. + + + + BitBake obeys all inter-recipe dependencies. + + + + Recipe files must reside in locations found in the + BBFILES + variable. + +
+ +
+ Append (<filename>.bbappend</filename>) Files + + + Append files, which are files that have the + .bbappend file extension, add or + extend build information to an existing + recipe file. + + + + BitBake expects every append file to have a corresponding recipe file. + Furthermore, the append file and corresponding recipe file + must use the same root filename. + The filenames can differ only in the file type suffix used + (e.g. formfactor_0.0.bb and + formfactor_0.0.bbappend). + + + + Information in append files overrides the information in the + similarly-named recipe file. + +
+
+ +
+ BitBake + + + The OpenEmbedded build system uses BitBake to produce images. + BitBake consists of several functional areas. + This section takes a closer look at each of those areas. + + +
+ Source Fetching + + + The first stages of building a recipe are to fetch and unpack + the source code: + + +figures/ + + The do_fetch and + do_unpack tasks fetch the source files + and unpack them into the work directory. + By default, everything is accomplished in the + build directory, + which has a defined structure. + + + + Unpacked source files are pointed to by a variable. + For example, in the Yocto Project and OpenEmbedded build systems, + the S variable points to these source files. + Each recipe has an area in the Build Directory where the + unpacked source code resides. + The name of that directory for any given recipe is defined from + several different variables. + You can see the variables that define these directories + by looking at the figure that shows the structure and variables + used in the Yocto Project: + + TMPDIR + + PACKAGE_ARCH + + TARGET_OS + + PN + + PV + + PR + + WORKDIR + + S + + + + + + Briefly, the S directory contains the + unpacked source files for a recipe. + The WORKDIR directory is where all the + building goes on for a given recipe. + +
+ +
+ Patching + + + Once source code is fetched and unpacked, BitBake locates + patch files and applies them to the source files: + + + + + The do_patch task processes recipes by + using the + SRC_URI + variable to locate applicable patch files, which by default + are *.patch or + *.diff files, or any file if + "apply=yes" is specified for the file in + SRC_URI. + + + + BitBake finds and applies multiple patches for a single recipe + in the order in which it finds the patches. + Patches are applied to the recipe's source files located in the + S directory. + + + + For more information on how the source directories are + created, see the + "Source Fetching" + section. + +
+ +
+ Configuration and Compilation + + + After source code is patched, BitBake executes tasks that + configure and compile the source code: + + + + + This step in the build process consists of three tasks: + + do_configure: + This task configures the source by enabling and + disabling any build-time and configuration options for + the software being built. + Configurations can come from the recipe itself as well + as from an inherited class. + Additionally, the software itself might configure itself + depending on the target for which it is being built. + + + The configurations handled by the + do_configure task are specific + to source code configuration for the source code + being built by the recipe. + + If you are using the Autotools class + (autotools.bbclass), + you can add additional configuration options by using + the EXTRA_OECONF + variable. + For information on how this variable works within + that class, see the + meta/classes/autotools.bbclass file. + + do_compile: + Once a configuration task has been satisfied, BitBake + compiles the source using the + do_compile task. + Compilation occurs in the directory pointed to by the + B + variable. + Realize that the B directory is, by + default, the same as the + S + directory. + do_install: + Once compilation is done, BitBake executes the + do_install task. + This task copies files from the B + directory and places them in a holding area pointed to + by the D variable. + + +
+ +
+ Package Splitting + + + After source code is configured and compiled, the + OpenEmbedded build system analyzes + the results and splits the output into packages: + + + + + The do_package and + do_packagedata tasks combine to analyze + the files found in the D directory + and split them into subsets based on available packages and + files. + The analyzing process involves the following as well as other + items: splitting out debugging symbols, + looking at shared library dependencies between packages, + and looking at package relationships. + The do_packagedata task creates package + metadata based on the analysis such that the + OpenEmbedded build system can generate the final packages. + Working, staged, and intermediate results of the analysis + and package splitting process use these areas: + + PKGD + + PKGDATA_DIR + + PKGDESTWORK + + PKGDEST + + + The FILES + variable defines the files that go into each package in + PACKAGES. + If you want details on how this is accomplished in the Yocto Project + for example, you can look at the package.bbclass + file in a Yocto tree. + + + + Depending on the type of packages being created (RPM, DEB, or + IPK), the do_package_write_* task + creates the actual packages and places them in the + Package Feed area, which is + ${TMPDIR}/deploy. + + Support for creating feeds directly from the + deploy/* directories does not exist. + Creating such feeds usually requires some kind of feed + maintenance mechanism that would upload the new packages + into an official package feed (e.g. the + Ångström distribution). + This functionality is highly distribution-specific + and thus is not provided out of the box. + + +
+ +
+ Image Generation + + + Once packages are split and stored in the Package Feeds area, + the OpenEmbedded build system uses BitBake to generate the + root filesystem image: + + + + + The image generation process consists of several stages and + depends on many variables. + The do_rootfs task uses these key variables + to help create the list of packages to actually install: + + IMAGE_INSTALL: + Lists out the base set of packages to install from + the Package Feeds area. + PACKAGE_EXCLUDE: + Specifies packages that should not be installed. + + IMAGE_FEATURES: + Specifies features to include in the image. + Most of these features map to additional packages for + installation. + PACKAGE_CLASSES: + Specifies the package backend to use and consequently + helps determine where to locate packages within the + Package Feeds area. + IMAGE_LINGUAS: + Determines the language(s) for which additional + language support packages are installed. + + + + + + Package installation is under control of the package manager + (e.g. smart/rpm, opkg, or apt/dpkg) regardless of whether or + not package management is enabled for the target. + At the end of the process, if package management is not + enabled for the target, the package manager's data files + are deleted from the root filesystem. + + + + During image generation, the build system attempts to run + all post-installation scripts. + Any that fail to run on the build host are run on the + target when the target system is first booted. + If you are using a + read-only root filesystem, + all the post installation scripts must succeed during the + package installation phase since the root filesystem cannot be + written into. + + + + During Optimization, optimizing processes are run across + the image. + These processes include mklibs and + prelink. + The mklibs process optimizes the size + of the libraries. + A prelink process optimizes the dynamic + linking of shared libraries to reduce start up time of + executables. + + + + Part of the image generation process includes compressing the + root filesystem image. + Compression is accomplished through several optimization + routines designed to reduce the overall size of the image. + + + + After the root filesystem has been constructed, the image + generation process turns everything into an image file or + a set of image files. + The formats used for the root filesystem depend on the + IMAGE_FSTYPES variable. + + + + The entire image generation process is run under Pseudo. + Running under Pseudo ensures that the files in the root + filesystem have correct ownership. + +
+ +
+ SDK Generation + + + The OpenEmbedded build system uses BitBake to generate the + Software Development Kit (SDK) installer script: + + + + + Like image generation, the SDK script process consists of + several stages and depends on many variables. + The do_populate_sdk task uses these + key variables to help create the list of packages to actually + install. + + + + The do_populate_sdk task handles two + parts: a target part and a host part. + The target part is the part built for the target hardware and + includes libraries and headers. + The host part is the part of the SDK that runs on the + SDKMACHINE. + + + + Once both parts are constructed, the + do_populate_sdk task performs some cleanup + on both parts. + After the cleanup, the task creates a cross-development + environment setup script and any configuration files that + might be needed. + + + + The final output of the task is the Cross-development + toolchain installation script (.sh file), + which includes the environment setup script. + +
+
+ diff --git a/bitbake/doc/user-manual/user-manual-metadata.xml b/bitbake/doc/user-manual/user-manual-metadata.xml index 796d27b151..8d37f6bee1 100644 --- a/bitbake/doc/user-manual/user-manual-metadata.xml +++ b/bitbake/doc/user-manual/user-manual-metadata.xml @@ -895,35 +895,6 @@ -
- Running a Task - - - Tasks can either be a shell task or a Python task. - For shell tasks, BitBake writes a shell script to - ${WORKDIR}/temp/run.do_taskname.pid - and then executes the script. - The generated shell script contains all the exported variables, - and the shell functions with all variables expanded. - Output from the shell script goes to the file - ${WORKDIR}/temp/log.do_taskname.pid. - Looking at the expanded shell functions in the run file and - the output in the log files is a useful debugging technique. - - - - For Python tasks, BitBake executes the task internally and logs - information to the controlling terminal. - Future versions of BitBake will write the functions to files - similar to the way shell tasks are handled. - Logging will be handled in a way similar to shell tasks as well. - - - - Once all the tasks have been completed BitBake exits. - -
-
Variable Flags @@ -1033,310 +1004,6 @@
-
- Parsing and Execution - -
- Parsing Overview - - - BitBake parses configuration files, classes, recipes, and append - files. - - - - The first thing BitBake does is look for the - bitbake.conf file. - This file resides in the conf - directory, which must be listed in - BBPATH. - - - - The bitbake.conf file lists other configuration - files to include from the conf directory below the - directories listed in BBPATH. - In general, the most important of these - configuration files from a user's perspective - is local.conf, which contains the user's - customized settings for the build environment. - - - - Other notable configuration files are the distribution configuration - file and the machine configuration file. - These configuration files are normally identified by - variables unique to the build systems using BitBake. - For example, the Yocto Project uses the - DISTRO and MACHINE - variables, respectively. - - - - After parsing of the configuration files, some standard classes are - included. - The base.bbclass file - is always included. - Other classes that are specified in the configuration using the - INHERIT - variable are also included. - BitBake searches for class files in a "classes" subdirectory under - the paths in BBPATH in the same way as - configuration files. - - - - After classes are included, the variable - BBFILES is set, usually in - local.conf, and defines the list of - places to search for recipe and append files. - Adding extra content to BBFILES is best - achieved through the use of BitBake layers. - - - - BitBake parses each recipe and append file located with - BBFILES and stores the values of various - variables into the datastore. - In summary, for each recipe and append file pairing, the configuration - plus the base class of variables are set, followed by the data in the - recipe file itself, followed by any inherit commands - that the recipe file might contain. - - - - Because parsing recipe and append files is a time consuming - process, a cache, referred to as the "setscene" - is kept to speed up subsequent parsing. - The setscene is invalid if the timestamps of a recipe changes, - any of the include files change, configuration files change, - or class files on which the recipe file depends change. - -
- -
- Configuration files - - - Prior to parsing configuration files, Bitbake looks - at certain variables, including: - - BB_ENV_WHITELIST - BB_PRESERVE_ENV - BB_ENV_EXTRAWHITE - BB_ORIGENV - PREFERRED_VERSION - PREFERRED_PROVIDERS - - - - - The first kind of metadata in BitBake is configuration metadata. - This metadata is global, and therefore affects all packages and - tasks that are executed. - - - - BitBake will first search the current working directory for an - optional conf/bblayers.conf configuration file. - This file is expected to contain a - BBLAYERS - variable that is a space delimited list of 'layer' directories. - For each directory in this list, a conf/layer.conf - file will be searched for and parsed with the - LAYERDIR - variable being set to the directory where the layer was found. - The idea is these files will setup - BBPATH - and other variables correctly for a given build directory automatically - for the user. - - - - BitBake will then expect to find conf/bitbake.conf - file somewhere in the user specified BBPATH. - That configuration file generally has include directives to pull - in any other metadata (generally files specific to architecture, - machine, local and so on). - - - - Only variable definitions and include directives are allowed - in .conf files. - The following variables include: - - - BITBAKE_UI - - - BBDEBUG - - - MULTI_PROVIDER_WHITELIST - - - BB_NUMBER_PARSE_THREADS - - - BBPKGS - - - BB_DEFAULT_TASK - - - TOPDIR - - - BB_VERBOSE_LOGS - - - BB_NICE_LEVEL - - - BBFILE_COLLECTIONS - - - ASSUME_PROVIDED - - - BB_DANGLINGAPPENDS_WARNONLY - - - BBINCLUDED - - - BBFILE_PRIORITY - - - BUILDNAME - - - BBMASK - - - - -
- Layers - - - Layers allow you to isolate different types of - customizations from each other. - While you might find it tempting to keep everything in one layer - when working on a single project, the more modular you organize - your metadata, the easier it is to cope with future changes. - - - - To illustrate how you can use layers to keep things modular, - consider machine customizations. - These types of customizations typically reside in a special layer, - rather than a general layer, called a Board Specific Package (BSP) Layer. - Furthermore, the machine customizations should be isolated from - recipes and metadata that support a new GUI environment, for - example. - This situation gives you a couple of layers: one for the machine - configurations and one for the GUI environment. - It is important to understand, however, that the BSP layer can still - make machine-specific additions to recipes within - the GUI environment layer without polluting the GUI layer itself - with those machine-specific changes. - You can accomplish this through a recipe that is a BitBake append - (.bbappend) file. - - - - There are certain variables specific to layers: - - - LAYERDEPENDS - - - LAYERVERSION - - - -
- -
- Schedulers - - - Variables specific to scheduling functionality exist: - - - BB_SCHEDULER - - - BB_SCHEDULERS - - - -
-
- -
- Classes - - - BitBake's rudimentary inheritance mechanism is accomplished using - classes. - As briefly mentioned in the metadata introduction, BitBake - parses a class when an inherit directive is encountered, and it - is located in the classes directory - relative to the directories in - BBPATH. - -
- -
- Recipe (<filename>.bb</filename>) Files - - - Recipe files, which are files that have the - .bb file extension, are logical units of - tasks for execution. - Normally, that logical unit is a package that needs to be - built. - - - - BitBake obeys all inter-recipe dependencies. - - - - Recipe files must reside in locations found in the - BBFILES - variable. - -
- -
- Append (<filename>.bbappend</filename>) Files - - - Append files, which are files that have the - .bbappend file extension, add or - extend build information to an existing - recipe file. - - - - BitBake expects every append file to have a corresponding recipe file. - Furthermore, the append file and corresponding recipe file - must use the same root filename. - The filenames can differ only in the file type suffix used - (e.g. formfactor_0.0.bb and - formfactor_0.0.bbappend). - - - - Information in append files overrides the information in the - similarly-named recipe file. - -
-
-
Events diff --git a/bitbake/doc/user-manual/user-manual.xml b/bitbake/doc/user-manual/user-manual.xml index 1e8777a4b3..ba690ab243 100644 --- a/bitbake/doc/user-manual/user-manual.xml +++ b/bitbake/doc/user-manual/user-manual.xml @@ -75,6 +75,8 @@ + + -- cgit v1.2.3-54-g00ecf