path: root/bitbake/doc/user-manual/user-manual-execution.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<chapter id="user-manual-execution">
    <title>Execution</title>

    <para>
        The primary purpose for running BitBake is to produce an
        image, which can be a kernel or a software development kit (SDK).
        Of course, you can execute the <filename>bitbake</filename>
        command with options that cause it to execute single tasks,
        compile single recipe files, capture or clear data, or simply
        return information about the execution environment.
    </para>

    <para>
        This chapter describes BitBake's execution process from start
        to finish when you use it to create an image.
        The execution process is launched using the following command
        form:
        <literallayout class='monospaced'>
     $ bitbake &lt;target&gt;
        </literallayout>
        For information on the BitBake command and its options,
        see the
        "<link linkend='user-manual-command'>BitBake Command</link>
        chapter.
    </para>

    <section id='parsing-the-base-configuration-metadata'>
        <title>Parsing the Base Configuration Metadata</title>

        <para>
            The first thing BitBake does is parse base configuration
            metadata.
            Base configuration metadata consists of the
            <filename>bblayers.conf</filename> file to determine what
            layers BitBake needs to recognize, all necessary
            <filename>layer.conf</filename> files (one from each layer),
            and <filename>bitbake.conf</filename>.
        </para>

        <para>
            The <filename>bitbake.conf</filename> file resides in the
            <filename>conf</filename> directory, which must be listed in
            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>.
            This configuratoin file lists and includes other configuration
            files from the <filename>conf</filename> directory below the
            directories listed in <filename>BBPATH</filename>.
            In general, the most important of these included
            configuration files from a user's perspective
            is <filename>local.conf</filename>, which contains the user's
            customized settings for the build environment.
        </para>

        <para>
            Other notable configuration files are the distribution configuration
            file and the machine configuration file.
            These configuration files are normally identified by
            variables unique to the build systems using BitBake.
            For example, the Yocto Project uses the
            <filename>DISTRO</filename> and <filename>MACHINE</filename>
            variables, respectively.
        </para>

        <para>
            Prior to parsing configuration files, Bitbake looks
            at certain variables, including:
            <itemizedlist>
                <listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem>
                <listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem>
                <listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem>
                <listitem><para><link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link></para></listitem>
                <listitem><para><link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link></para></listitem>
                <listitem><para><link linkend='var-PREFERRED_PROVIDERS'><filename>PREFERRED_PROVIDERS</filename></link></para></listitem>
            </itemizedlist>
        </para>

        <para>
            The base configuration metadata is global
            and therefore affects all packages and tasks that are executed.
        </para>

        <para>
            BitBake first searches the current working directory for an
            optional <filename>conf/bblayers.conf</filename> configuration file.
            This file is expected to contain a
            <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
            variable that is a space delimited list of 'layer' directories.
        </para>

        <para>
            For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
            file is searched for and parsed with the
            <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
            variable being set to the directory where the layer was found.
            The idea is these files automatically setup
            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
            and other variables correctly for a given build directory.
        </para>

        <para>
            BitBake then expects to find the <filename>conf/bitbake.conf</filename>
            file somewhere in the user-specified <filename>BBPATH</filename>.
            That configuration file generally has include directives to pull
            in any other metadata such as files specific to the architecture,
            the machine, the local environment, and so forth.
        </para>

        <para>
            Only variable definitions and include directives are allowed
            in <filename>.conf</filename> files.
            The following variables include:
            <itemizedlist>
                <listitem><para>
                    <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBDEBUG'><filename>BBDEBUG</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-MULTI_PROVIDER_WHITELIST'><filename>MULTI_PROVIDER_WHITELIST</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></link>
                    </para></listitem>
                <listitem><para>
                    <filename>BBPKGS</filename>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_DEFAULT_TASK'><filename>BB_DEFAULT_TASK</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-TOPDIR'><filename>TOPDIR</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_VERBOSE_LOGS'><filename>BB_VERBOSE_LOGS</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_NICE_LEVEL'><filename>BB_NICE_LEVEL</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBFILE_COLLECTIONS'><filename>BBFILE_COLLECTIONS</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_DANGLINGAPPENDS_WARNONLY'><filename>BB_DANGLINGAPPENDS_WARNONLY</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBINCLUDED'><filename>BBINCLUDED</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BUILDNAME'><filename>BUILDNAME</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBMASK'><filename>BBMASK</filename></link>
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            After parsing configuration files, BitBake uses its rudimentary
            inheritance mechanism, which is through class files, to inherit
            some standard classes.
            BitBake parses a class when the inherit directive responsible
            for getting that class is encountered.
        </para>

        <para>
            The <filename>base.bbclass</filename> file is always included.
            Other classes that are specified in the configuration using the
            <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
            variable are also included.
            BitBake searches for class files in a "classes" subdirectory under
            the paths in <filename>BBPATH</filename> in the same way as
            configuration files.
        </para>

        <para>
            A good way to get an idea of the configuration files and
            the class files used in your execution environment is to
            run the following BitBake command:
            <literallayout class='monospaced'>
     $ bitbake -e > mybb.log
            </literallayout>
            Examining the top of the <filename>mybb.log</filename>
            shows you the many configuration files and class files
            used in your execution environment.
        </para>
    </section>

    <section id='locating-recipes'>
        <title>Locating Recipes</title>

        <para>
            The <filename>BBFILES</filename> variable is how BitBake
            locates files.
            This variable is a space-separated list of files
            that are available, supports wildcards, and is set shortly
            after the parsing phase of BitBake's execution.
        </para>

        <para>
            <literallayout class='monospaced'>
     BBFILES = "/path/to/bbfiles/*.bb"
            </literallayout>
            With regard to dependencies, it expects the
            <filename>.bb</filename> to define a
            <filename>DEPENDS</filename> variable, which contains a
           space separated list of “package names”, which themselves
            are the <filename>PN</filename> variable. The
            <filename>PN</filename> variable is, in general,
            set to a component of the <filename>.bb</filename>
           filename by default.
        </para>
   </section>

    <section id='parsing-recipes'>
        <title>Parsing Recipes</title>

        <para>
            After classes are included, the variable
            <filename>BBFILES</filename> is set, usually in
            <filename>local.conf</filename>, and defines the list of
            places to search for recipe and append files.
            Adding extra content to <filename>BBFILES</filename> is best
            achieved through the use of BitBake layers.
        </para>

        <para>
            BitBake parses each recipe and append file located with
            <filename>BBFILES</filename> and stores the values of various
            variables into the datastore.
            In summary, for each recipe and append file pairing, the configuration
            plus the base class of variables are set, followed by the data in the
            recipe file itself, followed by any inherit commands
            that the recipe file might contain.
        </para>

        <para>
            Part of parsing a recipe is making sure that all the recipes
            that the recipe being parsed depends on are understood.
            These other recipes could be located in other layers or in
            a specific layer version.
            These two variables in a recipe can help with with these
            cases:
            <itemizedlist>
                <listitem><para>
                    <link linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link>
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            Because parsing recipe and append files is a time consuming
            process, a cache, referred to as the "setscene"
            is kept to speed up subsequent parsing.
            The setscene is invalid if the timestamps of a recipe changes,
            any of the include files change, configuration files change,
            or class files on which the recipe file depends change.
        </para>
    </section>

    <section id='executing-tasks'>
        <title>Executing Tasks</title>

        <para>
            Tasks can either be a shell task or a Python task.
            For shell tasks, BitBake writes a shell script to
            <filename>${WORKDIR}/temp/run.do_taskname.pid</filename>
            and then executes the script.
            The generated shell script contains all the exported variables,
            and the shell functions with all variables expanded.
            Output from the shell script goes to the file
            <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
            Looking at the expanded shell functions in the run file and
            the output in the log files is a useful debugging technique.
        </para>

        <para>
            For Python tasks, BitBake executes the task internally and logs
            information to the controlling terminal.
            Future versions of BitBake will write the functions to files
            similar to the way shell tasks are handled.
            Logging will be handled in a way similar to shell tasks as well.
        </para>

        <para>
            Variables specific to scheduling functionality exist:
            <itemizedlist>
                <listitem><para>
                    <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>

    <section id='source-fetching-dev-environment'>
        <title>Source Fetching</title>

        <para>
            The first stages of building a recipe are to fetch and unpack
            the source code:
            <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
        </para>

        <para>
            The <filename>do_fetch</filename> and
            <filename>do_unpack</filename> tasks fetch the source files
            and unpack them into the work directory.
            By default, everything is accomplished in the
            build directory,
            which has a defined structure.
        </para>

        <para>
            Unpacked source files are pointed to by a variable.
            For example, in the Yocto Project and OpenEmbedded build systems,
            the <filename>S</filename> variable points to these source files.
            Each recipe has an area in the Build Directory where the
            unpacked source code resides.
            The name of that directory for any given recipe is defined from
            several different variables.
            You can see the variables that define these directories
            by looking at the figure that shows the structure and variables
            used in the Yocto Project:
            <itemizedlist>
                <listitem><para><filename>TMPDIR</filename>
                    </para></listitem>
                <listitem><para><filename>PACKAGE_ARCH</filename>
                    </para></listitem>
                <listitem><para><filename>TARGET_OS</filename>
                    </para></listitem>
                <listitem><para><link linkend='var-PN'><filename>PN</filename></link>
                    </para></listitem>
                <listitem><para><link linkend='var-PV'><filename>PV</filename></link>
                    </para></listitem>
                <listitem><para><link linkend='var-PR'><filename>PR</filename></link>
                    </para></listitem>
                <listitem><para><filename>WORKDIR</filename>
                    </para></listitem>
                <listitem><para><filename>S</filename>
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            Briefly, the <filename>S</filename> directory contains the
            unpacked source files for a recipe.
            The <filename>WORKDIR</filename> directory is where all the
            building goes on for a given recipe.
        </para>
    </section>

    <section id='patching-dev-environment'>
        <title>Patching</title>

        <para>
            Once source code is fetched and unpacked, BitBake locates
            patch files and applies them to the source files:
            <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
        </para>

        <para>
            The <filename>do_patch</filename> task processes recipes by
            using the
            <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
            variable to locate applicable patch files, which by default
            are <filename>*.patch</filename> or
            <filename>*.diff</filename> files, or any file if
            "apply=yes" is specified for the file in
            <filename>SRC_URI</filename>.
        </para>

        <para>
            BitBake finds and applies multiple patches for a single recipe
            in the order in which it finds the patches.
            Patches are applied to the recipe's source files located in the
            <filename>S</filename> directory.
        </para>

        <para>
            For more information on how the source directories are
            created, see the
            "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
            section.
        </para>
    </section>

    <section id='configuration-and-compilation-dev-environment'>
        <title>Configuration and Compilation</title>

        <para>
            After source code is patched, BitBake executes tasks that
            configure and compile the source code:
            <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
        </para>

        <para>
            This step in the build process consists of three tasks:
            <itemizedlist>
                <listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
                    This task configures the source by enabling and
                    disabling any build-time and configuration options for
                    the software being built.
                    Configurations can come from the recipe itself as well
                    as from an inherited class.
                    Additionally, the software itself might configure itself
                    depending on the target for which it is being built.
                    </para>

                    <para>The configurations handled by the
                    <filename>do_configure</filename> task are specific
                    to source code configuration for the source code
                    being built by the recipe.</para>

                    <para>If you are using the Autotools class
                    (<filename>autotools.bbclass</filename>),
                    you can add additional configuration options by using
                    the <filename>EXTRA_OECONF</filename>
                    variable.
                    For information on how this variable works within
                    that class, see the
                    <filename>meta/classes/autotools.bbclass</filename> file.
                    </para></listitem>
                <listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
                    Once a configuration task has been satisfied, BitBake
                    compiles the source using the
                    <filename>do_compile</filename> task.
                    Compilation occurs in the directory pointed to by the
                    <link linkend='var-B'><filename>B</filename></link>
                    variable.
                    Realize that the <filename>B</filename> directory is, by
                    default, the same as the
                    <filename>S</filename>
                    directory.</para></listitem>
                <listitem><para><emphasis><filename>do_install</filename>:</emphasis>
                    Once compilation is done, BitBake executes the
                    <filename>do_install</filename> task.
                    This task copies files from the <filename>B</filename>
                    directory and places them in a holding area pointed to
                    by the <filename>D</filename> variable.</para></listitem>
            </itemizedlist>
        </para>
    </section>

    <section id='package-splitting-dev-environment'>
        <title>Package Splitting</title>

        <para>
            After source code is configured and compiled, the
            OpenEmbedded build system analyzes
            the results and splits the output into packages:
            <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
        </para>

        <para>
            The <filename>do_package</filename> and
            <filename>do_packagedata</filename> tasks combine to analyze
            the files found in the <filename>D</filename> directory
            and split them into subsets based on available packages and
            files.
            The analyzing process involves the following as well as other
            items: splitting out debugging symbols,
            looking at shared library dependencies between packages,
            and looking at package relationships.
            The <filename>do_packagedata</filename> task creates package
            metadata based on the analysis such that the
            OpenEmbedded build system can generate the final packages.
            Working, staged, and intermediate results of the analysis
            and package splitting process use these areas:
            <itemizedlist>
                <listitem><para><filename>PKGD</filename>
                    </para></listitem>
                <listitem><para><filename>PKGDATA_DIR</filename>
                    </para></listitem>
                <listitem><para><filename>PKGDESTWORK</filename>
                    </para></listitem>
                <listitem><para><filename>PKGDEST</filename>
                    </para></listitem>
            </itemizedlist>
            The <filename>FILES</filename>
            variable defines the files that go into each package in
            <link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>.
            If you want details on how this is accomplished in the Yocto Project
            for example, you can look at the <filename>package.bbclass</filename>
            file in a Yocto tree.
        </para>

        <para>
            Depending on the type of packages being created (RPM, DEB, or
            IPK), the <filename>do_package_write_*</filename> task
            creates the actual packages and places them in the
            Package Feed area, which is
            <filename>${TMPDIR}/deploy</filename>.
            <note>
                Support for creating feeds directly from the
                <filename>deploy/*</filename> directories does not exist.
                Creating such feeds usually requires some kind of feed
                maintenance mechanism that would upload the new packages
                into an official package feed (e.g. the
                Ångström distribution).
                This functionality is highly distribution-specific
                and thus is not provided out of the box.
            </note>
        </para>
    </section>

    <section id='image-generation-dev-environment'>
        <title>Image Generation</title>

        <para>
            Once packages are split and stored in the Package Feeds area,
            the OpenEmbedded build system uses BitBake to generate the
            root filesystem image:
            <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
        </para>

        <para>
            The image generation process consists of several stages and
            depends on many variables.
            The <filename>do_rootfs</filename> task uses these key variables
            to help create the list of packages to actually install:
            <itemizedlist>
                <listitem><para><filename>IMAGE_INSTALL</filename>:
                    Lists out the base set of packages to install from
                    the Package Feeds area.</para></listitem>
                <listitem><para><filename>PACKAGE_EXCLUDE</filename>:
                    Specifies packages that should not be installed.
                    </para></listitem>
                <listitem><para><filename>IMAGE_FEATURES</filename>:
                    Specifies features to include in the image.
                    Most of these features map to additional packages for
                    installation.</para></listitem>
                <listitem><para><filename>PACKAGE_CLASSES</filename>:
                    Specifies the package backend to use and consequently
                    helps determine where to locate packages within the
                    Package Feeds area.</para></listitem>
                <listitem><para><filename>IMAGE_LINGUAS</filename>:
                    Determines the language(s) for which additional
                    language support packages are installed.
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            Package installation is under control of the package manager
            (e.g. smart/rpm, opkg, or apt/dpkg) regardless of whether or
            not package management is enabled for the target.
            At the end of the process, if package management is not
            enabled for the target, the package manager's data files
            are deleted from the root filesystem.
        </para>

        <para>
            During image generation, the build system attempts to run
            all post-installation scripts.
            Any that fail to run on the build host are run on the
            target when the target system is first booted.
            If you are using a
            read-only root filesystem,
            all the post installation scripts must succeed during the
            package installation phase since the root filesystem cannot be
            written into.
        </para>

        <para>
            During Optimization, optimizing processes are run across
            the image.
            These processes include <filename>mklibs</filename> and
            <filename>prelink</filename>.
            The <filename>mklibs</filename> process optimizes the size
            of the libraries.
            A <filename>prelink</filename> process optimizes the dynamic
            linking of shared libraries to reduce start up time of
            executables.
        </para>

        <para>
            Part of the image generation process includes compressing the
            root filesystem image.
            Compression is accomplished through several optimization
            routines designed to reduce the overall size of the image.
        </para>

        <para>
            After the root filesystem has been constructed, the image
            generation process turns everything into an image file or
            a set of image files.
            The formats used for the root filesystem depend on the
            <filename>IMAGE_FSTYPES</filename> variable.
        </para>

        <note>
            The entire image generation process is run under Pseudo.
            Running under Pseudo ensures that the files in the root
            filesystem have correct ownership.
        </note>
    </section>

    <section id='sdk-generation-dev-environment'>
        <title>SDK Generation</title>

        <para>
            The OpenEmbedded build system uses BitBake to generate the
            Software Development Kit (SDK) installer script:
            <imagedata fileref="figures/sdk-generation.png" align="center" width="6in" depth="7in" />
        </para>

        <para>
            Like image generation, the SDK script process consists of
            several stages and depends on many variables.
            The <filename>do_populate_sdk</filename> task uses these
            key variables to help create the list of packages to actually
            install.
        </para>

        <para>
            The <filename>do_populate_sdk</filename> task handles two
            parts: a target part and a host part.
            The target part is the part built for the target hardware and
            includes libraries and headers.
            The host part is the part of the SDK that runs on the
            <filename>SDKMACHINE</filename>.
        </para>

        <para>
            Once both parts are constructed, the
            <filename>do_populate_sdk</filename> task performs some cleanup
            on both parts.
            After the cleanup, the task creates a cross-development
            environment setup script and any configuration files that
            might be needed.
        </para>

        <para>
            The final output of the task is the Cross-development
            toolchain installation script (<filename>.sh</filename> file),
            which includes the environment setup script.
        </para>
    </section>
</chapter>