path: root/bitbake/doc/user-manual/user-manual-execution.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<chapter id="user-manual-execution">
    <title>Execution</title>

    <para>
        The primary purpose for running BitBake is to produce some kind
        of output such as an image, a kernel, or a software development
        kit.
        Of course, you can execute the <filename>bitbake</filename>
        command with options that cause it to execute single tasks,
        compile single recipe files, capture or clear data, or simply
        return information about the execution environment.
    </para>

    <para>
        This chapter describes BitBake's execution process from start
        to finish when you use it to create an image.
        The execution process is launched using the following command
        form:
        <literallayout class='monospaced'>
     $ bitbake &lt;target&gt;
        </literallayout>
        For information on the BitBake command and its options,
        see the
        "<link linkend='user-manual-command'>BitBake Command</link>
        chapter.
    </para>

    <section id='parsing-the-base-configuration-metadata'>
        <title>Parsing the Base Configuration Metadata</title>

        <para>
            The first thing BitBake does is parse base configuration
            metadata.
            Base configuration metadata consists of the
            <filename>bblayers.conf</filename> file to determine what
            layers BitBake needs to recognize, all necessary
            <filename>layer.conf</filename> files (one from each layer),
            and <filename>bitbake.conf</filename>.
            The data itself is of various types:
            <itemizedlist>
                <listitem><para><emphasis>Recipes:</emphasis>
                    Details about particular pieces of software.
                    </para></listitem>
                <listitem><para><emphasis>Class Data:</emphasis>
                    An abstraction of common build information
                    (e.g. how to build a Linux kernel).
                    </para></listitem>
                <listitem><para><emphasis>Configuration Data:</emphasis>
                    Machine-specific settings, policy decisions,
                    and so forth.
                    Configuration data acts as the glue to bind everything
                    together.</para></listitem>
            </itemizedlist>
        </para>

        <para>
            The <filename>layer.conf</filename> files are used to
            construct key variables such as
            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
            and
            <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
            <filename>BBPATH</filename> is used to search for
            configuration and class files under
            <filename>conf/</filename> and <filename>class/</filename>
            directories, respectively.
            <filename>BBFILES</filename> is used to find recipe files
            (<filename>.bb</filename> and <filename>.bbappend</filename>).
            If there is no <filename>bblayers.conf</filename> file,
            it is assumed the user has set the <filename>BBPATH</filename>
            and <filename>BBFILES</filename> directly in the environment.
        </para>

        <para>
            Next, the <filename>bitbake.conf</filename> file is searched
            using the <filename>BBPATH</filename> variable that was
            just constructed.
            The <filename>bitbake.conf</filename> file usually indicates
            all the other key include files to parse.
            The usual convention is to have machine, distro, site, and local
            configurations.
            This means a user provides their own customizations
            through a <filename>local.conf</filename> file.
        </para>

        <para>
            As mentioned in the previous paragraph, two of the other notable
            configuration files are the distro and machine configuration
            files.
            These configuration files are normally identified by
            variables unique to the build systems using BitBake.
            For example, the Yocto Project uses the
            <filename>DISTRO</filename> and <filename>MACHINE</filename>
            variables, respectively.
        </para>

        <para>
            Prior to parsing configuration files, Bitbake looks
            at certain variables, including:
            <itemizedlist>
                <listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem>
                <listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem>
                <listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem>
                <listitem><para><link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link></para></listitem>
                <listitem><para><link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link></para></listitem>
                <listitem><para><link linkend='var-PREFERRED_PROVIDERS'><filename>PREFERRED_PROVIDERS</filename></link></para></listitem>
            </itemizedlist>
        </para>

        <para>
            The base configuration metadata is global
            and therefore affects all packages and tasks that are executed.
        </para>

        <para>
            BitBake first searches the current working directory for an
            optional <filename>conf/bblayers.conf</filename> configuration file.
            This file is expected to contain a
            <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
            variable that is a space delimited list of 'layer' directories.
            Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
            file then it is assumed the user has set the <filename>BBPATH</filename>
            and <filename>BBFILES</filename> directly in the environment.
        </para>

        <para>
            For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
            file is searched for and parsed with the
            <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
            variable being set to the directory where the layer was found.
            The idea is these files automatically setup
            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
            and other variables correctly for a given build directory.
        </para>

        <para>
            BitBake then expects to find the <filename>conf/bitbake.conf</filename>
            file somewhere in the user-specified <filename>BBPATH</filename>.
            That configuration file generally has include directives to pull
            in any other metadata such as files specific to the architecture,
            the machine, the local environment, and so forth.
        </para>

        <para>
            Only variable definitions and include directives are allowed
            in <filename>.conf</filename> files.
            The following variables include:
            <itemizedlist>
                <listitem><para>
                    <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBDEBUG'><filename>BBDEBUG</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-MULTI_PROVIDER_WHITELIST'><filename>MULTI_PROVIDER_WHITELIST</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></link>
                    </para></listitem>
                <listitem><para>
                    <filename>BBPKGS</filename>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_DEFAULT_TASK'><filename>BB_DEFAULT_TASK</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-TOPDIR'><filename>TOPDIR</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_VERBOSE_LOGS'><filename>BB_VERBOSE_LOGS</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_NICE_LEVEL'><filename>BB_NICE_LEVEL</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBFILE_COLLECTIONS'><filename>BBFILE_COLLECTIONS</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_DANGLINGAPPENDS_WARNONLY'><filename>BB_DANGLINGAPPENDS_WARNONLY</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBINCLUDED'><filename>BBINCLUDED</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BUILDNAME'><filename>BUILDNAME</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BBMASK'><filename>BBMASK</filename></link>
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            After parsing configuration files, BitBake uses its rudimentary
            inheritance mechanism, which is through class files, to inherit
            some standard classes.
            BitBake parses a class when the inherit directive responsible
            for getting that class is encountered.
        </para>

        <para>
            The <filename>base.bbclass</filename> file is always included.
            Other classes that are specified in the configuration using the
            <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
            variable are also included.
            BitBake searches for class files in a "classes" subdirectory under
            the paths in <filename>BBPATH</filename> in the same way as
            configuration files.
        </para>

        <para>
            A good way to get an idea of the configuration files and
            the class files used in your execution environment is to
            run the following BitBake command:
            <literallayout class='monospaced'>
     $ bitbake -e > mybb.log
            </literallayout>
            Examining the top of the <filename>mybb.log</filename>
            shows you the many configuration files and class files
            used in your execution environment.
        </para>
    </section>

    <section id='locating-and-parsing-recipes'>
        <title>Locating and Parsing Recipes</title>

        <para>
            During the configuration phase, BitBake will have
            set
            <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
            BitBake now uses it to construct a list of recipes to parse,
            along with any append files (<filename>.bbappend</filename>)
            to apply.
            <filename>BBFILES</filename> is a space-separated list of
            available files and supports wildcards.
            An example would be:
            <literallayout class='monospaced'>
     BBFILES = "/path/to/bbfiles/*.bb"
            </literallayout>
            BitBake parses each recipe and append file located
            with <filename>BBFILES</filename> and stores the values of
            various variables into the datastore.
            For each file, a fresh copy of the base configuration is
            made, then the recipe is parsed line by line.
            Any inherit statements cause BitBake to find and
            then parse class files (<filename>.bbclass</filename>)
            using
            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
            as the search path.
            Finally, BitBake parses in order any append files found in
            <filename>BBFILES</filename>.
        </para>

        <para>
            One common convention is to use the recipe filename to define
            pieces of metadata.
            For example, in <filename>bitbake.conf</filename> the recipe
            name and version set
            <link linkend='var-PN'><filename>PN</filename></link> and
            <link linkend='var-PV'><filename>PV</filename></link>:
            <literallayout class='monospaced'>
     PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}"
     PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or 'defaultpkgname'}"
            </literallayout>
            In this example, a recipe called "something_1.2.3.bb" sets
            <filename>PN</filename> to "something" and
            <filename>PV</filename> to "1.2.3".
        </para>

        <para>
            By the time parsing is complete for a recipe, BitBake
            has a list of tasks that the recipe defines and a set of
            data consisting of keys and values.
        </para>

        <para>
            BitBake does not need all this information.
            It only needs a small subset of the information to make
            decisions about the recipe.
            Consequently, BitBake caches the values in which it is
            interested.
        </para>

        <para>
            Subsequent BitBake commands then parse the base
            configuration and compute a checksum of that data.
            If that checksum matches what is in the cache, the
            recipe and class files have not changed.
            In this case, BitBake reloads the cached information
            about the recipe instead of reparsing it from scratch.
        </para>
    </section>

    <section id='bb-bitbake-providers'>
        <title>Preferences and Providers</title>

        <para>
            Assuming BitBake has been instructed to execute a target and
            that all the recipe files have been parsed, BitBake starts to
            build the target and look for providers of that target.
            Once a provider is selected, BitBake resolves all the dependencies for
            the target.
            As an example, suppose the target is
            <filename>core-image-sato</filename>.
            In this case, it would lead to
            <filename>packagegroup-core-x11-sato</filename>,
            which in turn leads to recipes like <filename>matchbox-terminal</filename>,
            <filename>pcmanfm</filename> and <filename>gthumb</filename>.
            These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
        </para>

        <para>
            Sometimes a target might have multiple providers.
            A common example is "virtual/kernel", which is provided by each kernel package.
            Each machine often selects the best kernel provider by using a line similar to the
            following in the machine configuration file:
        </para>

        <literallayout class='monospaced'>
     PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
        </literallayout>

        <para>
            The default
            <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
            is the provider with the same name as the target.
        </para>

        <para>
            Understanding how providers are chosen is made complicated by the fact
            that multiple versions might exist.
            BitBake defaults to the highest version of a provider.
            Version comparisons are made using the same method as Debian.
            You can use the
            <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
            variable to specify a particular version (usually in the distro configuration).
            You can influence the order by using the
            <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
            variable.
            By default, files have a preference of "0".
            Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
            package unlikely to be used unless it is explicitly referenced.
            Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
            <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
            <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
            versions until they have undergone sufficient testing to be considered stable.
        </para>

        <para>
            In summary, BitBake has created a list of providers, which is prioritized, for each target.
        </para>
    </section>

    <section id='bb-bitbake-dependencies'>
        <title>Dependencies</title>

        <para>
            Each target BitBake builds consists of multiple tasks such as
            <filename>fetch</filename>, <filename>unpack</filename>,
            <filename>patch</filename>, <filename>configure</filename>,
            and <filename>compile</filename>.
            For best performance on multi-core systems, BitBake considers each
            task as an independent
            entity with its own set of dependencies.
        </para>

        <para>
            Dependencies are defined through several variables.
            You can find information about variables BitBake uses in
            the <link linkend='ref-variables-glos'>Variables Glossary</link>
            near the end of this manual.
            At a basic level, it is sufficient to know that BitBake uses the
            <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
            <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
            calculating dependencies.
        </para>

        <para>
            For more information on how BitBake handles dependencies, see the
            "<link linkend='dependencies'>Dependencies</link>" section.
        </para>
    </section>

    <section id='ref-bitbake-tasklist'>
        <title>The Task List</title>

        <para>
            Based on the generated list of providers and the dependency information,
            BitBake can now calculate exactly what tasks it needs to run and in what
            order it needs to run them.
            The build now starts with BitBake forking off threads up to the limit set in the
            <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
            variable.
            BitBake continues to fork threads as long as there are tasks ready to run,
            those tasks have all their dependencies met, and the thread threshold has not been
            exceeded.
        </para>

        <para>
            It is worth noting that you can greatly speed up the build time by properly setting
            the <filename>BB_NUMBER_THREADS</filename> variable.
        </para>

        <para>
            As each task completes, a timestamp is written to the directory specified by the
            <link linkend='var-STAMP'><filename>STAMP</filename></link> variable.
            On subsequent runs, BitBake looks in the build directory within
            <filename>tmp/stamps</filename>and does not rerun
            tasks that are already completed unless a timestamp is found to be invalid.
            Currently, invalid timestamps are only considered on a per
            recipe file basis.
            So, for example, if the configure stamp has a timestamp greater than the
            compile timestamp for a given target, then the compile task would rerun.
            Running the compile task again, however, has no effect on other providers
            that depend on that target.
            This behavior could change or become configurable in future versions of BitBake.
        </para>

        <note>
            Some tasks are marked as "nostamp" tasks.
            No timestamp file is created when these tasks are run.
            Consequently, "nostamp" tasks are always rerun.
        </note>

        <para>
            For more information on tasks, see the
            "<link linkend='tasks'>Tasks</link>" section.
        </para>
    </section>

    <section id='executing-tasks'>
        <title>Executing Tasks</title>

        <para>
            Tasks can either be a shell task or a Python task.
            For shell tasks, BitBake writes a shell script to
            <filename>${WORKDIR}/temp/run.do_taskname.pid</filename>
            and then executes the script.
            The generated shell script contains all the exported variables,
            and the shell functions with all variables expanded.
            Output from the shell script goes to the file
            <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
            Looking at the expanded shell functions in the run file and
            the output in the log files is a useful debugging technique.
        </para>

        <para>
            For Python tasks, BitBake executes the task internally and logs
            information to the controlling terminal.
            Future versions of BitBake will write the functions to files
            similar to the way shell tasks are handled.
            Logging will be handled in a way similar to shell tasks as well.
        </para>

        <para>
            Variables specific to scheduling functionality exist:
            <itemizedlist>
                <listitem><para>
                    <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
                    </para></listitem>
                <listitem><para>
                    <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>

    <section id='setscene'>
        <title>Setscene</title>

        <para>
            This section needs to get the concept of the setscene across.
            The reader needs to know what it is and what it is used for during
            the build process.
        </para>

        <para>
            You can find more information on setscene metadata in the
            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
            section.
        </para>
    </section>

    <section id='checksums'>
        <title>Checksums (Signatures)</title>

        <para>
            A checksum is a unique signature of a task's inputs.
            The setscene code uses a checksum to determine if a task needs
            to be run.
            Because it is a change in a task's inputs that triggers running
            the task, the process needs to detect all the inputs to a given task.
            For shell tasks, this turns out to be fairly easy because
            BitBake generates a "run" shell script for each task and
            it is possible to create a checksum that gives you a good idea of when
            the task's data changes.
        </para>

        <para>
            To complicate the problem, some things should not be included in
            the checksum.
            First, there is the actual specific build path of a given task -
            the working directory.
            It does not matter if the working directory changes because it should not
            affect the output for target packages.
            The simplistic approach for excluding the working directory is to set
            it to some fixed value and create the checksum for the "run" script.
        </para>

        <para>
            Another problem results from the "run" scripts containing functions that
            might or might not get called.
            The incremental build solution contains code that figures out dependencies
            between shell functions.
            This code is used to prune the "run" scripts down to the minimum set,
            thereby alleviating this problem and making the "run" scripts much more
            readable as a bonus.
        </para>

        <para>
            So far we have solutions for shell scripts.
            What about Python tasks?
            The same approach applies even though these tasks are more difficult.
            The process needs to figure out what variables a Python function accesses
            and what functions it calls.
            Again, the incremental build solution contains code that first figures out
            the variable and function dependencies, and then creates a checksum for the data
            used as the input to the task.
        </para>

        <para>
            Like the working directory case, situations exist where dependencies
            should be ignored.
            For these cases, you can instruct the build process to ignore a dependency
            by using a line like the following:
            <literallayout class='monospaced'>
     PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
            </literallayout>
            This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
            depend on the value of <filename>MACHINE</filename>, even if it does reference it.
        </para>

        <para>
            Equally, there are cases where we need to add dependencies BitBake
            is not able to find.
            You can accomplish this by using a line like the following:
            <literallayout class='monospaced'>
      PACKAGE_ARCHS[vardeps] = "MACHINE"
            </literallayout>
            This example explicitly adds the <filename>MACHINE</filename> variable as a
            dependency for <filename>PACKAGE_ARCHS</filename>.
        </para>

        <para>
            Consider a case with in-line Python, for example, where BitBake is not
            able to figure out dependencies.
            When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
            produces output when it discovers something for which it cannot figure out
            dependencies.
        </para>

        <para>
            Thus far, this section has limited discussion to the direct inputs into a task.
            Information based on direct inputs is referred to as the "basehash" in the
            code.
            However, there is still the question of a task's indirect inputs - the
            things that were already built and present in the build directory.
            The checksum (or signature) for a particular task needs to add the hashes
            of all the tasks on which the particular task depends.
            Choosing which dependencies to add is a policy decision.
            However, the effect is to generate a master checksum that combines the basehash
            and the hashes of the task's dependencies.
        </para>

        <para>
            At the code level, there are a variety of ways both the basehash and the
            dependent task hashes can be influenced.
            Within the BitBake configuration file, we can give BitBake some extra information
            to help it construct the basehash.
            The following statement effectively results in a list of global variable
            dependency excludes - variables never included in any checksum.
            This example uses variables from OpenEmbedded to help illustrate
            the concept:
            <literallayout class='monospaced'>
     BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
         SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
         USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
         PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
         CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
            </literallayout>
            The previous example excludes the work directory, which is part of
            <filename>TMPDIR</filename>.
        </para>

        <para>
            The rules for deciding which hashes of dependent tasks to include through
            dependency chains are more complex and are generally accomplished with a
            Python function.
            The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
            of this and also illustrates how you can insert your own policy into the system
            if so desired.
            This file defines the two basic signature generators OpenEmbedded Core
            uses:  "OEBasic" and "OEBasicHash".
            By default, there is a dummy "noop" signature handler enabled in BitBake.
            This means that behavior is unchanged from previous versions.
            <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
            through this setting in the <filename>bitbake.conf</filename> file:
            <literallayout class='monospaced'>
     BB_SIGNATURE_HANDLER ?= "OEBasicHash"
            </literallayout>
            The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
            "OEBasic" version but adds the task hash to the stamp files.
            This results in any metadata change that changes the task hash, automatically
            causing the task to be run again.
            This removes the need to bump
            <link linkend='var-PR'><filename>PR</filename></link>
            values, and changes to metadata automatically ripple across the build.
        </para>

        <para>
            It is also worth noting that the end result of these signature generators is to
            make some dependency and hash information available to the build.
            This information includes:
            <itemizedlist>
                <listitem><para><filename>BB_BASEHASH_task-&lt;taskname&gt;</filename>:
                    The base hashes for each task in the recipe.
                    </para></listitem>
                <listitem><para><filename>BB_BASEHASH_&lt;filename:taskname&gt;</filename>:
                    The base hashes for each dependent task.
                    </para></listitem>
                <listitem><para><filename>BBHASHDEPS_&lt;filename:taskname&gt;</filename>:
                    The task dependencies for each task.
                    </para></listitem>
                <listitem><para><filename>BB_TASKHASH</filename>:
                    The hash of the currently running task.
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            You can find more information on checksum metadata in the
            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
            section.
        </para>
    </section>
</chapter>