path: root/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
    "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<chapter id="bitbake-user-manual-intro">
    <title>Overview</title>

    <para>
        Welcome to the BitBake User Manual.
        This manual provides information on the BitBake tool.
        The information attempts to be as independent as possible regarding
        systems that use BitBake, such as OpenEmbedded and the
        Yocto Project.
        In some cases, scenarios or examples within the context of
        a build system are used in the manual to help with understanding.
        For these cases, the manual clearly states the context.
    </para>

    <section id="intro">
        <title>Introduction</title>

        <para>
            Fundamentally, BitBake is a generic task execution
            engine that allows shell and Python tasks to be run
            efficiently and in parallel while working within
            complex inter-task dependency constraints.
            One of BitBake's main users, OpenEmbedded, takes this core
            and builds embedded Linux software stacks using
            a task-oriented approach.
        </para>

        <para>
            Conceptually, BitBake is similar to GNU Make in
            some regards but has significant differences:
            <itemizedlist>
                <listitem><para>
                    BitBake executes tasks according to provided
                    metadata that builds up the tasks.
                    Metadata is stored in recipe (<filename>.bb</filename>)
                    and related recipe "append" (<filename>.bbappend</filename>)
                    files, configuration (<filename>.conf</filename>) and
                    underlying include (<filename>.inc</filename>) files, and
                    in class (<filename>.bbclass</filename>) files.
                    The metadata provides
                    BitBake with instructions on what tasks to run and
                    the dependencies between those tasks.
                    </para></listitem>
                <listitem><para>
                    BitBake includes a fetcher library for obtaining source
                    code from various places such as local files, source control
                    systems, or websites.
                    </para></listitem>
                <listitem><para>
                    The instructions for each unit to be built (e.g. a piece
                    of software) are known as "recipe" files and
                    contain all the information about the unit
                    (dependencies, source file locations, checksums, description
                    and so on).
                    </para></listitem>
                <listitem><para>
                    BitBake includes a client/server abstraction and can
                    be used from a command line or used as a service over
                    XML-RPC and has several different user interfaces.
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>

    <section id="history-and-goals">
        <title>History and Goals</title>

        <para>
            BitBake was originally a part of the OpenEmbedded project.
            It was inspired by the Portage package management system
            used by the Gentoo Linux distribution.
            On December 7, 2004, OpenEmbedded project team member
            Chris Larson split the project into two distinct pieces:
            <itemizedlist>
                <listitem><para>BitBake, a generic task executor</para></listitem>
                <listitem><para>OpenEmbedded, a metadata set utilized by
                    BitBake</para></listitem>
            </itemizedlist>
            Today, BitBake is the primary basis of the
            <ulink url="http://www.openembedded.org/">OpenEmbedded</ulink>
            project, which is being used to build and maintain Linux
            distributions such as the
            <ulink url='http://www.angstrom-distribution.org/'>Angstrom Distribution</ulink>,
            and which is also being used as the build tool for Linux projects
            such as the
            <ulink url='http://www.yoctoproject.org'>Yocto Project</ulink>.
        </para>

        <para>
            Prior to BitBake, no other build tool adequately met the needs of
            an aspiring embedded Linux distribution.
            All of the build systems used by traditional desktop Linux
            distributions lacked important functionality, and none of the
            ad hoc Buildroot-based systems, prevalent in the
            embedded space, were scalable or maintainable.
        </para>

        <para>
            Some important original goals for BitBake were:
            <itemizedlist>
                <listitem><para>
                    Handle cross-compilation.
                    </para></listitem>
                <listitem><para>
                    Handle inter-package dependencies (build time on
                    target architecture, build time on native
                    architecture, and runtime).
                    </para></listitem>
                <listitem><para>
                    Support running any number of tasks within a given
                    package, including, but not limited to, fetching
                    upstream sources, unpacking them, patching them,
                    configuring them, and so forth.
                    </para></listitem>
                <listitem><para>
                    Be Linux distribution agnostic for both build and
                    target systems.
                    </para></listitem>
                <listitem><para>
                    Be architecture agnostic.
                    </para></listitem>
                <listitem><para>
                    Support multiple build and target operating systems
                    (e.g. Cygwin, the BSDs, and so forth).
                    </para></listitem>
                <listitem><para>
                    Be self contained, rather than tightly
                    integrated into the build machine's root
                    filesystem.
                    </para></listitem>
                <listitem><para>
                    Handle conditional metadata on the target architecture,
                    operating system, distribution, and machine.
                    </para></listitem>
                <listitem><para>
                    Be easy to use the tools to supply local metadata and packages
                    against which to operate.
                    </para></listitem>
                <listitem><para>
                    Be easy to use BitBake to collaborate between multiple
                    projects for their builds.
                    </para></listitem>
                <listitem><para>
                    Provide an inheritance mechanism to share
                    common metadata between many packages.
                    </para></listitem>
            </itemizedlist>
            Over time it became apparent that some further requirements
            were necessary:
            <itemizedlist>
                <listitem><para>
                    Handle variants of a base recipe (e.g. native, sdk,
                    and multilib).
                    </para></listitem>
                <listitem><para>
                    Split metadata into layers and allow layers
                    to enhance or override other layers.
                    </para></listitem>
                <listitem><para>
                    Allow representation of a given set of input variables
                    to a task as a checksum.
                    Based on that checksum, allow acceleration of builds
                    with prebuilt components.
                    </para></listitem>
            </itemizedlist>
            BitBake satisfies all the original requirements and many more
            with extensions being made to the basic functionality to
            reflect the additional requirements.
            Flexibility and power have always been the priorities.
            BitBake is highly extensible and supports embedded Python code and
            execution of any arbitrary tasks.
        </para>
    </section>

    <section id="Concepts">
        <title>Concepts</title>

        <para>
            BitBake is a program written in the Python language.
            At the highest level, BitBake interprets metadata, decides
            what tasks are required to run, and executes those tasks.
            Similar to GNU Make, BitBake controls how software is
            built.
            GNU Make achieves its control through "makefiles", while
            BitBake uses "recipes".
        </para>

        <para>
            BitBake extends the capabilities of a simple
            tool like GNU Make by allowing for the definition of much more
            complex tasks, such as assembling entire embedded Linux
            distributions.
        </para>

        <para>
            The remainder of this section introduces several concepts
            that should be understood in order to better leverage
            the power of BitBake.
        </para>

        <section id='recipes'>
            <title>Recipes</title>

            <para>
                BitBake Recipes, which are denoted by the file extension
                <filename>.bb</filename>, are the most basic metadata files.
                These recipe files provide BitBake with the following:
                <itemizedlist>
                    <listitem><para>Descriptive information about the
                        package (author, homepage, license, and so on)</para></listitem>
                    <listitem><para>The version of the recipe</para></listitem>
                    <listitem><para>Existing dependencies (both build
                        and runtime dependencies)</para></listitem>
                    <listitem><para>Where the source code resides and
                        how to fetch it</para></listitem>
                    <listitem><para>Whether the source code requires
                        any patches, where to find them, and how to apply
                        them</para></listitem>
                    <listitem><para>How to configure and compile the
                        source code</para></listitem>
                    <listitem><para>Where on the target machine to install the
                        package or packages created</para></listitem>
                </itemizedlist>
            </para>

            <para>
                Within the context of BitBake, or any project utilizing BitBake
                as its build system, files with the <filename>.bb</filename>
                extension are referred to as recipes.
                <note>
                    The term "package" is also commonly used to describe recipes.
                    However, since the same word is used to describe packaged
                    output from a project, it is best to maintain a single
                    descriptive term - "recipes".
                    Put another way, a single "recipe" file is quite capable
                    of generating a number of related but separately installable
                    "packages".
                    In fact, that ability is fairly common.
                </note>
            </para>
        </section>

        <section id='configuration-files'>
            <title>Configuration Files</title>

            <para>
                Configuration files, which are denoted by the
                <filename>.conf</filename> extension, define
                various configuration variables that govern the project's build
                process.
                These files fall into several areas that define
                machine configuration options, distribution configuration
                options, compiler tuning options, general common
                configuration options, and user configuration options.
                The main configuration file is the sample
                <filename>bitbake.conf</filename> file, which is
                located within the BitBake source tree
                <filename>conf</filename> directory.
            </para>
        </section>

        <section id='classes'>
            <title>Classes</title>

            <para>
                Class files, which are denoted by the
                <filename>.bbclass</filename> extension, contain
                information that is useful to share between metadata files.
                The BitBake source tree currently comes with one class metadata file
                called <filename>base.bbclass</filename>.
                You can find this file in the
                <filename>classes</filename> directory.
                The <filename>base.bbclass</filename> class files is special since it
                is always included automatically for all recipes
                and classes.
                This class contains definitions for standard basic tasks such
                as fetching, unpacking, configuring (empty by default),
                compiling (runs any Makefile present), installing (empty by
                default) and packaging (empty by default).
                These tasks are often overridden or extended by other classes
                added during the project development process.
            </para>
        </section>

        <section id='layers'>
            <title>Layers</title>

            <para>
                Layers allow you to isolate different types of
                customizations from each other.
                While you might find it tempting to keep everything in one layer
                when working on a single project, the more modular you organize
                your metadata, the easier it is to cope with future changes.
            </para>

            <para>
                To illustrate how you can use layers to keep things modular,
                consider customizations you might make to support a specific target machine.
                These types of customizations typically reside in a special layer,
                rather than a general layer, called a Board Support Package (BSP)
                Layer.
                Furthermore, the machine customizations should be isolated from
                recipes and metadata that support a new GUI environment, for
                example.
                This situation gives you a couple of layers: one for the machine
                configurations and one for the GUI environment.
                It is important to understand, however, that the BSP layer can still
                make machine-specific additions to recipes within
                the GUI environment layer without polluting the GUI layer itself
                with those machine-specific changes.
                You can accomplish this through a recipe that is a BitBake append
                (<filename>.bbappend</filename>) file.
            </para>
        </section>

        <section id='append-bbappend-files'>
            <title>Append Files</title>

            <para>
                Append files, which are files that have the
                <filename>.bbappend</filename> file extension, extend or
                override information in an existing recipe file.
            </para>

            <para>
                BitBake expects every append file to have a corresponding recipe file.
                Furthermore, the append file and corresponding recipe file
                must use the same root filename.
                The filenames can differ only in the file type suffix used
                (e.g. <filename>formfactor_0.0.bb</filename> and
                <filename>formfactor_0.0.bbappend</filename>).
            </para>

            <para>
                Information in append files extends or
                overrides the information in the underlying,
                similarly-named recipe files.
            </para>

            <para>
                When you name an append file, you can use the
                wildcard character (%) to allow for matching recipe names.
                For example, suppose you have an append file named
                as follows:
                <literallayout class='monospaced'>
     busybox_1.21.%.bbappend
                </literallayout>
                That append file would match any <filename>busybox_1.21.x.bb</filename>
                version of the recipe.
                So, the append file would match the following recipe names:
                <literallayout class='monospaced'>
     busybox_1.21.1.bb
     busybox_1.21.2.bb
     busybox_1.21.3.bb
                </literallayout>
                If the <filename>busybox</filename> recipe was updated to
                <filename>busybox_1.3.0.bb</filename>, the append name would not
                match.
                However, if you named the append file
                <filename>busybox_1.%.bbappend</filename>, then you would have a match.
            </para>

            <para>
                In the most general case, you could name the append file something as
                simple as <filename>busybox_%.bbappend</filename> to be entirely
                version independent.
            </para>
        </section>
    </section>

    <section id='obtaining-bitbake'>
        <title>Obtaining BitBake</title>

        <para>
            You can obtain BitBake several different ways:
            <itemizedlist>
                <listitem><para><emphasis>Cloning BitBake:</emphasis>
                    Using Git to clone the BitBake source code repository
                    is the recommended method for obtaining BitBake.
                    Cloning the repository makes it easy to get bug fixes
                    and have access to stable branches and the master
                    branch.
                    Once you have cloned BitBake, you should use
                    the latest stable
                    branch for development since the master branch is for
                    BitBake development and might contain less stable changes.
                    </para>
                    <para>You usually need a version of BitBake
                    that matches the metadata you are using.
                    The metadata is generally backwards compatible but
                    not forward compatible.</para>
                    <para>Here is an example that clones the BitBake repository:
                    <literallayout class='monospaced'>
     $ git clone git://git.openembedded.org/bitbake
                    </literallayout>
                    This command clones the BitBake Git repository into a
                    directory called <filename>bitbake</filename>.
                    Alternatively, you can
                    designate a directory after the
                    <filename>git clone</filename> command
                    if you want to call the new directory something
                    other than <filename>bitbake</filename>.
                    Here is an example that names the directory
                    <filename>bbdev</filename>:
                    <literallayout class='monospaced'>
     $ git clone git://git.openembedded.org/bitbake bbdev
                    </literallayout></para></listitem>
                <listitem><para><emphasis>Installation using your Distribution
                    Package Management System:</emphasis>
                    This method is not
                    recommended because the BitBake version that is
                    provided by your distribution, in most cases,
                    is several
                    releases behind a snapshot of the BitBake repository.
                    </para></listitem>
                <listitem><para><emphasis>Taking a snapshot of BitBake:</emphasis>
                    Downloading a snapshot of BitBake from the
                    source code repository gives you access to a known
                    branch or release of BitBake.
                    <note>
                         Cloning the Git repository, as described earlier,
                         is the preferred method for getting BitBake.
                         Cloning the repository makes it easier to update as
                         patches are added to the stable branches.
                    </note></para>
                    <para>The following example downloads a snapshot of
                    BitBake version 1.17.0:
                    <literallayout class='monospaced'>
     $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz
     $ tar zxpvf bitbake-1.17.0.tar.gz
                    </literallayout>
                    After extraction of the tarball using the tar utility,
                    you have a directory entitled
                    <filename>bitbake-1.17.0</filename>.
                    </para></listitem>
                <listitem><para><emphasis>Using the BitBake that Comes With Your
                    Build Checkout:</emphasis>
                    A final possibility for getting a copy of BitBake is that it
                    already comes with your checkout of a larger Bitbake-based build
                    system, such as Poky or Yocto Project.
                    Rather than manually checking out individual layers and
                    gluing them together yourself, you can check
                    out an entire build system.
                    The checkout will already include a version of BitBake that
                    has been thoroughly tested for compatibility with the other
                    components.
                    For information on how to check out a particular BitBake-based
                    build system, consult that build system's supporting documentation.
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>

    <section id="bitbake-user-manual-command">
        <title>The BitBake Command</title>

        <para>
            The <filename>bitbake</filename> command is the primary interface
            to the BitBake tool.
            This section presents the BitBake command syntax and provides
            several execution examples.
        </para>

        <section id='usage-and-syntax'>
            <title>Usage and syntax</title>

            <para>
                Following is the usage and syntax for BitBake:
                <literallayout class='monospaced'>
     $ bitbake -h
     Usage: bitbake [options] [recipename/target recipe:do_task ...]

         Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
         It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
         will provide the layer, BBFILES and other configuration information.

     Options:
       --version             show program's version number and exit
       -h, --help            show this help message and exit
       -b BUILDFILE, --buildfile=BUILDFILE
                             Execute tasks from a specific .bb recipe directly.
                             WARNING: Does not handle any dependencies from other
                             recipes.
       -k, --continue        Continue as much as possible after an error. While the
                             target that failed and anything depending on it cannot
                             be built, as much as possible will be built before
                             stopping.
       -a, --tryaltconfigs   Continue with builds by trying to use alternative
                             providers where possible.
       -f, --force           Force the specified targets/task to run (invalidating
                             any existing stamp file).
       -c CMD, --cmd=CMD     Specify the task to execute. The exact options
                             available depend on the metadata. Some examples might
                             be 'compile' or 'populate_sysroot' or 'listtasks' may
                             give a list of the tasks available.
       -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
                             Invalidate the stamp for the specified task such as
                             'compile' and then run the default task for the
                             specified target(s).
       -r PREFILE, --read=PREFILE
                             Read the specified file before bitbake.conf.
       -R POSTFILE, --postread=POSTFILE
                             Read the specified file after bitbake.conf.
       -v, --verbose         Output more log message data to the terminal.
       -D, --debug           Increase the debug level. You can specify this more
                             than once.
       -n, --dry-run         Don't execute, just go through the motions.
       -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
                             Dump out the signature construction information, with
                             no task execution. The SIGNATURE_HANDLER parameter is
                             passed to the handler. Two common values are none and
                             printdiff but the handler may define more/less. none
                             means only dump the signature, printdiff means compare
                             the dumped signature with the cached one.
       -p, --parse-only      Quit after parsing the BB recipes.
       -s, --show-versions   Show current and preferred versions of all recipes.
       -e, --environment     Show the global or per-recipe environment complete
                             with information about where variables were
                             set/changed.
       -g, --graphviz        Save dependency tree information for the specified
                             targets in the dot syntax.
       -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
                             Assume these dependencies don't exist and are already
                             provided (equivalent to ASSUME_PROVIDED). Useful to
                             make dependency graphs more appealing
       -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
                             Show debug logging for the specified logging domains
       -P, --profile         Profile the command and save reports.
       -u UI, --ui=UI        The user interface to use (depexp, goggle, hob, knotty
                             or ncurses - default knotty).
       -t SERVERTYPE, --servertype=SERVERTYPE
                             Choose which server type to use (process or xmlrpc -
                             default process).
       --token=XMLRPCTOKEN   Specify the connection token to be used when
                             connecting to a remote server.
       --revisions-changed   Set the exit code depending on whether upstream
                             floating revisions have changed or not.
       --server-only         Run bitbake without a UI, only starting a server
                             (cooker) process.
       -B BIND, --bind=BIND  The name/address for the bitbake server to bind to.
       --no-setscene         Do not run any setscene tasks. sstate will be ignored
                             and everything needed, built.
       --remote-server=REMOTE_SERVER
                             Connect to the specified server.
       -m, --kill-server     Terminate the remote server.
       --observe-only        Connect to a server as an observing-only client.
       --status-only         Check the status of the remote bitbake server.
       -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG
                             Writes the event log of the build to a bitbake event
                             json file. Use '' (empty string) to assign the name
                             automatically.
                </literallayout>
            </para>
        </section>

        <section id='bitbake-examples'>
            <title>Examples</title>

            <para>
                This section presents some examples showing how to use BitBake.
            </para>

            <section id='example-executing-a-task-against-a-single-recipe'>
                <title>Executing a Task Against a Single Recipe</title>

                <para>
                    Executing tasks for a single recipe file is relatively simple.
                    You specify the file in question, and BitBake parses
                    it and executes the specified task.
                    If you do not specify a task, BitBake executes the default
                    task, which is "build”.
                    BitBake obeys inter-task dependencies when doing
                    so.
                </para>

                <para>
                    The following command runs the build task, which is
                    the default task, on the <filename>foo_1.0.bb</filename>
                    recipe file:
                    <literallayout class='monospaced'>
     $ bitbake -b foo_1.0.bb
                    </literallayout>
                    The following command runs the clean task on the
                    <filename>foo.bb</filename> recipe file:
                    <literallayout class='monospaced'>
     $ bitbake -b foo.bb -c clean
                    </literallayout>
                    <note>
                        The "-b" option explicitly does not handle recipe
                        dependencies.
                        Other than for debugging purposes, it is instead
                        recommended that you use the syntax presented in the
                        next section.
                    </note>
                </para>
            </section>

            <section id='executing-tasks-against-a-set-of-recipe-files'>
                <title>Executing Tasks Against a Set of Recipe Files</title>

                <para>
                    There are a number of additional complexities introduced
                    when one wants to manage multiple <filename>.bb</filename>
                    files.
                    Clearly there needs to be a way to tell BitBake what
                    files are available and, of those, which you
                    want to execute.
                    There also needs to be a way for each recipe
                    to express its dependencies, both for build-time and
                    runtime.
                    There must be a way for you to express recipe preferences
                    when multiple recipes provide the same functionality, or when
                    there are multiple versions of a recipe.
                </para>

                <para>
                    The <filename>bitbake</filename> command, when not using
                    "--buildfile" or "-b" only accepts a "PROVIDES".
                    You cannot provide anything else.
                    By default, a recipe file generally "PROVIDES" its
                    "packagename" as shown in the following example:
                    <literallayout class='monospaced'>
     $ bitbake foo
                    </literallayout>
                    This next example "PROVIDES" the package name and also uses
                    the "-c" option to tell BitBake to just execute the
                    <filename>do_clean</filename> task:
                    <literallayout class='monospaced'>
     $ bitbake -c clean foo
                    </literallayout>
                </para>
            </section>

            <section id='executing-a-list-of-task-and-recipe-combinations'>
                <title>Executing a List of Task and Recipe Combinations</title>

                <para>
                    The BitBake command line supports specifying different
                    tasks for individual targets when you specify multiple
                    targets.
                    For example, suppose you had two targets (or recipes)
                    <filename>myfirstrecipe</filename> and
                    <filename>mysecondrecipe</filename> and you needed
                    BitBake to run <filename>taskA</filename> for the first
                    recipe and <filename>taskB</filename> for the second
                    recipe:
                    <literallayout class='monospaced'>
     $ bitbake myfirstrecipe recipe:do_taskA mysecondrecipe recipe:do_taskB
                    </literallayout>
                </para>
            </section>

            <section id='generating-dependency-graphs'>
                <title>Generating Dependency Graphs</title>

                <para>
                    BitBake is able to generate dependency graphs using
                    the <filename>dot</filename> syntax.
                    You can convert these graphs into images using the
                    <filename>dot</filename> tool from
                    <ulink url='http://www.graphviz.org'>Graphviz</ulink>.
                </para>

                <para>
                    When you generate a dependency graph, BitBake writes four files
                    to the current working directory:
                    <itemizedlist>
                        <listitem><para><emphasis><filename>package-depends.dot</filename>:</emphasis>
                            Shows BitBake's knowledge of dependencies between
                            runtime targets.
                            </para></listitem>
                        <listitem><para><emphasis><filename>pn-depends.dot</filename>:</emphasis>
                            Shows dependencies between build-time targets
                            (i.e. recipes).
                            </para></listitem>
                        <listitem><para><emphasis><filename>task-depends.dot</filename>:</emphasis>
                            Shows dependencies between tasks.
                            </para></listitem>
                        <listitem><para><emphasis><filename>pn-buildlist</filename>:</emphasis>
                            Shows a simple list of targets that are to be built.
                            </para></listitem>
                    </itemizedlist>
                </para>

                <para>
                    To stop depending on common depends, use the "-I" depend
                    option and BitBake omits them from the graph.
                    Leaving this information out can produce more readable graphs.
                    This way, you can remove from the graph
                    <filename>DEPENDS</filename> from inherited classes
                    such as <filename>base.bbclass</filename>.
                </para>

                <para>
                    Here are two examples that create dependency graphs.
                    The second example omits depends common in OpenEmbedded from
                    the graph:
                    <literallayout class='monospaced'>
     $ bitbake -g foo

     $ bitbake -g -I virtual/kernel -I eglibc foo
                    </literallayout>
                </para>
            </section>
        </section>
    </section>
</chapter>