path: root/documentation/ref-manual/usingpoky.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >

<chapter id='usingpoky'>
<title>Using the Yocto Project</title>

    <para>
        This chapter describes common usage for the Yocto Project.
        The information is introductory in nature as other manuals in the Yocto Project
        documentation set provide more details on how to use the Yocto Project.
    </para>

<section id='usingpoky-build'>
    <title>Running a Build</title>

    <para>
        This section provides a summary of the build process and provides information
        for less obvious aspects of the build process.
        For general information on how to build an image using the OpenEmbedded build
        system, see the
        "<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>"
        section of the Yocto Project Quick Start.
    </para>

    <section id='build-overview'>
        <title>Build Overview</title>

        <para>
            In the development environment you will need to build an image whenever you change hardware
            support, add or change system libraries, or add or change services that have dependencies.
        </para>

        <mediaobject>
            <imageobject>
                <imagedata fileref="figures/building-an-image.png" format="PNG" align='center' scalefit='1'/>
            </imageobject>
            <caption>
                <para>Building an Image</para>
            </caption>
        </mediaobject>

        <para>
            The first thing you need to do is set up the OpenEmbedded build
            environment by sourcing an environment setup script
            (i.e.
            <link linkend='structure-core-script'><filename>&OE_INIT_FILE;</filename></link>
            or
            <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>).
            Here is an example:
            <literallayout class='monospaced'>
     $ source &OE_INIT_FILE; [<replaceable>build_dir</replaceable>]
            </literallayout>
        </para>

        <para>
            The <replaceable>build_dir</replaceable> argument is optional and specifies the directory the
            OpenEmbedded build system uses for the build -
            the <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
            If you do not specify a Build Directory, it defaults to a directory
            named <filename>build</filename> in your current working directory.
            A common practice is to use a different Build Directory for different targets.
            For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
            target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
        </para>

        <para>
            Once the build environment is set up, you can build a target using:
            <literallayout class='monospaced'>
     $ bitbake <replaceable>target</replaceable>
            </literallayout>
        </para>

        <para>
            The <replaceable>target</replaceable> is the name of the recipe you want to build.
            Common targets are the images in <filename>meta/recipes-core/images</filename>,
            <filename>meta/recipes-sato/images</filename>, etc. all found in the
            <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
            Or, the target can be the name of a recipe for a specific piece of software such as
            BusyBox.
            For more details about the images the OpenEmbedded build system supports, see the
            "<link linkend="ref-images">Images</link>" chapter.
        </para>

        <note>
            Building an image without GNU General Public License Version
            3 (GPLv3), or similarly licensed, components is supported for
            only minimal and base images.
            See the "<link linkend='ref-images'>Images</link>" chapter for more information.
        </note>
    </section>

    <section id='building-an-image-using-gpl-components'>
        <title>Building an Image Using GPL Components</title>

        <para>
            When building an image using GPL components, you need to maintain your original
            settings and not switch back and forth applying different versions of the GNU
            General Public License.
            If you rebuild using different versions of GPL, dependency errors might occur
            due to some components not being rebuilt.
        </para>
    </section>
</section>

<section id='usingpoky-install'>
    <title>Installing and Using the Result</title>

    <para>
        Once an image has been built, it often needs to be installed.
        The images and kernels built by the OpenEmbedded build system are placed in the
        <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink> in
        <filename class="directory">tmp/deploy/images</filename>.
        For information on how to run pre-built images such as <filename>qemux86</filename>
        and <filename>qemuarm</filename>, see the
        <ulink url='&YOCTO_DOCS_SDK_URL;#sdk-manual'>Yocto Project Software Development Kit (SDK) Developer's Guide</ulink>.
        For information about how to install these images, see the documentation for your
        particular board or machine.
    </para>
</section>

<section id='usingpoky-debugging-tools-and-techniques'>
    <title>Debugging Tools and Techniques</title>

    <para>
        The exact method for debugging build failures depends on the nature of
        the problem and on the system's area from which the bug originates.
        Standard debugging practices such as comparison against the last
        known working version with examination of the changes and the
        re-application of steps to identify the one causing the problem are
        valid for the Yocto Project just as they are for any other system.
        Even though it is impossible to detail every possible potential failure,
        this section provides some general tips to aid in debugging.
    </para>

    <para>
        A useful feature for debugging is the error reporting tool.
        Configuring the Yocto Project to use this tool causes the
        OpenEmbedded build system to produce error reporting commands as
        part of the console output.
        You can enter the commands after the build completes
        to log error information
        into a common database, that can help you figure out what might be
        going wrong.
        For information on how to enable and use this feature, see the
        "<ulink url='&YOCTO_DOCS_DEV_URL;#using-the-error-reporting-tool'>Using the Error Reporting Tool</ulink>"
        section in the Yocto Project Development Manual.
    </para>

    <para>
        For discussions on debugging, see the
        "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-gdb-remotedebug'>Debugging With the GNU Project Debugger (GDB) Remotely</ulink>" section
        in the Yocto Project Developer's Manual
        and the
        "<ulink url='&YOCTO_DOCS_SDK_URL;#adt-eclipse'>Working within Eclipse</ulink>"
        section in the Yocto Project Software Development Kit (SDK) Developer's Guide.
    </para>

    <note>
        The remainder of this section presents many examples of the
        <filename>bitbake</filename> command.
        You can learn about BitBake by reading the
        <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>.
    </note>

    <section id='usingpoky-debugging-viewing-logs-from-failed-tasks'>
        <title>Viewing Logs from Failed Tasks</title>

        <para>
            You can find the log for a task in the file
            <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/temp/log.do_</filename><replaceable>taskname</replaceable>.
            For example, the log for the
            <link linkend='ref-tasks-compile'><filename>do_compile</filename></link>
            task of the QEMU minimal image for the x86 machine
            (<filename>qemux86</filename>) might be in
            <filename>tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile</filename>.
            To see the commands
            <ulink url='&YOCTO_DOCS_DEV_URL;#bitbake-term'>BitBake</ulink> ran
            to generate a log, look at the corresponding
            <filename>run.do_</filename><replaceable>taskname</replaceable>
            file in the same directory.
        </para>

        <para>
            <filename>log.do_</filename><replaceable>taskname</replaceable> and
            <filename>run.do_</filename><replaceable>taskname</replaceable>
            are actually symbolic links to
            <filename>log.do_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>
            and
            <filename>log.run_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>,
            where <replaceable>pid</replaceable> is the PID the task had when
            it ran.
            The symlinks always point to the files corresponding to the most
            recent run.
        </para>
    </section>

    <section id='usingpoky-debugging-viewing-variable-values'>
        <title>Viewing Variable Values</title>
        <para>
            BitBake's <filename>-e</filename> option is used to display
            variable values after parsing.
            The following command displays the variable values after the
            configuration files (i.e. <filename>local.conf</filename>,
            <filename>bblayers.conf</filename>,
            <filename>bitbake.conf</filename> and so forth) have been
            parsed:
            <literallayout class='monospaced'>
     $ bitbake -e
            </literallayout>
            The following command displays variable values after a specific
            recipe has been parsed.
            The variables include those from the configuration as well:
            <literallayout class='monospaced'>
     $ bitbake -e recipename
            </literallayout>
            <note><para>
                Each recipe has its own private set of variables (datastore).
                Internally, after parsing the configuration, a copy of the
                resulting datastore is made prior to parsing each recipe.
                This copying implies that variables set in one recipe will
                not be visible to other recipes.</para>

                <para>Likewise, each task within a recipe gets a private
                datastore based on the recipe datastore, which means that
                variables set within one task will not be visible to
                other tasks.</para>
            </note>
        </para>

        <para>
            In the output of <filename>bitbake -e</filename>, each variable is
            preceded by a description of how the variable got its value,
            including temporary values that were later overriden.
            This description also includes variable flags (varflags) set on
            the variable.
            The output can be very helpful during debugging.
        </para>

        <para>
            Variables that are exported to the environment are preceded by
            <filename>export</filename> in the output of
            <filename>bitbake -e</filename>.
            See the following example:
            <literallayout class='monospaced'>
     export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
            </literallayout>
        </para>

        <para>
            In addition to variable values, the output of the
            <filename>bitbake -e</filename> and
            <filename>bitbake -e</filename>&nbsp;<replaceable>recipe</replaceable>
            commands includes the following information:
            <itemizedlist>
                <listitem><para>
                    The output starts with a tree listing all configuration
                    files and classes included globally, recursively listing
                    the files they include or inherit in turn.
                    Much of the behavior of the OpenEmbedded build system
                    (including the behavior of the
                    <link linkend='normal-recipe-build-tasks'>normal recipe build tasks</link>)
                    is implemented in the
                    <link linkend='ref-classes-base'><filename>base</filename></link>
                    class and the classes it inherits, rather than being built
                    into BitBake itself.
                    </para></listitem>
                <listitem><para>
                    After the variable values, all functions appear in the
                    output.
                    For shell functions, variables referenced within the
                    function body are expanded.
                    If a function has been modified using overrides or
                    using override-style operators like
                    <filename>_append</filename> and
                    <filename>_prepend</filename>, then the final assembled
                    function body appears in the output.
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>

    <section id='viewing-package-information-with-oe-pkgdata-util'>
        <title>Viewing Package Information with <filename>oe-pkgdata-util</filename></title>

        <para>
            You can use the <filename>oe-pkgdata-util</filename> command-line
            utility to query
            <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
            and display various package-related information.
            When you use the utility, you must use it to view information
            on packages that have already been built.
        </para>

        <para>
            Following are a few of the available
            <filename>oe-pkgdata-util</filename> subcommands.
            <note>
                You can use the standard * and ? globbing wildcards as part of
                package names and paths.
            </note>
            <itemizedlist>
                <listitem><para>
                    <filename>oe-pkgdata-util list-pkgs [</filename><replaceable>pattern</replaceable><filename>]</filename>:
                    Lists all packages that have been built, optionally
                    limiting the match to packages that match
                    <replaceable>pattern</replaceable>.
                    </para></listitem>
                <listitem><para>
                    <filename>oe-pkgdata-util list-pkg-files&nbsp;</filename><replaceable>package</replaceable><filename>&nbsp;...</filename>:
                    Lists the files and directories contained in the given
                    packages.
                    <note>
                        <para>
                        A different way to view the contents of a package is
                        to look at the
                        <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/packages-split</filename>
                        directory of the recipe that generates the
                        package.
                        This directory is created by the
                        <link linkend='ref-tasks-package'><filename>do_package</filename></link>
                        task and has one subdirectory for each package the
                        recipe generates, which contains the files stored in
                        that package.</para>
                        <para>
                        If you want to inspect the
                        <filename>${WORKDIR}/packages-split</filename>
                        directory, make sure that
                        <link linkend='ref-classes-rm-work'><filename>rm_work</filename></link>
                        is not enabled when you build the recipe.
                        </para>
                        </note>
                    </para></listitem>
                <listitem><para>
                    <filename>oe-pkgdata-util find-path&nbsp;</filename><replaceable>path</replaceable><filename>&nbsp;...</filename>:
                    Lists the names of the packages that contain the given
                    paths.
                    For example, the following tells us that
                    <filename>/usr/share/man/man1/make.1</filename>
                    is contained in the <filename>make-doc</filename>
                    package:
                    <literallayout class='monospaced'>
     $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
     make-doc: /usr/share/man/man1/make.1
                    </literallayout>
                    </para></listitem>
                <listitem><para>
                    <filename>oe-pkgdata-util lookup-recipe&nbsp;</filename><replaceable>package</replaceable><filename>&nbsp;...</filename>:
                    Lists the name of the recipes that
                    produce the given packages.
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            For more information on the <filename>oe-pkgdata-util</filename>
            command, use the help facility:
            <literallayout class='monospaced'>
     $ oe-pkgdata-util &dash;&dash;help
     $ oe-pkgdata-util <replaceable>subcommand</replaceable> --help
            </literallayout>
        </para>
    </section>

    <section id='usingpoky-viewing-dependencies-between-recipes-and-tasks'>
        <title>Viewing Dependencies Between Recipes and Tasks</title>

        <para>
            Sometimes it can be hard to see why BitBake wants to build other
            recipes before the one you have specified.
            Dependency information can help you understand why a recipe is
            built.
        </para>

        <para>
            To generate dependency information for a recipe, run the following
            command:
            <literallayout class='monospaced'>
     $ bitbake -g <replaceable>recipename</replaceable>
            </literallayout>
            This command writes the following files in the current directory:
            <itemizedlist>
                <listitem><para>
                    <filename>pn-buildlist</filename>: A list of
                    recipes/targets involved in building
                    <replaceable>recipename</replaceable>.
                    "Involved" here means that at least one task from the
                     recipe needs to run when building
                    <replaceable>recipename</replaceable> from scratch.
                    Targets that are in
                    <link linkend='var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></link>
                    are not listed.
                    </para></listitem>
                <listitem><para>
                    <filename>pn-depends.dot</filename>: A graph showing
                    dependencies between build-time targets (recipes).
                    </para></listitem>
                <listitem><para>
                    <filename>package-depends.dot</filename>: A graph showing
                    known dependencies between runtime targets.
                    </para></listitem>
                <listitem><para>
                    <filename>task-depends.dot</filename>: A graph showing
                    dependencies between tasks.
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            The graphs are in
            <ulink url='https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29'>DOT</ulink>
            format and can be converted to images (e.g. using the
            <filename>dot</filename> tool from
            <ulink url='http://www.graphviz.org/'>Graphviz</ulink>).
            <note><title>Notes</title>
                <itemizedlist>
                    <listitem><para>
                        DOT files use a plain text format.
                        The graphs generated using the
                        <filename>bitbake -g</filename> command are often so
                        large as to be difficult to read without special
                        pruning (e.g. with Bitbake's
                        <filename>-I</filename> option) and processing.
                        Despite the form and size of the graphs, the
                        corresponding <filename>.dot</filename> files can still
                        be possible to read and provide useful information.
                        </para>

                        <para>As an example, the
                        <filename>task-depends.dot</filename> file contains
                        lines such as the following:
                        <literallayout class='monospaced'>
     "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
                        </literallayout>
                        The above example line reveals that the
                        <link linkend='ref-tasks-configure'><filename>do_configure</filename></link>
                        task in <filename>libxslt</filename> depends on the
                        <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
                        task in <filename>libxml2</filename>, which is a normal
                        <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
                        dependency between the two recipes.
                        </para></listitem>
                    <listitem><para>
                        For an example of how <filename>.dot</filename> files
                        can be processed, see the
                        <filename>scripts/contrib/graph-tool</filename> Python
                        script, which finds and displays paths between graph
                        nodes.
                        </para></listitem>
                </itemizedlist>
            </note>
        </para>

        <para>
            You can use a different method to view dependency information
            by using the following command:
            <literallayout class='monospaced'>
     $ bitbake -g -u depexp <replaceable>recipename</replaceable>
            </literallayout>
            This command displays a GUI window from which you can view
            build-time and runtime dependencies for the recipes involved in
            building <replaceable>recipename</replaceable>.
        </para>
    </section>

    <section id='usingpoky-viewing-task-variable-dependencies'>
        <title>Viewing Task Variable Dependencies</title>

        <para>
            As mentioned in the
            "<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>Checksums (Signatures)</ulink>"
            section of the BitBake User Manual, BitBake tries to automatically
            determine what variables a task depends on so that it can rerun
            the task if any values of the variables change.
            This determination is usually reliable.
            However, if you do things like construct variable names at runtime,
            then you might have to manually declare dependencies on those
            variables using <filename>vardeps</filename> as described in the
            "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
            section of the BitBake User Manual.
        </para>

        <para>
            If you are unsure whether a variable dependency is being picked up
            automatically for a given task, you can list the variable
            dependencies BitBake has determined by doing the following:
            <orderedlist>
                <listitem><para>
                    Build the recipe containing the task:
                    <literallayout class='monospaced'>
     $ bitbake <replaceable>recipename</replaceable>
                    </literallayout>
                    </para></listitem>
                <listitem><para>
                    Inside the
                    <link linkend='var-STAMPS_DIR'><filename>STAMPS_DIR</filename></link>
                    directory, find the signature data
                    (<filename>sigdata</filename>) file that corresponds to the
                    task.
                    The <filename>sigdata</filename> files contain a pickled
                    Python database of all the metadata that went into creating
                    the input checksum for the task.
                    As an example, for the
                    <link linkend='ref-tasks-fetch'><filename>do_fetch</filename></link>
                    task of the <filename>db</filename> recipe, the
                    <filename>sigdata</filename> file might be found in the
                    following location:
                    <literallayout class='monospaced'>
     ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
                    </literallayout>
                    For tasks that are accelerated through the shared state
                    (<link linkend='shared-state-cache'>sstate</link>)
                    cache, an additional <filename>siginfo</filename> file is
                    written into
                    <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
                    along with the cached task output.
                    The <filename>siginfo</filename> files contain exactly the
                    same information as <filename>sigdata</filename> files.
                    </para></listitem>
                <listitem><para>
                    Run <filename>bitbake-dumpsig</filename> on the
                    <filename>sigdata</filename> or
                    <filename>siginfo</filename> file.
                    Here is an example:
                    <literallayout class='monospaced'>
     $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
                    </literallayout>
                    In the output of the above command, you will find a line
                    like the following, which lists all the (inferred) variable
                    dependencies for the task.
                    This list also includes indirect dependencies from
                    variables depending on other variables, recursively.
                    <literallayout class='monospaced'>
     Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
                    </literallayout>
                    <note>
                        Functions (e.g. <filename>base_do_fetch</filename>)
                        also count as variable dependencies.
                        These functions in turn depend on the variables they
                        reference.
                    </note>
                    The output of <filename>bitbake-dumpsig</filename> also includes
                    the value each variable had, a list of dependencies for each
                    variable, and
                    <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></ulink>
                    information.
                    </para></listitem>
            </orderedlist>
        </para>

        <para>
            There is also a <filename>bitbake-diffsigs</filename> command for
            comparing two <filename>siginfo</filename> or
            <filename>sigdata</filename> files.
            This command can be helpful when trying to figure out what changed
            between two versions of a task.
            If you call <filename>bitbake-diffsigs</filename> with just one
            file, the command behaves like
            <filename>bitbake-dumpsig</filename>.
        </para>

        <para>
            You can also use BitBake to dump out the signature construction
            information without executing tasks by using either of the
            following BitBake command-line options:
            <literallayout class='monospaced'>
     &dash;&dash;dump-signatures=<replaceable>SIGNATURE_HANDLER</replaceable>
     -S <replaceable>SIGNATURE_HANDLER</replaceable>
            </literallayout>
            <note>
                Two common values for
                <replaceable>SIGNATURE_HANDLER</replaceable> are "none" and
                "printdiff", which dump only the signature or compare the
                dumped signature with the cached one, respectively.
            </note>
            Using BitBake with either of these options causes BitBake to dump
            out <filename>sigdata</filename> files in the
            <filename>stamps</filename> directory for every task it would have
            executed instead of building the specified target package.
        </para>
    </section>

    <section id='usingpoky-debugging-taskrunning'>
        <title>Running Specific Tasks</title>

        <para>
            Any given recipe consists of a set of tasks.
            The standard BitBake behavior in most cases is:
            <filename>do_fetch</filename>,
            <filename>do_unpack</filename>,
            <filename>do_patch</filename>, <filename>do_configure</filename>,
            <filename>do_compile</filename>, <filename>do_install</filename>,
            <filename>do_package</filename>,
            <filename>do_package_write_*</filename>, and
            <filename>do_build</filename>.
            The default task is <filename>do_build</filename> and any tasks
            on which it depends build first.
            Some tasks, such as <filename>do_devshell</filename>, are not part
            of the default build chain.
            If you wish to run a task that is not part of the default build
            chain, you can use the <filename>-c</filename> option in BitBake.
            Here is an example:
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop -c devshell
            </literallayout>
        </para>

        <para>
            If you wish to rerun a task, use the <filename>-f</filename> force
            option.
            For example, the following sequence forces recompilation after
            changing files in the work directory.
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop
               .
               .
        <replaceable>make some changes to the source code in the work directory</replaceable>
               .
               .
     $ bitbake matchbox-desktop -c compile -f
     $ bitbake matchbox-desktop
            </literallayout>
        </para>

        <para>
            This sequence first builds and then recompiles
            <filename>matchbox-desktop</filename>.
            The last command reruns all tasks (basically the packaging tasks)
            after the compile.
            BitBake recognizes that the <filename>do_compile</filename>
            task was rerun and therefore understands that the other tasks
            also need to be run again.
        </para>

        <para>
            You can view a list of tasks in a given package by running the
            <filename>do_listtasks</filename> task as follows:
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop -c listtasks
            </literallayout>
            The results appear as output to the console and are also in the
            file <filename>${WORKDIR}/temp/log.do_listtasks</filename>.
        </para>
    </section>

    <section id='checking-for-missing-build-time-dependencies'>
        <title>Checking for Missing Build-Time Dependencies</title>

        <para>
            A recipe might build successfully even though some of its
            build-time dependencies are missing from
            <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>.
            Following are the two most common ways in which that can happen:
            <itemizedlist>
                <listitem><para>
                    The build-time dependency just happens to already exist in
                    the staging sysroot
                    (<link linkend='var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></link>)
                    by the time the recipe is built.
                    This situation occurs when the build-time dependency is
                    built earlier during recipe processing.
                    </para></listitem>
                <listitem><para>
                    The component built by the recipe conditionally enables
                    functionality depending on whether it can find the
                    build-time dependency in the staging sysroot.
                    If the build-time dependency is missing, the corresponding
                    functionality is disabled.
                    This condition is known as a "floating dependency".
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            Because dealing with the second case is more complex, focus will
            be on the first case.
            The
            <link linkend='ref-classes-insane'><filename>build-deps</filename></link>
            QA check checks that every library the component linked against is
            declared as a build-time dependency.
            If that is not the case, then the first situation described in the
            previous list exists, and <filename>build-deps</filename> reports
            a missing build-time dependency.
        </para>

        <para>
            Another, more manual, way to check a recipe for missing build-time
            dependencies of the first type is to build with an empty staging
            sysroot.
            This method can also find missing build-time dependencies
            that are not in the form of libraries, which the
            <filename>build-deps</filename> QA check is unable to find.
        </para>

        <para>
            An easy way to empty the staging sysroots is to simply remove
            <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>,
            which is usually
            <filename>${</filename><link linkend='var-BUILDDIR'><filename>BUILDDIR</filename></link><filename>}/tmp</filename>,
            as it includes the staging sysroots.
            Another, faster method to empty the staging sysroots is to use the
            <filename>scripts/wipe-sysroot</filename> script,
            which removes just the staging sysroots and keeps everything else
            in <filename>TMPDIR</filename>.
            <note>
                The <filename>scripts/</filename> directory appears in
                <filename>PATH</filename> after running the build environment
                initialization script (i.e.
                <link linkend='structure-core-script'><filename>oe-init-build-env</filename></link>
                or
                <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>),
                which results in the ability to to run
                <filename>wipe-sysroot</filename> immediately.
            </note>
        </para>

    </section>

    <section id='usingpoky-debugging-bitbake'>
        <title>General BitBake Problems</title>

        <para>
            You can see debug output from BitBake by using the <filename>-D</filename> option.
            The debug output gives more information about what BitBake
            is doing and the reason behind it.
            Each <filename>-D</filename> option you use increases the logging level.
            The most common usage is <filename>-DDD</filename>.
        </para>

        <para>
            The output from <filename>bitbake -DDD -v</filename> <replaceable>targetname</replaceable> can reveal why
            BitBake chose a certain version of a package or why BitBake
            picked a certain provider.
            This command could also help you in a situation where you think BitBake did something
            unexpected.
        </para>
    </section>

    <section id='development-host-system-issues'>
        <title>Development Host System Issues</title>

        <para>
            Sometimes issues on the host development system can cause your
            build to fail.
            Following are known, host-specific problems.
            Be sure to always consult the
            <ulink url='&YOCTO_RELEASE_NOTES;'>Release Notes</ulink>
            for a look at all release-related issues.
            <itemizedlist>
                <listitem><para><emphasis><filename>glibc-initial</filename> fails to build</emphasis>:
                    If your development host system has the unpatched
                    <filename>GNU Make 3.82</filename>,
                    the
                    <link linkend='ref-tasks-install'><filename>do_install</filename></link>
                    task fails for <filename>glibc-initial</filename> during
                    the build.</para>
                    <para>Typically, every distribution that ships
                    <filename>GNU Make 3.82</filename> as
                    the default already has the patched version.
                    However, some distributions, such as Debian, have
                    <filename>GNU Make 3.82</filename> as an option, which
                    is unpatched.
                    You will see this error on these types of distributions.
                    Switch to <filename>GNU Make 3.81</filename> or patch
                    your <filename>make</filename> to solve the problem.
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>

    <section id='usingpoky-debugging-buildfile'>
        <title>Building with No Dependencies</title>
        <para>
            To build a specific recipe (<filename>.bb</filename> file),
            you can use the following command form:
            <literallayout class='monospaced'>
     $ bitbake -b <replaceable>somepath</replaceable>/<replaceable>somerecipe</replaceable>.bb
            </literallayout>
            This command form does not check for dependencies.
            Consequently, you should use it
            only when you know existing dependencies have been met.
            <note>
                You can also specify fragments of the filename.
                In this case, BitBake checks for a unique match.
            </note>
        </para>
    </section>

    <section id='recipe-logging-mechanisms'>
        <title>Recipe Logging Mechanisms</title>
        <para>
            The Yocto Project provides several logging functions for producing
            debugging output and reporting errors and warnings.
            For Python functions, the following logging functions exist.
            All of these functions log to
            <filename>${T}/log.do_</filename><replaceable>task</replaceable>,
            and can also log to standard output (stdout) with the right
            settings:
            <itemizedlist>
                <listitem><para>
                    <filename>bb.plain(</filename><replaceable>msg</replaceable><filename>)</filename>:
                    Writes <replaceable>msg</replaceable> as is to the log while
                    also logging to stdout.
                    </para></listitem>
                <listitem><para>
                    <filename>bb.note(</filename><replaceable>msg</replaceable><filename>)</filename>:
                    Writes "NOTE: <replaceable>msg</replaceable>" to the log.
                    Also logs to stdout if BitBake is called with "-v".
                    </para></listitem>
                <listitem><para>
                    <filename>bb.debug(</filename><replaceable>level</replaceable><filename>,&nbsp;</filename><replaceable>msg</replaceable><filename>)</filename>:
                    Writes "DEBUG: <replaceable>msg</replaceable>" to the log.
                    Also logs to stdout if the log level is greater than or
                    equal to <replaceable>level</replaceable>.
                    See the
                    "<ulink url='&YOCTO_DOCS_BB_URL;#usage-and-syntax'>-D</ulink>"
                    option in the BitBake User Manual for more information.
                    </para></listitem>
                <listitem><para>
                    <filename>bb.warn(</filename><replaceable>msg</replaceable><filename>)</filename>:
                    Writes "WARNING: <replaceable>msg</replaceable>" to the log
                    while also logging to stdout.
                    </para></listitem>
                <listitem><para>
                    <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>:
                    Writes "ERROR: <replaceable>msg</replaceable>" to the log
                    while also logging to stdout.
                    <note>
                        Calling this function does not cause the task to fail.
                    </note>
                    </para></listitem>
                <listitem><para>
                    <filename>bb.fatal(</filename><replaceable>msg</replaceable><filename>)</filename>:
                    This logging function is similar to
                    <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>
                    but also causes the calling task to fail.
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            The same logging functions are also available in shell functions,
            under the names
            <filename>bbplain</filename>, <filename>bbnote</filename>,
            <filename>bbdebug</filename>, <filename>bbwarn</filename>,
            <filename>bberror</filename>, and <filename>bbfatal</filename>.
            The
            <link linkend='ref-classes-logging'><filename>logging</filename></link>
            class implements these functions.
            See that class in the
            <filename>meta/classes</filename> folder of the
            <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>
            for information.
        </para>

        <section id='logging-with-python'>
            <title>Logging With Python</title>
            <para>
                When creating recipes using Python and inserting code that handles build logs,
                keep in mind the goal is to have informative logs while keeping the console as
                "silent" as possible.
                Also, if you want status messages in the log, use the "debug" loglevel.
            </para>

            <para>
                Following is an example written in Python.
                The code handles logging for a function that determines the
                number of tasks needed to be run.
                See the
                "<link linkend='ref-tasks-listtasks'><filename>do_listtasks</filename></link>"
                section for additional information:
                <literallayout class='monospaced'>
     python do_listtasks() {
         bb.debug(2, "Starting to figure out the task list")
         if noteworthy_condition:
             bb.note("There are 47 tasks to run")
         bb.debug(2, "Got to point xyz")
         if warning_trigger:
             bb.warn("Detected warning_trigger, this might be a problem later.")
         if recoverable_error:
             bb.error("Hit recoverable_error, you really need to fix this!")
         if fatal_error:
             bb.fatal("fatal_error detected, unable to print the task list")
         bb.plain("The tasks present are abc")
         bb.debug(2, "Finished figuring out the tasklist")
     }
                </literallayout>
            </para>
        </section>

        <section id='logging-with-bash'>
            <title>Logging With Bash</title>
            <para>
                When creating recipes using Bash and inserting code that handles build
                logs, you have the same goals - informative with minimal console output.
                The syntax you use for recipes written in Bash is similar to that of
                recipes written in Python described in the previous section.
            </para>

            <para>
                Following is an example written in Bash.
                The code logs the progress of the <filename>do_my_function</filename> function.
                <literallayout class='monospaced'>
     do_my_function() {
         bbdebug 2 "Running do_my_function"
         if [ exceptional_condition ]; then
             bbnote "Hit exceptional_condition"
         fi
         bbdebug 2  "Got to point xyz"
         if [ warning_trigger ]; then
             bbwarn "Detected warning_trigger, this might cause a problem later."
         fi
         if [ recoverable_error ]; then
             bberror "Hit recoverable_error, correcting"
         fi
         if [ fatal_error ]; then
             bbfatal "fatal_error detected"
         fi
         bbdebug 2 "Completed do_my_function"
     }
                </literallayout>
            </para>
        </section>
    </section>

    <section id='usingpoky-debugging-others'>
        <title>Other Tips</title>

        <para>
            Here are some other tips that you might find useful:
            <itemizedlist>
                <listitem><para>
                    When adding new packages, it is worth watching for
                    undesirable items making their way into compiler command
                    lines.
                    For example, you do not want references to local system
                    files like
                    <filename>/usr/lib/</filename> or
                    <filename>/usr/include/</filename>.
                    </para></listitem>
                <listitem><para>
                    If you want to remove the <filename>psplash</filename>
                    boot splashscreen,
                    add <filename>psplash=false</filename> to  the kernel
                    command line.
                    Doing so prevents <filename>psplash</filename> from loading
                    and thus allows you to see the console.
                    It is also possible to switch out of the splashscreen by
                    switching the virtual console (e.g. Fn+Left or Fn+Right
                    on a Zaurus).
                    </para></listitem>
                <listitem><para>
                    Removing
                    <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
                    (usually <filename>tmp/</filename>, within the
                    <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>)
                    can often fix temporary build issues.
                    Removing <filename>TMPDIR</filename> is usually a
                    relatively cheap operation, because task output will be
                    cached in
                    <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
                    (usually <filename>sstate-cache/</filename>, which is
                    also in the Build Directory).
                    <note>
                        Removing <filename>TMPDIR</filename> might be a
                        workaround rather than a fix.
                        Consequently, trying to determine the underlying cause
                        of an issue before removing the directory is a good
                        ideal.
                    </note>
                    </para></listitem>
                <listitem><para>
                    Understanding how a feature is used in practice within
                    existing recipes can be very helpful.
                    It is recommended that you configure some method that
                    allows you to quickly search through files.</para>

                    <para>Using GNU Grep, you can use the following shell
                    function to recursively search through common
                    recipe-related files, skipping binary files,
                    <filename>.git</filename> directories, and the
                    Build Directory (assuming its name starts with
                    "build"):
                    <literallayout class='monospaced'>
     g() {
         grep -Ir \
              --exclude-dir=.git \
              --exclude-dir='build*' \
              --include='*.bb*' \
              --include='*.inc*' \
              --include='*.conf*' \
              --include='*.py*' \
              "$@"
     }
                    </literallayout>
                    Following are some usage examples:
                    <literallayout class='monospaced'>
     $ g FOO    # Search recursively for "FOO"
     $ g -i foo # Search recursively for "foo", ignoring case
     $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
                    </literallayout>
                    If figuring out how some feature works requires a lot of
                    searching, it might indicate that the documentation should
                    be extended or improved.
                    In such cases, consider filing a documentation bug using
                    the Yocto Project implementation of
                    <ulink url='https://bugzilla.yoctoproject.org/'>Bugzilla</ulink>.
                    For general information on how to submit a bug against
                    the Yocto Project, see the
                    "<ulink url='&YOCTO_DOCS_DEV_URL;#tracking-bugs'>Tracking Bugs</ulink>"
                    section in the Yocto Project Development Manual.
                    <note>
                        The manuals might not be the right place to document
                        variables that are purely internal and have a limited
                        scope (e.g. internal variables used to implement a
                        single <filename>.bbclass</filename> file).
                    </note>
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>
</section>

<section id='maintaining-build-output-quality'>
    <title>Maintaining Build Output Quality</title>

    <para>
        Many factors can influence the quality of a build.
        For example, if you upgrade a recipe to use a new version of an upstream software
        package or you experiment with some new configuration options, subtle changes
        can occur that you might not detect until later.
        Consider the case where your recipe is using a newer version of an upstream package.
        In this case, a new version of a piece of software might introduce an optional
        dependency on another library, which is auto-detected.
        If that library has already been built when the software is building,
        the software will link to the built library and that library will be pulled
        into your image along with the new software even if you did not want the
        library.
    </para>

    <para>
        The
        <link linkend='ref-classes-buildhistory'><filename>buildhistory</filename></link>
        class exists to help you maintain
        the quality of your build output.
        You can use the class to highlight unexpected and possibly unwanted
        changes in the build output.
        When you enable build history, it records information about the contents of
        each package and image and then commits that information to a local Git
        repository where you can examine the information.
    </para>

    <para>
        The remainder of this section describes the following:
        <itemizedlist>
           <listitem><para>How you can enable and disable
               build history</para></listitem>
           <listitem><para>How to understand what the build history contains
               </para></listitem>
           <listitem><para>How to limit the information used for build history
               </para></listitem>
           <listitem><para>How to examine the build history from both a
               command-line and web interface</para></listitem>
       </itemizedlist>
    </para>

    <section id='enabling-and-disabling-build-history'>
        <title>Enabling and Disabling Build History</title>

        <para>
            Build history is disabled by default.
            To enable it, add the following <filename>INHERIT</filename>
            statement and set the
            <link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></link>
            variable to "1" at the end of your
            <filename>conf/local.conf</filename> file found in the
            <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>:
            <literallayout class='monospaced'>
     INHERIT += "buildhistory"
     BUILDHISTORY_COMMIT = "1"
            </literallayout>
            Enabling build history as previously described
            causes the build process to collect build
            output information and commit it to a local
            <ulink url='&YOCTO_DOCS_DEV_URL;#git'>Git</ulink> repository.
            <note>
                Enabling build history increases your build times slightly,
                particularly for images, and increases the amount of disk
                space used during the build.
            </note>
        </para>

        <para>
            You can disable build history by removing the previous statements
            from your <filename>conf/local.conf</filename> file.
        </para>
    </section>

    <section id='understanding-what-the-build-history-contains'>
        <title>Understanding What the Build History Contains</title>

        <para>
            Build history information is kept in
            <filename>${</filename><link linkend='var-TOPDIR'><filename>TOPDIR</filename></link><filename>}/buildhistory</filename>
            in the Build Directory as defined by the
            <link linkend='var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></link>
            variable.
            The following is an example abbreviated listing:
            <imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" />
        </para>

        <para>
            At the top level, there is a <filename>metadata-revs</filename> file
            that lists the revisions of the repositories for the layers enabled
            when the build was produced.
            The rest of the data splits into separate
            <filename>packages</filename>, <filename>images</filename> and
            <filename>sdk</filename> directories, the contents of which are
            described below.
        </para>

        <section id='build-history-package-information'>
            <title>Build History Package Information</title>

            <para>
                The history for each package contains a text file that has
                name-value pairs with information about the package.
                For example, <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename>
                contains the following:
                <literallayout class='monospaced'>
     PV = 1.22.1
     PR = r32
     RPROVIDES =
     RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
     RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
     PKGSIZE = 540168
     FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
        /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
        /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
        /usr/share/pixmaps /usr/share/applications /usr/share/idl \
        /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
     FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
        /etc/busybox.links.nosuid /etc/busybox.links.suid
                </literallayout>
                Most of these name-value pairs correspond to variables used
                to produce the package.
                The exceptions are <filename>FILELIST</filename>, which is the
                actual list of files in the package, and
                <filename>PKGSIZE</filename>, which is the total size of files
                in the package in bytes.
            </para>

            <para>
                There is also a file corresponding to the recipe from which the
                package came (e.g.
                <filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>):
                <literallayout class='monospaced'>
     PV = 1.22.1
     PR = r32
     DEPENDS = initscripts kern-tools-native update-rc.d-native \
        virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
        virtual/libc virtual/update-alternatives
     PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
        busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
        busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
                </literallayout>
            </para>

            <para>
                Finally, for those recipes fetched from a version control
                system (e.g., Git), a file exists that lists source revisions
                that are specified in the recipe and lists the actual revisions
                used during the build.
                Listed and actual revisions might differ when
                <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
                is set to
                <filename>${<link linkend='var-AUTOREV'>AUTOREV</link>}</filename>.
                Here is an example assuming
                <filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>):
                <literallayout class='monospaced'>
     # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
     SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
     # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
     SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
                </literallayout>
                You can use the <filename>buildhistory-collect-srcrevs</filename>
                command with the <filename>-a</filename> option to
                collect the stored <filename>SRCREV</filename> values
                from build history and report them in a format suitable for
                use in global configuration (e.g.,
                <filename>local.conf</filename> or a distro include file) to
                override floating <filename>AUTOREV</filename> values to a
                fixed set of revisions.
                Here is some example output from this command:
                <literallayout class='monospaced'>
     $ buildhistory-collect-srcrevs -a
     # i586-poky-linux
     SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
     SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
     SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
     SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
     # x86_64-linux
     SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
     SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
     SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
     SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
     SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
     SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
     SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
     SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
     SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
     # qemux86-poky-linux
     SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
     SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
     # all-poky-linux
     SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
                </literallayout>
                <note>
                    Here are some notes on using the
                    <filename>buildhistory-collect-srcrevs</filename> command:
                    <itemizedlist>
                        <listitem><para>By default, only values where the
                            <filename>SRCREV</filename> was
                            not hardcoded (usually when <filename>AUTOREV</filename>
                            was used) are reported.
                            Use the <filename>-a</filename> option to see all
                            <filename>SRCREV</filename> values.
                            </para></listitem>
                        <listitem><para>The output statements might not have any effect
                            if overrides are applied elsewhere in the build system
                            configuration.
                            Use the <filename>-f</filename> option to add the
                            <filename>forcevariable</filename> override to each output line
                            if you need to work around this restriction.
                            </para></listitem>
                        <listitem><para>The script does apply special handling when
                            building for multiple machines.
                            However, the script does place a
                            comment before each set of values that specifies
                            which triplet to which they belong as shown above
                            (e.g., <filename>i586-poky-linux</filename>).
                            </para></listitem>
                    </itemizedlist>
                </note>
            </para>
        </section>

        <section id='build-history-image-information'>
            <title>Build History Image Information</title>

            <para>
                The files produced for each image are as follows:
                <itemizedlist>
                    <listitem><para><filename>image-files:</filename>
                        A directory containing selected files from the root
                        filesystem.
                        The files are defined by
                        <link linkend='var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></link>.
                        </para></listitem>
                    <listitem><para><filename>build-id.txt:</filename>
                        Human-readable information about the build configuration
                        and metadata source revisions.
                        This file contains the full build header as printed
                        by BitBake.</para></listitem>
                    <listitem><para><filename>*.dot:</filename>
                        Dependency graphs for the image that are
                        compatible with <filename>graphviz</filename>.
                        </para></listitem>
                    <listitem><para><filename>files-in-image.txt:</filename>
 	                    A list of files in the image with permissions,
                        owner, group, size, and symlink information.
                        </para></listitem>
                    <listitem><para><filename>image-info.txt:</filename>
                        A text file containing name-value pairs with information
                        about the image.
                        See the following listing example for more information.
                        </para></listitem>
                    <listitem><para><filename>installed-package-names.txt:</filename>
                        A list of installed packages by name only.</para></listitem>
                    <listitem><para><filename>installed-package-sizes.txt:</filename>
                        A list of installed packages ordered by size.
                        </para></listitem>
                    <listitem><para><filename>installed-packages.txt:</filename>
                        A list of installed packages with full package
                        filenames.</para></listitem>
                </itemizedlist>
                <note>
                    Installed package information is able to be gathered and
                    produced even if package management is disabled for the final
                    image.
                </note>
            </para>

            <para>
                Here is an example of <filename>image-info.txt</filename>:
                <literallayout class='monospaced'>
     DISTRO = poky
     DISTRO_VERSION = 1.7
     USER_CLASSES = buildstats image-mklibs image-prelink
     IMAGE_CLASSES = image_types
     IMAGE_FEATURES = debug-tweaks
     IMAGE_LINGUAS =
     IMAGE_INSTALL = packagegroup-core-boot run-postinsts
     BAD_RECOMMENDATIONS =
     NO_RECOMMENDATIONS =
     PACKAGE_EXCLUDE =
     ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
        write_image_manifest ; buildhistory_list_installed_image ; \
        buildhistory_get_image_installed ; ssh_allow_empty_password;  \
        postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
     IMAGE_POSTPROCESS_COMMAND =   buildhistory_get_imageinfo ;
     IMAGESIZE = 6900
                </literallayout>
                Other than <filename>IMAGESIZE</filename>, which is the
                total size of the files in the image in Kbytes, the
                name-value pairs are variables that may have influenced the
                content of the image.
                This information is often useful when you are trying to determine
                why a change in the package or file listings has occurred.
            </para>
        </section>

        <section id='using-build-history-to-gather-image-information-only'>
            <title>Using Build History to Gather Image Information Only</title>

            <para>
                As you can see, build history produces image information,
                including dependency graphs, so you can see why something
                was pulled into the image.
                If you are just interested in this information and not
                interested in collecting specific package or SDK information,
                you can enable writing only image information without
                any history by adding the following to your
                <filename>conf/local.conf</filename> file found in the
                <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>:
                <literallayout class='monospaced'>
     INHERIT += "buildhistory"
     BUILDHISTORY_COMMIT = "0"
     BUILDHISTORY_FEATURES = "image"
                </literallayout>
                Here, you set the
                <link linkend='var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></link>
                variable to use the image feature only.
            </para>
        </section>

        <section id='build-history-sdk-information'>
            <title>Build History SDK Information</title>

            <para>
                Build history collects similar information on the contents
                of SDKs
                (e.g. <filename>bitbake -c populate_sdk imagename</filename>)
                as compared to information it collects for images.
                Furthermore, this information differs depending on whether an
                extensible or standard SDK is being produced.
            </para>

            <para>
                The following list shows the files produced for SDKs:
                <itemizedlist>
                    <listitem><para><filename>files-in-sdk.txt:</filename>
                        A list of files in the SDK with permissions,
                        owner, group, size, and symlink information.
                        This list includes both the host and target parts
                        of the SDK.
                        </para></listitem>
                    <listitem><para><filename>sdk-info.txt:</filename>
                        A text file containing name-value pairs with information
                        about the SDK.
                        See the following listing example for more information.
                        </para></listitem>
                    <listitem><para><filename>sstate-task-sizes.txt:</filename>
                        A text file containing name-value pairs with information
                        about task group sizes
                        (e.g. <filename>do_populate_sysroot</filename> tasks
                        have a total size).
                        The <filename>sstate-task-sizes.txt</filename> file
                        exists only when an extensible SDK is created.
                        </para></listitem>
                    <listitem><para><filename>sstate-package-sizes.txt:</filename>
                        A text file containing name-value pairs with information
                        for the shared-state packages and sizes in the SDK.
                        The <filename>sstate-package-sizes.txt</filename> file
                        exists only when an extensible SDK is created.
                        </para></listitem>
                    <listitem><para><filename>sdk-files:</filename>
                        A folder that contains copies of the files mentioned in
                        <filename>BUILDHISTORY_SDK_FILES</filename> if the
                        files are present in the output.
                        Additionally, the default value of
                        <filename>BUILDHISTORY_SDK_FILES</filename> is specific
                        to the extensible SDK although you can set it
                        differently if you would like to pull in specific files
                        from the standard SDK.</para>
                        <para>The default files are
                        <filename>conf/local.conf</filename>,
                        <filename>conf/bblayers.conf</filename>,
                        <filename>conf/auto.conf</filename>,
                        <filename>conf/locked-sigs.inc</filename>, and
                        <filename>conf/devtool.conf</filename>.
                        Thus, for an extensible SDK, these files get copied
                        into the <filename>sdk-files</filename> directory.
                        </para></listitem>
                    <listitem><para>The following information appears under
                        each of the <filename>host</filename>
                        and <filename>target</filename> directories
                        for the portions of the SDK that run on the host and
                        on the target, respectively:
                        <note>
                            The following files for the most part are empty
                            when producing an extensible SDK because this
                            type of SDK is not constructed from packages as is
                            the standard SDK.
                        </note>
                        <itemizedlist>
                            <listitem><para><filename>depends.dot:</filename>
                                Dependency graph for the SDK that is
                                compatible with <filename>graphviz</filename>.
                                </para></listitem>
                            <listitem><para><filename>installed-package-names.txt:</filename>
                                A list of installed packages by name only.
                                </para></listitem>
                            <listitem><para><filename>installed-package-sizes.txt:</filename>
                                A list of installed packages ordered by size.
                                </para></listitem>
                            <listitem><para><filename>installed-packages.txt:</filename>
                                A list of installed packages with full package
                                filenames.</para></listitem>
                            </itemizedlist>
                        </para></listitem>
                </itemizedlist>
            </para>

            <para>
                Here is an example of <filename>sdk-info.txt</filename>:
                <literallayout class='monospaced'>
     DISTRO = poky
     DISTRO_VERSION = 1.3+snapshot-20130327
     SDK_NAME = poky-glibc-i686-arm
     SDK_VERSION = 1.3+snapshot
     SDKMACHINE =
     SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
     BAD_RECOMMENDATIONS =
     SDKSIZE = 352712
                </literallayout>
                Other than <filename>SDKSIZE</filename>, which is the
                total size of the files in the SDK in Kbytes, the
                name-value pairs are variables that might have influenced the
                content of the SDK.
                This information is often useful when you are trying to
                determine why a change in the package or file listings
                has occurred.
            </para>
        </section>

        <section id='examining-build-history-information'>
            <title>Examining Build History Information</title>

            <para>
                You can examine build history output from the command line or
                from a web interface.
            </para>

            <para>
                To see any changes that have occurred (assuming you have
                <link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT = "1"</filename></link>),
                you can simply
                use any Git command that allows you to view the history of
                a repository.
                Here is one method:
                <literallayout class='monospaced'>
      $ git log -p
                </literallayout>
                You need to realize, however, that this method does show
                changes that are not significant (e.g. a package's size
                changing by a few bytes).
            </para>

            <para>
                A command-line tool called <filename>buildhistory-diff</filename>
                does exist, though, that queries the Git repository and prints just
                the differences that might be significant in human-readable form.
                Here is an example:
                <literallayout class='monospaced'>
     $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
     Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
        /etc/anotherpkg.conf was added
        /sbin/anotherpkg was added
        * (installed-package-names.txt):
        *   anotherpkg was added
     Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
        anotherpkg was added
     packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
        * PR changed from "r0" to "r1"
        * PV changed from "0.1.10" to "0.1.12"
     packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
        * PR changed from "r0" to "r1"
        * PV changed from "0.1.10" to "0.1.12"
                </literallayout>
                <note>
                    The <filename>buildhistory-diff</filename> tool requires
                    the <filename>GitPython</filename> package.
                    Be sure to install it using Pip3 as follows:
                    <literallayout class='monospaced'>
   $ pip3 install GitPython --user
                    </literallayout>
                    Alternatively, you can install
                    <filename>python3-git</filename> using the appropriate
                    distribution package manager (e.g.
                    <filename>apt-get</filename>, <filename>dnf</filename>, or
                    <filename>zipper</filename>).
                </note>
            </para>

            <para>
                To see changes to the build history using a web interface, follow
                the instruction in the <filename>README</filename> file here.
                <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>.
            </para>

            <para>
                Here is a sample screenshot of the interface:
                <imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" />
            </para>
        </section>
    </section>
</section>

<section id='speeding-up-the-build'>
    <title>Speeding Up the Build</title>

    <para>
        Build time can be an issue.
        By default, the build system uses simple controls to try and maximize
        build efficiency.
        In general, the default settings for all the following variables
        result in the most efficient build times when dealing with single
        socket systems (i.e. a single CPU).
        If you have multiple CPUs, you might try increasing the default
        values to gain more speed.
        See the descriptions in the glossary for each variable for more
        information:
        <itemizedlist>
            <listitem><para>
                <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename>:</link>
                The maximum number of threads BitBake simultaneously executes.
                </para></listitem>
            <listitem><para>
                <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename>:</ulink>
                The number of threads BitBake uses during parsing.
                </para></listitem>
            <listitem><para>
                <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename>:</link>
                Extra options passed to the <filename>make</filename> command
                during the
                <link linkend='ref-tasks-compile'><filename>do_compile</filename></link>
                task in order to specify parallel compilation on the
                local build host.
                </para></listitem>
            <listitem><para>
                <link linkend='var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename>:</link>
                Extra options passed to the <filename>make</filename> command
                during the
                <link linkend='ref-tasks-install'><filename>do_install</filename></link>
                task in order to specify parallel installation on the
                local build host.
                </para></listitem>
        </itemizedlist>
        As mentioned, these variables all scale to the number of processor
        cores available on the build system.
        For single socket systems, this auto-scaling ensures that the build
        system fundamentally takes advantage of potential parallel operations
        during the build based on the build machine's capabilities.
    </para>

    <para>
        Following are additional factors that can affect build speed:
        <itemizedlist>
            <listitem><para>
                File system type:
                The file system type that the build is being performed on can
                also influence performance.
                Using <filename>ext4</filename> is recommended as compared
                to <filename>ext2</filename> and <filename>ext3</filename>
                due to <filename>ext4</filename> improved features
                such as extents.
                </para></listitem>
            <listitem><para>
                Disabling the updating of access time using
                <filename>noatime</filename>:
                The <filename>noatime</filename> mount option prevents the
                build system from updating file and directory access times.
                </para></listitem>
            <listitem><para>
                Setting a longer commit:
                Using the "commit=" mount option increases the interval
                in seconds between disk cache writes.
                Changing this interval from the five second default to
                something longer increases the risk of data loss but decreases
                the need to write to the disk, thus increasing the build
                performance.
                </para></listitem>
            <listitem><para>
                Choosing the packaging backend:
                Of the available packaging backends, IPK is the fastest.
                Additionally, selecting a singular packaging backend also
                helps.
                </para></listitem>
            <listitem><para>
                Using <filename>tmpfs</filename> for
                <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
                as a temporary file system:
                While this can help speed up the build, the benefits are
                limited due to the compiler using
                <filename>-pipe</filename>.
                The build system goes to some lengths to avoid
                <filename>sync()</filename> calls into the
                file system on the principle that if there was a significant
                failure, the
                <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
                contents could easily be rebuilt.
                </para></listitem>
            <listitem><para>
                Inheriting the
                <link linkend='ref-classes-rm-work'><filename>rm_work</filename></link>
                class:
                Inheriting this class has shown to speed up builds due to
                significantly lower amounts of data stored in the data
                cache as well as on disk.
                Inheriting this class also makes cleanup of
                <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
                faster, at the expense of being easily able to dive into the
                source code.
                File system maintainers have recommended that the fastest way
                to clean up large numbers of files is to reformat partitions
                rather than delete files due to the linear nature of partitions.
                This, of course, assumes you structure the disk partitions and
                file systems in a way that this is practical.
                </para></listitem>
        </itemizedlist>
        Aside from the previous list, you should keep some trade offs in
        mind that can help you speed up the build:
        <itemizedlist>
            <listitem><para>
                Remove items from
                <link linkend='var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></link>
                that you might not need.
                </para></listitem>
            <listitem><para>
                Exclude debug symbols and other debug information:
                If you do not need these symbols and other debug information,
                disabling the <filename>*-dbg</filename> package generation
                can speed up the build.
                You can disable this generation by setting the
                <link linkend='var-INHIBIT_PACKAGE_DEBUG_SPLIT'><filename>INHIBIT_PACKAGE_DEBUG_SPLIT</filename></link>
                variable to "1".
                </para></listitem>
            <listitem><para>
                Disable static library generation for recipes derived from
                <filename>autoconf</filename> or <filename>libtool</filename>:
                Following is an example showing how to disable static
                libraries and still provide an override to handle exceptions:
                <literallayout class='monospaced'>
     STATICLIBCONF = "--disable-static"
     STATICLIBCONF_sqlite3-native = ""
     EXTRA_OECONF += "${STATICLIBCONF}"
                </literallayout>
                <note><title>Notes</title>
                    <itemizedlist>
                        <listitem><para>
                            Some recipes need static libraries in order to work
                            correctly (e.g. <filename>pseudo-native</filename>
                            needs <filename>sqlite3-native</filename>).
                            Overrides, as in the previous example, account for
                            these kinds of exceptions.
                            </para></listitem>
                        <listitem><para>
                            Some packages have packaging code that assumes the
                            presence of the static libraries.
                            If so, you might need to exclude them as well.
                            </para></listitem>
                    </itemizedlist>
                </note>
            </para></listitem>
        </itemizedlist>
    </para>
</section>
</chapter>
<!--
vim: expandtab tw=80 ts=4
-->