path: root/documentation/poky-ref-manual/usingpoky.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<chapter id='usingpoky'>
<title>Using the Yocto Project</title>

    <para>
        This section gives an overview of the components that make up the Yocto Project 
        followed by information about Yocto Project builds and dealing with any
        problems that might arise.
    </para>

<section id='usingpoky-components'>
    <title>Yocto Project Components</title>

    <para>
        The BitBake task executor together with various types of configuration files form the 
        Yocto Project core.
        This section overviews the BitBake task executor and the
        configuration files by describing what they are used for and how they interact.
    </para>
    
    <para>  
        BitBake handles the parsing and execution of the data files. 
        The data itself is of various types:
    <itemizedlist>
        <listitem><para><emphasis>Recipes:</emphasis>  Provides details about particular 
            pieces of software</para></listitem>
        <listitem><para><emphasis>Class Data:</emphasis>  An abstraction of common build 
            information (e.g. how to build a Linux kernel).</para></listitem>
        <listitem><para><emphasis>Configuration Data:</emphasis>  Defines machine-specific settings, 
            policy decisions, etc.
            Configuration data acts as the glue to bind everything together.</para></listitem>
    </itemizedlist>
        For more information on data, see the
        <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#yocto-project-terms'>
        Yocto Project Terms</ulink> section in 
        <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html'>
        The Yocto Project Development Manual</ulink>.
    </para>

    <para> 
        BitBake knows how to combine multiple data sources together and refers to each data source
        as a <link linkend='usingpoky-changes-layers'>'layer'</link>.
    </para>

    <para>
        Following are some brief details on these core components.
        For more detailed information on these components see the 
        <link linkend='ref-structure'>'Reference: Directory Structure'</link>
        appendix.
    </para>

    <section id='usingpoky-components-bitbake'>
        <title>BitBake</title>

        <para>
            BitBake is the tool at the heart of the Yocto Project and is responsible
            for parsing the metadata, generating a list of tasks from it,
            and then executing those tasks. 
            To see a list of the options BitBake supports, use the following help command:
            <literallayout class='monospaced'>
     $ bitbake --help
            </literallayout>
        </para>

        <para>
            The most common usage for BitBake is <filename>bitbake &lt;packagename&gt;</filename>, where
            <filename>packagename</filename> is the name of the package you want to build 
            (referred to as the "target" in this manual). 
            The target often equates to the first part of a <filename>.bb</filename> filename.
            So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
            might type the following:
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop
            </literallayout>
            Several different versions of <filename>matchbox-desktop</filename> might exist.
            BitBake chooses the one selected by the distribution configuration.
            You can get more details about how BitBake chooses between different 
            target versions and providers in the 
            <link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
        </para>

        <para>
            BitBake also tries to execute any dependent tasks first.
            So for example, before building <filename>matchbox-desktop</filename>, BitBake
            would build a cross compiler and <filename>eglibc</filename> if they had not already 
            been built.
            <note>This release of the Yocto Project does not support the <filename>glibc</filename>
                GNU version of the Unix standard C library.  By default, the Yocto Project builds with
                <filename>eglibc</filename>.</note>
        </para>

        <para>
            A useful BitBake option to consider is the <filename>-k</filename> or 
            <filename>--continue</filename> option.  
            This option instructs BitBake to try and continue processing the job as much 
            as possible even after encountering an error.  
            When an error occurs, the target that
            failed and those that depend on it cannot be remade.  
            However, when you use this option other dependencies can still be processed.
        </para>
    </section>

    <section id='usingpoky-components-metadata'>
        <title>Metadata (Recipes)</title>

        <para>
            The <filename>.bb</filename> files are usually referred to as "recipes." 
            In general, a recipe contains information about a single piece of software.
            The information includes the location from which to download the source patches 
            (if any are needed), which special configuration options to apply, 
            how to compile the source files, and how to package the compiled output. 
        </para>

        <para>
            The term "package" can also be used to describe recipes.
            However, since the same word is used for the packaged output from the Yocto 
            Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), 
            this document avoids using the term "package" to refer to recipes.
        </para>
    </section>

    <section id='usingpoky-components-classes'>
        <title>Classes</title>

        <para>
            Class files (<filename>.bbclass</filename>) contain information that is useful to share
            between metadata files. 
            An example is the Autotools class, which contains
            common settings for any application that Autotools uses.
            The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
            about common classes and how to use them.
        </para>
    </section>

    <section id='usingpoky-components-configuration'>
        <title>Configuration</title>

        <para>
            The configuration files (<filename>.conf</filename>) define various configuration variables
            that govern the Yocto Project build process. 
            These files fall into several areas that define machine configuration options, 
            distribution configuration options, compiler tuning options, general common configuration
            options and user configuration options (<filename>local.conf</filename>, which is found
            in the Yocto Project files build directory).
        </para>
    </section>
</section>


<section id='usingpoky-build'>
    <title>Running a Build</title>

    <para>
        You can find general information on how to build an image using the 
        Yocto Project in the 
        <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html#building-image'>
        Building an Image</ulink> section of the 
        <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html'>
        Yocto Project Quick Start</ulink>.
        This section provides a summary of the build process and provides information
        for less obvious aspects of the build process.
    </para>

    <section id='build-overview'>
        <title>Build Overview</title>

        <para>
            The first thing you need to do is set up the Yocto Project build environment by sourcing
            the environment setup script as follows:
            <literallayout class='monospaced'>
     $ source oe-init-build-env [build_dir]
            </literallayout>
        </para>

        <para>
            The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
            uses for the build.
            If you do not specify a build directory it defaults to <filename>build</filename>
            in your current working directory.
            A common practice is to use a different build directory for different targets. 
            For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
            target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
            See <link linkend="structure-core-script">oe-init-build-env</link>
            for more information on this script.
        </para>

        <para>
            Once the Yocto Project build environment is set up, you can build a target using:
            <literallayout class='monospaced'>
     $ bitbake &lt;target&gt;
            </literallayout>
        </para>

        <para>
            The <filename>target</filename> is the name of the recipe you want to build. 
            Common targets are the images in <filename>meta/recipes-core/images</filename>,
            <filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project
            files.
            Or, the target can be the name of a recipe for a specific piece of software such as 
            <application>busybox</application>. 
            For more details about the images Yocto Project supports, see the 
            <link linkend="ref-images">'Reference: Images'</link> appendix.
        </para>

        <note>
            Building an image without GNU Public License Version 3 (GPLv3) components is 
            only supported for minimal and base images.
            See <link linkend='ref-images'>'Reference: Images'</link> for more information.
        </note>
    </section>

    <section id='building-an-image-using-gpl-components'>
        <title>Building an Image Using GPL Components</title>

        <para>
            When building an image using GPL components, you need to maintain your original 
            settings and not switch back and forth applying different versions of the GNU
            Public License.  
            If you rebuild using different versions of GPL, dependency errors might occur
            due to some components not being rebuilt.
        </para>
    </section>

    <section id="considering-shared-state-cache">
        <title>Considering Shared State Cache</title>

        <para>
            By design, the Yocto Project builds everything from scratch unless it can determine that
            a given task's inputs have not changed.  
            While building from scratch ensures that everything is current, it does also
            mean that a lot of time could be spent rebuiding things that don't necessarily need built.
        </para>

        <para>
            The Yocto Project build process uses a shared state caching scheme to avoid having to 
            rebuild software when it is not necessary.  
            Because the build time for a Yocto image can be significant, it is helpful to try and 
            determine what really needs built and what can be skipped given a particular project's
            build process.
        </para>

        <para>
            The scheme that the Yocto Project uses involves checksum generation and comparison for
            a task's inputs.
            The scheme also employs an area of memory called the shared state cache that is 
            pointed to by the <filename>SSTATE_DIR</filename> variable.
            This area contains task output generated from a previous build.  
            If a given task's checksum matches the checksum of a previous build for the same  
            task, the build process uses the state of the cache rather than rerunning that 
            task.
        </para>

        <para>
            The previous paragraph is a simplistic explanation of how the build process 
            uses checksums and shared state memory cache to avoide building tasks that 
            don't need built.
            If you want a bit more explanation on the topic, 
            see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
            State - What does it mean and why should I care?</ulink>" from the Yocto 
            Project discussion archives.
        </para>

        <para>
            As with all schemes, this one has some drawbacks.
            It is possible that you could make implicit changes that are not factored into the checksum 
            calculation, but do affect a task's output. 
            A good example is perhaps when a tool changes its output.
            Let's say that the output of <filename>rpmdeps</filename> needed to change.
            The result of the change should be that all the "package", "package_write_rpm",
            and "package_deploy-rpm" sstate-cache items would become invalid.
            But, because this is a change that is external to the code and therefore implicit,
            the associated sstate-cache items do not become invalidated.
            In this case, the build process would use the cache items rather than running the 
            task again. 
            Obviously, these types of implicit changes can cause problems.
        </para>

        <para>
            To avoid these problems during the build, you need to understand the effects of any
            change you make.
            Note that any changes you make directly to a function automatically are factored into 
            the checksum calculation and thus, will invalidate the associated area of sstate cache.
            You need to be aware of any implicit changes that are not obvious changes to the 
            code and could affect the output of a given task. 
            Once you are aware of such a change, you can take steps to invalidate the cache 
            and force the task to run. 
            The step to take is as simple as changing a function's comments in the source code. 
            For example, to invalidate package sstate files, change the comment statments
            of <filename>do_package</filename> or one of the functions it calls.
            The change is purely cosmetic, but it causes the checksum to be recalculated and  
            forces the task to be run again.
        </para>

        <note>
            For an example of a commit that makes a cosmetic change to invalidate an sstate, 
            see this
            <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
        </note>
    </section>
</section>

<section id='usingpoky-install'>
    <title>Installing and Using the Result</title>

    <para>
        Once an image has been built, it often needs to be installed. 
        The images and kernels built by the Yocto Project are placed in the build directory in 
        <filename class="directory">tmp/deploy/images</filename>. 
        For information on how to run pre-built images such as <filename>qemux86</filename> 
        and <filename>qemuarm</filename>, see the
        <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html#using-pre-built'>
        Using Pre-Built Binaries and QEMU</ulink> section in the 
        <ulink url='http://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html'>
        Yocto Project Quick Start</ulink>.
        For information about how to install these images, see the documentation for your
        particular board/machine.
    </para>
</section>

<section id='usingpoky-debugging'>
    <title>Debugging Build Failures</title>

    <para>
        The exact method for debugging Yocto Project build failures depends on the nature of the 
        problem and on the system's area from which the bug originates. 
        Standard debugging practices such as comparison against the last 
        known working version with examination of the changes and the re-application of steps 
        to identify the one causing the problem are
        valid for Yocto Project just as they are for any other system. 
        Even though it is impossible to detail every possible potential failure, 
        this section provides some general tips to aid in debugging.
    </para>

    <section id='usingpoky-debugging-taskfailures'>
        <title>Task Failures</title>

        <para>The log file for shell tasks is available in 
            <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>. 
            For example, the <filename>compile</filename> task for the QEMU minimal image for the x86
            machine (<filename>qemux86</filename>) might be 
            <filename>tmp/work/qemux86-poky-linux/core-image-minimal-1.0-r0/temp/log.do_compile.20830</filename>.
            To see what BitBake runs to generate that log, look at the corresponding 
            <filename>run.do_taskname.pid</filename> file located in the same directory.
        </para>

        <para>
            Presently, the output from Python tasks is sent directly to the console.
        </para>
    </section>

    <section id='usingpoky-debugging-taskrunning'>
        <title>Running Specific Tasks</title>

        <para>
            Any given package consists of a set of tasks.  
            The standard BitBake behavior in most cases is: <filename>fetch</filename>, 
            <filename>unpack</filename>, 
            <filename>patch</filename>, <filename>configure</filename>,
            <filename>compile</filename>, <filename>install</filename>, <filename>package</filename>,
            <filename>package_write</filename>, and <filename>build</filename>. 
            The default task is <filename>build</filename> and any tasks on which it depends 
            build first.
            Some tasks exist, such as <filename>devshell</filename>, that are not part of the
            default build chain.  
            If you wish to run a task that is not part of the default build chain, you can use the 
            <filename>-c</filename> option in BitBake as follows:
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop -c devshell
            </literallayout>
        </para>

        <para>
            If you wish to rerun a task, use the <filename>-f</filename> force option. 
            For example, the following sequence forces recompilation after changing files in the 
            working directory.
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop
               .
               .
        [make some changes to the source code in the working directory]
               .
               .
     $ bitbake matchbox-desktop -c compile -f
     $ bitbake matchbox-desktop
            </literallayout>
        </para>

        <para>
            This sequence first builds <filename>matchbox-desktop</filename> and then recompiles it.
            The last command reruns all tasks (basically the packaging tasks) after the compile.
            BitBake recognizes that the <filename>compile</filename> task was rerun and therefore 
            understands that the other tasks also need to be run again.
        </para>

        <para>
            You can view a list of tasks in a given package by running the
            <filename>listtasks</filename> task as follows:
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop -c listtasks
            </literallayout>
            The results are in the file <filename>${WORKDIR}/temp/log.do_listtasks</filename>.
        </para>
    </section>

    <section id='usingpoky-debugging-dependencies'>
        <title>Dependency Graphs</title>

        <para>
            Sometimes it can be hard to see why BitBake wants to build some other packages before a given 
            package you have specified.
            The <filename>bitbake -g targetname</filename> command creates the 
            <filename>depends.dot</filename> and <filename>task-depends.dot</filename> files 
            in the current directory. 
            These files show the package and task dependencies and are useful for debugging problems.
            You can use the <filename>bitbake -g -u depexp targetname</filename> command to 
            display the results in a more human-readable form.
        </para>
    </section>

    <section id='usingpoky-debugging-bitbake'>
        <title>General BitBake Problems</title>

        <para>
            You can see debug output from BitBake by using the <filename>-D</filename> option.
            The debug output gives more information about what BitBake
            is doing and the reason behind it. 
            Each <filename>-D</filename> option you use increases the logging level.
            The most common usage is <filename>-DDD</filename>.
        </para>

        <para>
            The output from <filename>bitbake -DDD -v targetname</filename> can reveal why
            BitBake chose a certain version of a package or why BitBake
            picked a certain provider.
            This command could also help you in a situation where you think BitBake did something 
            unexpected.
        </para>
    </section>

    <section id='usingpoky-debugging-buildfile'>
        <title>Building with No Dependencies</title>
        <para>
            If you really want to build a specific <filename>.bb</filename> file, you can use
            the command form <filename>bitbake -b &lt;somepath/somefile.bb&gt;</filename>. 
            This command form does not check for dependencies so you should use it
            only when you know its dependencies already exist. 
            You can also specify fragments of the filename.
            In this case, BitBake checks for a unique match.
        </para>
    </section>

    <section id='usingpoky-debugging-variables'>
        <title>Variables</title>
        <para>
            The <filename>-e</filename> option dumps the resulting environment for
            either the configuration (no package specified) or for a
            specific package when specified; or <filename>-b recipename</filename>
            to show the environment from parsing a single recipe file only.
        </para>
    </section>
    
    <section id='recipe-logging-mechanisms'>
        <title>Recipe Logging Mechanisms</title>
        <para>
            Best practices exist while writing recipes that both log build progress and 
            act on build conditions such as warnings and errors. 
            Both Python and Bash language bindings exist for the logging mechanism:
            <itemizedlist>
                <listitem><para><emphasis>Python:</emphasis> For Python functions, BitBake
                    supports several loglevels: <filename>bb.fatal</filename>, 
                    <filename>bb.error</filename>, <filename>bb.warn</filename>,
                    <filename>bb.note</filename>, <filename>bb.plain</filename>,
                    and <filename>bb.debug</filename>.</para></listitem>
                <listitem><para><emphasis>Bash:</emphasis> For Bash functions, the same set 
                    of loglevels exist and are accessed with a similar syntax:
                    <filename>bbfatal</filename>, <filename>bberror</filename>, 
                    <filename>bbwarn</filename>, <filename>bbnote</filename>, 
                    <filename>bbplain</filename>, and <filename>bbdebug</filename>.</para></listitem>
            </itemizedlist>
        </para>

        <para>
            For guidance on how logging is handled 
            in both Python and Bash recipes, see the 
            <filename>logging.bbclass</filename> file in the 
            <filename>meta/classes</filename> directory of the Yocto Project files.
        </para>

        <section id='logging-with-python'>
            <title>Logging With Python</title>
            <para>
                When creating recipes using Python and inserting code that handles build logs
                keep in mind the goal is to have informative logs while keeping the console as 
                "silent" as possible. 
                Also, if you want status messages in the log use the "debug" loglevel.
            </para>

            <para>
                Following is an example written in Python.
                The code handles logging for a function that determines the number of tasks 
                needed to be run:
                <literallayout class='monospaced'>
     python do_listtasks() {
         bb.debug(2, "Starting to figure out the task list")
         if noteworthy_condition:
             bb.note("There are 47 tasks to run")
         bb.debug(2, "Got to point xyz")
         if warning_trigger:
             bb.warn("Detected warning_trigger, this might be a problem later.")
         if recoverable_error:
             bb.error("Hit recoverable_error, you really need to fix this!")
         if fatal_error:
             bb.fatal("fatal_error detected, unable to print the task list")
         bb.plain("The tasks present are abc")
         bb.debug(2, "Finished figureing out the tasklist")
     }
                </literallayout>
            </para>
        </section>

        <section id='logging-with-bash'>
            <title>Logging With Bash</title>
            <para>
                When creating recipes using Bash and inserting code that handles build
                logs you have the same goals - informative with minimal console output. 
                The syntax you use for recipes written in Bash is similar to that of 
                recipes written in Python described in the previous section.
            </para>

            <para>
                Following is an example written in Bash.
                The code logs the progress of the <filename>do_my_function</filename> function.
                <literallayout class='monospaced'>
     do_my_function() {
         bbdebug 2 "Running do_my_function"
         if [ exceptional_condition ]; then
             bbnote "Hit exceptional_condition"
         fi
         bbdebug 2  "Got to point xyz"
         if [ warning_trigger ]; then
             bbwarn "Detected warning_trigger, this might cause a problem later."
         fi
         if [ recoverable_error ]; then
             bberror "Hit recoverable_error, correcting"
         fi
         if [ fatal_error ]; then
             bbfatal "fatal_error detected"
         fi
         bbdebug 2 "Completed do_my_function"
     }
                </literallayout>
            </para>
        </section>
    </section>
    
    <section id='usingpoky-debugging-others'>
        <title>Other Tips</title>

        <para>
            Here are some other tips that you might find useful:
            <itemizedlist>
                <listitem><para>When adding new packages, it is worth watching for 
                    undesirable items making their way into compiler command lines.
                    For example, you do not want references to local system files like 
                    <filename>/usr/lib/</filename> or <filename>/usr/include/</filename>.
                    </para></listitem>
                <listitem><para>If you want to remove the psplash boot splashscreen, 
                    add <filename>psplash=false</filename> to  the kernel command line.
                    Doing so prevents psplash from loading and thus allows you to see the console.
                    It is also possible to switch out of the splashscreen by 
                    switching the virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
                    </para></listitem>
            </itemizedlist>
        </para>
    </section>
</section>

</chapter>
<!-- 
vim: expandtab tw=80 ts=4 
-->