path: root/documentation/ref-manual/technical-details.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >

<chapter id='technical-details'>
<title>Technical Details</title>

    <para>
        This chapter provides technical details for various parts of the
        Yocto Project.
        Currently, topics include Yocto Project components,
        cross-toolchain generation, shared state (sstate) cache,
        x32, Wayland support, and Licenses.
    </para>

<section id='fakeroot-and-pseudo'>
    <title>Fakeroot and Pseudo</title>

    <para>
        Some tasks are easier to implement when allowed to perform certain
        operations that are normally reserved for the root user.
        For example, the
        <link linkend='ref-tasks-install'><filename>do_install</filename></link>
        task benefits from being able to set the UID and GID of installed files
        to arbitrary values.
    </para>

    <para>
        One approach to allowing tasks to perform root-only operations
        would be to require BitBake to run as root.
        However, this method is cumbersome and has security issues.
        The approach that is actually used is to run tasks that benefit from
        root privileges in a "fake" root environment.
        Within this environment, the task and its child processes believe that
        they are running as the root user, and see an internally consistent
        view of the filesystem.
        As long as generating the final output (e.g. a package or an image)
        does not require root privileges, the fact that some earlier steps ran
        in a fake root environment does not cause problems.
    </para>

    <para>
        The capability to run tasks in a fake root environment is known as
        "fakeroot", which is derived from the BitBake keyword/variable
        flag that requests a fake root environment for a task.
        In current versions of the OpenEmbedded build system,
        the program that implements fakeroot is known as Pseudo.
    </para>

    <para>
        Pseudo overrides system calls through the
        <filename>LD_PRELOAD</filename> mechanism to give the
        illusion of running as root.
        To keep track of "fake" file ownership and permissions resulting from
        operations that require root permissions, an sqlite3
        database is used.
        This database is stored in
        <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/pseudo/files.db</filename>
        for individual recipes.
        Storing the database in a file as opposed to in memory
        gives persistence between tasks, and even between builds.
        <note><title>Caution</title>
            If you add your own task that manipulates the same files or
            directories as a fakeroot task, then that task should also run
            under fakeroot.
            Otherwise, the task will not be able to run root-only operations,
            and will not see the fake file ownership and permissions set by the
            other task.
            You should also add a dependency on
            <filename>virtual/fakeroot-native:do_populate_sysroot</filename>,
            giving the following:
            <literallayout class='monospaced'>
       fakeroot do_mytask () {
           ...
       }
       do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
            </literallayout>
        </note>
        For more information, see the
        <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>
        variables in the BitBake User Manual.
        You can also reference this
        <ulink url='http://www.ibm.com/developerworks/opensource/library/os-aapseudo1/index.html'>Pseudo</ulink>
        article.
    </para>
</section>

<section id="wayland">
    <title>Wayland</title>

    <para>
        <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink>
        is a computer display server protocol that
        provides a method for compositing window managers to communicate
        directly with applications and video hardware and expects them to
        communicate with input hardware using other libraries.
        Using Wayland with supporting targets can result in better control
        over graphics frame rendering than an application might otherwise
        achieve.
    </para>

    <para>
        The Yocto Project provides the Wayland protocol libraries and the
        reference
        <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink>
        compositor as part of its release.
        This section describes what you need to do to implement Wayland and
        use the compositor when building an image for a supporting target.
    </para>

    <section id="wayland-support">
        <title>Support</title>

        <para>
            The Wayland protocol libraries and the reference Weston compositor
            ship as integrated packages in the <filename>meta</filename> layer
            of the
            <link linkend='source-directory'>Source Directory</link>.
            Specifically, you can find the recipes that build both Wayland
            and Weston at <filename>meta/recipes-graphics/wayland</filename>.
        </para>

        <para>
            You can build both the Wayland and Weston packages for use only
            with targets that accept the
            <ulink url='http://dri.freedesktop.org/wiki/'>Mesa 3D and Direct Rendering Infrastructure</ulink>,
            which is also known as Mesa DRI.
            This implies that you cannot build and use the packages if your
            target uses, for example, the
            <trademark class='registered'>Intel</trademark> Embedded Media and
            Graphics Driver (<trademark class='registered'>Intel</trademark>
            EMGD) that overrides Mesa DRI.
        </para>

        <note>
            Due to lack of EGL support, Weston 1.0.3 will not run directly on
            the emulated QEMU hardware.
            However, this version of Weston will run under X emulation without
            issues.
        </note>
    </section>

    <section id="enabling-wayland-in-an-image">
        <title>Enabling Wayland in an Image</title>

        <para>
            To enable Wayland, you need to enable it to be built and enable
            it to be included in the image.
        </para>

        <section id="enable-building">
            <title>Building</title>

            <para>
                To cause Mesa to build the <filename>wayland-egl</filename>
                platform and Weston to build Wayland with Kernel Mode
                Setting
                (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>)
                support, include the "wayland" flag in the
                <link linkend="var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></link>
                statement in your <filename>local.conf</filename> file:
                <literallayout class='monospaced'>
     DISTRO_FEATURES_append = " wayland"
                </literallayout>
            </para>

            <note>
                If X11 has been enabled elsewhere, Weston will build Wayland
                with X11 support
            </note>
        </section>

        <section id="enable-installation-in-an-image">
            <title>Installing</title>

            <para>
                To install the Wayland feature into an image, you must
                include the following
                <link linkend='var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></link>
                statement in your <filename>local.conf</filename> file:
                <literallayout class='monospaced'>
     CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
                </literallayout>
            </para>
        </section>
    </section>

    <section id="running-weston">
        <title>Running Weston</title>

        <para>
            To run Weston inside X11, enabling it as described earlier and
            building a Sato image is sufficient.
            If you are running your image under Sato, a Weston Launcher appears
            in the "Utility" category.
        </para>

        <para>
            Alternatively, you can run Weston through the command-line
            interpretor (CLI), which is better suited for development work.
            To run Weston under the CLI, you need to do the following after
            your image is built:
            <orderedlist>
                <listitem><para>Run these commands to export
                    <filename>XDG_RUNTIME_DIR</filename>:
                    <literallayout class='monospaced'>
     mkdir -p /tmp/$USER-weston
     chmod 0700 /tmp/$USER-weston
     export XDG_RUNTIME_DIR=/tmp/$USER-weston
                    </literallayout></para></listitem>
                <listitem><para>Launch Weston in the shell:
                    <literallayout class='monospaced'>
     weston
                    </literallayout></para></listitem>
            </orderedlist>
        </para>
    </section>
</section>

<section id="licenses">
    <title>Licenses</title>

    <para>
        This section describes the mechanism by which the OpenEmbedded build system
        tracks changes to licensing text.
        The section also describes how to enable commercially licensed recipes,
        which by default are disabled.
    </para>

    <para>
        For information that can help you maintain compliance with various open
        source licensing during the lifecycle of the product, see the
        "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>"
        section in the Yocto Project Development Tasks Manual.
    </para>

    <section id="usingpoky-configuring-LIC_FILES_CHKSUM">
        <title>Tracking License Changes</title>

        <para>
            The license of an upstream project might change in the future.
            In order to prevent these changes going unnoticed, the
            <filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename>
            variable tracks changes to the license text. The checksums are validated at the end of the
            configure step, and if the checksums do not match, the build will fail.
        </para>

        <section id="usingpoky-specifying-LIC_FILES_CHKSUM">
            <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>

            <para>
                The <filename>LIC_FILES_CHKSUM</filename>
                variable contains checksums of the license text in the source
                code for the recipe.
                Following is an example of how to specify
                <filename>LIC_FILES_CHKSUM</filename>:
                <literallayout class='monospaced'>
     LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
                         file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
                         file://licfile2.txt;endline=50;md5=zzzz \
                         ..."
                </literallayout>
                <note><title>Notes</title>
                    <itemizedlist>
                        <listitem><para>
                            When using "beginline" and "endline", realize that
                            line numbering begins with one and not zero.
                            Also, the included lines are inclusive (i.e. lines
                            five through and including 29 in the previous
                            example for <filename>licfile1.txt</filename>).
                            </para></listitem>
                        <listitem><para>
                            When a license check fails, the selected license
                            text is included as part of the QA message.
                            Using this output, you can determine the exact
                            start and finish for the needed license text.
                            </para></listitem>
                    </itemizedlist>
                </note>
            </para>

            <para>
                The build system uses the
                <filename><link linkend='var-S'>S</link></filename> variable as
                the default directory when searching files listed in
                <filename>LIC_FILES_CHKSUM</filename>.
                The previous example employs the default directory.
            </para>

            <para>
                Consider this next example:
                <literallayout class='monospaced'>
     LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
                                         md5=bb14ed3c4cda583abc85401304b5cd4e"
     LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
                </literallayout>
            </para>

            <para>
                The first line locates a file in
                <filename>${S}/src/ls.c</filename> and isolates lines five
                through 16 as license text.
                The second line refers to a file in
                <filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>.
            </para>
            <para>
                Note that <filename>LIC_FILES_CHKSUM</filename> variable is
                mandatory for all recipes, unless the
                <filename>LICENSE</filename> variable is set to "CLOSED".
            </para>
        </section>

        <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
            <title>Explanation of Syntax</title>
            <para>
                As mentioned in the previous section, the
                <filename>LIC_FILES_CHKSUM</filename> variable lists all the
                important files that contain the license text for the source code.
                It is possible to specify a checksum for an entire file, or a specific section of a
                file (specified by beginning and ending line numbers with the "beginline" and "endline"
                parameters, respectively).
                The latter is useful for source files with a license notice header,
                README documents, and so forth.
                If you do not use the "beginline" parameter, then it is assumed that the text begins on the
                first line of the file.
                Similarly, if you do not use the "endline" parameter, it is assumed that the license text
                ends with the last line of the file.
            </para>

            <para>
                The "md5" parameter stores the md5 checksum of the license text.
                If the license text changes in any way as compared to this parameter
                then a mismatch occurs.
                This mismatch triggers a build failure and notifies the developer.
                Notification allows the developer to review and address the license text changes.
                Also note that if a mismatch occurs during the build, the correct md5
                checksum is placed in the build log and can be easily copied to the recipe.
            </para>

            <para>
                There is no limit to how many files you can specify using the
                <filename>LIC_FILES_CHKSUM</filename> variable.
                Generally, however, every project requires a few specifications for license tracking.
                Many projects have a "COPYING" file that stores the license information for all the source
                code files.
                This practice allows you to just track the "COPYING" file as long as it is kept up to date.
            </para>

            <tip>
                If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match
                error and displays the correct "md5" parameter value during the build.
                The correct parameter is also captured in the build log.
            </tip>

            <tip>
                If the whole file contains only license text, you do not need to use the "beginline" and
                "endline" parameters.
            </tip>
        </section>
    </section>

    <section id="enabling-commercially-licensed-recipes">
        <title>Enabling Commercially Licensed Recipes</title>

        <para>
            By default, the OpenEmbedded build system disables
            components that have commercial or other special licensing
            requirements.
            Such requirements are defined on a
            recipe-by-recipe basis through the
            <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
            variable definition in the affected recipe.
            For instance, the
            <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
            recipe contains the following statement:
            <literallayout class='monospaced'>
     LICENSE_FLAGS = "commercial"
            </literallayout>
            Here is a slightly more complicated example that contains both an
            explicit recipe name and version (after variable expansion):
            <literallayout class='monospaced'>
     LICENSE_FLAGS = "license_${PN}_${PV}"
            </literallayout>
	        In order for a component restricted by a <filename>LICENSE_FLAGS</filename>
	        definition to be enabled and included in an image, it
	        needs to have a matching entry in the global
	        <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link>
            variable, which is a variable
	        typically defined in your <filename>local.conf</filename> file.
            For example, to enable
	        the <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
	        package, you could add either the string
	        "commercial_gst-plugins-ugly" or the more general string
	        "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.
            See the
            "<link linkend='license-flag-matching'>License Flag Matching</link>" section
            for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works.
            Here is the example:
            <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
            </literallayout>
	        Likewise, to additionally enable the package built from the recipe containing
	        <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming
	        that the actual recipe name was <filename>emgd_1.10.bb</filename>,
	        the following string would enable that package as well as
	        the original <filename>gst-plugins-ugly</filename> package:
            <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
            </literallayout>
	        As a convenience, you do not need to specify the complete license string
	        in the whitelist for every package.
            You can use an abbreviated form, which consists
	        of just the first portion or portions of the license string before
	        the initial underscore character or characters.
            A partial string will match
	        any license that contains the given string as the first
	        portion of its license.
            For example, the following
	        whitelist string will also match both of the packages
	        previously mentioned as well as any other packages that have
	        licenses starting with "commercial" or "license".
            <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial license"
            </literallayout>
        </para>

        <section id="license-flag-matching">
            <title>License Flag Matching</title>

            <para>
		        License flag matching allows you to control what recipes the
                OpenEmbedded build system includes in the build.
                Fundamentally, the build system attempts to match
                <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
                strings found in recipes against
                <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link>
                strings found in the whitelist.
                A match causes the build system to include a recipe in the
                build, while failure to find a match causes the build system to
                exclude a recipe.
            </para>

            <para>
                In general, license flag matching is simple.
                However, understanding some concepts will help you
                correctly and effectively use matching.
            </para>

            <para>
                Before a flag
                defined by a particular recipe is tested against the
                contents of the whitelist, the expanded string
                <filename>_${PN}</filename> is appended to the flag.
                This expansion makes each <filename>LICENSE_FLAGS</filename>
                value recipe-specific.
                After expansion, the string is then matched against the
                whitelist.
                Thus, specifying
                <filename>LICENSE_FLAGS = "commercial"</filename>
                in recipe "foo", for example, results in the string
                <filename>"commercial_foo"</filename>.
                And, to create a match, that string must appear in the
                whitelist.
            </para>

            <para>
                Judicious use of the <filename>LICENSE_FLAGS</filename>
                strings and the contents of the
                <filename>LICENSE_FLAGS_WHITELIST</filename> variable
                allows you a lot of flexibility for including or excluding
                recipes based on licensing.
                For example, you can broaden the matching capabilities by
                using license flags string subsets in the whitelist.
                <note>When using a string subset, be sure to use the part of
                    the expanded string that precedes the appended underscore
                    character (e.g. <filename>usethispart_1.3</filename>,
                    <filename>usethispart_1.4</filename>, and so forth).
                </note>
                For example, simply specifying the string "commercial" in
                the whitelist matches any expanded
                <filename>LICENSE_FLAGS</filename> definition that starts with
                the string "commercial" such as "commercial_foo" and
                "commercial_bar", which are the strings the build system
                automatically generates for hypothetical recipes named
                "foo" and "bar" assuming those recipes simply specify the
                following:
                <literallayout class='monospaced'>
     LICENSE_FLAGS = "commercial"
                </literallayout>
                Thus, you can choose to exhaustively
                enumerate each license flag in the whitelist and
                allow only specific recipes into the image, or
                you can use a string subset that causes a broader range of
                matches to allow a range of recipes into the image.
            </para>

            <para>
                This scheme works even if the
                <filename>LICENSE_FLAGS</filename> string already
                has <filename>_${PN}</filename> appended.
                For example, the build system turns the license flag
                "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would
                match both the general "commercial" and the specific
                "commercial_1.2_foo" strings found in the whitelist, as
                expected.
            </para>

            <para>
                Here are some other scenarios:
                <itemizedlist>
                    <listitem><para>You can specify a versioned string in the
                        recipe such as "commercial_foo_1.2" in a "foo" recipe.
                        The build system expands this string to
                        "commercial_foo_1.2_foo".
                        Combine this license flag with a whitelist that has
                        the string "commercial" and you match the flag along
                        with any other flag that starts with the string
                        "commercial".</para></listitem>
                    <listitem><para>Under the same circumstances, you can
                        use "commercial_foo" in the whitelist and the
                        build system not only matches "commercial_foo_1.2" but
                        also matches any license flag with the string
                        "commercial_foo", regardless of the version.
                        </para></listitem>
                    <listitem><para>You can be very specific and use both the
                        package and version parts in the whitelist (e.g.
                        "commercial_foo_1.2") to specifically match a
                        versioned recipe.</para></listitem>
                </itemizedlist>
            </para>
        </section>

        <section id="other-variables-related-to-commercial-licenses">
            <title>Other Variables Related to Commercial Licenses</title>

            <para>
                Other helpful variables related to commercial
                license handling exist and are defined in the
                <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file:
                <literallayout class='monospaced'>
     COMMERCIAL_AUDIO_PLUGINS ?= ""
     COMMERCIAL_VIDEO_PLUGINS ?= ""
                </literallayout>
                If you want to enable these components, you can do so by making sure you have
                statements similar to the following
                in your <filename>local.conf</filename> configuration file:
                <literallayout class='monospaced'>
     COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
        gst-plugins-ugly-mpegaudioparse"
     COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
        gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
                </literallayout>
                Of course, you could also create a matching whitelist
                for those components using the more general "commercial"
                in the whitelist, but that would also enable all the
                other packages with
                <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
                containing "commercial", which you may or may not want:
                <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial"
                </literallayout>
            </para>

            <para>
                Specifying audio and video plug-ins as part of the
                <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
                <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements
                (along with the enabling
                <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the
                plug-ins or components into built images, thus adding
                support for media formats or components.
            </para>
        </section>
    </section>
</section>
</chapter>
<!--
vim: expandtab tw=80 ts=4
-->