path: root/documentation/ref-manual/technical-details.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >

<chapter id='technical-details'>
<title>Technical Details</title>

    <para>
        This chapter provides technical details for various parts of the
        Yocto Project.
        Currently, topics include Yocto Project components,
        cross-toolchain generation, shared state (sstate) cache,
        x32, Wayland support, and Licenses.
    </para>

<section id="licenses">
    <title>Licenses</title>

    <para>
        This section describes the mechanism by which the OpenEmbedded build system
        tracks changes to licensing text.
        The section also describes how to enable commercially licensed recipes,
        which by default are disabled.
    </para>

    <para>
        For information that can help you maintain compliance with various open
        source licensing during the lifecycle of the product, see the
        "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>"
        section in the Yocto Project Development Tasks Manual.
    </para>

    <section id="usingpoky-configuring-LIC_FILES_CHKSUM">
        <title>Tracking License Changes</title>

        <para>
            The license of an upstream project might change in the future.
            In order to prevent these changes going unnoticed, the
            <filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename>
            variable tracks changes to the license text. The checksums are validated at the end of the
            configure step, and if the checksums do not match, the build will fail.
        </para>

        <section id="usingpoky-specifying-LIC_FILES_CHKSUM">
            <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>

            <para>
                The <filename>LIC_FILES_CHKSUM</filename>
                variable contains checksums of the license text in the source
                code for the recipe.
                Following is an example of how to specify
                <filename>LIC_FILES_CHKSUM</filename>:
                <literallayout class='monospaced'>
     LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
                         file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
                         file://licfile2.txt;endline=50;md5=zzzz \
                         ..."
                </literallayout>
                <note><title>Notes</title>
                    <itemizedlist>
                        <listitem><para>
                            When using "beginline" and "endline", realize that
                            line numbering begins with one and not zero.
                            Also, the included lines are inclusive (i.e. lines
                            five through and including 29 in the previous
                            example for <filename>licfile1.txt</filename>).
                            </para></listitem>
                        <listitem><para>
                            When a license check fails, the selected license
                            text is included as part of the QA message.
                            Using this output, you can determine the exact
                            start and finish for the needed license text.
                            </para></listitem>
                    </itemizedlist>
                </note>
            </para>

            <para>
                The build system uses the
                <filename><link linkend='var-S'>S</link></filename> variable as
                the default directory when searching files listed in
                <filename>LIC_FILES_CHKSUM</filename>.
                The previous example employs the default directory.
            </para>

            <para>
                Consider this next example:
                <literallayout class='monospaced'>
     LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
                                         md5=bb14ed3c4cda583abc85401304b5cd4e"
     LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
                </literallayout>
            </para>

            <para>
                The first line locates a file in
                <filename>${S}/src/ls.c</filename> and isolates lines five
                through 16 as license text.
                The second line refers to a file in
                <filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>.
            </para>
            <para>
                Note that <filename>LIC_FILES_CHKSUM</filename> variable is
                mandatory for all recipes, unless the
                <filename>LICENSE</filename> variable is set to "CLOSED".
            </para>
        </section>

        <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
            <title>Explanation of Syntax</title>
            <para>
                As mentioned in the previous section, the
                <filename>LIC_FILES_CHKSUM</filename> variable lists all the
                important files that contain the license text for the source code.
                It is possible to specify a checksum for an entire file, or a specific section of a
                file (specified by beginning and ending line numbers with the "beginline" and "endline"
                parameters, respectively).
                The latter is useful for source files with a license notice header,
                README documents, and so forth.
                If you do not use the "beginline" parameter, then it is assumed that the text begins on the
                first line of the file.
                Similarly, if you do not use the "endline" parameter, it is assumed that the license text
                ends with the last line of the file.
            </para>

            <para>
                The "md5" parameter stores the md5 checksum of the license text.
                If the license text changes in any way as compared to this parameter
                then a mismatch occurs.
                This mismatch triggers a build failure and notifies the developer.
                Notification allows the developer to review and address the license text changes.
                Also note that if a mismatch occurs during the build, the correct md5
                checksum is placed in the build log and can be easily copied to the recipe.
            </para>

            <para>
                There is no limit to how many files you can specify using the
                <filename>LIC_FILES_CHKSUM</filename> variable.
                Generally, however, every project requires a few specifications for license tracking.
                Many projects have a "COPYING" file that stores the license information for all the source
                code files.
                This practice allows you to just track the "COPYING" file as long as it is kept up to date.
            </para>

            <tip>
                If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match
                error and displays the correct "md5" parameter value during the build.
                The correct parameter is also captured in the build log.
            </tip>

            <tip>
                If the whole file contains only license text, you do not need to use the "beginline" and
                "endline" parameters.
            </tip>
        </section>
    </section>

    <section id="enabling-commercially-licensed-recipes">
        <title>Enabling Commercially Licensed Recipes</title>

        <para>
            By default, the OpenEmbedded build system disables
            components that have commercial or other special licensing
            requirements.
            Such requirements are defined on a
            recipe-by-recipe basis through the
            <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
            variable definition in the affected recipe.
            For instance, the
            <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
            recipe contains the following statement:
            <literallayout class='monospaced'>
     LICENSE_FLAGS = "commercial"
            </literallayout>
            Here is a slightly more complicated example that contains both an
            explicit recipe name and version (after variable expansion):
            <literallayout class='monospaced'>
     LICENSE_FLAGS = "license_${PN}_${PV}"
            </literallayout>
	        In order for a component restricted by a <filename>LICENSE_FLAGS</filename>
	        definition to be enabled and included in an image, it
	        needs to have a matching entry in the global
	        <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link>
            variable, which is a variable
	        typically defined in your <filename>local.conf</filename> file.
            For example, to enable
	        the <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
	        package, you could add either the string
	        "commercial_gst-plugins-ugly" or the more general string
	        "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.
            See the
            "<link linkend='license-flag-matching'>License Flag Matching</link>" section
            for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works.
            Here is the example:
            <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
            </literallayout>
	        Likewise, to additionally enable the package built from the recipe containing
	        <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming
	        that the actual recipe name was <filename>emgd_1.10.bb</filename>,
	        the following string would enable that package as well as
	        the original <filename>gst-plugins-ugly</filename> package:
            <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
            </literallayout>
	        As a convenience, you do not need to specify the complete license string
	        in the whitelist for every package.
            You can use an abbreviated form, which consists
	        of just the first portion or portions of the license string before
	        the initial underscore character or characters.
            A partial string will match
	        any license that contains the given string as the first
	        portion of its license.
            For example, the following
	        whitelist string will also match both of the packages
	        previously mentioned as well as any other packages that have
	        licenses starting with "commercial" or "license".
            <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial license"
            </literallayout>
        </para>

        <section id="license-flag-matching">
            <title>License Flag Matching</title>

            <para>
		        License flag matching allows you to control what recipes the
                OpenEmbedded build system includes in the build.
                Fundamentally, the build system attempts to match
                <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
                strings found in recipes against
                <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link>
                strings found in the whitelist.
                A match causes the build system to include a recipe in the
                build, while failure to find a match causes the build system to
                exclude a recipe.
            </para>

            <para>
                In general, license flag matching is simple.
                However, understanding some concepts will help you
                correctly and effectively use matching.
            </para>

            <para>
                Before a flag
                defined by a particular recipe is tested against the
                contents of the whitelist, the expanded string
                <filename>_${PN}</filename> is appended to the flag.
                This expansion makes each <filename>LICENSE_FLAGS</filename>
                value recipe-specific.
                After expansion, the string is then matched against the
                whitelist.
                Thus, specifying
                <filename>LICENSE_FLAGS = "commercial"</filename>
                in recipe "foo", for example, results in the string
                <filename>"commercial_foo"</filename>.
                And, to create a match, that string must appear in the
                whitelist.
            </para>

            <para>
                Judicious use of the <filename>LICENSE_FLAGS</filename>
                strings and the contents of the
                <filename>LICENSE_FLAGS_WHITELIST</filename> variable
                allows you a lot of flexibility for including or excluding
                recipes based on licensing.
                For example, you can broaden the matching capabilities by
                using license flags string subsets in the whitelist.
                <note>When using a string subset, be sure to use the part of
                    the expanded string that precedes the appended underscore
                    character (e.g. <filename>usethispart_1.3</filename>,
                    <filename>usethispart_1.4</filename>, and so forth).
                </note>
                For example, simply specifying the string "commercial" in
                the whitelist matches any expanded
                <filename>LICENSE_FLAGS</filename> definition that starts with
                the string "commercial" such as "commercial_foo" and
                "commercial_bar", which are the strings the build system
                automatically generates for hypothetical recipes named
                "foo" and "bar" assuming those recipes simply specify the
                following:
                <literallayout class='monospaced'>
     LICENSE_FLAGS = "commercial"
                </literallayout>
                Thus, you can choose to exhaustively
                enumerate each license flag in the whitelist and
                allow only specific recipes into the image, or
                you can use a string subset that causes a broader range of
                matches to allow a range of recipes into the image.
            </para>

            <para>
                This scheme works even if the
                <filename>LICENSE_FLAGS</filename> string already
                has <filename>_${PN}</filename> appended.
                For example, the build system turns the license flag
                "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would
                match both the general "commercial" and the specific
                "commercial_1.2_foo" strings found in the whitelist, as
                expected.
            </para>

            <para>
                Here are some other scenarios:
                <itemizedlist>
                    <listitem><para>You can specify a versioned string in the
                        recipe such as "commercial_foo_1.2" in a "foo" recipe.
                        The build system expands this string to
                        "commercial_foo_1.2_foo".
                        Combine this license flag with a whitelist that has
                        the string "commercial" and you match the flag along
                        with any other flag that starts with the string
                        "commercial".</para></listitem>
                    <listitem><para>Under the same circumstances, you can
                        use "commercial_foo" in the whitelist and the
                        build system not only matches "commercial_foo_1.2" but
                        also matches any license flag with the string
                        "commercial_foo", regardless of the version.
                        </para></listitem>
                    <listitem><para>You can be very specific and use both the
                        package and version parts in the whitelist (e.g.
                        "commercial_foo_1.2") to specifically match a
                        versioned recipe.</para></listitem>
                </itemizedlist>
            </para>
        </section>

        <section id="other-variables-related-to-commercial-licenses">
            <title>Other Variables Related to Commercial Licenses</title>

            <para>
                Other helpful variables related to commercial
                license handling exist and are defined in the
                <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file:
                <literallayout class='monospaced'>
     COMMERCIAL_AUDIO_PLUGINS ?= ""
     COMMERCIAL_VIDEO_PLUGINS ?= ""
                </literallayout>
                If you want to enable these components, you can do so by making sure you have
                statements similar to the following
                in your <filename>local.conf</filename> configuration file:
                <literallayout class='monospaced'>
     COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
        gst-plugins-ugly-mpegaudioparse"
     COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
        gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
                </literallayout>
                Of course, you could also create a matching whitelist
                for those components using the more general "commercial"
                in the whitelist, but that would also enable all the
                other packages with
                <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
                containing "commercial", which you may or may not want:
                <literallayout class='monospaced'>
     LICENSE_FLAGS_WHITELIST = "commercial"
                </literallayout>
            </para>

            <para>
                Specifying audio and video plug-ins as part of the
                <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
                <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements
                (along with the enabling
                <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the
                plug-ins or components into built images, thus adding
                support for media formats or components.
            </para>
        </section>
    </section>
</section>
</chapter>
<!--
vim: expandtab tw=80 ts=4
-->