path: root/documentation/kernel-dev/kernel-dev-advanced.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >

<chapter id='kernel-dev-advanced'>

<title>Working with Advanced Metadata</title>

    <para>
        In addition to configuration fragments and patches, the
        Yocto Project kernel tools support rich metadata that you can
        use to define complex policies and BSP support.
        The purpose of the metadata and the tools to manage it, known as
        the kern-tools (<filename>kern-tools-native_git.bb</filename>), is
        to assist in managing the complexity of the configuration and sources
        in support of multiple Board Support Packages (BSPs) and Linux kernel
        types.
    </para>

    <para>
        In particular, the kernel tools allow you to specify only what you
        must, and nothing more.
        Where a complete Linux kernel <filename>.config</filename> includes
        all the automatically selected <filename>CONFIG</filename> options,
        the configuration fragments only need to contain the highest level
        visible <filename>CONFIG</filename> options as presented by the Linux
        kernel <filename>menuconfig</filename> system.
        This reduces your maintenance effort and allows you
        to further separate your configuration in ways that make sense for
        your project.
        A common split is policy and hardware.
        For example, all your kernels might support
        the <filename>proc</filename> and <filename>sys</filename> filesystems,
        but only specific boards will require sound, USB, or specific drivers.
        Specifying these individually allows you to aggregate them
        together as needed, but maintain them in only one place.
        Similar logic applies to source changes.
    </para>

    <para>
        Original Text:
        <literallayout class='monospaced'>
In addition to configuration fragments and patches, the Yocto Project kernel
tools support rich metadata which you can use to define complex policies and
BSP support. The purpose of the metadata and the tools to manage it, known as
the kern-tools (kern-tools-native_git.bb), is to assist in managing the
complexity of the configuration and sources in support of multiple BSPs and
Linux kernel types.

In particular, the kernel tools allow you to specify only what you must, and
nothing more.  Where a complete Linux kernel .config includes all the
automatically selected CONFIG options, the configuration fragments only need to
contain the highest level visible CONFIG options as presented by the Linux
kernel menuconfig system. This reduces your maintenance effort and allows you
to further separate your configuration in ways that make sense for your project.
A common split is policy and hardware. For example, all your kernels may support
the proc and sys filesystems, but only specific boards will require sound, usb,
or specific drivers. Specifying these individually allows you to aggregate them
together as needed, but maintain them in only one place. Similar logic applies
to source changes.
        </literallayout>
    </para>

<section id='using-metadata-in-a-recipe'>
    <title>Using Metadata in a Recipe</title>

    <para>
        The metadata provided with any linux-yocto style Linux kernel sources
        must define a BSP that corresponds to the definition laid out in the
        recipe.
        A BSP consists of an aggregation of kernel policy and hardware specific
        feature enablement.
        This can be influenced from within the recipe.
    </para>

    <para>
        Every linux-yocto style recipe must define the following variable:
        <literallayout class='monospaced'>
     KMACHINE
        </literallayout>
        <filename>KMACHINE</filename> is typically set to the same value as
        used within the recipe-space BSP definition, such as "routerstationpro"
        or "fri2".
        However, multiple BSPs can reuse the same <filename>KMACHINE</filename>
        name if they are built using the same BSP description.
        See section 3.3.5 for more information.
        The <filename>meta-intel</filename> "fri2" and "fri2-noemgd" are good
        examples of such a situation where each specifies
        <filename>KMACHINE</filename> as "fri2".
    </para>

    <para>
        They may optionally define the following variables:
        <literallayout class='monospaced'>
     KBRANCH
     KERNEL_FEATURES
     KBRANCH_DEFAULT
     LINUX_KERNEL_TYPE
        </literallayout>
        <filename>KBRANCH_DEFAULT</filename> defines the default source branch
        within the Linux kernel source repository to be used to build the
        Linux kernel.
        It is used as the default value for <filename>KBRANCH</filename> which
        may define an alternate branch, typically with a machine override,
        such as:
        <literallayout class='monospaced'>
     KBRANCH_fri2 = "standard/fri2"
        </literallayout>
        Unless you specify otherwise, <filename>KBRANCH_DEFAULT</filename>
        is initialized to "master".
    </para>

    <para>
        <filename>LINUX_KERNEL_TYPE</filename> defines the kernel type to be
        used in assembling the configuration and defaults to "standard"
        if you do not specify otherwise.
        Together with <filename>KMACHINE</filename>, this defines the search
        arguments used by the Yocto Project Linux kernel tools to find the
        appropriate description within the metadata with which to build out
        the sources and configuration.
        The linux-yocto recipes define "standard", "tiny", and "preempt-rt"
        kernel types.
        See section 3.3.4 for more inforation on kernel types.
    </para>

    <para>
        During the build, the kern-tools will search for the BSP description
        file that most closely matches the <filename>KMACHINE</filename>
        and <filename>LINUX_KERNEL_TYPE</filename> passed in from the
        recipe.
        It will use the first BSP description it finds matching both variables.
        Failing that it will issue a warning such as the following:
        <literallayout class='monospaced'>
     WARNING: Can't find any BSP hardware or required configuration fragments.
     WARNING: Looked at meta/cfg/broken/fri2-broken/hdw_frags.txt and
              meta/cfg/broken/fri2-broken/required_frags.txt in directory:
              meta/cfg/broken/fri2-broken
        </literallayout>
        In this example, <filename>KMACHINE</filename> was set to "fri2-broken"
        and <filename>LINUX_KERNEL_TYPE</filename> was set to "broken".
    </para>

    <para>
        It will then search first for the <filename>KMACHINE</filename> and
        then for the <filename>LINUX_KERNEL_TYPE</filename>.
        If it cannot find a partial match, it will use the
        sources from the <filename>KBRANCH</filename> and any configuration
        specified in the <filename>SRC_URI</filename>.
    </para>

    <para>
        <filename>KERNEL_FEATURES</filename> can be used to include features
        (configuration fragments, patches, or both) that are not already
        included by the <filename>KMACHINE</filename> and
        <filename>LINUX_KERNEL_TYPE</filename> combination.
        To include a feature specified as "features/netfilter.scc" for example,
        specify:
        <literallayout class='monospaced'>
     KERNEL_FEATURES += "features/netfilter.scc"
        </literallayout>
        To include a feature called "cfg/sound.scc" just for the
        <filename>qemux86</filename> machine, specify:
        <literallayout class='monospaced'>
     KERNEL_FEATURES_append_qemux86 = "cfg/sound.scc"
        </literallayout>
        The value of the entries in <filename>KERNEL_FEATURES</filename>
        are dependent on their location within the metadata itself.
        The examples here are taken from the
        <filename>linux-yocto-3.4</filename> repository where "features"
        and "cfg" are subdirectories of the <filename>metadata</filename>
        directory.
        For details, see section 3.3.
        <note>
            The processing of the these variables has evolved some between the
	        0.9 and 1.3 releases of the Yocto Project and associated
	        kern-tools sources.
            The descriptions in this section are accurate for 1.3 and later
	        releases of the Yocto Project.
        </note>
    </para>

    <para>
        Original Text.
        <literallayout class='monospaced'>
The metadata provided with any linux-yocto style Linux kernel sources must
define a BSP that corresponds to the definition laid out in the recipe. A BSP
consists of an aggregation of kernel policy and hardware specific feature
enablement. This can be influenced from within the recipe.

Every linux-yocto style recipe must define the following variables:

	KMACHINE

KMACHINE is typically set to the same value as used within the recipe-space BSP
definition, such as "routerstationpro" or "fri2". However, multiple BSPs can
reuse the same KMACHINE name if they are built using the same BSP description
(see 3.3.5). The meta-intel "fri2" and "fri2-noemgd" are good examples of such
a situation where each specifies KMACHINE as "fri2".

They may optionally define the following variables:
	KBRANCH
	KERNEL_FEATURES
	KBRANCH_DEFAULT
	LINUX_KERNEL_TYPE

KBRANCH_DEFAULT defines the default source branch within the Linux kernel source
repository to be used to build the Linux kernel. It is used as the default value
for KBRANCH which may define an alternate branch, typically with a machine
override, such as:

KBRANCH_fri2 = "standard/fri2"

Unless you specify otherwise, KBRANCH_DEFAULT is initialized to "master".

LINUX_KERNEL_TYPE defines the kernel type to be used in assembling the
configuration and defaults to "standard" if you do not specify otherwise.
Together with KMACHINE, this defines the search arguments used by the Yocto
Project Linux kernel tools to find the appropriate description within the
metadata with which to build out the sources and configuration. The linux-yocto
recipes define "standard", "tiny", and "preempt-rt" kernel types. See 3.3.4 for
more inforation on kernel types.

During the build, the kern-tools will search for the BSP description file that
most closely matches the KMACHINE and LINUX_KERNEL_TYPE passed in from the
recipe. It will use the first BSP description it finds matching both variables.
Failing that it will issue a warning such as the following:

	WARNING: Can't find any BSP hardware or required configuration fragments.
	WARNING: Looked at meta/cfg/broken/fri2-broken/hdw_frags.txt and
	         meta/cfg/broken/fri2-broken/required_frags.txt in directory:
	         meta/cfg/broken/fri2-broken

	In this example KMACHINE was set to "fri2-broken" and LINUX_KERNEL_TYPE
	was set to "broken".

It will then search first for the KMACHINE and then
for the LINUX_KERNEL_TYPE. If it cannot find a partial match, it will use the
sources from the KBRANCH and any configuration specified in the SRC_URI.

KERNEL_FEATURES can be used to include features (configuration fragments,
patches, or both) that are not already included by the KMACHINE and
LINUX_KERNEL_TYPE combination. To include a feature specified as
"features/netfilter.scc" for example, specify:

KERNEL_FEATURES += "features/netfilter.scc"

To include a feature called "cfg/sound.scc" just for the qemux86 machine,
specify:

KERNEL_FEATURES_append_qemux86 = "cfg/sound.scc"

The value of the entries in KERNEL_FEATURES are dependent on their location
within the metadata itself. The examples here are taken from the
linux-yocto-3.4 repository where "features" and "cfg" are subdirectories of the
metadata directory. For details, see 3.3.

	Note: The processing of the these variables has evolved some between the
	      0.9 and 1.3 releases of the Yocto Project and associated
	      kern-tools sources. The above is accurate for 1.3 and later
	      releases of the Yocto Project.
        </literallayout>
    </para>
</section>

<section id='metadata-location'>
    <title>Metadata Location</title>

    <para>
        This metadata can be defined along with the Linux kernel
        recipe (recipe-space) as partially described in the
        "<link linkend='modifying-an-existing-recipe'>Modifying an Existing Recipe</link>"
        section as well as within the Linux kernel sources themselves
        (in-tree).
    </para>

    <para>
        Where you choose to store the metadata depends on what you want
        to do and how you intend to work.
        If you are unfamiliar with the Linux kernel and only wish
        to apply a config and possibly a couple of patches provided to
        you by others, you may find the recipe-space mechanism to be easier
        to work with.
        This is also a good approach if you are working with Linux kernel
        sources you do not control or if you just don't want to maintain a
        Linux kernel git repository on your own.
    </para>

    <para>
        If you are doing active kernel development and are already
        maintaining a Linux kernel git repository of your own, you may find
        it more convenient to work with the metadata in the same
        repository as the Linux kernel sources.
        This can make iterative development of the Linux kernel more efficient
        outside of the BitBake environment.

    </para>

    <para>
        Regardless of where the meta-data is stored, the syntax as
        described in the following sections applies equally.
    </para>

    <para>
        Original Text:
        <literallayout class='monospaced'>
This meta-data can be defined along with the Linux kernel recipe (recipe-space)
as partially described in section 2.2 as well as within the Linux kernel sources
themselves (in-tree).

Where you choose to store the meta-data depends on what you want to do and how
you intend to work. If you are unfamiliar with the Linux kernel and only wish
to apply a config and possibly a couple of patches provided to you by others,
you may find the recipe-space mechanism to be easier to work with. This is also
a good approach if you are working with Linux kernel sources you do not control
or if you just don't want to maintain a Linux kernel git repository on your own.

If you are doing active kernel development and are already maintaining a Linux
kernel git repository of your own, you may find it more convenient to work with
the meta-data in the same repository as the Linux kernel sources. This can make
iterative development of the Linux kernel more efficient outside of the bitbake
environment.

Regardless of where the meta-data is stored, the syntax as
described in the following sections applies equally.
        </literallayout>
    </para>

    <section id='recipe-space-metadata'>
        <title>Recipe-Space Metadata</title>

        <para>
            When stored in recipe-space, the metadata files reside in a
            directory hierarchy below
            <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink>,
            which is typically set to
            <filename>${THISDIR}/${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename>
            for a linux-yocto or linux-yocto-custom derived Linux kernel
            recipe.
            See the "<link linkend='modifying-an-existing-recipe'>Modifying an Existing Recipe</link>"
            section for more information.
        </para>

        <para>
            By way of example, a trivial tree of metadata stored in
            recipe-space within a BSP layer might look like the following:
            <literallayout class='monospaced'>
     meta/
     `-- recipes-kernel
         `-- linux
             `-- linux-yocto
                 |-- bsp-standard.scc
                 |-- bsp.cfg
                 `-- standard.cfg
            </literallayout>
        </para>

        <para>
            When the metadata is stored in recipe-space, you must take
            steps to ensure BitBake has the necessary information to decide
            which files to fetch and when they need to be fetched again.
        </para>

        <para>
            It is only necessary to specify the <filename>.scc</filename>
            files on the
            <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>.
            BitBake parses them and fetches any files referenced in the
            <filename>.scc</filename> files by the <filename>include</filename>,
            <filename>patch</filename>, or <filename>kconf</filename> commands.
            Because of this, it is necessary to bump the recipe
            <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
            value when changing the content of files not explicitly listed
            in the SRC_URI.
        </para>

        <para>
            Original text:
            <literallayout class='monospaced'>
When stored in recipe-space, the meta-data files reside in a directory hierarchy
below FILESEXTRAPATHS, which is typically set to ${THISDIR}/${PN} for a
linux-yocto or linux-yocto-custom derived Linux kernel recipe. See 2.2.

By way of example, a trivial tree of meta-data stored in recipe-space within a
BSP layer might look like the following:

meta/
`-- recipes-kernel
    `-- linux
        `-- linux-yocto
            |-- bsp-standard.scc
            |-- bsp.cfg
            `-- standard.cfg

When the meta-data is stored in recipe-space, you must take steps to ensure
bitbake has the necessary information to decide which files to fetch and when
they need to be fetched again.

It is only necessary to specify the .scc files on the SRC_URI; bitbake will
parse them and fetch any files referenced in the .scc files by the include,
patch, or kconf commands. Because of this, it is necessary to bump the recipe PR
value when changing the content of files not explicitly listed in the SRC_URI.
            </literallayout>
        </para>
    </section>

    <section id='in-tree-metadata'>
        <title>In-Tree Metadata</title>

        <para>
            When stored in-tree, the metadata files reside in the
            "meta" directory of the Linux kernel sources.
            They may be present in the same branch as the sources,
            such as "master", or in their own orphan branch,
            typically named "meta".
            An orphan branch in Git is a branch with unique history and
            content to the other branches in the repository.
            This is useful to track metadata changes independently from the
            sources of the Linux kernel, while still keeping them
            together in the same repository.
            For the purposes of this document, we will discuss all
            in-tree metadata as residing below the
            <filename>meta/cfg/kernel-cache</filename> directory.
        </para>

        <para>
            By way of example, a trivial tree of metadata stored in a custom
            Linux kernel Git repository might look like the following:
            <literallayout class='monospaced'>
     meta/
     `-- cfg
         `-- kernel-cache
             |-- bsp-standard.scc
             |-- bsp.cfg
             `-- standard.cfg
            </literallayout>
        </para>

        <para>
            To use a specific branch for the metadata, specify the branch
            in the <filename>KMETA</filename> variable in your Linux kernel
            recipe, for example:
            <literallayout class='monospaced'>
     KMETA = "meta"
            </literallayout>
            To use the same branch as the sources, set
            <filename>KMETA</filename> to the empty string:
            <literallayout class='monospaced'>
     KMETA = ""
            </literallayout>
            If you are working with your own sources and want to create an
            orphan <filename>meta</filename> branch, you can do so using the
            following commands from within your Linux kernel Git repository:
            <literallayout class='monospaced'>
     $ git checkout --orphan meta
     $ git rm -rf .
     $ git commit --allow-empty -m "Create orphan meta branch"
            </literallayout>
        </para>

        <para>
            Original text:
            <literallayout class='monospaced'>
When stored in-tree, the meta-data files reside in the "meta" directory of the
Linux kernel sources. They may be present in the same branch as the sources,
such as "master", or in their own orphan branch, typically named "meta". An
orphan branch in git is a branch with unique history and content to the other
branches in the repository. This is useful to track meta-data changes
independently from the sources of the Linux kernel, while still keeping them
together in the same repository. For the purposes of this document, we will
discuss all in-tree meta-data as residing below the "meta/cfg/kernel-cache"
directory.

By way of example, a trivial tree of meta-data stored in a custom Linux kernel
git repository might look like the following:

meta/
`-- cfg
    `-- kernel-cache
        |-- bsp-standard.scc
        |-- bsp.cfg
        `-- standard.cfg

To use a specific branch for the meta-data, specify the branch in the KMETA
variable in your Linux kernel recipe, for example:

	KMETA = "meta"

To use the same branch as the sources, set KMETA to the empty string:

	KMETA = ""

If you are working with your own sources and want to create an orphan meta
branch, you can do so using the following commands from within your Linux kernel
git repository:

	$ git checkout --orphan meta
	$ git rm -rf .
	$ git commit --allow-empty -m "Create orphan meta branch"
            </literallayout>
        </para>
    </section>
</section>

<section id='metadata-syntax'>
    <title>Metadata Syntax</title>

    <para>
        The Yocto Project Linux kernel tools metadata consists of three
        primary types of files: <filename>scc</filename>
        <footnote>
            <para>
                <filename>scc</filename> stands for Series Configuration
                Control, but the naming has less significance in the
                current implementation of the tooling than it had in the
                past.
                Consider it to be a description file.
            </para>
        </footnote>
        description files, configuration fragments, and patches.
        The <filename>scc</filename> files define variables and include or
        otherwise reference any of the three file types.
        The description files are used to aggregate all types of metadata into
        what ultimately describes the sources and the configuration required
        to build a Linux kernel tailored to a specific machine.
    </para>

    <para>
        The <filename>scc</filename> description files are used to define two
        fundamental types of metadata:
        <itemizedlist>
            <listitem><para>Features</para></listitem>
            <listitem><para>Board Support Packages (BSPs)</para></listitem>
        </itemizedlist>
    </para>

    <para>
        Features aggregate sources in the form of patches and configuration
        in the form of configuration fragments into a modular reusable unit.
        Features are used to implement conceptually separate metadata
        descriptions like pure configuration fragments, simple patches,
        complex features, and kernel types (ktypes).
        Kernel types define general kernel features and policy to be reused
        in the BSPs.
    </para>

    <para>
        BSPs define hardware-specific features and aggregate them with kernel
        types to form the final description of what will be assembled and built.
    </para>

    <para>
        While the metadata syntax does not enforce any logical separation of
        configuration fragments, patches, features or kernel types, best
        practices dictate a logical separation of these types of meta-data.
        The following metadata file hierarchy is recommended:
        <literallayout class='monospaced'>
     &lt;base&gt;/
        bsp/
        cfg/
        features/
        ktypes/
        patches/
        </literallayout>
    </para>

    <para>
        The <filename>bsp</filename> directory should contain the
        BSP descriptions, described in detail in section 3.3.5.
        The remaining directories all contain "features"; the separation
        is meant to aid in conceptualizing their intended usage.
        A simple guide to determine where your <filename>scc</filename>
        description file should go is as follows.
        If it contains only configuration fragments, it belongs in
        <filename>cfg</filename>.
        If it contains only source-code fixes, it belongs in
        <filename>patches</filename>.
        If it encapsulates a major feature, often combining sources and
        configurations, it belongs in <filename>features</filename>.
        If it aggregates non-hardware configuration and patches
        in order to define a base kernel policy or major kernel type to
        be reused across multiple BSPs, it belongs in
        <filename>ktypes</filename>.
    </para>

    <para>
        The lines between these can easily become blurred, especially as
        out-of-tree features are slowly merged upstream over time.
        Also remember that this is purely logical organization and has
        no impact on the functionality of the metadata as
        all of <filename>cfg</filename>, <filename>features</filename>,
        <filename>patches</filename>, and <filename>ktypes</filename>,
        contain "features" as far as the Yocto Project Linux kernel
        tools are concerned.
    </para>

    <para>
        Paths used in metadata files are relative to
        <filename>&lt;base&gt;</filename>, which is either
        <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink>
        if you are creating metadata in recipe-space as described in
        section "<link linkend='recipe-space-metadata'>Recipe-Space Metadata</link>",
        or <filename>meta/cfg/kernel-cache/</filename> if you are creating
        metadata in-tree as described in
        the "<link linkend='in-tree-metadata'>In-Tree Metadata</link>" section.
    </para>            

    <para>
        Original text:
        <literallayout class='monospaced'>
The Yocto Project Linux kernel tools meta-data consists of three primary types
of files: scc* description files, configuration fragments, and patches. The scc
files define variables and include or otherwise reference any of the three file
types. The description files are used to aggregate all types of meta-data into
what ultimately describes the sources and the configuration required to build a
Linux kernel tailored to a specific machine.

The scc description files are used to define two fundamental types of meta-data:
	o Features
	o BSPs

Features aggregate sources in the form of patches and configuration in the form
of configuration fragments into a modular reusable unit. Features are used to
implement conceptually separate meta-data descriptions like pure configuration
fragments, simple patches, complex features, and kernel types (ktypes). Kernel
types define general kernel features and policy to be reused in the BSPs.

BSPs define hardware-specific features and aggregate them with kernel types to
form the final description of what will be assembled and built.

While the meta-data syntax does not enforce any logical separation of
configuration fragments, patches, features or kernel types, best practices
dictate a logical separation of these types of meta-data. The following
meta-data file hierarchy is recommended:

	&lt;base&gt;/
		bsp/
		cfg/
		features/
		ktypes/
		patches/

The bsp directory should contain the BSP descriptions, described in detail in
3.3.5. The remaining directories all contain "features"; the separation is meant
to aid in conceptualizing their intended usage.  A simple guide to determine
where your scc description file should go is as follows. If it contains only
configuration fragments, it belongs in cfg. If it contains only source-code
fixes, it belongs in patches. If it encapsulates a major feature, often
combining sources and configurations, it belongs in features. If it aggregates
non-hardware configuration and patches in order to define a base kernel policy
or major kernel type to be reused across multiple BSPs, it belongs in ktypes.
The line between these can easily become blurred, especially as out-of-tree
features are slowly merged upstream over time. Also remember that this is purely
logical organization and has no impact on the functionality of the meta-data as
all of cfg, features, patches, and ktypes, contain "features" as far as the
Yocto Project Linux kernel tools are concerned.

Paths used in meta-data files are relative to &lt;base&gt; which is either
FILESEXTRAPATHS if you are creating meta-data in recipe-space (see 3.2.1), or
meta/cfg/kernel-cache/ if you are creating meta-data in-tree (see 3.2.2).

* scc stands for Series Configuration Control, but the naming has less
  significance in the current implementation of the tooling than it had in the
  past. Consider it to be a description file.
        </literallayout>
    </para>




</section>

</chapter>
<!--
vim: expandtab tw=80 ts=4
-->