path: root/documentation/kernel-dev/kernel-dev-concepts-appx.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >

<appendix id='kernel-dev-concepts-appx'>
<title>Advanced Kernel Concepts</title>

    <section id='kernel-big-picture'>
        <title>Yocto Project Kernel Development and Maintenance</title>

        <para>
            Kernels available through the Yocto Project (Linux Yocto kernels),
            like other kernels, are based off the Linux kernel releases from
            <ulink url='http://www.kernel.org'></ulink>.
            At the beginning of a major Linux kernel development cycle, the
            Yocto Project team chooses a Linux kernel based on factors such as
            release timing, the anticipated release timing of final upstream
            <filename>kernel.org</filename> versions, and Yocto Project
            feature requirements.
            Typically, the Linux kernel chosen is in the final stages of
            development by the Linux community.
            In other words, the Linux kernel is in the release candidate
            or "rc" phase and has yet to reach final release.
            But, by being in the final stages of external development, the
            team knows that the <filename>kernel.org</filename> final release
            will clearly be within the early stages of the Yocto Project
            development window.
        </para>

        <para>
            This balance allows the Yocto Project team to deliver the most
            up-to-date Linux Yocto kernel possible, while still ensuring that
            the team has a stable official release for the baseline Linux
            kernel version.
        </para>

        <para>
            As implied earlier, the ultimate source for Linux Yocto kernels
            are released kernels from <filename>kernel.org</filename>.
            In addition to a foundational kernel from
            <filename>kernel.org</filename>, the available Yocto Linux kernels
            contain a mix of important new mainline developments, non-mainline
            developments (when there is no alternative), Board Support Package
            (BSP) developments, and custom features.
            These additions result in a commercially released Yocto
            Project Linux kernel that caters to specific embedded designer
            needs for targeted hardware.
        </para>

        <para>
            You can find a web interface to the Yocto Linux kernels in the
            <ulink url='&YOCTO_DOCS_REF_URL;#source-repositories'>Source Repositories</ulink>
            at
            <ulink url='&YOCTO_GIT_URL;'></ulink>.
            If you look at the interface, you will see to the left a
            grouping of Git repositories titled "Yocto Linux Kernel".
            Within this group, you will find several Linux Yocto kernels
            developed and included with Yocto Project releases:
            <itemizedlist>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.1</filename>:</emphasis>
                    The stable Yocto Project kernel to use with the Yocto
                    Project Release 2.0.
                    This kernel is based on the Linux 4.1 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.4</filename>:</emphasis>
                    The stable Yocto Project kernel to use with the Yocto
                    Project Release 2.1.
                    This kernel is based on the Linux 4.4 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.9</filename>:</emphasis>
                    The stable Yocto Project kernel to use with the Yocto
                    Project Release 2.3.
                    This kernel is based on the Linux 4.9 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.10</filename>:</emphasis>
                    The default stable Yocto Project kernel to use with the
                    Yocto Project Release 2.3.
                    This kernel is based on the Linux 4.10 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-dev</filename>:</emphasis>
                    A development kernel based on the latest upstream release
                    candidate available.
                    </para></listitem>
            </itemizedlist>
            <note><title>Notes</title>
                Long Term Support Initiative (LTSI) for Yocto Linux
                kernels is as follows:
                <itemizedlist>
                    <listitem><para>
                        For Yocto Project releases 1.7, 1.8, and 2.0,
                        the LTSI kernel is
                        <filename>linux-yocto-3.14</filename>.
                        </para></listitem>
                    <listitem><para>
                        For Yocto Project releases 2.1, 2.2, and 2.3,
                        the LTSI kernel is <filename>linux-yocto-4.1</filename>.
                        </para></listitem>
                    <listitem><para>
                        <filename>linux-yocto-4.1</filename>,
                        <filename>linux-yocto-4.4</filename>, and
                        <filename>linux-yocto-4.9</filename> are all LTS
                        kernels.
                        </para></listitem>
                </itemizedlist>
            </note>
        </para>

        <para>
            Once a Yocto Linux kernel is officially released, the Yocto
            Project team goes into their next development cycle, or upward
            revision (uprev) cycle, while still continuing maintenance on the
            released kernel.
            It is important to note that the most sustainable and stable way
            to include feature development upstream is through a kernel uprev
            process.
            Back-porting hundreds of individual fixes and minor features from
            various kernel versions is not sustainable and can easily
            compromise quality.
        </para>

        <para>
            During the uprev cycle, the Yocto Project team uses an ongoing
            analysis of Linux kernel development, BSP support, and release
            timing to select the best possible <filename>kernel.org</filename>
            Linux kernel version on which to base subsequent Yocto Linux
            kernel development.
            The team continually monitors Linux community kernel development
            to look for significant features of interest.
            The team does consider back-porting large features if they have a
            significant advantage.
            User or community demand can also trigger a back-port or creation
            of new functionality in the Yocto Project baseline kernel during
            the uprev cycle.
        </para>

        <para>
            Generally speaking, every new Linux kernel both adds features and
            introduces new bugs.
            These consequences are the basic properties of upstream
            Linux kernel development and are managed by the Yocto Project
            team's Linux Yocto kernel development strategy.
            It is the Yocto Project team's policy to not back-port minor
            features to the released Linux Yocto kernel.
            They only consider back-porting significant technological
            jumps &dash; and, that is done after a complete gap analysis.
            The reason for this policy is that back-porting any small to
            medium sized change from an evolving Linux kernel can easily
            create mismatches, incompatibilities and very subtle errors.
        </para>

        <para>
            The policies described in this section result in both a stable
            and a cutting edge Linux Yocto kernel that mixes forward ports of
            existing Linux kernel features and significant and critical new
            functionality.
            Forward porting Linux kernel functionality into the Linux Yocto
            kernels available through the Yocto Project can be thought of as
            a "micro uprev."
            The many “micro uprevs” produce a Linux Yocto kernel version with
            a mix of important new mainline, non-mainline, BSP developments
            and feature integrations.
            This Yocto Linux kernel gives insight into new features and
            allows focused amounts of testing to be done on the kernel,
            which prevents surprises when selecting the next major uprev.
            The quality of these cutting edge Linux Yocto kernels is evolving
            and the kernels are used in leading edge feature and BSP
            development.
        </para>
    </section>

    <section id='kernel-architecture'>
        <title>Kernel Architecture</title>

        <para>
            This section describes the architecture of the Yocto Linux kernels
            available through the Yocto Project and provides information
            on the mechanisms used to achieve that architecture.
        </para>

        <section id='architecture-overview'>
            <title>Overview</title>

            <para>
                As mentioned earlier, a key goal of the Yocto Project is
                to present the developer with a kernel that has a clear and
                continuous history that is visible to the user.
                The architecture and mechanisms used achieve that goal in a
                manner similar to upstream Linux kernel development in
                <filename>kernel.org</filename>.
            </para>

            <para>
                You can think of a Yocto Linux kernel as consisting of a
                baseline Linux kernel with added features logically structured
                on top of the baseline.
                The features are tagged and organized by way of a branching
                strategy implemented by the Yocto Project team using the
                Source Code Manager (SCM) Git.
                <note><title>Notes</title>
                    <itemizedlist>
                        <listitem><para>
                            Git is the obvious SCM for meeting the Yocto Linux
                            kernel organizational and structural goals
                            described in this section.
                            Not only is Git the SCM for Linux kernel
                            development in <filename>kernel.org</filename> but,
                            Git continues to grow in popularity and supports
                            many different work flows, front-ends and
                            management techniques.
                            </para></listitem>
                        <listitem><para>
                            You can find documentation on Git at
                            <ulink url='http://git-scm.com/documentation'></ulink>.
                            You can also get an introduction to Git as it
                            applies to the Yocto Project in the
                            "<ulink url='&YOCTO_DOCS_REF_URL;#git'>Git</ulink>"
                            section in the Yocto Project Reference Manual.
                            The latter reference provides an overview of
                            Git and presents a minimal set of Git commands
                            that allows you to be functional using Git.
                            You can use as much, or as little, of what Git
                            has to offer to accomplish what you need for your
                            project.
                            You do not have to be a "Git Expert" in order to
                            use it with the Yocto Project.
                            </para></listitem>
                    </itemizedlist>
                </note>
            </para>

            <para>
                The result is that the user has the ability to see the added
                features and the commits that make up those features.
                In addition to being able to see added features, the user
                can also view the history of what made up the baseline
                Linux kernel.
            </para>

            <para>
                The following illustration shows the conceptual Yocto
                Linux kernel.
                <imagedata fileref="figures/kernel-architecture-overview.png" width="6in" depth="7in" align="center" scale="100" />
            </para>

            <para>
                In the illustration, the "Kernel.org Branch Point" marks the
                specific spot (or Linux kernel release) from which the
                Yocto Linux kernel is created.
                From this point forward in the tree, features and differences
                are organized and tagged.
            </para>

            <para>
                The "Yocto Project Baseline Kernel" contains functionality that
                is common to every kernel type and BSP that is organized
                further along in the tree.
                Placing these common features in the tree this way means
                features do not have to be duplicated along individual
                branches of the tree structure.
            </para>

            <para>
                From the Yocto Project Baseline Kernel, branch points represent
                specific functionality for individual Board Support Packages
                (BSPs) as well as real-time kernels.
                The illustration represents this through three BSP-specific
                branches and a real-time kernel branch.
                Each branch represents some unique functionality for the BSP
                or for a real-time Yocto Linux kernel.
            </para>

            <para>
                In this example structure, the real-time kernel branch has
                common features for all real-time Yocto Linux kernels and
                contains more branches for individual BSP-specific real-time
                kernels.
                The illustration shows three branches as an example.
                Each branch points the way to specific, unique features for a
                respective real-time kernel as they apply to a given BSP.
            </para>

            <para>
                The resulting tree structure presents a clear path of markers
                (or branches) to the developer that, for all practical
                purposes, is the Yocto Linux kernel needed for any given set of
                requirements.
                <note>
                    Keep in mind the figure does not take into account all the
                    supported Yocto Linux kernels, but rather shows a single
                    generic kernel just for conceptual purposes.
                    Also keep in mind that this structure represents the Yocto
                    Project
                    <ulink url='&YOCTO_DOCS_REF_URL;#source-repositories'>Source Repositories</ulink>
                    that are either pulled from during the build or established
                    on the host development system prior to the build by either
                    cloning a particular kernel's Git repository or by
                    downloading and unpacking a tarball.
                </note>
            </para>
        </section>

        <section id='branching-and-workflow'>
            <title>Branching Strategy and Workflow</title>
            <para>
                The Yocto Project team creates kernel branches at points where functionality is
                no longer shared and thus, needs to be isolated.
                For example, board-specific incompatibilities would require different functionality
                and would require a branch to separate the features.
                Likewise, for specific kernel features, the same branching strategy is used.
            </para>
            <para>
                This branching strategy results in a tree that has features organized to be specific
                for particular functionality, single kernel types, or a subset of kernel types.
                This strategy also results in not having to store the same feature twice
                internally in the tree.
                Rather, the kernel team stores the unique differences required to apply the
                feature onto the kernel type in question.
                <note>
                    The Yocto Project team strives to place features in the tree such that they can be
                    shared by all boards and kernel types where possible.
                    However, during development cycles or when large features are merged,
                    the team cannot always follow this practice.
                    In those cases, the team uses isolated branches to merge features.
                </note>
            </para>
            <para>
                BSP-specific code additions are handled in a similar manner to kernel-specific additions.
                Some BSPs only make sense given certain kernel types.
                So, for these types, the team creates branches off the end of that kernel type for all
                of the BSPs that are supported on that kernel type.
                From the perspective of the tools that create the BSP branch, the BSP is really no
                different than a feature.
                Consequently, the same branching strategy applies to BSPs as it does to features.
                So again, rather than store the BSP twice, the team only stores the unique
                differences for the BSP across the supported multiple kernels.
            </para>
            <para>
                While this strategy can result in a tree with a significant number of branches, it is
                important to realize that from the developer's point of view, there is a linear
                path that travels from the baseline <filename>kernel.org</filename>, through a select
                group of features and ends with their BSP-specific commits.
                In other words, the divisions of the kernel are transparent and are not relevant
                to the developer on a day-to-day basis.
                From the developer's perspective, this path is the "master" branch.
                The developer does not need to be aware of the existence of any other branches at all.
                Of course, there is value in the existence of these branches
                in the tree, should a person decide to explore them.
                For example, a comparison between two BSPs at either the commit level or at the line-by-line
                code <filename>diff</filename> level is now a trivial operation.
            </para>
            <para>
                Working with the kernel as a structured tree follows recognized community best practices.
                In particular, the kernel as shipped with the product, should be
                considered an "upstream source" and viewed as a series of
                historical and documented modifications (commits).
                These modifications represent the development and stabilization done
                by the Yocto Project kernel development team.
            </para>
            <para>
                Because commits only change at significant release points in the product life cycle,
                developers can work on a branch created
                from the last relevant commit in the shipped Yocto Project kernel.
                As mentioned previously, the structure is transparent to the developer
                because the kernel tree is left in this state after cloning and building the kernel.
            </para>
        </section>
    </section>
</appendix>
<!--
vim: expandtab tw=80 ts=4
-->