path: root/documentation/kernel-manual/kernel-concepts.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<chapter id='kernel-concepts'>

<title>Yocto Project Kernel Concepts</title>

<section id='concepts-org'>
    <title>Introduction</title>
    <para>
        This chapter provides conceptual information about the Yocto Project kernel:
        <itemizedlist>
            <listitem><para>Kernel Goals</para></listitem>
            <listitem><para>Yocto Project Kernel Development and Maintenance Overview</para></listitem>
            <listitem><para>Kernel Architecture</para></listitem>
            <listitem><para>Kernel Tools</para></listitem>
        </itemizedlist>
    </para>
</section>

    <section id='kernel-goals'>
        <title>Kernel Goals</title>
        <para>
            The complexity of embedded kernel design has increased dramatically. 
            Whether it is managing multiple implementations of a particular feature or tuning and
            optimizing board specific features, flexibility and maintainability are key concerns. 
            The Yocto Project Linux kernel is presented with the embedded
            developer's needs in mind and has evolved to assist in these key concerns. 
            For example, prior methods such as applying hundreds of patches to an extracted
            tarball have been replaced with proven techniques that allow easy inspection,
            bisection and analysis of changes. 
            Application of these techniques also creates a platform for performing integration and 
            collaboration with the thousands of upstream development projects.
        </para>
        <para>
            With all these considerations in mind, the Yocto Project kernel and development team
            strives to attain these goals:
        <itemizedlist>
            <listitem><para>Allow the end user to leverage community best practices to seamlessly 
            manage the development, build and debug cycles.</para></listitem>
            <listitem><para>Create a platform for performing integration and collaboration with the 
            thousands of upstream development projects that exist.</para></listitem>
            <listitem><para>Provide mechanisms that support many different work flows, front-ends and 
            management techniques.</para></listitem>
            <listitem><para>Deliver the most up-to-date kernel possible while still ensuring that 
            the baseline kernel is the most stable official release.</para></listitem>
            <listitem><para>Include major technological features as part of Yocto Project's up-rev 
            strategy.</para></listitem>
            <listitem><para>Present a git tree, that just like the upstream kernel.org tree, has a 
            clear and continuous history.</para></listitem>
            <listitem><para>Deliver a key set of supported kernel types, where each type is tailored 
            to a specific use case (i.g. networking, consumer, devices, and so forth).</para></listitem>
            <listitem><para>Employ a git branching strategy that from a customer's point of view
            results in a linear path from the baseline kernel.org, through a select group of features and
            ends with their BSP-specific commits.</para></listitem>
        </itemizedlist>
        </para>
    </section>

    <section id='kernel-big-picture'>
        <title>Yocto Project Kernel Development and Maintenance Overview</title>
        <para>
            Yocto Project kernel, like other kernels, is based off the Linux kernel release
            from <ulink url='http://www.kernel.org'></ulink>.  
            At the beginning of our major development cycle, we choose our Yocto Project kernel 
            based on factors like release timing, the anticipated release timing of "final" (i.e. non "rc")
            upstream kernel.org versions, and Yocto Project feature requirements.
            Typically this will be a kernel that is in the
            final stages of development by the community (i.e. still in the release
            candidate or "rc" phase) and not yet a final release. 
            But by being in the final stages of external development, we know that the 
            kernel.org final release will clearly land within the early stages of 
            the Yocto Project development window.
        </para>
        <para>
            This balance allows us to deliver the most up-to-date kernel
            as possible, while still ensuring that we have a stable official release as
            our baseline kernel version.
        </para>
        <para>
            The ultimate source for the Yocto Project kernel is a released kernel 
            from kernel.org.
            In addition to a foundational kernel from kernel.org the released 
            Yocto Project kernel contains a mix of important new mainline
            developments, non-mainline developments (when there is no alternative),
            Board Support Package (BSP) developments,
            and custom features.
            These additions result in a commercially released Yocto Project kernel that caters 
            to specific embedded designer needs for targeted hardware.
        </para>
<!--        <para>
            The following figure represents the overall place the Yocto Project kernel fills.
        </para>
        <para>
        <imagedata fileref="figures/kernel-big-picture.png" width="6in" depth="6in" align="center" scale="100" />
        </para>
        <para>
            In the figure the ultimate source for the Yocto Project kernel is a released kernel 
            from kernel.org.
            In addition to a foundational kernel from kernel.org the commercially released 
            Yocto Project kernel contains a mix of important new mainline
            developments, non-mainline developments, Board Support Package (BSP) developments,
            and custom features.
            These additions result in a commercially released Yocto Project kernel that caters 
            to specific embedded designer needs for targeted hardware. 
        </para> -->
        <para>
            Once a Yocto Project kernel is officially released the Yocto Project team goes into 
            their next development cycle, or "uprev" cycle while continuing maintenance on the 
            released kernel.
            It is important to note that the most sustainable and stable way
            to include feature development upstream is through a kernel uprev process.
            Back-porting of hundreds of individual fixes and minor features from various
            kernel versions is not sustainable and can easily compromise quality. 
            During the uprev cycle, the Yocto Project team uses an ongoing analysis of
            kernel development, BSP support, and release timing to select the best
            possible kernel.org version.
            The team continually monitors community kernel
            development to look for significant features of interest.
<!--            The illustration depicts this by showing the team looking back to kernel.org for new features, 
            BSP features, and significant bug fixes. -->
            The team does consider back-porting large features if they have a significant advantage. 
            User or community demand can also trigger a back-port or creation of new
            functionality in the Yocto Project baseline kernel during the uprev cycle. 
        </para>
        <para>
            Generally speaking, every new kernel both adds features and introduces new bugs.
            These consequences are the basic properties of upstream kernel development and are
            managed by the Yocto Project team's kernel strategy. 
            It is the Yocto Project team's policy to not back-port minor features to the released kernel. 
            They only consider back-porting significant technological jumps - and, that is done 
            after a complete gap analysis. 
            The reason for this policy is that simply back-porting any small to medium sized change 
            from an evolving kernel can easily create mismatches, incompatibilities and very 
            subtle errors.
        </para>
        <para>
            These policies result in both a stable and a cutting
            edge kernel that mixes forward ports of existing features and significant and critical 
            new functionality. 
            Forward porting functionality in the Yocto Project kernel can be thought of as a
            "micro uprev."
            The many “micro uprevs” produce a kernel version with a mix of 
            important new mainline, non-mainline, BSP developments and feature integrations. 
            This kernel gives insight into new features and allows focused
            amounts of testing to be done on the kernel, which prevents
            surprises when selecting the next major uprev. 
            The quality of these cutting edge kernels is evolving and the kernels are used in leading edge 
            feature and BSP development.
        </para>
    </section>

    <section id='kernel-architecture'>
        <title>Kernel Architecture</title>
        <para>
            This section describes the architecture of the Yocto Project kernel and provides information
            on the mechanisms used to achieve that architecture.
        </para>
        
        <section id='architecture-overview'>
            <title>Overview</title>
            <para>
                As mentioned earlier, a key goal of Yocto Project is to present the developer with 
                a kernel that has a clear and continuous history that is visible to the user. 
                The architecture and mechanisms used achieve that goal in a manner similar to the 
                upstream kernel.org.
                
            </para>
            <para>
                You can think of the Yocto Project kernel as consisting of a baseline kernel with
                added features logically structured on top of the baseline.
                The features are tagged and organized by way of a branching strategy implemented by the 
                source code manager (SCM) git. 
                The result is that the user has the ability to see the added features and 
                the commits that make up those features.
                In addition to being able to see added features, the user can also view the history of what 
                made up the baseline kernel as well.
            </para>
            <para>
                The following illustration shows the conceptual Yocto Project kernel.
            </para>
            <para>
                <imagedata fileref="figures/kernel-architecture-overview.png" width="6in" depth="7in" align="center" scale="100" />
            </para>
            <para>
                In the illustration, the "kernel.org Branch Point" marks the specific spot (or release) from 
                which the Yocto Project kernel is created.  From this point "up" in the tree features and 
                differences are organized and tagged.
            </para>
            <para>
                The "Yocto Project Baseline Kernel" contains functionality that is common to every kernel
                type and BSP that is organized further up the tree.  Placing these common features in the 
                tree this way means features don't have to be duplicated along individual branches of the 
                structure.
            </para>
            <para>
                From the Yocto Project Baseline Kernel branch points represent specific functionality
                for individual BSPs as well as real-time kernels.
                The illustration represents this through three BSP-specific branches and a real-time 
                kernel branch.  
                Each branch represents some unique functionality for the BSP or a real-time kernel.
            </para>
            <para>
                In this example structure, the real-time kernel branch has common features for all 
                real-time kernels and contains
                more branches for individual BSP-specific real-time kernels.  
                The illustration shows three branches as an example. 
                Each branch points the way to specific, unique features for a respective real-time
                kernel as they apply to a given BSP.
            </para>
            <para>
                The resulting tree structure presents a clear path of markers (or branches) to the user
                that for all practical purposes is the kernel needed for any given set of requirements.
            </para>
        </section>
 
        <section id='branching-and-workflow'>
            <title>Branching Strategy and Workflow</title>
            <para>
                The Yocto Project team creates kernel branches at points where functionality is 
                no longer shared and thus, needs to be isolated.
                For example, board-specific incompatibilities would require different functionality
                and would require a branch to separate the features. 
                Likewise, for specific kernel features the same branching strategy is used.
                This branching strategy results in a tree that has features organized to be specific 
                for particular functionality, single kernel types, or a subset of kernel types.  
                This strategy results in not having to store the same feature twice internally in the 
                tree.
                Rather we store the unique differences required to apply the feature onto the kernel type 
                in question.
            </para>
            <note><para>
                The Yocto Project team strives to place features in the tree such that they can be 
                shared by all boards and kernel types where possible.
                However, during development cycles or when large features are merged this practice 
                cannot always be followed. 
                In those cases isolated branches are used for feature merging.
            </para></note>
            <para>
                BSP-specific code additions are handled in a similar manner to kernel-specific additions. 
                Some BSPs only make sense given certain kernel types.
                So, for these types, we create branches off the end of that kernel type for all 
                of the BSPs that are supported on that kernel type.  
                From the perspective of the tools that create the BSP branch, the BSP is really no 
                different than a feature.
                Consequently, the same branching strategy applies to BSPs as it does to features.
                So again, rather than store the BSP twice, only the unique differences for the BSP across
                the supported multiple kernels are uniquely stored.
            </para>
            <para>
                While this strategy can result in a tree with a significant number of branches, it is
                important to realize that from the user's point of view, there is a linear
                path that travels from the baseline kernel.org, through a select group of features and
                ends with their BSP-specific commits.
                In other words, the divisions of the kernel are transparent and are not relevant 
                to the developer on a day-to-day basis.  
                From the user's perspective, this is the "master" branch.
                They do not need not be aware of the existence of any other branches at all.  
                Of course there is value in the existence of these branches
                in the tree, should a person decide to explore them. 
                For example, a comparison between two BSPs at either the commit level or at the line-by-line 
                code diff level is now a trivial operation.
            </para>
            <para>
                Working with the kernel as a structured tree follows recognized community best practices. 
                In particular, the kernel as shipped with the product should be
                considered an 'upstream source' and viewed as a series of
                historical and documented modifications (commits). 
                These modifications represent the development and stabilization done
                by the Yocto Project kernel development team.
            </para>
            <para>
                Because commits only change at significant release points in the product life cycle,
                developers can work on a branch created
                from the last relevant commit in the shipped Yocto Project kernel. 
                As mentioned previously, the structure is transparent to the user
                because the kernel tree is left in this state after cloning and building the kernel.
            </para>
        </section>
     
        <section id='source-code-manager-git'>
            <title>Source Code Manager - git</title>
            <para>
                The Source Code Manager (SCM) is git and it is the obvious mechanism for meeting the 
                previously mentioned goals.  
                Not only is it the SCM for kernel.org but git continues to grow in popularity and
                supports many different work flows, front-ends and management techniques.
            </para>
            <note><para> 
                It should be noted that you can use as much, or as little, of what git has to offer 
                as is appropriate to your project.
            </para></note>
        </section>
    </section>

    <section id='kernel-tools'>
        <title>Kernel Tools</title>
        <para>
Since most standard workflows involve moving forward with an existing tree by
continuing to add and alter the underlying baseline, the tools that manage
Yocto Project's kernel construction are largely hidden from the developer to
present a simplified view of the kernel for ease of use.
</para>
<para>
The fundamental properties of the tools that manage and construct the
kernel are:
<itemizedlist>
    <listitem><para>the ability to group patches into named, reusable features</para></listitem>
    <listitem><para>to allow top down control of included features</para></listitem>
    <listitem><para>the binding of kernel configuration to kernel patches/features</para></listitem>
    <listitem><para>the presentation of a seamless git repository that blends Yocto Project value with the kernel.org history and development</para></listitem>
</itemizedlist>
</para>
<!--<para>
The tools that construct a kernel tree will be discussed later in this 
document. The following tools form the foundation of the Yocto Project 
kernel toolkit:
<itemizedlist>
    <listitem><para>git  : distributed revision control system created by Linus Torvalds</para></listitem>
    <listitem><para>guilt: quilt on top of git</para></listitem>
    <listitem><para>*cfg : kernel configuration management and classification</para></listitem>
    <listitem><para>kgit*: Yocto Project kernel tree creation and management tools</para></listitem>
    <listitem><para>scc  : series &amp; configuration compiler</para></listitem>
</itemizedlist>
</para> -->
    </section> 





</chapter>
<!--
vim: expandtab tw=80 ts=4
-->