summaryrefslogtreecommitdiffstats
path: root/documentation/kernel-dev/kernel-dev-concepts-appx.xml
blob: 60d67d64fe8ea191e1f548c6cbd2fec7c9dcca36 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >

<appendix id='kernel-dev-concepts-appx'>
<title>Advanced Kernel Concepts</title>

    <section id='kernel-big-picture'>
        <title>Yocto Project Kernel Development and Maintenance</title>

        <para>
            Kernels available through the Yocto Project (Linux Yocto kernels),
            like other kernels, are based off the Linux kernel releases from
            <ulink url='http://www.kernel.org'></ulink>.
            At the beginning of a major Linux kernel development cycle, the
            Yocto Project team chooses a Linux kernel based on factors such as
            release timing, the anticipated release timing of final upstream
            <filename>kernel.org</filename> versions, and Yocto Project
            feature requirements.
            Typically, the Linux kernel chosen is in the final stages of
            development by the Linux community.
            In other words, the Linux kernel is in the release candidate
            or "rc" phase and has yet to reach final release.
            But, by being in the final stages of external development, the
            team knows that the <filename>kernel.org</filename> final release
            will clearly be within the early stages of the Yocto Project
            development window.
        </para>

        <para>
            This balance allows the Yocto Project team to deliver the most
            up-to-date Linux Yocto kernel possible, while still ensuring that
            the team has a stable official release for the baseline Linux
            kernel version.
        </para>

        <para>
            As implied earlier, the ultimate source for Linux Yocto kernels
            are released kernels from <filename>kernel.org</filename>.
            In addition to a foundational kernel from
            <filename>kernel.org</filename>, the available Yocto Linux kernels
            contain a mix of important new mainline developments, non-mainline
            developments (when there is no alternative), Board Support Package
            (BSP) developments, and custom features.
            These additions result in a commercially released Yocto
            Project Linux kernel that caters to specific embedded designer
            needs for targeted hardware.
        </para>

        <para>
            You can find a web interface to the Yocto Linux kernels in the
            <ulink url='&YOCTO_DOCS_REF_URL;#source-repositories'>Source Repositories</ulink>
            at
            <ulink url='&YOCTO_GIT_URL;'></ulink>.
            If you look at the interface, you will see to the left a
            grouping of Git repositories titled "Yocto Linux Kernel".
            Within this group, you will find several Linux Yocto kernels
            developed and included with Yocto Project releases:
            <itemizedlist>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.1</filename>:</emphasis>
                    The stable Yocto Project kernel to use with the Yocto
                    Project Release 2.0.
                    This kernel is based on the Linux 4.1 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.4</filename>:</emphasis>
                    The stable Yocto Project kernel to use with the Yocto
                    Project Release 2.1.
                    This kernel is based on the Linux 4.4 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.9</filename>:</emphasis>
                    The stable Yocto Project kernel to use with the Yocto
                    Project Release 2.3.
                    This kernel is based on the Linux 4.9 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-4.10</filename>:</emphasis>
                    The default stable Yocto Project kernel to use with the
                    Yocto Project Release 2.3.
                    This kernel is based on the Linux 4.10 released kernel.
                    </para></listitem>
                <listitem><para>
                    <emphasis><filename>linux-yocto-dev</filename>:</emphasis>
                    A development kernel based on the latest upstream release
                    candidate available.
                    </para></listitem>
            </itemizedlist>
            <note><title>Notes</title>
                Long Term Support Initiative (LTSI) for Yocto Linux
                kernels is as follows:
                <itemizedlist>
                    <listitem><para>
                        For Yocto Project releases 1.7, 1.8, and 2.0,
                        the LTSI kernel is
                        <filename>linux-yocto-3.14</filename>.
                        </para></listitem>
                    <listitem><para>
                        For Yocto Project releases 2.1, 2.2, and 2.3,
                        the LTSI kernel is <filename>linux-yocto-4.1</filename>.
                        </para></listitem>
                    <listitem><para>
                        <filename>linux-yocto-4.1</filename>,
                        <filename>linux-yocto-4.4</filename>, and
                        <filename>linux-yocto-4.9</filename> are all LTS
                        kernels.
                        </para></listitem>
                </itemizedlist>
            </note>
        </para>

        <para>
            Once a Yocto Linux kernel is officially released, the Yocto
            Project team goes into their next development cycle, or upward
            revision (uprev) cycle, while still continuing maintenance on the
            released kernel.
            It is important to note that the most sustainable and stable way
            to include feature development upstream is through a kernel uprev
            process.
            Back-porting hundreds of individual fixes and minor features from
            various kernel versions is not sustainable and can easily
            compromise quality.
        </para>

        <para>
            During the uprev cycle, the Yocto Project team uses an ongoing
            analysis of Linux kernel development, BSP support, and release
            timing to select the best possible <filename>kernel.org</filename>
            Linux kernel version on which to base subsequent Yocto Linux
            kernel development.
            The team continually monitors Linux community kernel development
            to look for significant features of interest.
            The team does consider back-porting large features if they have a
            significant advantage.
            User or community demand can also trigger a back-port or creation
            of new functionality in the Yocto Project baseline kernel during
            the uprev cycle.
        </para>

        <para>
            Generally speaking, every new Linux kernel both adds features and
            introduces new bugs.
            These consequences are the basic properties of upstream
            Linux kernel development and are managed by the Yocto Project
            team's Linux Yocto kernel development strategy.
            It is the Yocto Project team's policy to not back-port minor
            features to the released Linux Yocto kernel.
            They only consider back-porting significant technological
            jumps &dash; and, that is done after a complete gap analysis.
            The reason for this policy is that back-porting any small to
            medium sized change from an evolving Linux kernel can easily
            create mismatches, incompatibilities and very subtle errors.
        </para>

        <para>
            The policies described in this section result in both a stable
            and a cutting edge Linux Yocto kernel that mixes forward ports of
            existing Linux kernel features and significant and critical new
            functionality.
            Forward porting Linux kernel functionality into the Linux Yocto
            kernels available through the Yocto Project can be thought of as
            a "micro uprev."
            The many “micro uprevs” produce a Linux Yocto kernel version with
            a mix of important new mainline, non-mainline, BSP developments
            and feature integrations.
            This Yocto Linux kernel gives insight into new features and
            allows focused amounts of testing to be done on the kernel,
            which prevents surprises when selecting the next major uprev.
            The quality of these cutting edge Linux Yocto kernels is evolving
            and the kernels are used in leading edge feature and BSP
            development.
        </para>
    </section>

    <section id='yocto-linux-kernel-architecture-and-branching-strategies'>
        <title>Yocto Linux Kernel Architecture and Branching Strategies</title>

        <para>
            As mentioned earlier, a key goal of the Yocto Project is
            to present the developer with a kernel that has a clear and
            continuous history that is visible to the user.
            The architecture and mechanisms, in particular the branching
            strategies, used achieve that goal in a manner similar to
            upstream Linux kernel development in
            <filename>kernel.org</filename>.
        </para>

        <para>
            You can think of a Yocto Linux kernel as consisting of a
            baseline Linux kernel with added features logically structured
            on top of the baseline.
            The features are tagged and organized by way of a branching
            strategy implemented by the Yocto Project team using the
            Source Code Manager (SCM) Git.
            <note><title>Notes</title>
                <itemizedlist>
                    <listitem><para>
                        Git is the obvious SCM for meeting the Yocto Linux
                        kernel organizational and structural goals
                        described in this section.
                        Not only is Git the SCM for Linux kernel
                        development in <filename>kernel.org</filename> but,
                        Git continues to grow in popularity and supports
                        many different work flows, front-ends and
                        management techniques.
                        </para></listitem>
                    <listitem><para>
                        You can find documentation on Git at
                        <ulink url='http://git-scm.com/documentation'></ulink>.
                        You can also get an introduction to Git as it
                        applies to the Yocto Project in the
                        "<ulink url='&YOCTO_DOCS_REF_URL;#git'>Git</ulink>"
                        section in the Yocto Project Reference Manual.
                        The latter reference provides an overview of
                        Git and presents a minimal set of Git commands
                        that allows you to be functional using Git.
                        You can use as much, or as little, of what Git
                        has to offer to accomplish what you need for your
                        project.
                        You do not have to be a "Git Expert" in order to
                        use it with the Yocto Project.
                        </para></listitem>
                </itemizedlist>
            </note>
        </para>

        <para>
            Using Git's tagging and branching features, the Yocto Project
            team creates kernel branches at points where functionality is
            no longer shared and thus, needs to be isolated.
            For example, board-specific incompatibilities would require
            different functionality and would require a branch to
            separate the features.
            Likewise, for specific kernel features, the same branching
            strategy is used.
        </para>

        <para>
            This "tree-like" architecture results in a tree that has
            features organized to be specific for particular
            functionality, single kernel types, or a subset of kernel
            types.
            Thus, the user has the ability to see the added
            features and the commits that make up those features.
            In addition to being able to see added features, the user
            can also view the history of what made up the baseline
            Linux kernel.
        </para>

        <para>
            Another consequence of this strategy results in not having to
            store the same feature twice internally in the tree.
            Rather, the kernel team stores the unique differences required
            to apply the feature onto the kernel type in question.
            <note>
                The Yocto Project team strives to place features in the tree
                such that features can be shared by all boards and kernel
                types where possible.
                However, during development cycles or when large features
                are merged, the team cannot always follow this practice.
                In those cases, the team uses isolated branches to merge
                features.
            </note>
        </para>

        <para>
            BSP-specific code additions are handled in a similar manner to
            kernel-specific additions.
            Some BSPs only make sense given certain kernel types.
            So, for these types, the team creates branches off the end
            of that kernel type for all of the BSPs that are supported on
            that kernel type.
            From the perspective of the tools that create the BSP branch,
            the BSP is really no different than a feature.
            Consequently, the same branching strategy applies to BSPs as
            it does to kernel features.
            So again, rather than store the BSP twice, the team only
            stores the unique differences for the BSP across the supported
            multiple kernels.
        </para>

        <para>
            While this strategy can result in a tree with a significant number
            of branches, it is important to realize that from the developer's
            point of view, there is a linear path that travels from the
            baseline <filename>kernel.org</filename>, through a select
            group of features and ends with their BSP-specific commits.
            In other words, the divisions of the kernel are transparent and
            are not relevant to the developer on a day-to-day basis.
            From the developer's perspective, this path is the "master" branch
            in Git terms.
            The developer does not need to be aware of the existence of any
            other branches at all.
            Of course, value exists in the having these branches in the tree,
            should a person decide to explore them.
            For example, a comparison between two BSPs at either the commit
            level or at the line-by-line code <filename>diff</filename> level
            is now a trivial operation.
        </para>

        <para>
            The following illustration shows the conceptual Yocto
            Linux kernel.
            <imagedata fileref="figures/kernel-architecture-overview.png" width="6in" depth="7in" align="center" scale="100" />
        </para>

        <para>
            In the illustration, the "Kernel.org Branch Point" marks the
            specific spot (or Linux kernel release) from which the
            Yocto Linux kernel is created.
            From this point forward in the tree, features and differences
            are organized and tagged.
        </para>

        <para>
            The "Yocto Project Baseline Kernel" contains functionality that
            is common to every kernel type and BSP that is organized
            further along in the tree.
            Placing these common features in the tree this way means
            features do not have to be duplicated along individual
            branches of the tree structure.
        </para>

        <para>
            From the "Yocto Project Baseline Kernel", branch points represent
            specific functionality for individual Board Support Packages
            (BSPs) as well as real-time kernels.
            The illustration represents this through three BSP-specific
            branches and a real-time kernel branch.
            Each branch represents some unique functionality for the BSP
            or for a real-time Yocto Linux kernel.
        </para>

        <para>
            In this example structure, the "Real-time (rt) Kernel" branch has
            common features for all real-time Yocto Linux kernels and
            contains more branches for individual BSP-specific real-time
            kernels.
            The illustration shows three branches as an example.
            Each branch points the way to specific, unique features for a
            respective real-time kernel as they apply to a given BSP.
        </para>

        <para>
            The resulting tree structure presents a clear path of markers
            (or branches) to the developer that, for all practical
            purposes, is the Yocto Linux kernel needed for any given set of
            requirements.
            <note>
                Keep in mind the figure does not take into account all the
                supported Yocto Linux kernels, but rather shows a single
                generic kernel just for conceptual purposes.
                Also keep in mind that this structure represents the Yocto
                Project
                <ulink url='&YOCTO_DOCS_REF_URL;#source-repositories'>Source Repositories</ulink>
                that are either pulled from during the build or established
                on the host development system prior to the build by either
                cloning a particular kernel's Git repository or by
                downloading and unpacking a tarball.
            </note>
        </para>

        <para>
            Working with the kernel as a structured tree follows recognized
            community best practices.
            In particular, the kernel as shipped with the product, should be
            considered an "upstream source" and viewed as a series of
            historical and documented modifications (commits).
            These modifications represent the development and stabilization
            done by the Yocto Project kernel development team.
        </para>

        <para>
            Because commits only change at significant release points in the
            product life cycle, developers can work on a branch created
            from the last relevant commit in the shipped Yocto Project kernel.
            As mentioned previously, the structure is transparent to the
            developer because the kernel tree is left in this state after
            cloning and building the kernel.
        </para>
    </section>

    <section id='kernel-build-file-hierarchy'>
        <title>Kernel Build File Hierarchy</title>

        <para>
            Upstream storage of all the available kernel source code is
            one thing, while representing and using the code on your host
            development system is another.
            Conceptually, you can think of the kernel source repositories
            as all the source files necessary for all the supported
            Yocto Linux kernels.
            As a developer, you are just interested in the source files
            for the kernel on which you are working.
            And, furthermore, you need them available on your host system.
        </para>

        <para>
            Kernel source code is available on your host system several
            different ways:
            <itemizedlist>
                <listitem><para>
                    <emphasis>Files Accessed While using <filename>devtool</filename>:</emphasis>
                    <filename>devtool</filename>, which is available with the
                    Yocto Project, is the preferred method by which to
                    modify the kernel.
                    See the
                    "<link linkend='kernel-modification-workflow'>Kernel Modification Workflow</link>"
                    section.
                    </para></listitem>
                <listitem><para>
                    <emphasis>Cloned Repository:</emphasis>
                    If you are working in the kernel all the time, you probably
                    would want to set up your own local Git repository of the
                    Yocto Linux kernel tree.
                    For information on how to clone a Yocto Linux kernel
                    Git repository, see the
                    "<ulink url='&YOCTO_DOCS_DEV_URL;#local-kernel-files'>Setting Up to Work On a Kernel</ulink>"
                    section in the Yocto Project Development Manual.
                    </para></listitem>
                <listitem><para>
                    <emphasis>Temporary Source Files from a Build:</emphasis>
                    If you just need to make some patches to the kernel using
                    a traditional BitBake workflow (i.e. not using the
                    <filename>devtool</filename>), you can access temporary
                    kernel source files that were extracted and used during
                    a kernel build.
                    </para></listitem>
            </itemizedlist>
        </para>

        <para>
            The temporary kernel source files resulting from a build using
            BitBake have a particular hierarchy.
            When you build the kernel on your development system, all files
            needed for the build are taken from the source repositories
            pointed to by the
            <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
            variable and gathered in a temporary work area where they are
            subsequently used to create the unique kernel.
            Thus, in a sense, the process constructs a local source tree
            specific to your kernel from which to generate the new kernel
            image.
        </para>

        <para>
            The following figure shows the temporary file structure
            created on your host system when you build the kernel using
            Bitbake.
            This
            <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
            contains all the source files used during the build.
            <imagedata fileref="figures/kernel-overview-2-generic.png"
                width="6in" depth="5in" align="center" scale="100" />
        </para>

        <para>
            Again, for additional information on the Yocto Project kernel's
            architecture and its branching strategy, see the
            "<link linkend='yocto-linux-kernel-architecture-and-branching-strategies'>Yocto Linux Kernel Architecture and Branching Strategies</link>"
            section.
            You can also reference the
            "<link linkend='using-devtool-to-patch-the-kernel'>Using <filename>devtool</filename> to Patch the Kernel</link>"
            section for a detailed example that modifies the kernel.
        </para>
    </section>
</appendix>
<!--
vim: expandtab tw=80 ts=4
-->