diff options
Diffstat (limited to 'documentation/kernel-dev/kernel-dev-concepts-appx.rst')
-rw-r--r-- | documentation/kernel-dev/kernel-dev-concepts-appx.rst | 426 |
1 files changed, 426 insertions, 0 deletions
diff --git a/documentation/kernel-dev/kernel-dev-concepts-appx.rst b/documentation/kernel-dev/kernel-dev-concepts-appx.rst new file mode 100644 index 0000000000..04cb1172b2 --- /dev/null +++ b/documentation/kernel-dev/kernel-dev-concepts-appx.rst | |||
@@ -0,0 +1,426 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | ************************ | ||
4 | Advanced Kernel Concepts | ||
5 | ************************ | ||
6 | |||
7 | .. _kernel-big-picture: | ||
8 | |||
9 | Yocto Project Kernel Development and Maintenance | ||
10 | ================================================ | ||
11 | |||
12 | Kernels available through the Yocto Project (Yocto Linux kernels), like | ||
13 | other kernels, are based off the Linux kernel releases from | ||
14 | http://www.kernel.org. At the beginning of a major Linux kernel | ||
15 | development cycle, the Yocto Project team chooses a Linux kernel based | ||
16 | on factors such as release timing, the anticipated release timing of | ||
17 | final upstream ``kernel.org`` versions, and Yocto Project feature | ||
18 | requirements. Typically, the Linux kernel chosen is in the final stages | ||
19 | of development by the Linux community. In other words, the Linux kernel | ||
20 | is in the release candidate or "rc" phase and has yet to reach final | ||
21 | release. But, by being in the final stages of external development, the | ||
22 | team knows that the ``kernel.org`` final release will clearly be within | ||
23 | the early stages of the Yocto Project development window. | ||
24 | |||
25 | This balance allows the Yocto Project team to deliver the most | ||
26 | up-to-date Yocto Linux kernel possible, while still ensuring that the | ||
27 | team has a stable official release for the baseline Linux kernel | ||
28 | version. | ||
29 | |||
30 | As implied earlier, the ultimate source for Yocto Linux kernels are | ||
31 | released kernels from ``kernel.org``. In addition to a foundational | ||
32 | kernel from ``kernel.org``, the available Yocto Linux kernels contain a | ||
33 | mix of important new mainline developments, non-mainline developments | ||
34 | (when no alternative exists), Board Support Package (BSP) developments, | ||
35 | and custom features. These additions result in a commercially released | ||
36 | Yocto Project Linux kernel that caters to specific embedded designer | ||
37 | needs for targeted hardware. | ||
38 | |||
39 | You can find a web interface to the Yocto Linux kernels in the | ||
40 | :ref:`overview-manual/overview-manual-development-environment:yocto project source repositories` | ||
41 | at :yocto_git:`/`. If you look at the interface, you will see to | ||
42 | the left a grouping of Git repositories titled "Yocto Linux Kernel". | ||
43 | Within this group, you will find several Linux Yocto kernels developed | ||
44 | and included with Yocto Project releases: | ||
45 | |||
46 | - *linux-yocto-4.1:* The stable Yocto Project kernel to use with | ||
47 | the Yocto Project Release 2.0. This kernel is based on the Linux 4.1 | ||
48 | released kernel. | ||
49 | |||
50 | - *linux-yocto-4.4:* The stable Yocto Project kernel to use with | ||
51 | the Yocto Project Release 2.1. This kernel is based on the Linux 4.4 | ||
52 | released kernel. | ||
53 | |||
54 | - *linux-yocto-4.6:* A temporary kernel that is not tied to any | ||
55 | Yocto Project release. | ||
56 | |||
57 | - *linux-yocto-4.8:* The stable yocto Project kernel to use with | ||
58 | the Yocto Project Release 2.2. | ||
59 | |||
60 | - *linux-yocto-4.9:* The stable Yocto Project kernel to use with | ||
61 | the Yocto Project Release 2.3. This kernel is based on the Linux 4.9 | ||
62 | released kernel. | ||
63 | |||
64 | - *linux-yocto-4.10:* The default stable Yocto Project kernel to | ||
65 | use with the Yocto Project Release 2.3. This kernel is based on the | ||
66 | Linux 4.10 released kernel. | ||
67 | |||
68 | - *linux-yocto-4.12:* The default stable Yocto Project kernel to | ||
69 | use with the Yocto Project Release 2.4. This kernel is based on the | ||
70 | Linux 4.12 released kernel. | ||
71 | |||
72 | - *yocto-kernel-cache:* The ``linux-yocto-cache`` contains patches | ||
73 | and configurations for the linux-yocto kernel tree. This repository | ||
74 | is useful when working on the linux-yocto kernel. For more | ||
75 | information on this "Advanced Kernel Metadata", see the | ||
76 | ":doc:`kernel-dev-advanced`" Chapter. | ||
77 | |||
78 | - *linux-yocto-dev:* A development kernel based on the latest | ||
79 | upstream release candidate available. | ||
80 | |||
81 | .. note:: | ||
82 | |||
83 | Long Term Support Initiative (LTSI) for Yocto Linux kernels is as | ||
84 | follows: | ||
85 | |||
86 | - For Yocto Project releases 1.7, 1.8, and 2.0, the LTSI kernel is | ||
87 | ``linux-yocto-3.14``. | ||
88 | |||
89 | - For Yocto Project releases 2.1, 2.2, and 2.3, the LTSI kernel is | ||
90 | ``linux-yocto-4.1``. | ||
91 | |||
92 | - For Yocto Project release 2.4, the LTSI kernel is | ||
93 | ``linux-yocto-4.9`` | ||
94 | |||
95 | - ``linux-yocto-4.4`` is an LTS kernel. | ||
96 | |||
97 | Once a Yocto Linux kernel is officially released, the Yocto Project team | ||
98 | goes into their next development cycle, or upward revision (uprev) | ||
99 | cycle, while still continuing maintenance on the released kernel. It is | ||
100 | important to note that the most sustainable and stable way to include | ||
101 | feature development upstream is through a kernel uprev process. | ||
102 | Back-porting hundreds of individual fixes and minor features from | ||
103 | various kernel versions is not sustainable and can easily compromise | ||
104 | quality. | ||
105 | |||
106 | During the uprev cycle, the Yocto Project team uses an ongoing analysis | ||
107 | of Linux kernel development, BSP support, and release timing to select | ||
108 | the best possible ``kernel.org`` Linux kernel version on which to base | ||
109 | subsequent Yocto Linux kernel development. The team continually monitors | ||
110 | Linux community kernel development to look for significant features of | ||
111 | interest. The team does consider back-porting large features if they | ||
112 | have a significant advantage. User or community demand can also trigger | ||
113 | a back-port or creation of new functionality in the Yocto Project | ||
114 | baseline kernel during the uprev cycle. | ||
115 | |||
116 | Generally speaking, every new Linux kernel both adds features and | ||
117 | introduces new bugs. These consequences are the basic properties of | ||
118 | upstream Linux kernel development and are managed by the Yocto Project | ||
119 | team's Yocto Linux kernel development strategy. It is the Yocto Project | ||
120 | team's policy to not back-port minor features to the released Yocto | ||
121 | Linux kernel. They only consider back-porting significant technological | ||
122 | jumps DASH and, that is done after a complete gap analysis. The reason | ||
123 | for this policy is that back-porting any small to medium sized change | ||
124 | from an evolving Linux kernel can easily create mismatches, | ||
125 | incompatibilities and very subtle errors. | ||
126 | |||
127 | The policies described in this section result in both a stable and a | ||
128 | cutting edge Yocto Linux kernel that mixes forward ports of existing | ||
129 | Linux kernel features and significant and critical new functionality. | ||
130 | Forward porting Linux kernel functionality into the Yocto Linux kernels | ||
131 | available through the Yocto Project can be thought of as a "micro | ||
132 | uprev." The many "micro uprevs" produce a Yocto Linux kernel version | ||
133 | with a mix of important new mainline, non-mainline, BSP developments and | ||
134 | feature integrations. This Yocto Linux kernel gives insight into new | ||
135 | features and allows focused amounts of testing to be done on the kernel, | ||
136 | which prevents surprises when selecting the next major uprev. The | ||
137 | quality of these cutting edge Yocto Linux kernels is evolving and the | ||
138 | kernels are used in leading edge feature and BSP development. | ||
139 | |||
140 | Yocto Linux Kernel Architecture and Branching Strategies | ||
141 | ======================================================== | ||
142 | |||
143 | As mentioned earlier, a key goal of the Yocto Project is to present the | ||
144 | developer with a kernel that has a clear and continuous history that is | ||
145 | visible to the user. The architecture and mechanisms, in particular the | ||
146 | branching strategies, used achieve that goal in a manner similar to | ||
147 | upstream Linux kernel development in ``kernel.org``. | ||
148 | |||
149 | You can think of a Yocto Linux kernel as consisting of a baseline Linux | ||
150 | kernel with added features logically structured on top of the baseline. | ||
151 | The features are tagged and organized by way of a branching strategy | ||
152 | implemented by the Yocto Project team using the Source Code Manager | ||
153 | (SCM) Git. | ||
154 | |||
155 | .. note:: | ||
156 | |||
157 | - Git is the obvious SCM for meeting the Yocto Linux kernel | ||
158 | organizational and structural goals described in this section. Not | ||
159 | only is Git the SCM for Linux kernel development in ``kernel.org`` | ||
160 | but, Git continues to grow in popularity and supports many | ||
161 | different work flows, front-ends and management techniques. | ||
162 | |||
163 | - You can find documentation on Git at | ||
164 | http://git-scm.com/documentation. You can also get an | ||
165 | introduction to Git as it applies to the Yocto Project in the | ||
166 | ":ref:`overview-manual/overview-manual-development-environment:git`" section in the Yocto Project | ||
167 | Overview and Concepts Manual. The latter reference provides an | ||
168 | overview of Git and presents a minimal set of Git commands that | ||
169 | allows you to be functional using Git. You can use as much, or as | ||
170 | little, of what Git has to offer to accomplish what you need for | ||
171 | your project. You do not have to be a "Git Expert" in order to use | ||
172 | it with the Yocto Project. | ||
173 | |||
174 | Using Git's tagging and branching features, the Yocto Project team | ||
175 | creates kernel branches at points where functionality is no longer | ||
176 | shared and thus, needs to be isolated. For example, board-specific | ||
177 | incompatibilities would require different functionality and would | ||
178 | require a branch to separate the features. Likewise, for specific kernel | ||
179 | features, the same branching strategy is used. | ||
180 | |||
181 | This "tree-like" architecture results in a structure that has features | ||
182 | organized to be specific for particular functionality, single kernel | ||
183 | types, or a subset of kernel types. Thus, the user has the ability to | ||
184 | see the added features and the commits that make up those features. In | ||
185 | addition to being able to see added features, the user can also view the | ||
186 | history of what made up the baseline Linux kernel. | ||
187 | |||
188 | Another consequence of this strategy results in not having to store the | ||
189 | same feature twice internally in the tree. Rather, the kernel team | ||
190 | stores the unique differences required to apply the feature onto the | ||
191 | kernel type in question. | ||
192 | |||
193 | .. note:: | ||
194 | |||
195 | The Yocto Project team strives to place features in the tree such | ||
196 | that features can be shared by all boards and kernel types where | ||
197 | possible. However, during development cycles or when large features | ||
198 | are merged, the team cannot always follow this practice. In those | ||
199 | cases, the team uses isolated branches to merge features. | ||
200 | |||
201 | BSP-specific code additions are handled in a similar manner to | ||
202 | kernel-specific additions. Some BSPs only make sense given certain | ||
203 | kernel types. So, for these types, the team creates branches off the end | ||
204 | of that kernel type for all of the BSPs that are supported on that | ||
205 | kernel type. From the perspective of the tools that create the BSP | ||
206 | branch, the BSP is really no different than a feature. Consequently, the | ||
207 | same branching strategy applies to BSPs as it does to kernel features. | ||
208 | So again, rather than store the BSP twice, the team only stores the | ||
209 | unique differences for the BSP across the supported multiple kernels. | ||
210 | |||
211 | While this strategy can result in a tree with a significant number of | ||
212 | branches, it is important to realize that from the developer's point of | ||
213 | view, there is a linear path that travels from the baseline | ||
214 | ``kernel.org``, through a select group of features and ends with their | ||
215 | BSP-specific commits. In other words, the divisions of the kernel are | ||
216 | transparent and are not relevant to the developer on a day-to-day basis. | ||
217 | From the developer's perspective, this path is the "master" branch in | ||
218 | Git terms. The developer does not need to be aware of the existence of | ||
219 | any other branches at all. Of course, value exists in the having these | ||
220 | branches in the tree, should a person decide to explore them. For | ||
221 | example, a comparison between two BSPs at either the commit level or at | ||
222 | the line-by-line code ``diff`` level is now a trivial operation. | ||
223 | |||
224 | The following illustration shows the conceptual Yocto Linux kernel. | ||
225 | |||
226 | .. image:: figures/kernel-architecture-overview.png | ||
227 | :align: center | ||
228 | |||
229 | In the illustration, the "Kernel.org Branch Point" marks the specific | ||
230 | spot (or Linux kernel release) from which the Yocto Linux kernel is | ||
231 | created. From this point forward in the tree, features and differences | ||
232 | are organized and tagged. | ||
233 | |||
234 | The "Yocto Project Baseline Kernel" contains functionality that is | ||
235 | common to every kernel type and BSP that is organized further along in | ||
236 | the tree. Placing these common features in the tree this way means | ||
237 | features do not have to be duplicated along individual branches of the | ||
238 | tree structure. | ||
239 | |||
240 | From the "Yocto Project Baseline Kernel", branch points represent | ||
241 | specific functionality for individual Board Support Packages (BSPs) as | ||
242 | well as real-time kernels. The illustration represents this through | ||
243 | three BSP-specific branches and a real-time kernel branch. Each branch | ||
244 | represents some unique functionality for the BSP or for a real-time | ||
245 | Yocto Linux kernel. | ||
246 | |||
247 | In this example structure, the "Real-time (rt) Kernel" branch has common | ||
248 | features for all real-time Yocto Linux kernels and contains more | ||
249 | branches for individual BSP-specific real-time kernels. The illustration | ||
250 | shows three branches as an example. Each branch points the way to | ||
251 | specific, unique features for a respective real-time kernel as they | ||
252 | apply to a given BSP. | ||
253 | |||
254 | The resulting tree structure presents a clear path of markers (or | ||
255 | branches) to the developer that, for all practical purposes, is the | ||
256 | Yocto Linux kernel needed for any given set of requirements. | ||
257 | |||
258 | .. note:: | ||
259 | |||
260 | Keep in mind the figure does not take into account all the supported | ||
261 | Yocto Linux kernels, but rather shows a single generic kernel just | ||
262 | for conceptual purposes. Also keep in mind that this structure | ||
263 | represents the Yocto Project | ||
264 | Source Repositories | ||
265 | that are either pulled from during the build or established on the | ||
266 | host development system prior to the build by either cloning a | ||
267 | particular kernel's Git repository or by downloading and unpacking a | ||
268 | tarball. | ||
269 | |||
270 | Working with the kernel as a structured tree follows recognized | ||
271 | community best practices. In particular, the kernel as shipped with the | ||
272 | product, should be considered an "upstream source" and viewed as a | ||
273 | series of historical and documented modifications (commits). These | ||
274 | modifications represent the development and stabilization done by the | ||
275 | Yocto Project kernel development team. | ||
276 | |||
277 | Because commits only change at significant release points in the product | ||
278 | life cycle, developers can work on a branch created from the last | ||
279 | relevant commit in the shipped Yocto Project Linux kernel. As mentioned | ||
280 | previously, the structure is transparent to the developer because the | ||
281 | kernel tree is left in this state after cloning and building the kernel. | ||
282 | |||
283 | Kernel Build File Hierarchy | ||
284 | =========================== | ||
285 | |||
286 | Upstream storage of all the available kernel source code is one thing, | ||
287 | while representing and using the code on your host development system is | ||
288 | another. Conceptually, you can think of the kernel source repositories | ||
289 | as all the source files necessary for all the supported Yocto Linux | ||
290 | kernels. As a developer, you are just interested in the source files for | ||
291 | the kernel on which you are working. And, furthermore, you need them | ||
292 | available on your host system. | ||
293 | |||
294 | Kernel source code is available on your host system several different | ||
295 | ways: | ||
296 | |||
297 | - *Files Accessed While using devtool:* ``devtool``, which is | ||
298 | available with the Yocto Project, is the preferred method by which to | ||
299 | modify the kernel. See the ":ref:`kernel-dev/kernel-dev-intro:kernel modification workflow`" section. | ||
300 | |||
301 | - *Cloned Repository:* If you are working in the kernel all the time, | ||
302 | you probably would want to set up your own local Git repository of | ||
303 | the Yocto Linux kernel tree. For information on how to clone a Yocto | ||
304 | Linux kernel Git repository, see the | ||
305 | ":ref:`kernel-dev/kernel-dev-common:preparing the build host to work on the kernel`" | ||
306 | section. | ||
307 | |||
308 | - *Temporary Source Files from a Build:* If you just need to make some | ||
309 | patches to the kernel using a traditional BitBake workflow (i.e. not | ||
310 | using the ``devtool``), you can access temporary kernel source files | ||
311 | that were extracted and used during a kernel build. | ||
312 | |||
313 | The temporary kernel source files resulting from a build using BitBake | ||
314 | have a particular hierarchy. When you build the kernel on your | ||
315 | development system, all files needed for the build are taken from the | ||
316 | source repositories pointed to by the | ||
317 | :term:`SRC_URI` variable and gathered | ||
318 | in a temporary work area where they are subsequently used to create the | ||
319 | unique kernel. Thus, in a sense, the process constructs a local source | ||
320 | tree specific to your kernel from which to generate the new kernel | ||
321 | image. | ||
322 | |||
323 | The following figure shows the temporary file structure created on your | ||
324 | host system when you build the kernel using Bitbake. This | ||
325 | :term:`Build Directory` contains all the | ||
326 | source files used during the build. | ||
327 | |||
328 | .. image:: figures/kernel-overview-2-generic.png | ||
329 | :align: center | ||
330 | |||
331 | Again, for additional information on the Yocto Project kernel's | ||
332 | architecture and its branching strategy, see the | ||
333 | ":ref:`kernel-dev/kernel-dev-concepts-appx:yocto linux kernel architecture and branching strategies`" | ||
334 | section. You can also reference the | ||
335 | ":ref:`kernel-dev/kernel-dev-common:using \`\`devtool\`\` to patch the kernel`" | ||
336 | and | ||
337 | ":ref:`kernel-dev/kernel-dev-common:using traditional kernel development to patch the kernel`" | ||
338 | sections for detailed example that modifies the kernel. | ||
339 | |||
340 | Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase | ||
341 | ======================================================================================= | ||
342 | |||
343 | This section describes part of the kernel configuration audit phase that | ||
344 | most developers can ignore. For general information on kernel | ||
345 | configuration including ``menuconfig``, ``defconfig`` files, and | ||
346 | configuration fragments, see the | ||
347 | ":ref:`kernel-dev/kernel-dev-common:configuring the kernel`" section. | ||
348 | |||
349 | During this part of the audit phase, the contents of the final | ||
350 | ``.config`` file are compared against the fragments specified by the | ||
351 | system. These fragments can be system fragments, distro fragments, or | ||
352 | user-specified configuration elements. Regardless of their origin, the | ||
353 | OpenEmbedded build system warns the user if a specific option is not | ||
354 | included in the final kernel configuration. | ||
355 | |||
356 | By default, in order to not overwhelm the user with configuration | ||
357 | warnings, the system only reports missing "hardware" options as they | ||
358 | could result in a boot failure or indicate that important hardware is | ||
359 | not available. | ||
360 | |||
361 | To determine whether or not a given option is "hardware" or | ||
362 | "non-hardware", the kernel Metadata in ``yocto-kernel-cache`` contains | ||
363 | files that classify individual or groups of options as either hardware | ||
364 | or non-hardware. To better show this, consider a situation where the | ||
365 | ``yocto-kernel-cache`` contains the following files: | ||
366 | :: | ||
367 | |||
368 | yocto-kernel-cache/features/drm-psb/hardware.cfg | ||
369 | yocto-kernel-cache/features/kgdb/hardware.cfg | ||
370 | yocto-kernel-cache/ktypes/base/hardware.cfg | ||
371 | yocto-kernel-cache/bsp/mti-malta32/hardware.cfg | ||
372 | yocto-kernel-cache/bsp/qemu-ppc32/hardware.cfg | ||
373 | yocto-kernel-cache/bsp/qemuarma9/hardware.cfg | ||
374 | yocto-kernel-cache/bsp/mti-malta64/hardware.cfg | ||
375 | yocto-kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg | ||
376 | yocto-kernel-cache/bsp/common-pc/hardware.cfg | ||
377 | yocto-kernel-cache/bsp/common-pc-64/hardware.cfg | ||
378 | yocto-kernel-cache/features/rfkill/non-hardware.cfg | ||
379 | yocto-kernel-cache/ktypes/base/non-hardware.cfg | ||
380 | yocto-kernel-cache/features/aufs/non-hardware.kcf | ||
381 | yocto-kernel-cache/features/ocf/non-hardware.kcf | ||
382 | yocto-kernel-cache/ktypes/base/non-hardware.kcf | ||
383 | yocto-kernel-cache/ktypes/base/hardware.kcf | ||
384 | yocto-kernel-cache/bsp/qemu-ppc32/hardware.kcf | ||
385 | |||
386 | The following list | ||
387 | provides explanations for the various files: | ||
388 | |||
389 | - ``hardware.kcf``: Specifies a list of kernel Kconfig files that | ||
390 | contain hardware options only. | ||
391 | |||
392 | - ``non-hardware.kcf``: Specifies a list of kernel Kconfig files that | ||
393 | contain non-hardware options only. | ||
394 | |||
395 | - ``hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options that | ||
396 | are hardware, regardless of whether or not they are within a Kconfig | ||
397 | file specified by a hardware or non-hardware Kconfig file (i.e. | ||
398 | ``hardware.kcf`` or ``non-hardware.kcf``). | ||
399 | |||
400 | - ``non-hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options | ||
401 | that are not hardware, regardless of whether or not they are within a | ||
402 | Kconfig file specified by a hardware or non-hardware Kconfig file | ||
403 | (i.e. ``hardware.kcf`` or ``non-hardware.kcf``). | ||
404 | |||
405 | Here is a specific example using the | ||
406 | ``kernel-cache/bsp/mti-malta32/hardware.cfg``: | ||
407 | :: | ||
408 | |||
409 | CONFIG_SERIAL_8250 | ||
410 | CONFIG_SERIAL_8250_CONSOLE | ||
411 | CONFIG_SERIAL_8250_NR_UARTS | ||
412 | CONFIG_SERIAL_8250_PCI | ||
413 | CONFIG_SERIAL_CORE | ||
414 | CONFIG_SERIAL_CORE_CONSOLE | ||
415 | CONFIG_VGA_ARB | ||
416 | |||
417 | The kernel configuration audit automatically detects | ||
418 | these files (hence the names must be exactly the ones discussed here), | ||
419 | and uses them as inputs when generating warnings about the final | ||
420 | ``.config`` file. | ||
421 | |||
422 | A user-specified kernel Metadata repository, or recipe space feature, | ||
423 | can use these same files to classify options that are found within its | ||
424 | ``.cfg`` files as hardware or non-hardware, to prevent the OpenEmbedded | ||
425 | build system from producing an error or warning when an option is not in | ||
426 | the final ``.config`` file. | ||