From 2a993a0dac62a5b2ed36f710771fa73ad7a84094 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Mon, 29 Nov 2010 07:30:27 -0800 Subject: documentation/kernel-manual: Added these files so the book could have chapters. Signed-off-by: Scott Rifenbark --- documentation/kernel-manual/kernel-how-to.xml | 1954 +++++++++++++++++++++++++ 1 file changed, 1954 insertions(+) create mode 100644 documentation/kernel-manual/kernel-how-to.xml (limited to 'documentation/kernel-manual/kernel-how-to.xml') diff --git a/documentation/kernel-manual/kernel-how-to.xml b/documentation/kernel-manual/kernel-how-to.xml new file mode 100644 index 0000000000..7f3eac3d3b --- /dev/null +++ b/documentation/kernel-manual/kernel-how-to.xml @@ -0,0 +1,1954 @@ + + + + +Working with the Yocto Project Kernel + + +
+ Introduction + + This chapter describes how to accomplish tasks involving the kernel's tree structure. + The information covers the following: + + Tree construction + Build strategies + + Workflow examples + Source Code Manager (SCM) + + BSP creation + Patching + Updating BSP patches and configuration + + "dirty" string + + + +
+ +
+ Tree Construction + +The Yocto Project kernel repository, as shipped with the product, is created by +compiling and executing the set of feature descriptions for every BSP/feature +in the product. Those feature descriptions list all necessary patches, +configuration, branching, tagging and feature divisions found in the kernel. + + +The files used to describe all the valid features and BSPs in the Yocto Project +kernel can be found in any clone of the kernel git tree. The directory +wrs/cfg/kernel-cache/ is a snapshot of all the kernel configuration and +feature descriptions (.scc) that were used to build the kernel repository. +It should however be noted, that browsing the snapshot of feature +descriptions and patches is not an effective way to determine what is in a +particular kernel branch. Using git directly to get insight into the changes +in a branch is more efficient and a more flexible way to inspect changes to +the kernel. Examples of using git to inspect kernel commits are in the +following sections. + + +As a reminder, it is envisioned that a ground up reconstruction of the +complete kernel tree is an action only taken by Yocto Project team during an +active development cycle. When an end user creates a project, it takes +advantage of this complete tree in order to efficiently place a git tree +within their project. + + +The general flow of the project specific kernel tree construction is as follows: + + a top level kernel feature is passed to the kernel build subsystem, + normally this is a BSP for a particular kernel type. + + the file that describes the top level feature is located by searching + system directories: + + + the kernel-cache under linux/wrs/cfg/kernel-cache + + recipe SRC_URIs + + + + In a typical build a feature description of the format: + <bsp name>-<kernel type>.scc is the target of the search. + + + once located, the feature description is compiled into a simple script + of actions, or an existing equivalent script which was part of the + shipped kernel is located. + + extra features are appended to the top level feature description. Extra + features can come from the KERNEL_FEATURES variable in recipes. + + each extra feature is located, compiled and appended to the script from + step #3 + + the script is executed, and a meta-series is produced. The meta-series + is a description of all the branches, tags, patches and configuration that + need to be applied to the base git repository to completely create the + "bsp_name-kernel_type". + + the base repository is cloned, and the actions + listed in the meta-series are applied to the tree. + + the git repository is left with the desired branch checked out and any + required branching, patching and tagging has been performed. + + + + +The tree is now ready for configuration and compilation. Those two topics will +be covered below. + + +The end user generated meta-series adds to the kernel as shipped with + the Yocto Project release. Any add-ons and configuration data are applied + to the end of an existing branch. The full repository generation that + is found in the linux-2.6-windriver.git is the combination of all + supported boards and configurations. + + + +This technique is flexible and allows the seamless blending of an immutable +history with additional deployment specific patches. Any additions to the +kernel become an integrated part of the branches. + + + + + +A summary of end user tree construction activities follow: + + compile and link a full top-down kernel description from feature descriptions + execute the complete description to generate a meta-series + interpret the meta-series to create a customized git repository for the + board + migrate configuration fragments and configure the kernel + checkout the BSP branch and build + + +
+ +
+ Build Strategy + +There are some prerequisites that must be met before starting the compilation +phase of the kernel build system: + + + There must be a kernel git repository indicated in the SRC_URI. + There must be a branch <bsp name>-<kernel type>. + + + +These are typically met by running tree construction/patching phase of the +build system, but can be achieved by other means. Examples of alternate work +flows such as bootstrapping a BSP are provided below. + + +Before building a kernel it is configured by processing all of the +configuration "fragments" specified by the scc feature descriptions. As the +features are compiled, associated kernel configuration fragments are noted +and recorded in the meta-series in their compilation order. The +fragments are migrated, pre-processed and passed to the Linux Kernel +Configuration subsystem (lkc) as raw input in the form of a .config file. +The lkc uses its own internal dependency constraints to do the final +processing of that information and generates the final .config that will +be used during compilation. + + +Kernel compilation is started, using the board's architecture and other +relevant values from the board template, and a kernel image is produced. + + +The other thing that you will first see once you configure a kernel is that +it will generate a build tree that is separate from your git source tree. +This build dir will be called "linux-<BSPname>-<kerntype>-build" where +kerntype is one of standard kernel types. This functionality is done by making +use of the existing support that is within the kernel.org tree by default. + + +What this means, is that all the generated files (that includes the final +".config" itself, all ".o" and ".a" etc) are now in this directory. Since +the git source tree can contain any number of BSPs, all on their own branch, +you now can easily switch between builds of BSPs as well, since each one also +has their own separate build directory. + +
+ + + + + +
+ Workflow Examples + + +As previously noted, the Yocto Project kernel has built in git/guilt +integration, but these utilities are not the only way to work with the kernel +repository. Yocto Project has not made changes to git, or other tools that +invalidate alternate workflows. Additionally, the way the kernel repository +is constructed uses only core git functionality allowing any number of tools +or front ends to use the resulting tree. + +This section contains several workflow examples. + + +
+ Change Inspection: Kernel Changes/Commits + +A common question when working with a BSP/kernel is: "What changes have been applied to this tree?" + + +In some projects, where a collection of directories that +contained patches to the kernel, those patches could be inspected, grep'd or +otherwise used to get a general feeling for changes. This sort of patch +inspection is not an efficient way to determine what has been done to the +kernel, since there are many optional patches that are selected based on the +kernel type and feature description, not to mention patches that are actually +in directories that are not being searched. + + +A more effective way to determine what has changed in the kernel is to use +git and inspect / search the kernel tree. This is a full view of not only the +source code modifications, but the reasoning behind the changes. + +
+ What Changed in a BSP? + +These examples could continue for some time, since the Yocto Project git +repository doesn't break existing git functionality and there are nearly +endless permutations of those commands. Also note that unless a commit range +is given (<kernel type>..<bsp>-<kernel type>), kernel.org history is blended +with Yocto Project changes + + + # full description of the changes + > git whatchanged <kernel type>..<bsp>-<kernel type> + > eg: git whatchanged standard..common_pc-standard + + # summary of the changes + > git log ‐‐pretty=oneline ‐‐abbrev-commit <kernel type>..<bsp>-<kernel type> + + # source code changes (one combined diff) + > git diff <kernel type>..<bsp>-<kernel type> + > git show <kernel type>..<bsp>-<kernel type> + + # dump individual patches per commit + > git format-patch -o <dir> <kernel type>..<bsp>-<kernel type> + + # determine the change history of a particular file + > git whatchanged <path to file> + + # determine the commits which touch each line in a file + > git blame <path to file> + +
+ +
+ Show a Particular Feature or Branch Change + +Significant features or branches are tagged in the Yocto Project tree to divide +changes. Remember to first determine (or add) the tag of interest. Note: +there will be many tags, since each BSP branch is tagged, kernel.org tags and +feature tags are all present. + + + # show the changes tagged by a feature + > git show <tag> + > eg: git show yaffs2 + + # determine which branches contain a feature + > git branch ‐‐contains <tag> + + # show the changes in a kernel type + > git whatchanged wrs_base..<kernel type> + > eg: git whatchanged wrs_base..standard + + +Many other comparisons can be done to isolate BSP changes, such as comparing +to kernel.org tags (v2.6.27.18, etc), per subsystem comparisons (git +whatchanged mm) or many other types of checks. + +
+
+ +
+ Development: Saving Kernel Modifications + +Another common operation is to build a Yocto Project supplied BSP, make some +changes, rebuild and test. Those local changes often need to be exported, +shared or otherwise maintained. + + +Since the Yocto Project kernel source tree is backed by git, this activity is +greatly simplified and is much easier than in previous releases. git tracks +file modifications, additions and deletions, which allows the developer to +modify the code and later realize that the changes should be saved, and +easily determine what was changed. It also provides many tools to commit, +undo and export those modifications. + + +There are many ways to perform this action, and the technique employed +depends on the destination for the patches, which could be any of: + + bulk storage + internal sharing either through patches or using git + external submission + export for integration into another SCM + + + +The destination of the patches also incluences the method of gathering them +due to issues such as: + + bisectability + commit headers + division of subsystems for separate submission / review + + + +
+ Bulk Export + +If patches are simply being stored outside of the kernel source repository, +either permanently or temporarily, then there are several methods that can be +used. + + +Note the "bulk" in this discussion, these techniques are not appropriate for +full integration of upstream submission, since they do not properly divide +changes or provide an avenue for per-change commit messages. This example +assumes that changes have not been committed incrementally during development +and simply must be gathered and exported. + + # bulk export of ALL modifications without separation or division + # of the changes + + > git add . + > git commit -s -a -m >commit message< + or + > git commit -s -a # and interact with $EDITOR + + + +These operations have captured all the local changes in the project source +tree in a single git commit, and that commit is also stored in the project's +source tree. + + +Once exported, those changes can then be restored manually, via a template or +through integration with the default_kernel. Those topics are covered in +future sections. + +
+ +
+ Incremental/Planned Sharing + +Note: unlike the previous "bulk" section, the following examples assume that +changes have been incrementally committed to the tree during development and +now are being exported. + + +During development the following commands will be of interest, but for full +git documentation refer to the git man pages or an online resource such as +http://github.com + + # edit a file + > vi >path</file + # stage the change + > git add >path</file + # commit the change + > git commit -s + # remove a file + > git rm >path</file + # commit the change + > git commit -s + + ... etc. + + + +Distributed development with git is possible by having a universally agreed +upon unique commit identifier (set by the creator of the commit) mapping to a +specific changeset with a specific parent. This ID is created for you when +you create a commit, and will be re-created when you amend/alter or re-apply +a commit. As an individual in isolation, this is of no interest, but if you +intend to share your tree with normal git push/pull operations for +distributed development, you should consider the ramifications of changing a +commit that you've already shared with others. + + +Assuming that the changes have *not* been pushed upstream, or pulled into +another repository, both the commit content and commit messages associated +with development can be update via: + + > git add >path</file + > git commit ‐‐amend + > git rebase or git rebase -i + + + +Again, assuming that the changes have *not* been pushed upstream, and that +there are no pending works in progress (use "git status" to check) then +commits can be reverted (undone) via: + + # remove the commit, update working tree and remove all + # traces of the change + > git reset ‐‐hard HEAD^ + # remove the commit, but leave the files changed and staged for re-commit + > git reset ‐‐soft HEAD^ + # remove the commit, leave file change, but not staged for commit + > git reset ‐‐mixed HEAD^ + + + +Branches can be created, changes cherry-picked or any number of git +operations performed until the commits are in good order for pushing upstream +or pull requests. After a push or pull, commits are normally considered +'permanent' and should not be modified, only incrementally changed in new +commits. This is standard "git" workflow and Yocto Project recommends the +kernel.org best practices. + +It is recommend to tag or branch before adding changes to a Yocto Project + BSP (or creating a new one), since the branch or tag provides a + reference point to facilitate locating and exporting local changes. + + +
+ Export Internally Via Patches + +Committed changes can be extracted from a working directory by exporting them +as patches. Those patches can be used for upstream submission, placed in a +Yocto Project template for automatic kernel patching or many other common uses. + + + # >first commit> can be a tag if one was created before development + # began. It can also be the parent branch if a branch was created + # before development began. + + > git format-patch -o <dir> <first commit>..<last commit> + + + + + In other words: + + # identify commits of interest. + + # if the tree was tagged before development + > git format-patch -o <save dir> <tag> + + # if no tags are available + > git format-patch -o <save dir> HEAD^ # last commit + > git format-patch -o <save dir> HEAD^^ # last 2 commits + > git whatchanged # identify last commit + > git format-patch -o <save dir> <commit id> + > git format-patch -o <save dir> <rev-list> + + + + +The result is a directory with sequentially numbered patches, that when +applied to a repository using "git am", will reproduce the original commit +and all related information (author, date, commit log, etc) will be +preserved. Note that new commit IDs will be generated upon reapplication, +reflecting that the commit is now applied to an underlying commit with a +different ID. + + +
+ +
+ Export Internally Via git + +Committed changes can also be exported from a working directory by pushing +(or by making a pull request) the changes into a master repository. Those +same change can then be pulled into a new kernel build at a later time using this command form: + + git push ssh://<master server>/<path to repo> <local branch>:<remote branch> + +For example: + + > push ssh://git.mycompany.com/pub/git/kernel-2.6.27 common_pc-standard:common_pc-standard + +A pull request entails using "git request-pull" to compose an email to the +maintainer requesting that a branch be pulled into the master repository, see +http://github.com/guides/pull-requests for an example. + + +Other commands such as 'git stash' or branching can also be used to save +changes, but are not covered in this document. + + +See the section "importing from another SCM" for how a git push to the +default_kernel, can be used to automatically update the builds of all users +of a central git repository. + +
+
+ +
+ Export for External (Upstream) Submission + +If patches are to be sent for external submission, they can be done via a +pull request if the patch series is large or the maintainer prefers to pull +changes. But commonly, patches are sent as email series for easy review and +integration. + + +Before sending patches for review ensure that you understand the +standard of the community in question and follow their best practices. For +example, kernel patches should follow standards such as: + + + + Documentation/SubmittingPatches (in any linux kernel source tree) + + + +The messages used to commit changes are a large part of these standards, so +ensure that the headers for each commit have the required information. If the +initial commits were not properly documented or don't meet those standards +rebasing via git rebase -i offer an opportunity to manipulate the commits and +get them into the required format. Other techniques such as branching and +cherry picking commits are also viable options. + + +Once complete, patches are sent via email to the maintainer(s) or lists that +review and integrate changes. "git send-email" is commonly used to ensure +that patches are properly formatted for easy application and avoid mailer +induced patch damage. + + +An example of dumping patches for external submission follows: + + # dump the last 4 commits + > git format-patch ‐‐thread -n -o ~/rr/ HEAD^^^^ + > git send-email ‐‐compose ‐‐subject '[RFC 0/N] <patch series summary>' \ + ‐‐to foo@yoctoproject.org ‐‐to bar@yoctoproject.org \ + ‐‐cc list@yoctoproject.org ~/rr + # the editor is invoked for the 0/N patch, and when complete the entire + # series is sent via email for review + + +
+ +
+ Export for Import into Other SCM + +Using any one of the previously discussed techniques, commits can be exported +as patches for import into another SCM. Note however, that if those patches +are manually applied to a secondary tree and then that secondary tree is +checked into the SCM, then it often results in lost information (like commit +logs) and so it is not recommended. + + +Many SCMs can directly import git commits, or can translate git patches to +not lose information. Those facilities are SCM dependent and should be used +whenever possible. + +
+
+ +
+ SCM: Working with the Yocto Project Kernel in Another SCM + +This is not the same as the exporting of patches to another SCM, but instead +is concerned with kernel development that is done completely in another +environment, but built with the Yocto Project build system. In this scenario two +things must happen: + + The delivered Yocto Project kernel must be exported into the second + SCM. + Development must be exported from that secondary SCM into a + format that can be used by the Yocto Project build system. + + +
+ Exporting Delivered Kernel to SCM + +Depending on the SCM it may be possible to export the entire Yocto Project +kernel git repository, branches and all, into a new environment. This is the +preferred method, since it has the most flexibility and potential to maintain +the meta data associated with each commit. + + +When a direct import mechanism is not available, it is still possible to +export a branch (or series of branches) and check them into a new +repository. + + +The following commands illustrate some of the steps that could be used to +import the common_pc-standard kernel into a secondary SCM + + > git checkout common_pc-standard + > cd .. ; echo linux/.git > .cvsignore + > cvs import -m "initial import" linux MY_COMPANY start + +The CVS repo could now be relocated and used in a centralized manner. + + +The following commands illustrate how two BSPs could be condensed and merged +into a second SCM: + + > git checkout common_pc-standard + > git merge cav_ebt5800-standard + # resolve any conflicts and commit them + > cd .. ; echo linux/.git > .cvsignore + > cvs import -m "initial import" linux MY_COMPANY start + + +
+ +
+ Importing Changes for Build + +Once development has reached a suitable point in the second development +environment, changes can either be exported as patches or imported into git +directly (if a conversion/import mechanism is available for the SCM). + + +If changes are exported as patches, they can be placed in a recipe and +automatically applied to the kernel during patching. + + +
+
+ + + + + +
+ BSP: Creating + + This section provides an example for creating a BSP based on an existing, and hopefully, + similar one. + Follow these steps and keep in mind your particular situation and differences: + + Get a machine configuration file that matches your machine. + You can start with something in meta/conf/machine. + Or, meta-emenlow/conf/machine has an example in its own layer. + The most up-to-date machines that are probably most similar to yours and that you might want + to look at are meta/conf/machine/atom-pc.conf and + meta-emenlow/conf/machine/emenlow.conf. + Both of these were either just added or upgraded to use the Yocto Project kernel + at . + The main difference between them is that "emenlow" is in its own layer. + It is in its own layer because it needs extra machine-specific packages such as its + own video driver and other supporting packages. + The "atom-pc" is simpler and does not need any special packages - everything it needs can + be specified in the configuration file. + The "atom-pc" machine also supports all of Asus eee901, Acer Aspire One, Toshiba NB305, + and the Intel® Embedded Development Board 1-N450 with no changes. + If you want to make minor changes to support a slightly different machine, you can + create a new configuration file for it and add it alongside the others. + You might consider keeping the common stuff separate and including it. + Similarly, you can also use multiple configuration files for different machines even + if you do it as a separate layer like meta-emenlow. + As an example consider this: + + Copy meta-emenlow + Fix or remove anything you do not need. + For this example the only thing left was the kernel directory with a linux-yocto_git.bbappend + file (linux-yocto is the kernel listed in + meta-crownbay/conf/machine/crownbay.conf. + Finally, a new entry to the build/donf/bblayers.conf was added so the + new layer could be found by Bitbake. + + + Get an image with a working kernel built. + For the kernel to compile successfully, you need to create a branch in the git repository + specifically named for your machine. + So first create a bare clone of the Yocto Project git repository, and then create a + local clone of that: + + $ git clone ‐‐bare git://git.pokylinux.org/linux-2.6-windriver.git + linux-2.6-windriver.git + $ git clone linux-2.6-windriver.git linux-2.6-windriver + + + Now create a branch in the local clone and push it to the bare clone: + + $ git checkout -b crownbay-standard origin/standard $ git push origin crownbay-standard:crownbay-standard + + + At this point, your git tree should be set up well enough to compile. + Point the build at the new kernel git tree. + You can do this by commenting out the SRC_URI variable in + meta/recipes-kernel/linux/linux-yocto_git.bb and using a SRC_URI + that points to your new bare git tree. + You should also be able to do this in linux-yocto_git.bbappend in the layer: + + # To use a staged, on-disk bare clone of a Wind River Kernel, use a variant of the + # below SRC_URI = "git://///path/to/kernel/default_kernel.git;fullclone=1" + # + SRC_URI = "git://git.pokylinux.org/linux-2.6-windriver.git;protocol=git;fullclone=1;branch=${KBRANCH};name=machine +\ + git://git.pokylinux.org/linux-2.6-windriver.git;protocol=git;noclone=1;branch=wrs_meta;name=meta" + + + After doing that, select the machine in build/conf/local.conf: + + # + MACHINE ?= "crownbay" + # + + + You should now be able to build and boot an image with the new kernel: + + $ bitbake poky-image-sato-live + + + Of course, that will give you a kernel with the default config, which is probably + not what you want. + If you just want to set some kernel config options, you can do that by putting them in a files. + For example inserting the following into some .cfg file: + + CONFIG_NETDEV_1000=y + CONFIG_E1000E=y + + + And, another .cfg file would contain: + + CONFIG_LOG_BUF_SHIFT=18 + + http://git.pokylinux.org/cgit/cgit.cgi/linux-2.6-windriver/ + + SRC_URI_append_crownbay = " file://some.cfg \ + file://other.cfg \ + " + + + You could also add these directly to the git repo's wrs_meta branch as well. + However, the former method is probably easier. + If you're also adding patches to the kernel, you can do the same thing. + Put your patches in the SRC_URI as well (plus .cfg for their kernel config options if needed). + Practically speaking, to generate the patches, you'd go to the source in the build tree: + + build/tmp/work/crownbay-poky-linux/linux-yocto-2.6.34+git0+d1cd5c80ee97e81e130be8c3de3965b770f320d6_0+ +0431115c9d720fee5bb105f6a7411efb4f851d26-r13/linux + + + Then, modify the code there, using quilt to save the changes, and recompile + (bitbake -c compile -f) + until it works. + Once you have the final patch from quilt, copy it to the + SRC_URI location, and it should be + applied the next time you do a clean build. + Of course, since you have a branch for the BSP in git, it would be better to put it there instead. + For example, in this case, commit the patch to the crownbay-standard branch, and during the + next build it will be applied from there. + + +
+ + + + + + + + + + + + + + + + +
+ "-dirty" String + +If kernel images are being built with -dirty on the end of the version +string, this simply means that there are modification in the source +directory that haven't been committed. + + > git status + + + +The above git command will indicate modified, removed or added files. Those changes should +be committed to the tree (even if they will never be saved, or exported +for future use) and the kernel rebuilt. + + +To brute force pickup and commit all such pending changes enter the following: + + > git add . + > git commit -s -a -m "getting rid of -dirty" + + + +And then rebuild the kernel + +
+ +
+ Kernel: Transition Kernel Layer + +In order to temporarily use a different base kernel in Yocto Project +Linux 3.0 you need to do the following: + + Create a custom kernel layer. + Create a git repository of the transition kernel. + + + +Once those requirements are met multiple boards and kernels can +be built. The cost of setup is only paid once and then additional +BSPs and options can be added. + + +This creates a transition kernel layer to evaluate functionality +of some other kernel with the goal of easing transition to an +integrated and validated Yocto Project kernel. + + + + + + + + + +
+
+ + + + + + + + + + -- cgit v1.2.3-54-g00ecf