From 5dd34a717eb94f1934bf60842b1406349897c970 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Thu, 8 Dec 2011 10:34:44 -0800 Subject: documentation/poky-ref-manual: New chapter introduced Long-term strategy for the YP Reference Manual is that it contains reference material and not "how-to-information". A step in this direction is to isolate any discussions on components and other areas of YP that need talked about. So to start with, I have created a new chapter for now named "Technical Details" that so far has a discussion of some components and shared state. This is a step in the direction of making this manual a reference manual and not a "how to" manual. Changes included removing redundant material from the 'usingpoky.xml' chapter and also adding the new chapter 'technical-details' into the 'poky-ref-manual.xml' file used for the make. (From yocto-docs rev: a01477f787768230bc25da2d094326922be23dd4) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../poky-ref-manual/technical-details.xml | 229 +++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 documentation/poky-ref-manual/technical-details.xml (limited to 'documentation/poky-ref-manual/technical-details.xml') diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml new file mode 100644 index 0000000000..8f83ecb3a1 --- /dev/null +++ b/documentation/poky-ref-manual/technical-details.xml @@ -0,0 +1,229 @@ + + +Technical Details + + + This chapter provides technical details for various parts of the Yocto Project. + Currently, topics include Yocto Project components and shared state (sstate) cache. + + +
+ Yocto Project Components + + + The BitBake task executor together with various types of configuration files form the + Yocto Project core. + This section overviews the BitBake task executor and the + configuration files by describing what they are used for and how they interact. + + + + BitBake handles the parsing and execution of the data files. + The data itself is of various types: + + Recipes: Provides details about particular + pieces of software + Class Data: An abstraction of common build + information (e.g. how to build a Linux kernel). + Configuration Data: Defines machine-specific settings, + policy decisions, etc. + Configuration data acts as the glue to bind everything together. + + For more information on data, see the + + Yocto Project Terms section in + + The Yocto Project Development Manual. + + + + BitBake knows how to combine multiple data sources together and refers to each data source + as a 'layer'. + + + + Following are some brief details on these core components. + For more detailed information on these components see the + 'Reference: Directory Structure' + appendix. + + +
+ BitBake + + + BitBake is the tool at the heart of the Yocto Project and is responsible + for parsing the metadata, generating a list of tasks from it, + and then executing those tasks. + To see a list of the options BitBake supports, use the following help command: + + $ bitbake --help + + + + + The most common usage for BitBake is bitbake <packagename>, where + packagename is the name of the package you want to build + (referred to as the "target" in this manual). + The target often equates to the first part of a .bb filename. + So, to run the matchbox-desktop_1.2.3.bb file, you + might type the following: + + $ bitbake matchbox-desktop + + Several different versions of matchbox-desktop might exist. + BitBake chooses the one selected by the distribution configuration. + You can get more details about how BitBake chooses between different + target versions and providers in the + Preferences and Providers section. + + + + BitBake also tries to execute any dependent tasks first. + So for example, before building matchbox-desktop, BitBake + would build a cross compiler and eglibc if they had not already + been built. + This release of the Yocto Project does not support the glibc + GNU version of the Unix standard C library. By default, the Yocto Project builds with + eglibc. + + + + A useful BitBake option to consider is the -k or + --continue option. + This option instructs BitBake to try and continue processing the job as much + as possible even after encountering an error. + When an error occurs, the target that + failed and those that depend on it cannot be remade. + However, when you use this option other dependencies can still be processed. + +
+ +
+ Metadata (Recipes) + + + The .bb files are usually referred to as "recipes." + In general, a recipe contains information about a single piece of software. + The information includes the location from which to download the source patches + (if any are needed), which special configuration options to apply, + how to compile the source files, and how to package the compiled output. + + + + The term "package" can also be used to describe recipes. + However, since the same word is used for the packaged output from the Yocto + Project (i.e. .ipk or .deb files), + this document avoids using the term "package" to refer to recipes. + +
+ +
+ Classes + + + Class files (.bbclass) contain information that is useful to share + between metadata files. + An example is the Autotools class, which contains + common settings for any application that Autotools uses. + The Reference: Classes appendix provides details + about common classes and how to use them. + +
+ +
+ Configuration + + + The configuration files (.conf) define various configuration variables + that govern the Yocto Project build process. + These files fall into several areas that define machine configuration options, + distribution configuration options, compiler tuning options, general common configuration + options and user configuration options (local.conf, which is found + in the Yocto Project files build directory). + +
+
+ +
+ Considering Shared State Cache + + + By design, the Yocto Project builds everything from scratch unless it can determine that + a given task's inputs have not changed. + While building from scratch ensures that everything is current, it does also + mean that a lot of time could be spent rebuiding things that don't necessarily need built. + + + + The Yocto Project build process uses a shared state caching scheme to avoid having to + rebuild software when it is not necessary. + Because the build time for a Yocto image can be significant, it is helpful to try and + determine what really needs built and what can be skipped given a particular project's + build process. + + + + The scheme that the Yocto Project uses involves checksum generation and comparison for + a task's inputs. + The scheme also employs an area of memory called the shared state cache that is + pointed to by the SSTATE_DIR variable. + This area contains task output generated from a previous build. + If a given task's checksum matches the checksum of a previous build for the same + task, the build process uses the state of the cache rather than rerunning that + task. + + + + The previous paragraph is a simplistic explanation of how the build process + uses checksums and shared state memory cache to avoide building tasks that + don't need built. + If you want a bit more explanation on the topic, + see "Shared + State - What does it mean and why should I care?" from the Yocto + Project discussion archives. + + + + As with all schemes, this one has some drawbacks. + It is possible that you could make implicit changes that are not factored into the checksum + calculation, but do affect a task's output. + A good example is perhaps when a tool changes its output. + Let's say that the output of rpmdeps needed to change. + The result of the change should be that all the "package", "package_write_rpm", + and "package_deploy-rpm" sstate-cache items would become invalid. + But, because this is a change that is external to the code and therefore implicit, + the associated sstate-cache items do not become invalidated. + In this case, the build process would use the cache items rather than running the + task again. + Obviously, these types of implicit changes can cause problems. + + + + To avoid these problems during the build, you need to understand the effects of any + change you make. + Note that any changes you make directly to a function automatically are factored into + the checksum calculation and thus, will invalidate the associated area of sstate cache. + You need to be aware of any implicit changes that are not obvious changes to the + code and could affect the output of a given task. + Once you are aware of such a change, you can take steps to invalidate the cache + and force the task to run. + The step to take is as simple as changing a function's comments in the source code. + For example, to invalidate package sstate files, change the comment statments + of do_package or one of the functions it calls. + The change is purely cosmetic, but it causes the checksum to be recalculated and + forces the task to be run again. + + + + For an example of a commit that makes a cosmetic change to invalidate an sstate, + see this + commit. + +
+ + + -- cgit v1.2.3-54-g00ecf