summaryrefslogtreecommitdiffstats
path: root/documentation/poky-ref-manual/technical-details.xml
blob: b34179533b760b8991961db1bcf3769651935460 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<chapter id='technical-details'>
<title>Technical Details</title>

    <para>
        This chapter provides technical details for various parts of the Yocto Project. 
        Currently, topics include Yocto Project components and shared state (sstate) cache.
    </para>

<section id='usingpoky-components'>
    <title>Yocto Project Components</title>

    <para>
        The BitBake task executor together with various types of configuration files form the 
        Yocto Project core.
        This section overviews the BitBake task executor and the
        configuration files by describing what they are used for and how they interact.
    </para>
    
    <para>  
        BitBake handles the parsing and execution of the data files. 
        The data itself is of various types:
    <itemizedlist>
        <listitem><para><emphasis>Recipes:</emphasis>  Provides details about particular 
            pieces of software</para></listitem>
        <listitem><para><emphasis>Class Data:</emphasis>  An abstraction of common build 
            information (e.g. how to build a Linux kernel).</para></listitem>
        <listitem><para><emphasis>Configuration Data:</emphasis>  Defines machine-specific settings, 
            policy decisions, etc.
            Configuration data acts as the glue to bind everything together.</para></listitem>
    </itemizedlist>
        For more information on data, see the
        <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#yocto-project-terms'>
        Yocto Project Terms</ulink> section in 
        <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html'>
        The Yocto Project Development Manual</ulink>.
    </para>

    <para> 
        BitBake knows how to combine multiple data sources together and refers to each data source
        as a "<link linkend='usingpoky-changes-layers'>layer</link>".
    </para>

    <para>
        Following are some brief details on these core components.
        For more detailed information on these components see the 
        <link linkend='ref-structure'>'Reference: Directory Structure'</link>
        appendix.
    </para>

    <section id='usingpoky-components-bitbake'>
        <title>BitBake</title>

        <para>
            BitBake is the tool at the heart of the Yocto Project and is responsible
            for parsing the metadata, generating a list of tasks from it,
            and then executing those tasks. 
            To see a list of the options BitBake supports, use the following help command:
            <literallayout class='monospaced'>
     $ bitbake --help
            </literallayout>
        </para>

        <para>
            The most common usage for BitBake is <filename>bitbake &lt;packagename&gt;</filename>, where
            <filename>packagename</filename> is the name of the package you want to build 
            (referred to as the "target" in this manual). 
            The target often equates to the first part of a <filename>.bb</filename> filename.
            So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
            might type the following:
            <literallayout class='monospaced'>
     $ bitbake matchbox-desktop
            </literallayout>
            Several different versions of <filename>matchbox-desktop</filename> might exist.
            BitBake chooses the one selected by the distribution configuration.
            You can get more details about how BitBake chooses between different 
            target versions and providers in the 
            <link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
        </para>

        <para>
            BitBake also tries to execute any dependent tasks first.
            So for example, before building <filename>matchbox-desktop</filename>, BitBake
            would build a cross compiler and <filename>eglibc</filename> if they had not already 
            been built.
            <note>This release of the Yocto Project does not support the <filename>glibc</filename>
                GNU version of the Unix standard C library.  By default, the Yocto Project builds with
                <filename>eglibc</filename>.</note>
        </para>

        <para>
            A useful BitBake option to consider is the <filename>-k</filename> or 
            <filename>--continue</filename> option.  
            This option instructs BitBake to try and continue processing the job as much 
            as possible even after encountering an error.  
            When an error occurs, the target that
            failed and those that depend on it cannot be remade.  
            However, when you use this option other dependencies can still be processed.
        </para>
    </section>

    <section id='usingpoky-components-metadata'>
        <title>Metadata (Recipes)</title>

        <para>
            The <filename>.bb</filename> files are usually referred to as "recipes." 
            In general, a recipe contains information about a single piece of software.
            The information includes the location from which to download the source patches 
            (if any are needed), which special configuration options to apply, 
            how to compile the source files, and how to package the compiled output. 
        </para>

        <para>
            The term "package" can also be used to describe recipes.
            However, since the same word is used for the packaged output from the Yocto 
            Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), 
            this document avoids using the term "package" when refering to recipes.
        </para>
    </section>

    <section id='usingpoky-components-classes'>
        <title>Classes</title>

        <para>
            Class files (<filename>.bbclass</filename>) contain information that is useful to share
            between metadata files. 
            An example is the Autotools class, which contains
            common settings for any application that Autotools uses.
            The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
            about common classes and how to use them.
        </para>
    </section>

    <section id='usingpoky-components-configuration'>
        <title>Configuration</title>

        <para>
            The configuration files (<filename>.conf</filename>) define various configuration variables
            that govern the Yocto Project build process. 
            These files fall into several areas that define machine configuration options, 
            distribution configuration options, compiler tuning options, general common configuration
            options and user configuration options (<filename>local.conf</filename>, which is found
            in the Yocto Project files build directory).
        </para>
    </section>
</section>

<section id="shared-state-cache">
    <title>Shared State Cache</title>

    <para>
        By design, the Yocto Project builds everything from scratch unless it can determine that
        a given task's inputs have not changed.  
        While building from scratch ensures that everything is current, it does also
        mean that a lot of time could be spent rebuiding things that don't necessarily need built.
    </para>

    <para>
        The Yocto Project build process uses a shared state caching scheme to avoid having to 
        rebuild software when it is not necessary.  
        Because the build time for a Yocto image can be significant, it is helpful to try and 
        determine what really needs built and what can be skipped given a particular project's
        build process.
    </para>

    <para>
        The scheme that the Yocto Project uses involves checksum generation and comparison for
        a task's inputs.
        The scheme also employs an area of memory called the shared state cache that is 
        pointed to by the <filename>SSTATE_DIR</filename> variable.
        This area contains task output generated from a previous build.  
        If a given task's checksum matches the checksum of a previous build for the same  
        task, the build process uses the state of the cache rather than rerunning that 
        task.
    </para>

    <para>
        The previous paragraph is a simplistic explanation of how the build process 
        uses checksums and shared state memory cache to avoide building tasks that 
        don't need built.
        If you want a bit more explanation on the topic, 
        see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
        State - What does it mean and why should I care?</ulink>" from the Yocto 
        Project discussion archives.
    </para>

    <para>
        As with all schemes, this one has some drawbacks.
        It is possible that you could make implicit changes that are not factored into the checksum 
        calculation, but do affect a task's output. 
        A good example is perhaps when a tool changes its output.
        Let's say that the output of <filename>rpmdeps</filename> needed to change.
        The result of the change should be that all the "package", "package_write_rpm",
        and "package_deploy-rpm" sstate-cache items would become invalid.
        But, because this is a change that is external to the code and therefore implicit,
        the associated sstate-cache items do not become invalidated.
        In this case, the build process would use the cache items rather than running the 
        task again. 
        Obviously, these types of implicit changes can cause problems.
    </para>

    <para>
        To avoid these problems during the build, you need to understand the effects of any
        change you make.
        Note that any changes you make directly to a function automatically are factored into 
        the checksum calculation and thus, will invalidate the associated area of sstate cache.
        You need to be aware of any implicit changes that are not obvious changes to the 
        code and could affect the output of a given task. 
        Once you are aware of such a change, you can take steps to invalidate the cache 
        and force the task to run. 
        The step to take is as simple as changing a function's comments in the source code. 
        For example, to invalidate package sstate files, change the comment statments
        of <filename>do_package</filename> or one of the functions it calls.
        The change is purely cosmetic, but it causes the checksum to be recalculated and  
        forces the task to be run again.
    </para>

    <note>
        For an example of a commit that makes a cosmetic change to invalidate an sstate, 
        see this
        <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
    </note>
</section>

</chapter>
<!-- 
vim: expandtab tw=80 ts=4 
-->