summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual/ref-bitbake.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/ref-manual/ref-bitbake.xml')
-rw-r--r--documentation/ref-manual/ref-bitbake.xml419
1 files changed, 419 insertions, 0 deletions
diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml
new file mode 100644
index 0000000000..4d4b9d6188
--- /dev/null
+++ b/documentation/ref-manual/ref-bitbake.xml
@@ -0,0 +1,419 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='ref-bitbake'>
6
7 <title>BitBake</title>
8
9 <para>
10 BitBake is a program written in Python that interprets the metadata used by the OpenEmbedded
11 build system.
12 At some point, developers wonder what actually happens when you enter:
13 <literallayout class='monospaced'>
14 $ bitbake core-image-sato
15 </literallayout>
16 </para>
17
18 <para>
19 This chapter provides an overview of what happens behind the scenes from BitBake's perspective.
20 </para>
21
22 <note>
23 BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships.
24 As such, it has no real knowledge of what the tasks being executed actually do.
25 BitBake just considers a list of tasks with dependencies and handles metadata
26 that consists of variables in a certain format that get passed to the tasks.
27 </note>
28
29 <section id='ref-bitbake-parsing'>
30 <title>Parsing</title>
31
32 <para>
33 BitBake parses configuration files, classes, and <filename>.bb</filename> files.
34 </para>
35
36 <para>
37 The first thing BitBake does is look for the <filename>bitbake.conf</filename> file.
38 This file resides in the
39 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>
40 within the <filename>meta/conf/</filename> directory.
41 BitBake finds it by examining its
42 <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment
43 variable and looking for the <filename>meta/conf/</filename>
44 directory.
45 </para>
46
47 <para>
48 The <filename>bitbake.conf</filename> file lists other configuration
49 files to include from a <filename>conf/</filename>
50 directory below the directories listed in <filename>BBPATH</filename>.
51 In general, the most important configuration file from a user's perspective
52 is <filename>local.conf</filename>, which contains a user's customized
53 settings for the OpenEmbedded build environment.
54 Other notable configuration files are the distribution
55 configuration file (set by the
56 <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable)
57 and the machine configuration file
58 (set by the
59 <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable).
60 The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment
61 variables are both usually set in
62 the <filename>local.conf</filename> file.
63 Valid distribution
64 configuration files are available in the <filename>meta/conf/distro/</filename> directory
65 and valid machine configuration
66 files in the <filename>meta/conf/machine/</filename> directory.
67 Within the <filename>meta/conf/machine/include/</filename>
68 directory are various <filename>tune-*.inc</filename> configuration files that provide common
69 "tuning" settings specific to and shared between particular architectures and machines.
70 </para>
71
72 <para>
73 After the parsing of the configuration files, some standard classes are included.
74 The <filename>base.bbclass</filename> file is always included.
75 Other classes that are specified in the configuration using the
76 <filename><link linkend='var-INHERIT'>INHERIT</link></filename>
77 variable are also included.
78 Class files are searched for in a <filename>classes</filename> subdirectory
79 under the paths in <filename>BBPATH</filename> in the same way as
80 configuration files.
81 </para>
82
83 <para>
84 After classes are included, the variable
85 <filename><link linkend='var-BBFILES'>BBFILES</link></filename>
86 is set, usually in
87 <filename>local.conf</filename>, and defines the list of places to search for
88 <filename>.bb</filename> files.
89 By default, the <filename>BBFILES</filename> variable specifies the
90 <filename>meta/recipes-*/</filename> directory within Poky.
91 Adding extra content to <filename>BBFILES</filename> is best achieved through the use of
92 BitBake layers as described in the
93 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and
94 Creating Layers</ulink>" section of the Yocto Project Development Manual.
95 </para>
96
97 <para>
98 BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and
99 stores the values of various variables.
100 In summary, for each <filename>.bb</filename>
101 file the configuration plus the base class of variables are set, followed
102 by the data in the <filename>.bb</filename> file
103 itself, followed by any inherit commands that
104 <filename>.bb</filename> file might contain.
105 </para>
106
107 <para>
108 Because parsing <filename>.bb</filename> files is a time
109 consuming process, a cache is kept to speed up subsequent parsing.
110 This cache is invalid if the timestamp of the <filename>.bb</filename>
111 file itself changes, or if the timestamps of any of the include,
112 configuration or class files the <filename>.bb</filename>
113 file depends on changes.
114 </para>
115 </section>
116
117 <section id='ref-bitbake-providers'>
118 <title>Preferences and Providers</title>
119
120 <para>
121 Once all the <filename>.bb</filename> files have been
122 parsed, BitBake starts to build the target (<filename>core-image-sato</filename>
123 in the previous section's example) and looks for providers of that target.
124 Once a provider is selected, BitBake resolves all the dependencies for
125 the target.
126 In the case of <filename>core-image-sato</filename>, it would lead to
127 <filename>packagegroup-core-x11-sato</filename>,
128 which in turn leads to recipes like <filename>matchbox-terminal</filename>,
129 <filename>pcmanfm</filename> and <filename>gthumb</filename>.
130 These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
131 </para>
132
133 <para>
134 Sometimes a target might have multiple providers.
135 A common example is "virtual/kernel", which is provided by each kernel package.
136 Each machine often selects the best kernel provider by using a line similar to the
137 following in the machine configuration file:
138 </para>
139
140 <literallayout class='monospaced'>
141 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
142 </literallayout>
143
144 <para>
145 The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename>
146 is the provider with the same name as the target.
147 </para>
148
149 <para>
150 Understanding how providers are chosen is made complicated by the fact
151 that multiple versions might exist.
152 BitBake defaults to the highest version of a provider.
153 Version comparisons are made using the same method as Debian.
154 You can use the
155 <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
156 variable to specify a particular version (usually in the distro configuration).
157 You can influence the order by using the
158 <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename>
159 variable.
160 By default, files have a preference of "0".
161 Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
162 package unlikely to be used unless it is explicitly referenced.
163 Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
164 <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
165 <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
166 versions until they have undergone sufficient testing to be considered stable.
167 </para>
168
169 <para>
170 In summary, BitBake has created a list of providers, which is prioritized, for each target.
171 </para>
172 </section>
173
174 <section id='ref-bitbake-dependencies'>
175 <title>Dependencies</title>
176
177 <para>
178 Each target BitBake builds consists of multiple tasks such as
179 <filename>fetch</filename>, <filename>unpack</filename>,
180 <filename>patch</filename>, <filename>configure</filename>,
181 and <filename>compile</filename>.
182 For best performance on multi-core systems, BitBake considers each task as an independent
183 entity with its own set of dependencies.
184 </para>
185
186 <para>
187 Dependencies are defined through several variables.
188 You can find information about variables BitBake uses in the BitBake documentation,
189 which is found in the <filename>bitbake/doc/manual</filename> directory within the
190 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
191 At a basic level, it is sufficient to know that BitBake uses the
192 <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and
193 <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> variables when
194 calculating dependencies.
195 </para>
196 </section>
197
198 <section id='ref-bitbake-tasklist'>
199 <title>The Task List</title>
200
201 <para>
202 Based on the generated list of providers and the dependency information,
203 BitBake can now calculate exactly what tasks it needs to run and in what
204 order it needs to run them.
205 The build now starts with BitBake forking off threads up to the limit set in the
206 <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable.
207 BitBake continues to fork threads as long as there are tasks ready to run,
208 those tasks have all their dependencies met, and the thread threshold has not been
209 exceeded.
210 </para>
211
212 <para>
213 It is worth noting that you can greatly speed up the build time by properly setting
214 the <filename>BB_NUMBER_THREADS</filename> variable.
215 See the
216 "<ulink url='&YOCTO_DOCS_QS_URL;#building-image'>Building an Image</ulink>"
217 section in the Yocto Project Quick Start for more information.
218 </para>
219
220 <para>
221 As each task completes, a timestamp is written to the directory specified by the
222 <filename><link linkend='var-STAMP'>STAMP</link></filename> variable.
223 On subsequent runs, BitBake looks within the <filename>/build/tmp/stamps</filename>
224 directory and does not rerun
225 tasks that are already completed unless a timestamp is found to be invalid.
226 Currently, invalid timestamps are only considered on a per
227 <filename>.bb</filename> file basis.
228 So, for example, if the configure stamp has a timestamp greater than the
229 compile timestamp for a given target, then the compile task would rerun.
230 Running the compile task again, however, has no effect on other providers
231 that depend on that target.
232 This behavior could change or become configurable in future versions of BitBake.
233 </para>
234
235 <note>
236 Some tasks are marked as "nostamp" tasks.
237 No timestamp file is created when these tasks are run.
238 Consequently, "nostamp" tasks are always rerun.
239 </note>
240 </section>
241
242 <section id='ref-bitbake-runtask'>
243 <title>Running a Task</title>
244
245 <para>
246 Tasks can either be a shell task or a Python task.
247 For shell tasks, BitBake writes a shell script to
248 <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script.
249 The generated shell script contains all the exported variables, and the shell functions
250 with all variables expanded.
251 Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
252 Looking at the expanded shell functions in the run file and the output in the log files
253 is a useful debugging technique.
254 </para>
255
256 <para>
257 For Python tasks, BitBake executes the task internally and logs information to the
258 controlling terminal.
259 Future versions of BitBake will write the functions to files similar to the way
260 shell tasks are handled.
261 Logging will be handled in way similar to shell tasks as well.
262 </para>
263
264 <para>
265 Once all the tasks have been completed BitBake exits.
266 </para>
267
268 <para>
269 When running a task, BitBake tightly controls the execution environment
270 of the build tasks to make sure unwanted contamination from the build machine
271 cannot influence the build.
272 Consequently, if you do want something to get passed into the build
273 task's environment, you must take a few steps:
274 <orderedlist>
275 <listitem><para>Tell BitBake to load what you want from the environment
276 into the data store.
277 You can do so through the <filename>BB_ENV_EXTRAWHITE</filename>
278 variable.
279 For example, assume you want to prevent the build system from
280 accessing your <filename>$HOME/.ccache</filename> directory.
281 The following command tells BitBake to load
282 <filename>CCACHE_DIR</filename> from the environment into the data
283 store:
284 <literallayout class='monospaced'>
285 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR"
286 </literallayout></para></listitem>
287 <listitem><para>Tell BitBake to export what you have loaded into the
288 environment store to the task environment of every running task.
289 Loading something from the environment into the data store
290 (previous step) only makes it available in the datastore.
291 To export it to the task environment of every running task,
292 use a command similar to the following in your
293 <filename>local.conf</filename> or distro configuration file:
294 <literallayout class='monospaced'>
295 export CCACHE_DIR
296 </literallayout></para></listitem>
297 </orderedlist>
298 </para>
299
300 <note>
301 A side effect of the previous steps is that BitBake records the variable
302 as a dependency of the build process in things like the shared state
303 checksums.
304 If doing so results in unnecessary rebuilds of tasks, you can whitelist the
305 variable so that the shared state code ignores the dependency when it creates
306 checksums.
307 For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
308 example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section.
309 </note>
310 </section>
311
312 <section id='ref-bitbake-commandline'>
313 <title>BitBake Command Line</title>
314
315 <para>
316 Following is the BitBake help output:
317 </para>
318
319 <screen>
320$ bitbake --help
321Usage: bitbake [options] [package ...]
322
323Executes the specified task (default is 'build') for a given set of BitBake files.
324It expects that BBFILES is defined, which is a space separated list of files to
325be executed. BBFILES does support wildcards.
326Default BBFILES are the .bb files in the current directory.
327
328Options:
329 --version show program's version number and exit
330 -h, --help show this help message and exit
331 -b BUILDFILE, --buildfile=BUILDFILE
332 execute the task against this .bb file, rather than a
333 package from BBFILES. Does not handle any
334 dependencies.
335 -k, --continue continue as much as possible after an error. While the
336 target that failed, and those that depend on it,
337 cannot be remade, the other dependencies of these
338 targets can be processed all the same.
339 -a, --tryaltconfigs continue with builds by trying to use alternative
340 providers where possible.
341 -f, --force force run of specified cmd, regardless of stamp status
342 -c CMD, --cmd=CMD Specify task to execute. Note that this only executes
343 the specified task for the providee and the packages
344 it depends on, i.e. 'compile' does not implicitly call
345 stage for the dependencies (IOW: use only if you know
346 what you are doing). Depending on the base.bbclass a
347 listtasks tasks is defined and will show available
348 tasks
349 -r PREFILE, --read=PREFILE
350 read the specified file before bitbake.conf
351 -R POSTFILE, --postread=POSTFILE
352 read the specified file after bitbake.conf
353 -v, --verbose output more chit-chat to the terminal
354 -D, --debug Increase the debug level. You can specify this more
355 than once.
356 -n, --dry-run don't execute, just go through the motions
357 -S, --dump-signatures
358 don't execute, just dump out the signature
359 construction information
360 -p, --parse-only quit after parsing the BB files (developers only)
361 -s, --show-versions show current and preferred versions of all packages
362 -e, --environment show the global or per-package environment (this is
363 what used to be bbread)
364 -g, --graphviz emit the dependency trees of the specified packages in
365 the dot syntax
366 -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
367 Assume these dependencies don't exist and are already
368 provided (equivalent to ASSUME_PROVIDED). Useful to
369 make dependency graphs more appealing
370 -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
371 Show debug logging for the specified logging domains
372 -P, --profile profile the command and print a report
373 -u UI, --ui=UI userinterface to use
374 -t SERVERTYPE, --servertype=SERVERTYPE
375 Choose which server to use, none, process or xmlrpc
376 --revisions-changed Set the exit code depending on whether upstream
377 floating revisions have changed or not
378 </screen>
379 </section>
380
381 <section id='ref-bitbake-fetchers'>
382 <title>Fetchers</title>
383
384 <para>
385 BitBake also contains a set of "fetcher" modules that allow
386 retrieval of source code from various types of sources.
387 For example, BitBake can get source code from a disk with the metadata, from websites,
388 from remote shell accounts or from Source Code Management (SCM) systems
389 like <filename>cvs/subversion/git</filename>.
390 </para>
391
392 <para>
393 Fetchers are usually triggered by entries in
394 <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>.
395 You can find information about the options and formats of entries for specific
396 fetchers in the BitBake manual located in the
397 <filename>bitbake/doc/manual</filename> directory of the
398 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
399 </para>
400
401 <para>
402 One useful feature for certain Source Code Manager (SCM) fetchers is the ability to
403 "auto-update" when the upstream SCM changes version.
404 Since this ability requires certain functionality from the SCM, not all
405 systems support it.
406 Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update".
407 This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename>
408 variable.
409 See the
410 "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>" section
411 in the Yocto Project Development Manual for more information.
412 </para>
413
414 </section>
415
416</chapter>
417<!--
418vim: expandtab tw=80 ts=4 spell spelllang=en_gb
419-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-12 12:20:34 +0000