summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual/ref-bitbake.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/ref-manual/ref-bitbake.xml')
-rw-r--r--documentation/ref-manual/ref-bitbake.xml432
1 files changed, 432 insertions, 0 deletions
diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml
new file mode 100644
index 0000000..e24ea45
--- /dev/null
+++ b/documentation/ref-manual/ref-bitbake.xml
@@ -0,0 +1,432 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='ref-bitbake'>
6
7 <title>BitBake</title>
8
9 <para>
10 BitBake is a program written in Python that interprets the
11 <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> used by
12 the OpenEmbedded build system.
13 At some point, developers wonder what actually happens when you enter:
14 <literallayout class='monospaced'>
15 $ bitbake core-image-sato
16 </literallayout>
17 </para>
18
19 <para>
20 This chapter provides an overview of what happens behind the scenes from BitBake's perspective.
21 </para>
22
23 <note>
24 BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships.
25 As such, it has no real knowledge of what the tasks being executed actually do.
26 BitBake just considers a list of tasks with dependencies and handles
27 <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>
28 consisting of variables in a certain format that get passed to the tasks.
29 </note>
30
31 <section id='ref-bitbake-parsing'>
32 <title>Parsing</title>
33
34 <para>
35 BitBake parses configuration files, classes, and <filename>.bb</filename> files.
36 </para>
37
38 <para>
39 The first thing BitBake does is look for the <filename>bitbake.conf</filename> file.
40 This file resides in the
41 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>
42 within the <filename>meta/conf/</filename> directory.
43 BitBake finds it by examining its
44 <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment
45 variable and looking for the <filename>meta/conf/</filename>
46 directory.
47 </para>
48
49 <para>
50 The <filename>bitbake.conf</filename> file lists other configuration
51 files to include from a <filename>conf/</filename>
52 directory below the directories listed in <filename>BBPATH</filename>.
53 In general, the most important configuration file from a user's perspective
54 is <filename>local.conf</filename>, which contains a user's customized
55 settings for the OpenEmbedded build environment.
56 Other notable configuration files are the distribution
57 configuration file (set by the
58 <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable)
59 and the machine configuration file
60 (set by the
61 <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable).
62 The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment
63 variables are both usually set in
64 the <filename>local.conf</filename> file.
65 Valid distribution
66 configuration files are available in the <filename>meta/conf/distro/</filename> directory
67 and valid machine configuration
68 files in the <filename>meta/conf/machine/</filename> directory.
69 Within the <filename>meta/conf/machine/include/</filename>
70 directory are various <filename>tune-*.inc</filename> configuration files that provide common
71 "tuning" settings specific to and shared between particular architectures and machines.
72 </para>
73
74 <para>
75 After the parsing of the configuration files, some standard classes are included.
76 The <filename>base.bbclass</filename> file is always included.
77 Other classes that are specified in the configuration using the
78 <filename><link linkend='var-INHERIT'>INHERIT</link></filename>
79 variable are also included.
80 Class files are searched for in a <filename>classes</filename> subdirectory
81 under the paths in <filename>BBPATH</filename> in the same way as
82 configuration files.
83 </para>
84
85 <para>
86 After classes are included, the variable
87 <filename><link linkend='var-BBFILES'>BBFILES</link></filename>
88 is set, usually in
89 <filename>local.conf</filename>, and defines the list of places to search for
90 <filename>.bb</filename> files.
91 By default, the <filename>BBFILES</filename> variable specifies the
92 <filename>meta/recipes-*/</filename> directory within Poky.
93 Adding extra content to <filename>BBFILES</filename> is best achieved through the use of
94 BitBake layers as described in the
95 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and
96 Creating Layers</ulink>" section of the Yocto Project Development Manual.
97 </para>
98
99 <para>
100 BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and
101 stores the values of various variables.
102 In summary, for each <filename>.bb</filename>
103 file the configuration plus the base class of variables are set, followed
104 by the data in the <filename>.bb</filename> file
105 itself, followed by any inherit commands that
106 <filename>.bb</filename> file might contain.
107 </para>
108
109 <para>
110 Because parsing <filename>.bb</filename> files is a time
111 consuming process, a cache is kept to speed up subsequent parsing.
112 This cache is invalid if the timestamp of the <filename>.bb</filename>
113 file itself changes, or if the timestamps of any of the include,
114 configuration files or class files on which the
115 <filename>.bb</filename> file depends change.
116 </para>
117 </section>
118
119 <section id='ref-bitbake-providers'>
120 <title>Preferences and Providers</title>
121
122 <para>
123 Once all the <filename>.bb</filename> files have been
124 parsed, BitBake starts to build the target (<filename>core-image-sato</filename>
125 in the previous section's example) and looks for providers of that target.
126 Once a provider is selected, BitBake resolves all the dependencies for
127 the target.
128 In the case of <filename>core-image-sato</filename>, it would lead to
129 <filename>packagegroup-core-x11-sato</filename>,
130 which in turn leads to recipes like <filename>matchbox-terminal</filename>,
131 <filename>pcmanfm</filename> and <filename>gthumb</filename>.
132 These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
133 </para>
134
135 <para>
136 Sometimes a target might have multiple providers.
137 A common example is "virtual/kernel", which is provided by each kernel package.
138 Each machine often selects the best kernel provider by using a line similar to the
139 following in the machine configuration file:
140 </para>
141
142 <literallayout class='monospaced'>
143 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
144 </literallayout>
145
146 <para>
147 The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename>
148 is the provider with the same name as the target.
149 </para>
150
151 <para>
152 Understanding how providers are chosen is made complicated by the fact
153 that multiple versions might exist.
154 BitBake defaults to the highest version of a provider.
155 Version comparisons are made using the same method as Debian.
156 You can use the
157 <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
158 variable to specify a particular version (usually in the distro configuration).
159 You can influence the order by using the
160 <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename>
161 variable.
162 By default, files have a preference of "0".
163 Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
164 package unlikely to be used unless it is explicitly referenced.
165 Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
166 <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
167 <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
168 versions until they have undergone sufficient testing to be considered stable.
169 </para>
170
171 <para>
172 In summary, BitBake has created a list of providers, which is prioritized, for each target.
173 </para>
174 </section>
175
176 <section id='ref-bitbake-dependencies'>
177 <title>Dependencies</title>
178
179 <para>
180 Each target BitBake builds consists of multiple tasks such as
181 <filename>fetch</filename>, <filename>unpack</filename>,
182 <filename>patch</filename>, <filename>configure</filename>,
183 and <filename>compile</filename>.
184 For best performance on multi-core systems, BitBake considers each task as an independent
185 entity with its own set of dependencies.
186 </para>
187
188 <para>
189 Dependencies are defined through several variables.
190 You can find information about variables BitBake uses in the BitBake documentation,
191 which is found in the <filename>bitbake/doc/manual</filename> directory within the
192 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
193 At a basic level, it is sufficient to know that BitBake uses the
194 <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and
195 <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> variables when
196 calculating dependencies.
197 </para>
198 </section>
199
200 <section id='ref-bitbake-tasklist'>
201 <title>The Task List</title>
202
203 <para>
204 Based on the generated list of providers and the dependency information,
205 BitBake can now calculate exactly what tasks it needs to run and in what
206 order it needs to run them.
207 The build now starts with BitBake forking off threads up to the limit set in the
208 <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable.
209 BitBake continues to fork threads as long as there are tasks ready to run,
210 those tasks have all their dependencies met, and the thread threshold has not been
211 exceeded.
212 </para>
213
214 <para>
215 It is worth noting that you can greatly speed up the build time by properly setting
216 the <filename>BB_NUMBER_THREADS</filename> variable.
217 See the
218 "<ulink url='&YOCTO_DOCS_QS_URL;#building-image'>Building an Image</ulink>"
219 section in the Yocto Project Quick Start for more information.
220 </para>
221
222 <para>
223 As each task completes, a timestamp is written to the directory specified by the
224 <filename><link linkend='var-STAMP'>STAMP</link></filename> variable.
225 On subsequent runs, BitBake looks within the <filename>build/tmp/stamps</filename>
226 directory and does not rerun
227 tasks that are already completed unless a timestamp is found to be invalid.
228 Currently, invalid timestamps are only considered on a per
229 <filename>.bb</filename> file basis.
230 So, for example, if the configure stamp has a timestamp greater than the
231 compile timestamp for a given target, then the compile task would rerun.
232 Running the compile task again, however, has no effect on other providers
233 that depend on that target.
234 This behavior could change or become configurable in future versions of BitBake.
235 </para>
236
237 <note>
238 Some tasks are marked as "nostamp" tasks.
239 No timestamp file is created when these tasks are run.
240 Consequently, "nostamp" tasks are always rerun.
241 </note>
242 </section>
243
244 <section id='ref-bitbake-runtask'>
245 <title>Running a Task</title>
246
247 <para>
248 Tasks can either be a shell task or a Python task.
249 For shell tasks, BitBake writes a shell script to
250 <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script.
251 The generated shell script contains all the exported variables, and the shell functions
252 with all variables expanded.
253 Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
254 Looking at the expanded shell functions in the run file and the output in the log files
255 is a useful debugging technique.
256 </para>
257
258 <para>
259 For Python tasks, BitBake executes the task internally and logs information to the
260 controlling terminal.
261 Future versions of BitBake will write the functions to files similar to the way
262 shell tasks are handled.
263 Logging will be handled in a way similar to shell tasks as well.
264 </para>
265
266 <para>
267 Once all the tasks have been completed BitBake exits.
268 </para>
269
270 <para>
271 When running a task, BitBake tightly controls the execution environment
272 of the build tasks to make sure unwanted contamination from the build machine
273 cannot influence the build.
274 Consequently, if you do want something to get passed into the build
275 task's environment, you must take a few steps:
276 <orderedlist>
277 <listitem><para>Tell BitBake to load what you want from the environment
278 into the data store.
279 You can do so through the <filename>BB_ENV_EXTRAWHITE</filename>
280 variable.
281 For example, assume you want to prevent the build system from
282 accessing your <filename>$HOME/.ccache</filename> directory.
283 The following command tells BitBake to load
284 <filename>CCACHE_DIR</filename> from the environment into the data
285 store:
286 <literallayout class='monospaced'>
287 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR"
288 </literallayout></para></listitem>
289 <listitem><para>Tell BitBake to export what you have loaded into the
290 environment store to the task environment of every running task.
291 Loading something from the environment into the data store
292 (previous step) only makes it available in the datastore.
293 To export it to the task environment of every running task,
294 use a command similar to the following in your
295 <filename>local.conf</filename> or distro configuration file:
296 <literallayout class='monospaced'>
297 export CCACHE_DIR
298 </literallayout></para></listitem>
299 </orderedlist>
300 </para>
301
302 <note>
303 A side effect of the previous steps is that BitBake records the variable
304 as a dependency of the build process in things like the shared state
305 checksums.
306 If doing so results in unnecessary rebuilds of tasks, you can whitelist the
307 variable so that the shared state code ignores the dependency when it creates
308 checksums.
309 For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
310 example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section.
311 </note>
312 </section>
313
314 <section id='ref-bitbake-commandline'>
315 <title>BitBake Command Line</title>
316
317 <para>
318 Following is the BitBake help output:
319 </para>
320
321 <screen>
322$ bitbake --help
323Usage: bitbake [options] [recipename/target ...]
324
325 Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
326 It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
327 will provide the layer, BBFILES and other configuration information.
328
329Options:
330 --version show program's version number and exit
331 -h, --help show this help message and exit
332 -b BUILDFILE, --buildfile=BUILDFILE
333 Execute tasks from a specific .bb recipe directly.
334 WARNING: Does not handle any dependencies from other
335 recipes.
336 -k, --continue Continue as much as possible after an error. While the
337 target that failed and anything depending on it cannot
338 be built, as much as possible will be built before
339 stopping.
340 -a, --tryaltconfigs Continue with builds by trying to use alternative
341 providers where possible.
342 -f, --force Force the specified targets/task to run (invalidating
343 any existing stamp file).
344 -c CMD, --cmd=CMD Specify the task to execute. The exact options
345 available depend on the metadata. Some examples might
346 be 'compile' or 'populate_sysroot' or 'listtasks' may
347 give a list of the tasks available.
348 -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
349 Invalidate the stamp for the specified task such as
350 'compile' and then run the default task for the
351 specified target(s).
352 -r PREFILE, --read=PREFILE
353 Read the specified file before bitbake.conf.
354 -R POSTFILE, --postread=POSTFILE
355 Read the specified file after bitbake.conf.
356 -v, --verbose Output more log message data to the terminal.
357 -D, --debug Increase the debug level. You can specify this more
358 than once.
359 -n, --dry-run Don't execute, just go through the motions.
360 -S, --dump-signatures
361 Don't execute, just dump out the signature
362 construction information.
363 -p, --parse-only Quit after parsing the BB recipes.
364 -s, --show-versions Show current and preferred versions of all recipes.
365 -e, --environment Show the global or per-package environment complete
366 with information about where variables were
367 set/changed.
368 -g, --graphviz Save dependency tree information for the specified
369 targets in the dot syntax.
370 -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
371 Assume these dependencies don't exist and are already
372 provided (equivalent to ASSUME_PROVIDED). Useful to
373 make dependency graphs more appealing
374 -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
375 Show debug logging for the specified logging domains
376 -P, --profile Profile the command and save reports.
377 -u UI, --ui=UI The user interface to use (e.g. knotty, hob, depexp).
378 -t SERVERTYPE, --servertype=SERVERTYPE
379 Choose which server to use, process or xmlrpc.
380 --revisions-changed Set the exit code depending on whether upstream
381 floating revisions have changed or not.
382 --server-only Run bitbake without a UI, only starting a server
383 (cooker) process.
384 -B BIND, --bind=BIND The name/address for the bitbake server to bind to.
385 --no-setscene Do not run any setscene tasks. sstate will be ignored
386 and everything needed, built.
387 --remote-server=REMOTE_SERVER
388 Connect to the specified server.
389 -m, --kill-server Terminate the remote server.
390 --observe-only Connect to a server as an observing-only client.
391 </screen>
392 </section>
393
394 <section id='ref-bitbake-fetchers'>
395 <title>Fetchers</title>
396
397 <para>
398 BitBake also contains a set of "fetcher" modules that allow
399 retrieval of source code from various types of sources.
400 For example, BitBake can get source code from a disk with the metadata, from websites,
401 from remote shell accounts, or from Source Code Management (SCM) systems
402 like <filename>cvs/subversion/git</filename>.
403 </para>
404
405 <para>
406 Fetchers are usually triggered by entries in
407 <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>.
408 You can find information about the options and formats of entries for specific
409 fetchers in the BitBake manual located in the
410 <filename>bitbake/doc/manual</filename> directory of the
411 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
412 </para>
413
414 <para>
415 One useful feature for certain Source Code Manager (SCM) fetchers is the ability to
416 "auto-update" when the upstream SCM changes version.
417 Since this ability requires certain functionality from the SCM, not all
418 systems support it.
419 Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update".
420 This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename>
421 variable.
422 See the
423 "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>" section
424 in the Yocto Project Development Manual for more information.
425 </para>
426
427 </section>
428
429</chapter>
430<!--
431vim: expandtab tw=80 ts=4 spell spelllang=en_gb
432-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-29 03:20:26 +0000