summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual
diff options
context:
space:
mode:
authorScott Rifenbark <srifenbark@gmail.com>2018-04-05 09:55:23 -0700
committerRichard Purdie <richard.purdie@linuxfoundation.org>2018-05-24 17:16:07 +0100
commit35734315dde80207aaaee3cf5081dcfbc2b4bf2c (patch)
tree83288958be063f46befb7fe12efe72925856528b /documentation/ref-manual
parent21c0019f446e75007f177aad0d1151601fbf28b6 (diff)
downloadpoky-35734315dde80207aaaee3cf5081dcfbc2b4bf2c.tar.gz
ref-manual: Deleted the "ref-bitbake.xml" chapter
This information was merged into the BitBake User Manual. (From yocto-docs rev: eb68d4429aed652e4ca10c1ab55d3a815d453d6f) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/ref-manual')
-rw-r--r--documentation/ref-manual/ref-bitbake.xml479
1 files changed, 0 insertions, 479 deletions
diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml
deleted file mode 100644
index 9e3e9cf35d..0000000000
--- a/documentation/ref-manual/ref-bitbake.xml
+++ /dev/null
@@ -1,479 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='ref-bitbake'>
6
7 <title>BitBake</title>
8
9 <para>
10 BitBake is a program written in Python that interprets the
11 <link linkend='metadata'>Metadata</link> used by
12 the OpenEmbedded build system.
13 At some point, developers wonder what actually happens when you enter:
14 <literallayout class='monospaced'>
15 $ bitbake core-image-sato
16 </literallayout>
17 </para>
18
19 <para>
20 This chapter provides an overview of what happens behind the scenes from BitBake's perspective.
21 </para>
22
23 <note>
24 BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships.
25 As such, it has no real knowledge of what the tasks being executed actually do.
26 BitBake just considers a list of tasks with dependencies and handles
27 <link linkend='metadata'>Metadata</link>
28 consisting of variables in a certain format that get passed to the tasks.
29 </note>
30
31 <section id='ref-bitbake-parsing'>
32 <title>Parsing</title>
33
34 <para>
35 BitBake parses configuration files, classes, and <filename>.bb</filename> files.
36 </para>
37
38 <para>
39 The first thing BitBake does is look for the
40 <filename>bitbake.conf</filename> file.
41 This file resides in the
42 <link linkend='source-directory'>Source Directory</link>
43 within the <filename>meta/conf/</filename> directory.
44 BitBake finds it by examining its
45 <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment
46 variable and looking for the <filename>meta/conf/</filename>
47 directory.
48 </para>
49
50 <para>
51 The <filename>bitbake.conf</filename> file lists other configuration
52 files to include from a <filename>conf/</filename>
53 directory below the directories listed in <filename>BBPATH</filename>.
54 In general, the most important configuration file from a user's perspective
55 is <filename>local.conf</filename>, which contains a user's customized
56 settings for the OpenEmbedded build environment.
57 Other notable configuration files are the distribution
58 configuration file (set by the
59 <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable)
60 and the machine configuration file
61 (set by the
62 <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable).
63 The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment
64 variables are both usually set in
65 the <filename>local.conf</filename> file.
66 Valid distribution
67 configuration files are available in the <filename>meta/conf/distro/</filename> directory
68 and valid machine configuration
69 files in the <filename>meta/conf/machine/</filename> directory.
70 Within the <filename>meta/conf/machine/include/</filename>
71 directory are various <filename>tune-*.inc</filename> configuration files that provide common
72 "tuning" settings specific to and shared between particular architectures and machines.
73 </para>
74
75 <para>
76 After the parsing of the configuration files, some standard classes are included.
77 The <filename>base.bbclass</filename> file is always included.
78 Other classes that are specified in the configuration using the
79 <filename><link linkend='var-INHERIT'>INHERIT</link></filename>
80 variable are also included.
81 Class files are searched for in a <filename>classes</filename> subdirectory
82 under the paths in <filename>BBPATH</filename> in the same way as
83 configuration files.
84 </para>
85
86 <para>
87 After classes are included, the variable
88 <filename><link linkend='var-BBFILES'>BBFILES</link></filename>
89 is set, usually in
90 <filename>local.conf</filename>, and defines the list of places to search for
91 <filename>.bb</filename> files.
92 By default, the <filename>BBFILES</filename> variable specifies the
93 <filename>meta/recipes-*/</filename> directory within Poky.
94 Adding extra content to <filename>BBFILES</filename> is best achieved through the use of
95 BitBake layers as described in the
96 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
97 section of the Yocto Project Development Tasks Manual.
98 </para>
99
100 <para>
101 BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and
102 stores the values of various variables.
103 In summary, for each <filename>.bb</filename>
104 file the configuration plus the base class of variables are set, followed
105 by the data in the <filename>.bb</filename> file
106 itself, followed by any inherit commands that
107 <filename>.bb</filename> file might contain.
108 </para>
109
110 <para>
111 Because parsing <filename>.bb</filename> files is a time
112 consuming process, a cache is kept to speed up subsequent parsing.
113 This cache is invalid if the timestamp of the <filename>.bb</filename>
114 file itself changes, or if the timestamps of any of the include,
115 configuration files or class files on which the
116 <filename>.bb</filename> file depends change.
117 </para>
118
119 <note>
120 <para>
121 You need to be aware of how BitBake parses curly braces.
122 If a recipe uses a closing curly brace within the function and
123 the character has no leading spaces, BitBake produces a parsing
124 error.
125 If you use a pair of curly brace in a shell function, the
126 closing curly brace must not be located at the start of the line
127 without leading spaces.
128 </para>
129
130 <para>
131 Here is an example that causes BitBake to produce a parsing
132 error:
133 <literallayout class='monospaced'>
134 fakeroot create_shar() {
135 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
136 usage()
137 {
138 echo "test"
139 ###### The following "}" at the start of the line causes a parsing error ######
140 }
141 EOF
142 }
143 </literallayout>
144 Writing the recipe this way avoids the error:
145 <literallayout class='monospaced'>
146 fakeroot create_shar() {
147 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
148 usage()
149 {
150 echo "test"
151 ######The following "}" with a leading space at the start of the line avoids the error ######
152 }
153 EOF
154 }
155 </literallayout>
156 </para>
157 </note>
158 </section>
159
160 <section id='ref-bitbake-providers'>
161 <title>Preferences and Providers</title>
162
163 <para>
164 Once all the <filename>.bb</filename> files have been
165 parsed, BitBake starts to build the target (<filename>core-image-sato</filename>
166 in the previous section's example) and looks for providers of that target.
167 Once a provider is selected, BitBake resolves all the dependencies for
168 the target.
169 In the case of <filename>core-image-sato</filename>, it would lead to
170 <filename>packagegroup-core-x11-sato</filename>,
171 which in turn leads to recipes like <filename>matchbox-terminal</filename>,
172 <filename>pcmanfm</filename> and <filename>gthumb</filename>.
173 These recipes in turn depend on <filename>glibc</filename> and the toolchain.
174 </para>
175
176 <para>
177 Sometimes a target might have multiple providers.
178 A common example is "virtual/kernel", which is provided by each kernel package.
179 Each machine often selects the best kernel provider by using a line similar to the
180 following in the machine configuration file:
181 </para>
182
183 <literallayout class='monospaced'>
184 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
185 </literallayout>
186
187 <para>
188 The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename>
189 is the provider with the same name as the target.
190 </para>
191
192 <para>
193 Understanding how providers are chosen is made complicated by the fact
194 that multiple versions might exist.
195 BitBake defaults to the highest version of a provider.
196 Version comparisons are made using the same method as Debian.
197 You can use the
198 <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
199 variable to specify a particular version (usually in the distro configuration).
200 You can influence the order by using the
201 <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename>
202 variable.
203 By default, files have a preference of "0".
204 Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
205 package unlikely to be used unless it is explicitly referenced.
206 Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
207 <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
208 <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
209 versions until they have undergone sufficient testing to be considered stable.
210 </para>
211
212 <para>
213 In summary, BitBake has created a list of providers, which is prioritized, for each target.
214 </para>
215 </section>
216
217 <section id='ref-bitbake-dependencies'>
218 <title>Dependencies</title>
219
220 <para>
221 Each target BitBake builds consists of multiple tasks such as
222 <filename>fetch</filename>, <filename>unpack</filename>,
223 <filename>patch</filename>, <filename>configure</filename>,
224 and <filename>compile</filename>.
225 For best performance on multi-core systems, BitBake considers each task as an independent
226 entity with its own set of dependencies.
227 </para>
228
229 <para>
230 Dependencies are defined through several variables.
231 You can find information about variables BitBake uses in the
232 BitBake documentation, which is found in the
233 <filename>bitbake/doc/manual</filename> directory within the
234 <link linkend='source-directory'>Source Directory</link>.
235 At a basic level, it is sufficient to know that BitBake uses the
236 <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and
237 <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename>
238 variables when calculating dependencies.
239 </para>
240 </section>
241
242 <section id='ref-bitbake-tasklist'>
243 <title>The Task List</title>
244
245 <para>
246 Based on the generated list of providers and the dependency information,
247 BitBake can now calculate exactly what tasks it needs to run and in what
248 order it needs to run them.
249 The build now starts with BitBake forking off threads up to the limit set in the
250 <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable.
251 BitBake continues to fork threads as long as there are tasks ready to run,
252 those tasks have all their dependencies met, and the thread threshold has not been
253 exceeded.
254 </para>
255
256 <para>
257 It is worth noting that you can greatly speed up the build time by properly setting
258 the <filename>BB_NUMBER_THREADS</filename> variable.
259 See the
260 "<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>"
261 section in the Yocto Project Quick Start for more information.
262 </para>
263
264 <para>
265 As each task completes, a timestamp is written to the directory specified by the
266 <filename><link linkend='var-STAMP'>STAMP</link></filename> variable.
267 On subsequent runs, BitBake looks within the <filename>build/tmp/stamps</filename>
268 directory and does not rerun
269 tasks that are already completed unless a timestamp is found to be invalid.
270 Currently, invalid timestamps are only considered on a per
271 <filename>.bb</filename> file basis.
272 So, for example, if the configure stamp has a timestamp greater than the
273 compile timestamp for a given target, then the compile task would rerun.
274 Running the compile task again, however, has no effect on other providers
275 that depend on that target.
276 This behavior could change or become configurable in future versions of BitBake.
277 </para>
278
279 <note>
280 Some tasks are marked as "nostamp" tasks.
281 No timestamp file is created when these tasks are run.
282 Consequently, "nostamp" tasks are always rerun.
283 </note>
284 </section>
285
286 <section id='ref-bitbake-runtask'>
287 <title>Running a Task</title>
288
289 <para>
290 Tasks can either be a shell task or a Python task.
291 For shell tasks, BitBake writes a shell script to
292 <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script.
293 The generated shell script contains all the exported variables, and the shell functions
294 with all variables expanded.
295 Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
296 Looking at the expanded shell functions in the run file and the output in the log files
297 is a useful debugging technique.
298 </para>
299
300 <para>
301 For Python tasks, BitBake executes the task internally and logs information to the
302 controlling terminal.
303 Future versions of BitBake will write the functions to files similar to the way
304 shell tasks are handled.
305 Logging will be handled in a way similar to shell tasks as well.
306 </para>
307
308 <para>
309 Once all the tasks have been completed BitBake exits.
310 </para>
311
312 <para>
313 When running a task, BitBake tightly controls the execution environment
314 of the build tasks to make sure unwanted contamination from the build machine
315 cannot influence the build.
316 Consequently, if you do want something to get passed into the build
317 task's environment, you must take a few steps:
318 <orderedlist>
319 <listitem><para>Tell BitBake to load what you want from the environment
320 into the data store.
321 You can do so through the <filename>BB_ENV_EXTRAWHITE</filename>
322 variable.
323 For example, assume you want to prevent the build system from
324 accessing your <filename>$HOME/.ccache</filename> directory.
325 The following command tells BitBake to load
326 <filename>CCACHE_DIR</filename> from the environment into the data
327 store:
328 <literallayout class='monospaced'>
329 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR"
330 </literallayout></para></listitem>
331 <listitem><para>Tell BitBake to export what you have loaded into the
332 environment store to the task environment of every running task.
333 Loading something from the environment into the data store
334 (previous step) only makes it available in the datastore.
335 To export it to the task environment of every running task,
336 use a command similar to the following in your
337 <filename>local.conf</filename> or distro configuration file:
338 <literallayout class='monospaced'>
339 export CCACHE_DIR
340 </literallayout></para></listitem>
341 </orderedlist>
342 </para>
343
344 <note>
345 A side effect of the previous steps is that BitBake records the variable
346 as a dependency of the build process in things like the shared state
347 checksums.
348 If doing so results in unnecessary rebuilds of tasks, you can whitelist the
349 variable so that the shared state code ignores the dependency when it creates
350 checksums.
351 For information on this process, see the
352 <filename>BB_HASHBASE_WHITELIST</filename> example in the
353 "<ulink url='&YOCTO_DOCS_CM_URL;#overview-checksums'>Checksums (Signatures)</ulink>"
354 section in the Yocto Project Concepts Manual.
355 </note>
356 </section>
357
358 <section id='ref-bitbake-commandline'>
359 <title>BitBake Command Line</title>
360
361 <para>
362 Following is the BitBake help output:
363 </para>
364
365 <screen>
366$ bitbake --help
367Usage: bitbake [options] [recipename/target ...]
368
369 Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
370 It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
371 will provide the layer, BBFILES and other configuration information.
372
373Options:
374 --version show program's version number and exit
375 -h, --help show this help message and exit
376 -b BUILDFILE, --buildfile=BUILDFILE
377 Execute tasks from a specific .bb recipe directly.
378 WARNING: Does not handle any dependencies from other
379 recipes.
380 -k, --continue Continue as much as possible after an error. While the
381 target that failed and anything depending on it cannot
382 be built, as much as possible will be built before
383 stopping.
384 -a, --tryaltconfigs Continue with builds by trying to use alternative
385 providers where possible.
386 -f, --force Force the specified targets/task to run (invalidating
387 any existing stamp file).
388 -c CMD, --cmd=CMD Specify the task to execute. The exact options
389 available depend on the metadata. Some examples might
390 be 'compile' or 'populate_sysroot' or 'listtasks' may
391 give a list of the tasks available.
392 -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
393 Invalidate the stamp for the specified task such as
394 'compile' and then run the default task for the
395 specified target(s).
396 -r PREFILE, --read=PREFILE
397 Read the specified file before bitbake.conf.
398 -R POSTFILE, --postread=POSTFILE
399 Read the specified file after bitbake.conf.
400 -v, --verbose Output more log message data to the terminal.
401 -D, --debug Increase the debug level. You can specify this more
402 than once.
403 -n, --dry-run Don't execute, just go through the motions.
404 -S, --dump-signatures
405 Don't execute, just dump out the signature
406 construction information.
407 -p, --parse-only Quit after parsing the BB recipes.
408 -s, --show-versions Show current and preferred versions of all recipes.
409 -e, --environment Show the global or per-package environment complete
410 with information about where variables were
411 set/changed.
412 -g, --graphviz Save dependency tree information for the specified
413 targets in the dot syntax.
414 -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
415 Assume these dependencies don't exist and are already
416 provided (equivalent to ASSUME_PROVIDED). Useful to
417 make dependency graphs more appealing
418 -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
419 Show debug logging for the specified logging domains
420 -P, --profile Profile the command and save reports.
421 -u UI, --ui=UI The user interface to use (e.g. knotty and taskexp).
422 -t SERVERTYPE, --servertype=SERVERTYPE
423 Choose which server to use, process or xmlrpc.
424 --revisions-changed Set the exit code depending on whether upstream
425 floating revisions have changed or not.
426 --server-only Run bitbake without a UI, only starting a server
427 (cooker) process.
428 -B BIND, --bind=BIND The name/address for the bitbake server to bind to.
429 --no-setscene Do not run any setscene tasks. sstate will be ignored
430 and everything needed, built.
431 --remote-server=REMOTE_SERVER
432 Connect to the specified server.
433 -m, --kill-server Terminate the remote server.
434 --observe-only Connect to a server as an observing-only client.
435 </screen>
436 </section>
437
438 <section id='ref-bitbake-fetchers'>
439 <title>Fetchers</title>
440
441 <para>
442 BitBake also contains a set of "fetcher" modules that allow
443 retrieval of source code from various types of sources.
444 For example, BitBake can get source code from a disk with the metadata, from websites,
445 from remote shell accounts, or from Source Code Management (SCM) systems
446 like <filename>cvs/subversion/git</filename>.
447 </para>
448
449 <para>
450 Fetchers are usually triggered by entries in
451 <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>.
452 You can find information about the options and formats of entries for specific
453 fetchers in the BitBake manual located in the
454 <filename>bitbake/doc/manual</filename> directory of the
455 <link linkend='source-directory'>Source Directory</link>.
456 </para>
457
458 <para>
459 One useful feature for certain Source Code Manager (SCM) fetchers
460 is the ability to "auto-update" when the upstream SCM changes
461 version.
462 Since this ability requires certain functionality from the SCM,
463 not all systems support it.
464 Currently Subversion, Bazaar and to a limited extent, Git support
465 the ability to "auto-update".
466 This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename>
467 variable.
468 See the
469 "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>"
470 section in the Yocto Project Development Tasks Manual for more
471 information.
472 </para>
473
474 </section>
475
476</chapter>
477<!--
478vim: expandtab tw=80 ts=4 spell spelllang=en_gb
479-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-05 02:12:42 +0000