From 882e9cd2affb773eec8b1d387ab4e3b5cbdc0994 Mon Sep 17 00:00:00 2001 From: Richard Purdie Date: Tue, 26 Feb 2008 11:31:34 +0000 Subject: Add Poky handbook git-svn-id: https://svn.o-hand.com/repos/poky/trunk@3865 311d38ba-8fff-0310-9ca6-ca027cbcb966 --- handbook/ref-bitbake.xml | 340 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 340 insertions(+) create mode 100644 handbook/ref-bitbake.xml (limited to 'handbook/ref-bitbake.xml') diff --git a/handbook/ref-bitbake.xml b/handbook/ref-bitbake.xml new file mode 100644 index 0000000000..8652424466 --- /dev/null +++ b/handbook/ref-bitbake.xml @@ -0,0 +1,340 @@ + + + + + Reference: Bitbake + + + Bitbake a program written in Python which interprets the metadata + that makes up Poky. At some point, people wonder what actually happens + when you type bitbake poky-image-sato. This section + aims to give an overview of what happens behind the scenes from a + BitBake perspective. + + + + It is worth noting that bitbake aims to be a generic "task" executor + capable of handling complex dependency relationships. As such it has no + real knowledge of what the tasks its executing actually do. It just + considers a list of tasks with dependencies and handles metadata + consisting of variables in a certain format which get passed to the + tasks. + + +
+ Parsing + + + The first thing BitBake does is work out its configuration by + looking for a file called bitbake.conf. + Bitbake searches through the BBPATH environment + variable looking for a conf/ + directory containing a bitbake.conf file and + adds the first bitbake.conf file found in + BBPATH (similar to the PATH environment variable). + For Poky, bitbake.conf is found in meta/conf/. + + + + In Poky, bitbake.conf lists other configuration + files to include from a conf/ + directory below the directories listed in BBPATH. + In general the most important configuration file from a user's perspective + is local.conf, which contains a users customized + settings for Poky. Other notable configuration files are the distribution + configuration file (set by the + DISTRO variable) and the machine configuration file + (set by the MACHINE + variable). The + DISTRO and + MACHINE environment variables are both usually set in + the local.conf file. Valid distribution + configuration files are available in the + meta/conf/distro/ directory and valid machine configuration + files in the meta/conf/machine/ + directory. Within the + meta/conf/machine/include/ directory are various + tune-*.inc configuration files which provide common + "tuning" settings specific to and shared between particular + architectures and machines. + + + + After the parsing of the configuration files some standard classes + are included. In particular, base.bbclass is + always included, as will any other classes + specified in the configuration using the INHERIT + variable. Class files are searched for in a classes subdirectory + under the paths in BBPATH in the same way as + configuration files. + + + + After the parsing of the configuration files is complete, the + variable BBFILES + is set, usually in + local.conf, and defines the list of places to search for + .bb files. By + default this specifies the meta/packages/ + directory within Poky, but other directories such as + meta-extras/ can be included + too. If multiple directories are specified a system referred to as + "collections" is used to + determine which files have priority. + + + + Bitbake parses each .bb file in + BBFILES and + stores the values of various variables. In summary, for each + .bb + file the configuration + base class of variables are set, followed + by the data in the .bb file + itself, followed by any inherit commands that + .bb file might contain. + + + + Parsing .bb files is a time + consuming process, so a cache is kept to speed up subsequent parsing. + This cache is invalid if the timestamp of the .bb + file itself has changed, or if the timestamps of any of the include, + configuration or class files the .bb + file depends on have changed. + +
+ +
+ Preferences and Providers + + + Once all the .bb files have been + parsed, BitBake will proceed to build "poky-image-sato" (or whatever was + specified on the commandline) and looks for providers of that target. + Once a provider is selected, BitBake resolves all the dependencies for + the target. In the case of "poky-image-sato", it would lead to + task-oh.bb and task-base.bb + which in turn would lead to packages like Contacts, + Dates, BusyBox + and these in turn depend on glibc and the toolchain. + + + + Sometimes a target might have multiple providers and a common example + is "virtual/kernel" that is provided by each kernel package. Each machine + will often elect the best provider of its kernel with a line like the + following in the machine configuration file: + + PREFERRED_PROVIDER_virtual/kernel = "linux-rp" + + The default + PREFERRED_PROVIDER is the provider with the same name as + the target. + + + + Understanding how providers are chosen is complicated by the fact + multiple versions might be present. Bitbake defaults to the highest + version of a provider by default. Version comparisons are made using + the same method as Debian. The PREFERRED_VERSION + variable can be used to specify a particular version + (usually in the distro configuration) but the order can + also be influenced by the DEFAULT_PREFERENCE + variable. By default files + have a preference of "0". Setting the + DEFAULT_PREFERENCE to "-1" will + make a package unlikely to be used unless it was explicitly referenced and + "1" makes it likely the package will be used. + PREFERRED_VERSION overrides + any default preference. DEFAULT_PREFERENCE + is often used to mark more + experimental new versions of packages until they've undergone sufficient + testing to be considered stable. + + + + The end result is that internally, BitBake has now built a list of + providers for each target it needs in order of priority. + +
+ +
+ Dependencies + + + Each target BitBake builds consists of multiple tasks (e.g. fetch, + unpack, patch, configure, compile etc.). For best performance on + multi-core systems, BitBake considers each task as an independent + entity with a set of dependencies. There are many variables that + are used to signify these dependencies and more information can be found + found about these in the + BitBake manual. At a basic level it is sufficient to know + that BitBake uses the DEPENDS and + RDEPENDS variables when + calculating dependencies and descriptions of these variables are + available through the links. + + +
+ +
+ The Task List + + + Based on the generated list of providers and the dependency information, + BitBake can now calculate exactly which tasks it needs to run and in what + order. The build now starts with BitBake forking off threads up to + the limit set in the BB_NUMBER_THREADS variable + as long there are tasks ready to run, i.e. tasks with all their + dependencies met. + + + + As each task completes, a timestamp is written to the directory + specified by the STAMPS variable (usually + build/tmp/stamps/*/). On + subsequent runs, BitBake looks at the STAMPS + directory and will not rerun + tasks its already completed unless a timestamp is found to be invalid. + Currently, invalid timestamps are only considered on a per .bb file basis so if for example the configure stamp has a timestamp greater than the + compile timestamp for a given target the compile task would rerun but this + has no effect on other providers depending on that target. This could + change or become configurable in future versions of BitBake. Some tasks + are marked as "nostamp" tasks which means no timestamp file will be written + and the task will always rerun. + + + Once all the tasks have been completed BitBake exits. + +
+ +
+ Running a Task + + + It's worth noting what BitBake does to run a task. A task can either + be a shell task or a python task. For shell tasks, BitBake writes a + shell script to ${WORKDIR}/temp/run.do_taskname.pid + and then executes the script. The generated + shell script contains all the exported variables, and the shell functions + with all variables expanded. Output from the shell script is + sent to the file ${WORKDIR}/temp/log.do_taskname.pid. + Looking at the + expanded shell functions in the run file and the output in the log files + is a useful debugging technique. + + + + Python functions are executed internally to BitBake itself and + logging goes to the controlling terminal. Future versions of BitBake will + write the functions to files in a similar way to shell functions and + logging will also go to the log files in a similar way. + +
+ + +
+ Commandline + + + To quote from "bitbake --help": + + + Usage: bitbake [options] [package ...] + +Executes the specified task (default is 'build') for a given set of BitBake files. +It expects that BBFILES is defined, which is a space separated list of files to +be executed. BBFILES does support wildcards. +Default BBFILES are the .bb files in the current directory. + +Options: + --version show program's version number and exit + -h, --help show this help message and exit + -b BUILDFILE, --buildfile=BUILDFILE + execute the task against this .bb file, rather than a + package from BBFILES. + -k, --continue continue as much as possible after an error. While the + target that failed, and those that depend on it, + cannot be remade, the other dependencies of these + targets can be processed all the same. + -f, --force force run of specified cmd, regardless of stamp status + -i, --interactive drop into the interactive mode also called the BitBake + shell. + -c CMD, --cmd=CMD Specify task to execute. Note that this only executes + the specified task for the providee and the packages + it depends on, i.e. 'compile' does not implicitly call + stage for the dependencies (IOW: use only if you know + what you are doing). Depending on the base.bbclass a + listtasks tasks is defined and will show available + tasks + -r FILE, --read=FILE read the specified file before bitbake.conf + -v, --verbose output more chit-chat to the terminal + -D, --debug Increase the debug level. You can specify this more + than once. + -n, --dry-run don't execute, just go through the motions + -p, --parse-only quit after parsing the BB files (developers only) + -d, --disable-psyco disable using the psyco just-in-time compiler (not + recommended) + -s, --show-versions show current and preferred versions of all packages + -e, --environment show the global or per-package environment (this is + what used to be bbread) + -g, --graphviz emit the dependency trees of the specified packages in + the dot syntax + -I IGNORED_DOT_DEPS, --ignore-deps=IGNORED_DOT_DEPS + Stop processing at the given list of dependencies when + generating dependency graphs. This can help to make + the graph more appealing + -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS + Show debug logging for the specified logging domains + -P, --profile profile the command and print a report + +
+ +
+ Fetchers + + + As well as the containing the parsing and task/dependency handling + code, bitbake also contains a set of "fetcher" modules which allow + fetching of source code from various types of sources. Example + sources might be from disk with the metadata, from websites, from + remote shell accounts or from SCM systems like cvs/subversion/git. + + + + The fetchers are usually triggered by entries in + SRC_URI. Information about the + options and formats of entries for specific fetchers can be found in the + BitBake manual. + + + + One useful feature for certain SCM fetchers is the ability to + "auto-update" when the upstream SCM changes version. Since this + requires certain functionality from the SCM only certain systems + support it, currently Subversion, Bazaar and to a limited extent, Git. It + works using the SRCREV + variable. See the + developing with an external SCM based project section for more + information. + + +
+ + + -- cgit v1.2.3-54-g00ecf