handbook: Move into documentation directory

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>

1 files changed, 348 insertions, 0 deletions

diff --git a/documentation/poky-ref-manual/ref-bitbake.xml b/documentation/poky-ref-manual/ref-bitbake.xml
new file mode 100644
index 0000000000..eaf9467950
--- /dev/null
+++ b/documentation/poky-ref-manual/ref-bitbake.xml
@@ -0,0 +1,348 @@
+<!DOCTYPE appendix PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<appendix id='ref-bitbake'>
+
+    <title>Reference: Bitbake</title>
+
+    <para>
+        Bitbake is a program written in Python that interprets the metadata
+        that makes up Poky. At some point, people wonder what actually happens 
+        when you type <command>bitbake poky-image-sato</command>. This section 
+        aims to give an overview of what happens behind the scenes from a
+        BitBake perspective.
+    </para>
+
+    <para>
+        It is worth noting that bitbake aims to be a generic "task" executor 
+        capable of handling complex dependency relationships. As such it has no 
+        real knowledge of what the tasks it is executing actually do. It just 
+        considers a list of tasks with dependencies and handles metadata 
+        consisting of variables in a certain format which get passed to the 
+        tasks.
+    </para>
+
+    <section id='ref-bitbake-parsing'>
+        <title>Parsing</title>
+
+        <para>
+            The first thing BitBake does is that work out its configuration by 
+            looking for a file called <filename>bitbake.conf</filename>.
+            Bitbake searches through the <varname>BBPATH</varname> environment 
+            variable looking for a <filename class="directory">conf/</filename> 
+            directory containing a <filename>bitbake.conf</filename> file and 
+            adds the first <filename>bitbake.conf</filename> file found in
+            <varname>BBPATH</varname> (similar to the PATH environment variable). 
+            For Poky, <filename>bitbake.conf</filename> is found in <filename 
+            class="directory">meta/conf/</filename>. 
+        </para>
+
+        <para>
+            In Poky, <filename>bitbake.conf</filename> lists other configuration 
+            files to include from a <filename class="directory">conf/</filename> 
+            directory below the directories listed in <varname>BBPATH</varname>. 
+            In general the most important configuration file from a user's perspective 
+            is <filename>local.conf</filename>, which contains a users customized 
+            settings for Poky. Other notable configuration files are the distribution 
+            configuration file (set by the <glossterm><link linkend='var-DISTRO'>
+            DISTRO</link></glossterm> variable) and the machine configuration file 
+            (set by the <glossterm><link linkend='var-MACHINE'>MACHINE</link>
+            </glossterm> variable).  The <glossterm><link linkend='var-DISTRO'>
+            DISTRO</link></glossterm> and <glossterm><link linkend='var-MACHINE'>
+            MACHINE</link></glossterm> environment variables are both usually set in 
+            the <filename>local.conf</filename> file. Valid distribution 
+            configuration files are available in the <filename class="directory">
+            meta/conf/distro/</filename> directory and valid machine configuration 
+            files in the <filename class="directory">meta/conf/machine/</filename> 
+            directory. Within the <filename class="directory">
+            meta/conf/machine/include/</filename> directory are various <filename>
+            tune-*.inc</filename> configuration files which provide common 
+            "tuning" settings specific to and shared between particular 
+            architectures and machines.
+        </para>
+
+        <para>
+            After the parsing of the configuration files some standard classes 
+            are included. In particular, <filename>base.bbclass</filename> is 
+            always included, as will any other classes 
+            specified in the configuration using the <glossterm><link
+                    linkend='var-INHERIT'>INHERIT</link></glossterm>
+            variable. Class files are searched for in a classes subdirectory 
+            under the paths in <varname>BBPATH</varname> in the same way as
+            configuration files.
+        </para>
+
+        <para>
+            After the parsing of the configuration files is complete, the 
+            variable <glossterm><link linkend='var-BBFILES'>BBFILES</link></glossterm> 
+            is set, usually in
+            <filename>local.conf</filename>, and defines the list of places to search for 
+            <filename class="extension">.bb</filename> files. By 
+            default this specifies the <filename class="directory">meta/packages/
+            </filename> directory within Poky, but other directories such as
+            <filename class="directory">meta-extras/</filename> can be included 
+            too. Adding extra content to
+            <glossterm><link linkend='var-BBFILES'>BBFILES</link></glossterm> is best
+            acheived through the use of Bitbake
+            <link linkend='usingpoky-changes-layers'>"layers"</link>.
+        </para>
+
+        <para>
+            Bitbake parses each <filename class="extension">.bb</filename> file in 
+            <glossterm><link linkend='var-BBFILES'>BBFILES</link></glossterm> and 
+            stores the values of various variables.  In summary, for each 
+            <filename class="extension">.bb</filename> 
+            file the configuration + base class of variables are set, followed 
+            by the data in the <filename class="extension">.bb</filename> file 
+            itself, followed by any inherit commands that
+            <filename class="extension">.bb</filename> file might contain.
+        </para>
+
+        <para>
+            Parsing <filename class="extension">.bb</filename> files is a time 
+            consuming process, so a cache is kept to speed up subsequent parsing. 
+            This cache is invalid if the timestamp of the <filename class="extension">.bb</filename> 
+            file itself has changed, or if the timestamps of any of the include, 
+            configuration or class files the <filename class="extension">.bb</filename>
+            file depends on have changed.
+        </para>
+    </section>
+
+    <section id='ref-bitbake-providers'>
+        <title>Preferences and Providers</title>
+
+        <para>
+            Once all the <filename class="extension">.bb</filename> files have been 
+            parsed, BitBake will proceed to build "poky-image-sato" (or whatever was
+            specified on the commandline) and looks for providers of that target.
+            Once a provider is selected, BitBake resolves all the dependencies for  
+            the target. In the case of "poky-image-sato", it would lead to 
+            <filename>task-base.bb</filename>  
+            which in turn would lead to packages like <application>Contacts</application>, 
+            <application>Dates</application>, <application>BusyBox</application>
+            and these in turn depend on glibc and the toolchain.
+        </para>
+
+        <para>
+            Sometimes a target might have multiple providers and a common example 
+            is "virtual/kernel" that is provided by each kernel package. Each machine
+            will often elect the best provider of its kernel with a line like the 
+            following in the machine configuration file:
+        </para>
+        <programlisting><glossterm><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></glossterm>_virtual/kernel = "linux-rp"</programlisting>
+        <para>
+            The default <glossterm><link linkend='var-PREFERRED_PROVIDER'>
+            PREFERRED_PROVIDER</link></glossterm> is the provider with the same name as
+            the target.
+        </para>
+
+        <para>
+            Understanding how providers are chosen is complicated by the fact
+            multiple versions might be present. Bitbake defaults to the highest 
+            version of a provider by default. Version comparisons are made using 
+            the same method as Debian. The <glossterm><link 
+            linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></glossterm>
+            variable can be used to specify a particular version 
+            (usually in the distro configuration) but the order can 
+            also be influenced by the <glossterm><link
+                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm> 
+            variable. By default files 
+            have a preference of "0". Setting the
+            <glossterm><link
+                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm> to "-1" will 
+            make a package unlikely to be used unless it was explicitly referenced and
+            "1" makes it likely the package will be used. 
+            <glossterm><link
+                    linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></glossterm> overrides 
+            any <glossterm><link
+                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm>. <glossterm><link
+                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm> 
+            is often used to mark more 
+            experimental new versions of packages until they've undergone sufficient 
+            testing to be considered stable.
+        </para>
+
+        <para>
+            The end result is that internally, BitBake has now built a list of 
+            providers for each target it needs in order of priority.
+        </para>
+    </section>
+
+    <section id='ref-bitbake-dependencies'>
+        <title>Dependencies</title>
+
+        <para>
+            Each target BitBake builds consists of multiple tasks (e.g. fetch, 
+            unpack, patch, configure, compile etc.). For best performance on 
+            multi-core systems, BitBake considers each task as an independent 
+            entity with a set of dependencies. There are many variables that 
+            are used to signify these dependencies and more information can be found 
+            about these in the <ulink url='http://bitbake.berlios.de/manual/'>
+            BitBake manual</ulink>. At a basic level it is sufficient to know 
+        that BitBake uses the <glossterm><link
+                linkend='var-DEPENDS'>DEPENDS</link></glossterm> and 
+        <glossterm><link linkend='var-RDEPENDS'>RDEPENDS</link></glossterm> variables when 
+            calculating dependencies and descriptions of these variables are 
+            available through the links. 
+        </para>
+
+    </section>
+
+    <section id='ref-bitbake-tasklist'>
+        <title>The Task List</title>
+
+        <para>
+            Based on the generated list of providers and the dependency information, 
+            BitBake can now calculate exactly which tasks it needs to run and in what 
+            order. The build now starts with BitBake forking off threads up to
+            the limit set in the <glossterm><link
+                    linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></glossterm> variable
+            as long as there are tasks ready to run, i.e. tasks with all their
+            dependencies met.
+        </para>
+
+        <para>
+            As each task completes, a timestamp is written to the directory 
+            specified by the <glossterm><link
+                    linkend='var-STAMPS'>STAMPS</link></glossterm> variable (usually 
+            <filename class="directory">build/tmp/stamps/*/</filename>). On 
+            subsequent runs, BitBake looks at the <glossterm><link
+                    linkend='var-STAMPS'>STAMPS</link></glossterm>
+            directory and will not rerun 
+            tasks its already completed unless a timestamp is found to be invalid. 
+            Currently, invalid timestamps are only considered on a per <filename 
+            class="extension">.bb</filename> file basis so if for example the configure stamp has a timestamp greater than the 
+            compile timestamp for a given target the compile task would rerun but this 
+            has no effect on other providers depending on that target. This could 
+            change or become configurable in future versions of BitBake. Some tasks 
+            are marked as "nostamp" tasks which means no timestamp file will be written 
+            and the task will always rerun.
+        </para>
+
+        <para>Once all the tasks have been completed BitBake exits.</para> 
+
+    </section>
+
+    <section id='ref-bitbake-runtask'>
+        <title>Running a Task</title>
+
+        <para>
+            It's worth noting what BitBake does to run a task. A task can either
+            be a shell task or a python task. For shell tasks, BitBake writes a
+            shell script to <filename>${WORKDIR}/temp/run.do_taskname.pid</filename>
+            and then executes the script. The generated
+            shell script contains all the exported variables, and the shell functions 
+            with all variables expanded. Output from the shell script is 
+            sent to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
+            Looking at the
+            expanded shell functions in the run file and the output in the log files 
+            is a useful debugging technique.
+        </para>
+
+        <para>
+            Python functions are executed internally to BitBake itself and 
+            logging goes to the controlling terminal. Future versions of BitBake will 
+            write the functions to files in a similar way to shell functions and 
+            logging will also go to the log files in a similar way.
+        </para>
+    </section>
+
+
+    <section id='ref-bitbake-commandline'>
+        <title>Commandline</title>
+
+        <para>
+            To quote from "bitbake --help":
+        </para>
+
+        <screen>Usage: bitbake [options] [package ...]
+
+Executes the specified task (default is 'build') for a given set of BitBake files.
+It expects that BBFILES is defined, which is a space separated list of files to
+be executed.  BBFILES does support wildcards.
+Default BBFILES are the .bb files in the current directory.
+
+Options:
+  --version             show program's version number and exit
+  -h, --help            show this help message and exit
+  -b BUILDFILE, --buildfile=BUILDFILE
+                        execute the task against this .bb file, rather than a
+                        package from BBFILES.
+  -k, --continue        continue as much as possible after an error. While the
+                        target that failed, and those that depend on it,
+                        cannot be remade, the other dependencies of these
+                        targets can be processed all the same.
+  -a, --tryaltconfigs   continue with builds by trying to use alternative
+                        providers where possible.
+  -f, --force           force run of specified cmd, regardless of stamp status
+  -c CMD, --cmd=CMD     Specify task to execute. Note that this only executes
+                        the specified task for the providee and the packages
+                        it depends on, i.e. 'compile' does not implicitly call
+                        stage for the dependencies (IOW: use only if you know
+                        what you are doing). Depending on the base.bbclass a
+                        listtasks tasks is defined and will show available
+                        tasks
+  -r FILE, --read=FILE  read the specified file before bitbake.conf
+  -v, --verbose         output more chit-chat to the terminal
+  -D, --debug           Increase the debug level. You can specify this more
+                        than once.
+  -n, --dry-run         don't execute, just go through the motions
+  -S, --dump-signatures
+                        don't execute, just dump out the signature
+                        construction information
+  -p, --parse-only      quit after parsing the BB files (developers only)
+  -d, --disable-psyco   disable using the psyco just-in-time compiler (not
+                        recommended)
+  -s, --show-versions   show current and preferred versions of all packages
+  -e, --environment     show the global or per-package environment (this is
+                        what used to be bbread)
+  -g, --graphviz        emit the dependency trees of the specified packages in
+                        the dot syntax
+  -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
+                        Assume these dependencies don't exist and are already
+                        provided (equivalent to ASSUME_PROVIDED). Useful to
+                        make dependency graphs more appealing
+  -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
+                        Show debug logging for the specified logging domains
+  -P, --profile         profile the command and print a report
+  -u UI, --ui=UI        userinterface to use
+  --revisions-changed   Set the exit code depending on whether upstream
+                        floating revisions have changed or not</screen>
+
+    </section>
+
+    <section id='ref-bitbake-fetchers'>
+        <title>Fetchers</title>
+
+        <para>
+            As well as the containing the parsing and task/dependency handling 
+            code, bitbake also contains a set of "fetcher" modules which allow 
+            fetching of source code from various types of sources. Example 
+            sources might be from disk with the metadata, from websites, from 
+            remote shell accounts or from SCM systems like cvs/subversion/git. 
+        </para>
+
+        <para>
+            The fetchers are usually triggered by entries in 
+            <glossterm><link linkend='var-SRC_URI'>SRC_URI</link></glossterm>. Information about the 
+            options and formats of entries for specific fetchers can be found in the 
+            <ulink url='http://bitbake.berlios.de/manual/'>BitBake manual</ulink>.
+        </para>
+
+        <para>
+            One useful feature for certain SCM fetchers is the ability to 
+            "auto-update" when the upstream SCM changes version. Since this 
+            requires certain functionality from the SCM only certain systems 
+            support it, currently Subversion, Bazaar and to a limited extent, Git. It 
+            works using the <glossterm><link linkend='var-SRCREV'>SRCREV</link>
+            </glossterm> variable. See the <link linkend='platdev-appdev-srcrev'>
+            developing with an external SCM based project</link> section for more
+            information.
+        </para>
+
+    </section>
+
+</appendix>
+<!-- 
+vim: expandtab tw=80 ts=4 spell spelllang=en_gb
+-->