bitbake: user-manual-fetching.xml: Re-write of "File Download Support" chapter.

Basic re-write to clean up text and flow.

(Bitbake rev: 0f82dc4c22ce015bef87fd5aae0b6fa3e429016e)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

1 files changed, 271 insertions, 161 deletions

diff --git a/bitbake/doc/user-manual/user-manual-fetching.xml b/bitbake/doc/user-manual/user-manual-fetching.xml
index cdca64c801..499d4f5198 100644
--- a/bitbake/doc/user-manual/user-manual-fetching.xml
+++ b/bitbake/doc/user-manual/user-manual-fetching.xml
@@ -2,84 +2,110 @@
 "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
 
 <chapter>
-<title>File download support</title>
+<title>File Download Support</title>
+
+    <para>
+        BitBake's <filename>fetch</filename> and
+        <filename>fetch2</filename> modules support downloading
+        files.
+        This chapter provides an overview of the fetching process
+        and also presents sections on each of the fetchers BitBake
+        supports.
+        <note>
+            The original <filename>fetch</filename> code, for all
+            practical purposes, has been replaced by
+            <filename>fetch2</filename> code.
+            Consequently, the information in this chapter does not
+            apply to <filename>fetch</filename>.
+        </note>
+    </para>
 
     <section id='file-download-overview'>
         <title>Overview</title>
 
         <para>
-            BitBake provides support to download files.
-            This procedure is called fetching and is handled by the
-            fetch and fetch2 modules.
-            At this point, the original fetch code is considered to
-            be replaced by fetch2 and this manual is only related
-            to the fetch2 codebase.
+            When BitBake starts to execute, the very first thing
+            it does is to fetch the source files needed.
+            This section overviews the process.
+            For some additional information on fetching source files, see the
+            "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
+            section.
         </para>
 
         <para>
-            The <filename>SRC_URI</filename> is normally used to
-            tell BitBake which files to fetch.
-            The next sections describe the available fetchers and
-            their options.
-            Each fetcher honors a set of variables and per
-            URI parameters separated by a “;” consisting of a key and
-            a value.
-            The semantics of the variables and parameters are
-            defined by the fetcher.
-            BitBake tries to have consistent semantics between the
-            different fetchers.
+            When BitBake goes looking for source files, it follows a search
+            order:
+            <orderedlist>
+                <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
+                    BitBake first uses pre-mirrors to try and find source
+                    files.
+                    These locations are defined using the
+                    <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
+                    variable.
+                    </para></listitem>
+                <listitem><para><emphasis>Source URI:</emphasis>
+                    If pre-mirrors fail, BitBake uses
+                    <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
+                    </para></listitem>
+                <listitem><para><emphasis>Mirror Sites:</emphasis>
+                    If fetch failures occur using <filename>SRC_URI</filename>,
+                    BitBake next uses mirror location as defined by the
+                    <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
+                    variable.
+                    </para></listitem>
+            </orderedlist>
         </para>
 
         <para>
-            The overall fetch process first attempts to fetch from
-            <filename>PREMIRRORS</filename>.
-            If these fail, the original <filename>SRC_URI</filename>
-            is attempted.
-            If that fails, BitBake falls back to
-            <filename>MIRRORS</filename>.
             Because cross-URLs are supported, it is possible to mirror
             a Git repository on an HTTP server as a tarball.
             Here are some examples that show commonly used mirror
             definitions:
             <literallayout class='monospaced'>
-PREMIRRORS ?= "\
-    bzr://.*/.*   http://somemirror.org/sources/ \n \
-    cvs://.*/.*   http://somemirror.org/sources/ \n \
-    git://.*/.*   http://somemirror.org/sources/ \n \
-    hg://.*/.*    http://somemirror.org/sources/ \n \
-    osc://.*/.*   http://somemirror.org/sources/ \n \
-    p4://.*/.*    http://somemirror.org/sources/ \n \
-    svk://.*/.*   http://somemirror.org/sources/ \n \
-    svn://.*/.*   http://somemirror.org/sources/ \n"
-
-MIRRORS =+ "\
-    ftp://.*/.*      http://somemirror.org/sources/ \n \
-    http://.*/.*     http://somemirror.org/sources/ \n \
-    https://.*/.*    http://somemirror.org/sources/ \n"
+     PREMIRRORS ?= "\
+         bzr://.*/.*   http://somemirror.org/sources/ \n \
+         cvs://.*/.*   http://somemirror.org/sources/ \n \
+         git://.*/.*   http://somemirror.org/sources/ \n \
+         hg://.*/.*    http://somemirror.org/sources/ \n \
+         osc://.*/.*   http://somemirror.org/sources/ \n \
+         p4://.*/.*    http://somemirror.org/sources/ \n \
+         svk://.*/.*   http://somemirror.org/sources/ \n \
+         svn://.*/.*   http://somemirror.org/sources/ \n"
+
+     MIRRORS =+ "\
+         ftp://.*/.*      http://somemirror.org/sources/ \n \
+         http://.*/.*     http://somemirror.org/sources/ \n \
+         https://.*/.*    http://somemirror.org/sources/ \n"
             </literallayout>
         </para>
 
         <para>
-            Non-local downloaded output is placed
-            into the directory specified by the
-            <filename>DL_DIR</filename> variable.
-            For non local archive downloads, the code can verify
-            sha256 and md5 checksums for the download to ensure
-            the file has been downloaded correctly.
-            These can be specified in the following forms
-            for md5 and sha256 checksums, respectively:
+            Any source files that are not local (i.e. downloaded from
+            the Internet) are placed into the download directory,
+            which is specified by
+            <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>.
+        </para>
+
+        <para>
+            For non-local archive downloads, the fetcher code can verify
+            sha256 and md5 checksums to ensure
+            the archives have been downloaded correctly.
+            You can specify these checksums by using the
+            <filename>SRC_URI</filename> variable with the appropriate
+            varflags as follows:
             <literallayout class='monospaced'>
      SRC_URI[md5sum]
      SRC_URI[sha256sum]
             </literallayout>
-            You can also specify them as parameters on the
-            <filename>SRC_URI</filename>:
+            You can also specify the checksums as parameters on the
+            <filename>SRC_URI</filename> as shown below:
             <literallayout class='monospaced'>
      SRC_URI="http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
             </literallayout>
-            If <filename>BB_STRICT_CHECKSUM</filename> is set, any download
-            without a checksum will trigger an error message.
-            In cases where multiple files are listed in
+            If
+            <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
+            is set, any download without a checksum triggers an error message.
+            In cases where multiple files are listed using
             <filename>SRC_URI</filename>, the name parameter is used
             assign names to the URLs and these are then specified
             in the checksums using the following form:
@@ -89,142 +115,226 @@ MIRRORS =+ "\
         </para>
     </section>
 
-    <section id='local-file-fetcher'>
-        <title>Local file fetcher</title>
+    <section id='bb-fetchers'>
+        <title>Fetchers</title>
 
         <para>
-            The URN for the local file fetcher is file.
-            The filename can be either absolute or relative.
-            If the filename is relative,
-            <filename>FILESPATH</filename> and failing that
-            <filename>FILESDIR</filename> will be used to find the
-            appropriate relative file.
-            The metadata usually extend these variables to include
-            variations of the values in <filename>OVERRIDES</filename>.
-            Single files and complete directories can be specified.
-            <literallayout class='monospaced'>
+            As mentioned in the previous section, the
+            <filename>SRC_URI</filename> is normally used to
+            tell BitBake which files to fetch.
+            And, the fetcher BitBake uses depends on the how
+            <filename>SRC_URI</filename> is set.
+        </para>
+
+        <para>
+            These next few sections describe the available fetchers and
+            their options.
+            Each fetcher honors a set of variables URI parameters,
+            which are separated by semi-colon characters and consist
+            of a key and a value.
+            The semantics of the variables and parameters are
+            defined by the fetcher.
+            BitBake tries to have consistent semantics between the
+            different fetchers.
+        </para>
+
+        <section id='local-file-fetcher'>
+            <title>Local file fetcher</title>
+
+            <para>
+                The URN for the local file fetcher is file.
+            </para>
+
+            <para>
+                The filename can be either absolute or relative.
+                If the filename is relative,
+                <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
+                is used.
+                Failing that,
+                <link linkend='var-FILESDIR'><filename>FILESDIR</filename></link>
+                is used to find the appropriate relative file.
+            </para>
+
+            <para>
+                The metadata usually extend these variables to include
+                variations of the values in
+                <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
+                Single files and complete directories can be specified.
+                <literallayout class='monospaced'>
      SRC_URI = "file://relativefile.patch"
      SRC_URI = "file://relativefile.patch;this=ignored"
      SRC_URI = "file:///Users/ich/very_important_software"
-            </literallayout>
-        </para>
-    </section>
+                </literallayout>
+            </para>
+        </section>
 
-    <section id='cvs-fetcher'>
-        <title>CVS fetcher</title>
+        <section id='cvs-fetcher'>
+            <title>CVS fetcher</title>
 
-        <para>
-            The URN for the CVS fetcher is cvs.
-            This fetcher honors the variables <filename>CVSDIR</filename>,
-            <filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
-            <filename>UPDATECOMMAND_cvs</filename>.
-            <filename>DL_DIR</filename> specifies where a
-            temporary checkout is saved.
-            <filename>SRCDATE</filename> specifies which date to
-            use when doing the fetching (the special value of "now"
-            will cause the checkout to be updated on every build).
-            <filename>FETCHCOMMAND</filename> and
-            <filename>UPDATECOMMAND</filename> specify which executables
-            to use for the CVS checkout or update.
-        </para>
+            <para>
+                The URN for the CVS fetcher is cvs.
+            </para>
 
-        <para>
-            The supported parameters are module, tag, date,
-            method, localdir, rshand scmdata.
-            The module specifies which module to check out,
-            the tag describes which CVS TAG should be used for
-            the checkout.
-            By default, the TAG is empty.
-            A date can be specified to override the
-            <filename>SRCDATE</filename> of the
-            configuration to checkout a specific date.
-            The special value of "now" will cause the checkout to be
-            updated on every build.
-            method is by default pserver.
-            If ext is used the rsh parameter will be evaluated
-            and <filename>CVS_RSH</filename> will be set.
-            Finally, localdir is used to checkout into a special
-            directory relative to <filename>CVSDIR</filename>.
-            <literallayout class='monospaced'>
+            <para>
+                This fetcher honors the variables <filename>CVSDIR</filename>,
+                <filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
+                <filename>UPDATECOMMAND_cvs</filename>.
+                The
+                <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+                variable specifies where a
+                temporary checkout is saved.
+                The
+                <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
+                variable specifies which date to
+                use when doing the fetching.
+                The special value of "now" causes the checkout to be
+                updated on every build.
+                The <filename>FETCHCOMMAND</filename> and
+                <filename>UPDATECOMMAND</filename> variables specify the executables
+                to use for the CVS checkout or update.
+            </para>
+
+            <para>
+                The supported parameters are as follows:
+                <itemizedlist>
+                    <listitem><para><emphasis>"module":</emphasis>
+                        Specifies the module to check out.
+                        </para></listitem>
+                    <listitem><para><emphasis>"tag":</emphasis>
+                        Describes which CVS TAG should be used for
+                        the checkout.
+                        By default, the TAG is empty.
+                        </para></listitem>
+                    <listitem><para><emphasis>"date":</emphasis>
+                        Specifies a date.
+                        If no "date" is specified, the
+                        <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
+                        of the configuration is used to checkout a specific date.
+                        The special value of "now" causes the checkout to be
+                        updated on every build.
+                        </para></listitem>
+                    <listitem><para><emphasis>"method":</emphasis>
+                        By default <filename>pserver</filename>.
+                        If "method" is set to "ext", BitBake examines the "rsh"
+                        parameter and sets <filename>CVS_RSH</filename>.
+                        </para></listitem>
+                    <listitem><para><emphasis>"localdir":</emphasis>
+                        Used to checkout force into a special
+                        directory relative to <filename>CVSDIR</filename>.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rsh"</emphasis>
+                        Used in conjunction with the "method" parameter.
+                        </para></listitem>
+                    <listitem><para><emphasis>"scmdata":</emphasis>
+                        I need a description for this.
+                        </para></listitem>
+                </itemizedlist>
+                Following are two examples using cvs:
+                <literallayout class='monospaced'>
      SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
      SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
-            </literallayout>
-        </para>
-    </section>
+                </literallayout>
+            </para>
+        </section>
 
-    <section id='http-ftp-fetcher'>
-        <title>HTTP/FTP fetcher</title>
+        <section id='http-ftp-fetcher'>
+            <title>HTTP/FTP fetcher</title>
 
-        <para>
-            The URNs for the HTTP/FTP fetcher are http, https, and ftp.
-            This fetcher honors the variables
-            <filename>FETCHCOMMAND_wget</filename>.
-            <filename>FETCHCOMMAND</filename> contains the command used
-            for fetching.
-            “${URI}” and “${FILES}” will be replaced by the URI and
-            basename of the file to be fetched.
-            <literallayout class='monospaced'>
+            <para>
+                The URNs for the HTTP/FTP fetcher are http, https, and ftp.
+            </para>
+
+            <para>
+                This fetcher honors the variables
+                <filename>FETCHCOMMAND_wget</filename>.
+                The <filename>FETCHCOMMAND</filename> variable
+                contains the command used for fetching.
+                “${URI}” and “${FILES}” are replaced by the URI and
+                the base name of the file to be fetched.
+                <literallayout class='monospaced'>
      SRC_URI = "http://oe.handhelds.org/not_there.aac"
      SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
      SRC_URI = "ftp://you@oe.handheld.sorg/home/you/secret.plan"
-            </literallayout>
-        </para>
-    </section>
+                </literallayout>
+            </para>
+        </section>
 
-    <section id='svn-fetcher'>
-        <title>SVN Fetcher</title>
+        <section id='svn-fetcher'>
+            <title>SVN Fetcher</title>
 
-        <para>
-            The URN for the SVN fetcher is svn.
-        </para>
+            <para>
+                The URN for the SVN fetcher is svn.
+            </para>
 
-        <para>
-            This fetcher honors the variables
-            <filename>FETCHCOMMAND_svn</filename>,
-            <filename>SVNDIR</filename>,
-            and <filename>SRCREV</filename>.
-            <filename>FETCHCOMMAND</filename> contains the
-            subversion command.
-            <filename>SRCREV</filename> specifies which revision
-            to use when doing the fetching.
-        </para>
+            <para>
+                This fetcher honors the variables
+                <filename>FETCHCOMMAND_svn</filename>,
+                <filename>SVNDIR</filename>,
+                and
+                <link linkend='var-SRCREV'><filename>SRCREV</filename></link>.
+                The <filename>FETCHCOMMAND</filename> variable contains the
+                <filename>subversion</filename> command.
+                The <filename>SRCREV</filename> variable specifies which revision
+                to use when doing the fetching.
+            </para>
 
-        <para>
-            The supported parameters are proto, rev and scmdata.
-            proto is the Subversion protocol, rev is the
-            Subversion revision.
-            If scmdata is set to “keep”, the “.svn” directories will
-            be available during compile-time.
-            <literallayout class='monospaced'>
+            <para>
+                The supported parameters are as follows:
+                <itemizedlist>
+                    <listitem><para><emphasis>"proto":</emphasis>
+                        The Subversion protocol.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rev":</emphasis>
+                        The Subversion revision.
+                        </para></listitem>
+                    <listitem><para><emphasis>"scmdata":</emphasis>
+                        Set to "keep" causes the “.svn” directories
+                        to be available during compile-time.
+                        </para></listitem>
+                </itemizedlist>
+                Following are two examples using svn:
+                <literallayout class='monospaced'>
      SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
      SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
-            </literallayout>
-        </para>
-    </section>
+                </literallayout>
+            </para>
+        </section>
 
-    <section id='git-fetcher'>
-        <title>GIT Fetcher</title>
+        <section id='git-fetcher'>
+            <title>GIT Fetcher</title>
 
-        <para>
-            The URN for the GIT Fetcher is git.
-        </para>
+            <para>
+                The URN for the Git Fetcher is git.
+            </para>
 
-        <para>
-            The variable <filename>GITDIR</filename> will be used as the
-            base directory where the Git tree is cloned to.
-        </para>
+            <para>
+                The variable <filename>GITDIR</filename> is used as the
+                base directory in which the Git tree is cloned.
+            </para>
 
-        <para>
-            The parameters are tag, protocol, and scmdata.
-            The tag parameter is a Git tag, the default is “master”.
-            The protocol tag is the Git protocol to use and defaults to “git”
-            if a hostname is set, otherwise it is “file”.
-            If scmdata is set to “keep”, the “.git” directory will be available
-            during compile-time.
-            <literallayout class='monospaced'>
+            <para>
+                The supported parameters are as follows:
+                <itemizedlist>
+                    <listitem><para><emphasis>"tag":</emphasis>
+                        The Git tag.
+                        The default is "master".
+                        </para></listitem>
+                    <listitem><para><emphasis>"protocol":</emphasis>
+                        The Git protocol.
+                        The default is "git" when a hostname is set.
+                        If a hostname is not set, the Git protocol is "file".
+                        </para></listitem>
+                    <listitem><para><emphasis>"scmdata":</emphasis>
+                        When set to “keep”, the “.git” directory is available
+                        during compile-time.
+                        </para></listitem>
+                </itemizedlist>
+                Following are two examples using git:
+                <literallayout class='monospaced'>
      SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
      SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
-            </literallayout>
-        </para>
+                </literallayout>
+            </para>
+        </section>
     </section>
 </chapter>