summaryrefslogtreecommitdiffstats
path: root/bitbake/doc/user-manual/user-manual-fetching.xml
diff options
context:
space:
mode:
Diffstat (limited to 'bitbake/doc/user-manual/user-manual-fetching.xml')
-rw-r--r--bitbake/doc/user-manual/user-manual-fetching.xml432
1 files changed, 271 insertions, 161 deletions
diff --git a/bitbake/doc/user-manual/user-manual-fetching.xml b/bitbake/doc/user-manual/user-manual-fetching.xml
index cdca64c801..499d4f5198 100644
--- a/bitbake/doc/user-manual/user-manual-fetching.xml
+++ b/bitbake/doc/user-manual/user-manual-fetching.xml
@@ -2,84 +2,110 @@
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> 2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
3 3
4<chapter> 4<chapter>
5<title>File download support</title> 5<title>File Download Support</title>
6
7 <para>
8 BitBake's <filename>fetch</filename> and
9 <filename>fetch2</filename> modules support downloading
10 files.
11 This chapter provides an overview of the fetching process
12 and also presents sections on each of the fetchers BitBake
13 supports.
14 <note>
15 The original <filename>fetch</filename> code, for all
16 practical purposes, has been replaced by
17 <filename>fetch2</filename> code.
18 Consequently, the information in this chapter does not
19 apply to <filename>fetch</filename>.
20 </note>
21 </para>
6 22
7 <section id='file-download-overview'> 23 <section id='file-download-overview'>
8 <title>Overview</title> 24 <title>Overview</title>
9 25
10 <para> 26 <para>
11 BitBake provides support to download files. 27 When BitBake starts to execute, the very first thing
12 This procedure is called fetching and is handled by the 28 it does is to fetch the source files needed.
13 fetch and fetch2 modules. 29 This section overviews the process.
14 At this point, the original fetch code is considered to 30 For some additional information on fetching source files, see the
15 be replaced by fetch2 and this manual is only related 31 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
16 to the fetch2 codebase. 32 section.
17 </para> 33 </para>
18 34
19 <para> 35 <para>
20 The <filename>SRC_URI</filename> is normally used to 36 When BitBake goes looking for source files, it follows a search
21 tell BitBake which files to fetch. 37 order:
22 The next sections describe the available fetchers and 38 <orderedlist>
23 their options. 39 <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
24 Each fetcher honors a set of variables and per 40 BitBake first uses pre-mirrors to try and find source
25 URI parameters separated by a “;” consisting of a key and 41 files.
26 a value. 42 These locations are defined using the
27 The semantics of the variables and parameters are 43 <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
28 defined by the fetcher. 44 variable.
29 BitBake tries to have consistent semantics between the 45 </para></listitem>
30 different fetchers. 46 <listitem><para><emphasis>Source URI:</emphasis>
47 If pre-mirrors fail, BitBake uses
48 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
49 </para></listitem>
50 <listitem><para><emphasis>Mirror Sites:</emphasis>
51 If fetch failures occur using <filename>SRC_URI</filename>,
52 BitBake next uses mirror location as defined by the
53 <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
54 variable.
55 </para></listitem>
56 </orderedlist>
31 </para> 57 </para>
32 58
33 <para> 59 <para>
34 The overall fetch process first attempts to fetch from
35 <filename>PREMIRRORS</filename>.
36 If these fail, the original <filename>SRC_URI</filename>
37 is attempted.
38 If that fails, BitBake falls back to
39 <filename>MIRRORS</filename>.
40 Because cross-URLs are supported, it is possible to mirror 60 Because cross-URLs are supported, it is possible to mirror
41 a Git repository on an HTTP server as a tarball. 61 a Git repository on an HTTP server as a tarball.
42 Here are some examples that show commonly used mirror 62 Here are some examples that show commonly used mirror
43 definitions: 63 definitions:
44 <literallayout class='monospaced'> 64 <literallayout class='monospaced'>
45PREMIRRORS ?= "\ 65 PREMIRRORS ?= "\
46 bzr://.*/.* http://somemirror.org/sources/ \n \ 66 bzr://.*/.* http://somemirror.org/sources/ \n \
47 cvs://.*/.* http://somemirror.org/sources/ \n \ 67 cvs://.*/.* http://somemirror.org/sources/ \n \
48 git://.*/.* http://somemirror.org/sources/ \n \ 68 git://.*/.* http://somemirror.org/sources/ \n \
49 hg://.*/.* http://somemirror.org/sources/ \n \ 69 hg://.*/.* http://somemirror.org/sources/ \n \
50 osc://.*/.* http://somemirror.org/sources/ \n \ 70 osc://.*/.* http://somemirror.org/sources/ \n \
51 p4://.*/.* http://somemirror.org/sources/ \n \ 71 p4://.*/.* http://somemirror.org/sources/ \n \
52 svk://.*/.* http://somemirror.org/sources/ \n \ 72 svk://.*/.* http://somemirror.org/sources/ \n \
53 svn://.*/.* http://somemirror.org/sources/ \n" 73 svn://.*/.* http://somemirror.org/sources/ \n"
54 74
55MIRRORS =+ "\ 75 MIRRORS =+ "\
56 ftp://.*/.* http://somemirror.org/sources/ \n \ 76 ftp://.*/.* http://somemirror.org/sources/ \n \
57 http://.*/.* http://somemirror.org/sources/ \n \ 77 http://.*/.* http://somemirror.org/sources/ \n \
58 https://.*/.* http://somemirror.org/sources/ \n" 78 https://.*/.* http://somemirror.org/sources/ \n"
59 </literallayout> 79 </literallayout>
60 </para> 80 </para>
61 81
62 <para> 82 <para>
63 Non-local downloaded output is placed 83 Any source files that are not local (i.e. downloaded from
64 into the directory specified by the 84 the Internet) are placed into the download directory,
65 <filename>DL_DIR</filename> variable. 85 which is specified by
66 For non local archive downloads, the code can verify 86 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>.
67 sha256 and md5 checksums for the download to ensure 87 </para>
68 the file has been downloaded correctly. 88
69 These can be specified in the following forms 89 <para>
70 for md5 and sha256 checksums, respectively: 90 For non-local archive downloads, the fetcher code can verify
91 sha256 and md5 checksums to ensure
92 the archives have been downloaded correctly.
93 You can specify these checksums by using the
94 <filename>SRC_URI</filename> variable with the appropriate
95 varflags as follows:
71 <literallayout class='monospaced'> 96 <literallayout class='monospaced'>
72 SRC_URI[md5sum] 97 SRC_URI[md5sum]
73 SRC_URI[sha256sum] 98 SRC_URI[sha256sum]
74 </literallayout> 99 </literallayout>
75 You can also specify them as parameters on the 100 You can also specify the checksums as parameters on the
76 <filename>SRC_URI</filename>: 101 <filename>SRC_URI</filename> as shown below:
77 <literallayout class='monospaced'> 102 <literallayout class='monospaced'>
78 SRC_URI="http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" 103 SRC_URI="http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
79 </literallayout> 104 </literallayout>
80 If <filename>BB_STRICT_CHECKSUM</filename> is set, any download 105 If
81 without a checksum will trigger an error message. 106 <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
82 In cases where multiple files are listed in 107 is set, any download without a checksum triggers an error message.
108 In cases where multiple files are listed using
83 <filename>SRC_URI</filename>, the name parameter is used 109 <filename>SRC_URI</filename>, the name parameter is used
84 assign names to the URLs and these are then specified 110 assign names to the URLs and these are then specified
85 in the checksums using the following form: 111 in the checksums using the following form:
@@ -89,142 +115,226 @@ MIRRORS =+ "\
89 </para> 115 </para>
90 </section> 116 </section>
91 117
92 <section id='local-file-fetcher'> 118 <section id='bb-fetchers'>
93 <title>Local file fetcher</title> 119 <title>Fetchers</title>
94 120
95 <para> 121 <para>
96 The URN for the local file fetcher is file. 122 As mentioned in the previous section, the
97 The filename can be either absolute or relative. 123 <filename>SRC_URI</filename> is normally used to
98 If the filename is relative, 124 tell BitBake which files to fetch.
99 <filename>FILESPATH</filename> and failing that 125 And, the fetcher BitBake uses depends on the how
100 <filename>FILESDIR</filename> will be used to find the 126 <filename>SRC_URI</filename> is set.
101 appropriate relative file. 127 </para>
102 The metadata usually extend these variables to include 128
103 variations of the values in <filename>OVERRIDES</filename>. 129 <para>
104 Single files and complete directories can be specified. 130 These next few sections describe the available fetchers and
105 <literallayout class='monospaced'> 131 their options.
132 Each fetcher honors a set of variables URI parameters,
133 which are separated by semi-colon characters and consist
134 of a key and a value.
135 The semantics of the variables and parameters are
136 defined by the fetcher.
137 BitBake tries to have consistent semantics between the
138 different fetchers.
139 </para>
140
141 <section id='local-file-fetcher'>
142 <title>Local file fetcher</title>
143
144 <para>
145 The URN for the local file fetcher is file.
146 </para>
147
148 <para>
149 The filename can be either absolute or relative.
150 If the filename is relative,
151 <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
152 is used.
153 Failing that,
154 <link linkend='var-FILESDIR'><filename>FILESDIR</filename></link>
155 is used to find the appropriate relative file.
156 </para>
157
158 <para>
159 The metadata usually extend these variables to include
160 variations of the values in
161 <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
162 Single files and complete directories can be specified.
163 <literallayout class='monospaced'>
106 SRC_URI = "file://relativefile.patch" 164 SRC_URI = "file://relativefile.patch"
107 SRC_URI = "file://relativefile.patch;this=ignored" 165 SRC_URI = "file://relativefile.patch;this=ignored"
108 SRC_URI = "file:///Users/ich/very_important_software" 166 SRC_URI = "file:///Users/ich/very_important_software"
109 </literallayout> 167 </literallayout>
110 </para> 168 </para>
111 </section> 169 </section>
112 170
113 <section id='cvs-fetcher'> 171 <section id='cvs-fetcher'>
114 <title>CVS fetcher</title> 172 <title>CVS fetcher</title>
115 173
116 <para> 174 <para>
117 The URN for the CVS fetcher is cvs. 175 The URN for the CVS fetcher is cvs.
118 This fetcher honors the variables <filename>CVSDIR</filename>, 176 </para>
119 <filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
120 <filename>UPDATECOMMAND_cvs</filename>.
121 <filename>DL_DIR</filename> specifies where a
122 temporary checkout is saved.
123 <filename>SRCDATE</filename> specifies which date to
124 use when doing the fetching (the special value of "now"
125 will cause the checkout to be updated on every build).
126 <filename>FETCHCOMMAND</filename> and
127 <filename>UPDATECOMMAND</filename> specify which executables
128 to use for the CVS checkout or update.
129 </para>
130 177
131 <para> 178 <para>
132 The supported parameters are module, tag, date, 179 This fetcher honors the variables <filename>CVSDIR</filename>,
133 method, localdir, rshand scmdata. 180 <filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
134 The module specifies which module to check out, 181 <filename>UPDATECOMMAND_cvs</filename>.
135 the tag describes which CVS TAG should be used for 182 The
136 the checkout. 183 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
137 By default, the TAG is empty. 184 variable specifies where a
138 A date can be specified to override the 185 temporary checkout is saved.
139 <filename>SRCDATE</filename> of the 186 The
140 configuration to checkout a specific date. 187 <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
141 The special value of "now" will cause the checkout to be 188 variable specifies which date to
142 updated on every build. 189 use when doing the fetching.
143 method is by default pserver. 190 The special value of "now" causes the checkout to be
144 If ext is used the rsh parameter will be evaluated 191 updated on every build.
145 and <filename>CVS_RSH</filename> will be set. 192 The <filename>FETCHCOMMAND</filename> and
146 Finally, localdir is used to checkout into a special 193 <filename>UPDATECOMMAND</filename> variables specify the executables
147 directory relative to <filename>CVSDIR</filename>. 194 to use for the CVS checkout or update.
148 <literallayout class='monospaced'> 195 </para>
196
197 <para>
198 The supported parameters are as follows:
199 <itemizedlist>
200 <listitem><para><emphasis>"module":</emphasis>
201 Specifies the module to check out.
202 </para></listitem>
203 <listitem><para><emphasis>"tag":</emphasis>
204 Describes which CVS TAG should be used for
205 the checkout.
206 By default, the TAG is empty.
207 </para></listitem>
208 <listitem><para><emphasis>"date":</emphasis>
209 Specifies a date.
210 If no "date" is specified, the
211 <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
212 of the configuration is used to checkout a specific date.
213 The special value of "now" causes the checkout to be
214 updated on every build.
215 </para></listitem>
216 <listitem><para><emphasis>"method":</emphasis>
217 By default <filename>pserver</filename>.
218 If "method" is set to "ext", BitBake examines the "rsh"
219 parameter and sets <filename>CVS_RSH</filename>.
220 </para></listitem>
221 <listitem><para><emphasis>"localdir":</emphasis>
222 Used to checkout force into a special
223 directory relative to <filename>CVSDIR</filename>.
224 </para></listitem>
225 <listitem><para><emphasis>"rsh"</emphasis>
226 Used in conjunction with the "method" parameter.
227 </para></listitem>
228 <listitem><para><emphasis>"scmdata":</emphasis>
229 I need a description for this.
230 </para></listitem>
231 </itemizedlist>
232 Following are two examples using cvs:
233 <literallayout class='monospaced'>
149 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" 234 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
150 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" 235 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
151 </literallayout> 236 </literallayout>
152 </para> 237 </para>
153 </section> 238 </section>
154 239
155 <section id='http-ftp-fetcher'> 240 <section id='http-ftp-fetcher'>
156 <title>HTTP/FTP fetcher</title> 241 <title>HTTP/FTP fetcher</title>
157 242
158 <para> 243 <para>
159 The URNs for the HTTP/FTP fetcher are http, https, and ftp. 244 The URNs for the HTTP/FTP fetcher are http, https, and ftp.
160 This fetcher honors the variables 245 </para>
161 <filename>FETCHCOMMAND_wget</filename>. 246
162 <filename>FETCHCOMMAND</filename> contains the command used 247 <para>
163 for fetching. 248 This fetcher honors the variables
164 “${URI}” and “${FILES}” will be replaced by the URI and 249 <filename>FETCHCOMMAND_wget</filename>.
165 basename of the file to be fetched. 250 The <filename>FETCHCOMMAND</filename> variable
166 <literallayout class='monospaced'> 251 contains the command used for fetching.
252 “${URI}” and “${FILES}” are replaced by the URI and
253 the base name of the file to be fetched.
254 <literallayout class='monospaced'>
167 SRC_URI = "http://oe.handhelds.org/not_there.aac" 255 SRC_URI = "http://oe.handhelds.org/not_there.aac"
168 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac" 256 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
169 SRC_URI = "ftp://you@oe.handheld.sorg/home/you/secret.plan" 257 SRC_URI = "ftp://you@oe.handheld.sorg/home/you/secret.plan"
170 </literallayout> 258 </literallayout>
171 </para> 259 </para>
172 </section> 260 </section>
173 261
174 <section id='svn-fetcher'> 262 <section id='svn-fetcher'>
175 <title>SVN Fetcher</title> 263 <title>SVN Fetcher</title>
176 264
177 <para> 265 <para>
178 The URN for the SVN fetcher is svn. 266 The URN for the SVN fetcher is svn.
179 </para> 267 </para>
180 268
181 <para> 269 <para>
182 This fetcher honors the variables 270 This fetcher honors the variables
183 <filename>FETCHCOMMAND_svn</filename>, 271 <filename>FETCHCOMMAND_svn</filename>,
184 <filename>SVNDIR</filename>, 272 <filename>SVNDIR</filename>,
185 and <filename>SRCREV</filename>. 273 and
186 <filename>FETCHCOMMAND</filename> contains the 274 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>.
187 subversion command. 275 The <filename>FETCHCOMMAND</filename> variable contains the
188 <filename>SRCREV</filename> specifies which revision 276 <filename>subversion</filename> command.
189 to use when doing the fetching. 277 The <filename>SRCREV</filename> variable specifies which revision
190 </para> 278 to use when doing the fetching.
279 </para>
191 280
192 <para> 281 <para>
193 The supported parameters are proto, rev and scmdata. 282 The supported parameters are as follows:
194 proto is the Subversion protocol, rev is the 283 <itemizedlist>
195 Subversion revision. 284 <listitem><para><emphasis>"proto":</emphasis>
196 If scmdata is set to “keep”, the “.svn” directories will 285 The Subversion protocol.
197 be available during compile-time. 286 </para></listitem>
198 <literallayout class='monospaced'> 287 <listitem><para><emphasis>"rev":</emphasis>
288 The Subversion revision.
289 </para></listitem>
290 <listitem><para><emphasis>"scmdata":</emphasis>
291 Set to "keep" causes the “.svn” directories
292 to be available during compile-time.
293 </para></listitem>
294 </itemizedlist>
295 Following are two examples using svn:
296 <literallayout class='monospaced'>
199 SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667" 297 SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
200 SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126" 298 SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
201 </literallayout> 299 </literallayout>
202 </para> 300 </para>
203 </section> 301 </section>
204 302
205 <section id='git-fetcher'> 303 <section id='git-fetcher'>
206 <title>GIT Fetcher</title> 304 <title>GIT Fetcher</title>
207 305
208 <para> 306 <para>
209 The URN for the GIT Fetcher is git. 307 The URN for the Git Fetcher is git.
210 </para> 308 </para>
211 309
212 <para> 310 <para>
213 The variable <filename>GITDIR</filename> will be used as the 311 The variable <filename>GITDIR</filename> is used as the
214 base directory where the Git tree is cloned to. 312 base directory in which the Git tree is cloned.
215 </para> 313 </para>
216 314
217 <para> 315 <para>
218 The parameters are tag, protocol, and scmdata. 316 The supported parameters are as follows:
219 The tag parameter is a Git tag, the default is “master”. 317 <itemizedlist>
220 The protocol tag is the Git protocol to use and defaults to “git” 318 <listitem><para><emphasis>"tag":</emphasis>
221 if a hostname is set, otherwise it is “file”. 319 The Git tag.
222 If scmdata is set to “keep”, the “.git” directory will be available 320 The default is "master".
223 during compile-time. 321 </para></listitem>
224 <literallayout class='monospaced'> 322 <listitem><para><emphasis>"protocol":</emphasis>
323 The Git protocol.
324 The default is "git" when a hostname is set.
325 If a hostname is not set, the Git protocol is "file".
326 </para></listitem>
327 <listitem><para><emphasis>"scmdata":</emphasis>
328 When set to “keep”, the “.git” directory is available
329 during compile-time.
330 </para></listitem>
331 </itemizedlist>
332 Following are two examples using git:
333 <literallayout class='monospaced'>
225 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" 334 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
226 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http" 335 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
227 </literallayout> 336 </literallayout>
228 </para> 337 </para>
338 </section>
229 </section> 339 </section>
230</chapter> 340</chapter>