diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml | 738 |
1 files changed, 738 insertions, 0 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml new file mode 100644 index 0000000000..0dff736dea --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml | |||
@@ -0,0 +1,738 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> | ||
3 | |||
4 | <chapter> | ||
5 | <title>File Download Support</title> | ||
6 | |||
7 | <para> | ||
8 | BitBake's fetch module is a standalone piece of library code | ||
9 | that deals with the intricacies of downloading source code | ||
10 | and files from remote systems. | ||
11 | Fetching source code is one of the cornerstones of building software. | ||
12 | As such, this module forms an important part of BitBake. | ||
13 | </para> | ||
14 | |||
15 | <para> | ||
16 | The current fetch module is called "fetch2" and refers to the | ||
17 | fact that it is the second major version of the API. | ||
18 | The original version is obsolete and has been removed from the codebase. | ||
19 | Thus, in all cases, "fetch" refers to "fetch2" in this | ||
20 | manual. | ||
21 | </para> | ||
22 | |||
23 | <section id='the-download-fetch'> | ||
24 | <title>The Download (Fetch)</title> | ||
25 | |||
26 | <para> | ||
27 | BitBake takes several steps when fetching source code or files. | ||
28 | The fetcher codebase deals with two distinct processes in order: | ||
29 | obtaining the files from somewhere (cached or otherwise) | ||
30 | and then unpacking those files into a specific location and | ||
31 | perhaps in a specific way. | ||
32 | Getting and unpacking the files is often optionally followed | ||
33 | by patching. | ||
34 | Patching, however, is not covered by this module. | ||
35 | </para> | ||
36 | |||
37 | <para> | ||
38 | The code to execute the first part of this process, a fetch, | ||
39 | looks something like the following: | ||
40 | <literallayout class='monospaced'> | ||
41 | src_uri = (d.getVar('SRC_URI', True) or "").split() | ||
42 | fetcher = bb.fetch2.Fetch(src_uri, d) | ||
43 | fetcher.download() | ||
44 | </literallayout> | ||
45 | This code sets up an instance of the fetch class. | ||
46 | The instance uses a space-separated list of URLs from the | ||
47 | <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link> | ||
48 | variable and then calls the <filename>download</filename> | ||
49 | method to download the files. | ||
50 | </para> | ||
51 | |||
52 | <para> | ||
53 | The instantiation of the fetch class is usually followed by: | ||
54 | <literallayout class='monospaced'> | ||
55 | rootdir = l.getVar('WORKDIR', True) | ||
56 | fetcher.unpack(rootdir) | ||
57 | </literallayout> | ||
58 | This code unpacks the downloaded files to the | ||
59 | specified by <filename>WORKDIR</filename>. | ||
60 | <note> | ||
61 | For convenience, the naming in these examples matches | ||
62 | the variables used by OpenEmbedded. | ||
63 | If you want to see the above code in action, examine | ||
64 | the OpenEmbedded class file <filename>base.bbclass</filename>. | ||
65 | </note> | ||
66 | The <filename>SRC_URI</filename> and <filename>WORKDIR</filename> | ||
67 | variables are not hardcoded into the fetcher, since those fetcher | ||
68 | methods can be (and are) called with different variable names. | ||
69 | In OpenEmbedded for example, the shared state (sstate) code uses | ||
70 | the fetch module to fetch the sstate files. | ||
71 | </para> | ||
72 | |||
73 | <para> | ||
74 | When the <filename>download()</filename> method is called, | ||
75 | BitBake tries to resolve the URLs by looking for source files | ||
76 | in a specific search order: | ||
77 | <itemizedlist> | ||
78 | <listitem><para><emphasis>Pre-mirror Sites:</emphasis> | ||
79 | BitBake first uses pre-mirrors to try and find source files. | ||
80 | These locations are defined using the | ||
81 | <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link> | ||
82 | variable. | ||
83 | </para></listitem> | ||
84 | <listitem><para><emphasis>Source URI:</emphasis> | ||
85 | If pre-mirrors fail, BitBake uses the original URL (e.g from | ||
86 | <filename>SRC_URI</filename>). | ||
87 | </para></listitem> | ||
88 | <listitem><para><emphasis>Mirror Sites:</emphasis> | ||
89 | If fetch failures occur, BitBake next uses mirror locations as | ||
90 | defined by the | ||
91 | <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link> | ||
92 | variable. | ||
93 | </para></listitem> | ||
94 | </itemizedlist> | ||
95 | </para> | ||
96 | |||
97 | <para> | ||
98 | For each URL passed to the fetcher, the fetcher | ||
99 | calls the submodule that handles that particular URL type. | ||
100 | This behavior can be the source of some confusion when you | ||
101 | are providing URLs for the <filename>SRC_URI</filename> | ||
102 | variable. | ||
103 | Consider the following two URLs: | ||
104 | <literallayout class='monospaced'> | ||
105 | http://git.yoctoproject.org/git/poky;protocol=git | ||
106 | git://git.yoctoproject.org/git/poky;protocol=http | ||
107 | </literallayout> | ||
108 | In the former case, the URL is passed to the | ||
109 | <filename>wget</filename> fetcher, which does not | ||
110 | understand "git". | ||
111 | Therefore, the latter case is the correct form since the | ||
112 | Git fetcher does know how to use HTTP as a transport. | ||
113 | </para> | ||
114 | |||
115 | <para> | ||
116 | Here are some examples that show commonly used mirror | ||
117 | definitions: | ||
118 | <literallayout class='monospaced'> | ||
119 | PREMIRRORS ?= "\ | ||
120 | bzr://.*/.* http://somemirror.org/sources/ \n \ | ||
121 | cvs://.*/.* http://somemirror.org/sources/ \n \ | ||
122 | git://.*/.* http://somemirror.org/sources/ \n \ | ||
123 | hg://.*/.* http://somemirror.org/sources/ \n \ | ||
124 | osc://.*/.* http://somemirror.org/sources/ \n \ | ||
125 | p4://.*/.* http://somemirror.org/sources/ \n \ | ||
126 | svn://.*/.* http://somemirror.org/sources/ \n" | ||
127 | |||
128 | MIRRORS =+ "\ | ||
129 | ftp://.*/.* http://somemirror.org/sources/ \n \ | ||
130 | http://.*/.* http://somemirror.org/sources/ \n \ | ||
131 | https://.*/.* http://somemirror.org/sources/ \n" | ||
132 | </literallayout> | ||
133 | It is useful to note that BitBake supports | ||
134 | cross-URLs. | ||
135 | It is possible to mirror a Git repository on an HTTP | ||
136 | server as a tarball. | ||
137 | This is what the <filename>git://</filename> mapping in | ||
138 | the previous example does. | ||
139 | </para> | ||
140 | |||
141 | <para> | ||
142 | Since network accesses are slow, Bitbake maintains a | ||
143 | cache of files downloaded from the network. | ||
144 | Any source files that are not local (i.e. | ||
145 | downloaded from the Internet) are placed into the download | ||
146 | directory, which is specified by the | ||
147 | <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link> | ||
148 | variable. | ||
149 | </para> | ||
150 | |||
151 | <para> | ||
152 | File integrity is of key importance for reproducing builds. | ||
153 | For non-local archive downloads, the fetcher code can verify | ||
154 | SHA-256 and MD5 checksums to ensure the archives have been | ||
155 | downloaded correctly. | ||
156 | You can specify these checksums by using the | ||
157 | <filename>SRC_URI</filename> variable with the appropriate | ||
158 | varflags as follows: | ||
159 | <literallayout class='monospaced'> | ||
160 | SRC_URI[md5sum] = "value" | ||
161 | SRC_URI[sha256sum] = "value" | ||
162 | </literallayout> | ||
163 | You can also specify the checksums as parameters on the | ||
164 | <filename>SRC_URI</filename> as shown below: | ||
165 | <literallayout class='monospaced'> | ||
166 | SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" | ||
167 | </literallayout> | ||
168 | If multiple URIs exist, you can specify the checksums either | ||
169 | directly as in the previous example, or you can name the URLs. | ||
170 | The following syntax shows how you name the URIs: | ||
171 | <literallayout class='monospaced'> | ||
172 | SRC_URI = "http://example.com/foobar.tar.bz2;name=foo" | ||
173 | SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d | ||
174 | </literallayout> | ||
175 | After a file has been downloaded and has had its checksum checked, | ||
176 | a ".done" stamp is placed in <filename>DL_DIR</filename>. | ||
177 | BitBake uses this stamp during subsequent builds to avoid | ||
178 | downloading or comparing a checksum for the file again. | ||
179 | <note> | ||
180 | It is assumed that local storage is safe from data corruption. | ||
181 | If this were not the case, there would be bigger issues to worry about. | ||
182 | </note> | ||
183 | </para> | ||
184 | |||
185 | <para> | ||
186 | If | ||
187 | <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link> | ||
188 | is set, any download without a checksum triggers an | ||
189 | error message. | ||
190 | The | ||
191 | <link linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link> | ||
192 | variable can be used to make any attempted network access a fatal | ||
193 | error, which is useful for checking that mirrors are complete | ||
194 | as well as other things. | ||
195 | </para> | ||
196 | </section> | ||
197 | |||
198 | <section id='bb-the-unpack'> | ||
199 | <title>The Unpack</title> | ||
200 | |||
201 | <para> | ||
202 | The unpack process usually immediately follows the download. | ||
203 | For all URLs except Git URLs, BitBake uses the common | ||
204 | <filename>unpack</filename> method. | ||
205 | </para> | ||
206 | |||
207 | <para> | ||
208 | A number of parameters exist that you can specify within the | ||
209 | URL to govern the behavior of the unpack stage: | ||
210 | <itemizedlist> | ||
211 | <listitem><para><emphasis>unpack:</emphasis> | ||
212 | Controls whether the URL components are unpacked. | ||
213 | If set to "1", which is the default, the components | ||
214 | are unpacked. | ||
215 | If set to "0", the unpack stage leaves the file alone. | ||
216 | This parameter is useful when you want an archive to be | ||
217 | copied in and not be unpacked. | ||
218 | </para></listitem> | ||
219 | <listitem><para><emphasis>dos:</emphasis> | ||
220 | Applies to <filename>.zip</filename> and | ||
221 | <filename>.jar</filename> files and specifies whether to | ||
222 | use DOS line ending conversion on text files. | ||
223 | </para></listitem> | ||
224 | <listitem><para><emphasis>basepath:</emphasis> | ||
225 | Instructs the unpack stage to strip the specified | ||
226 | directories from the source path when unpacking. | ||
227 | </para></listitem> | ||
228 | <listitem><para><emphasis>subdir:</emphasis> | ||
229 | Unpacks the specific URL to the specified subdirectory | ||
230 | within the root directory. | ||
231 | </para></listitem> | ||
232 | </itemizedlist> | ||
233 | The unpack call automatically decompresses and extracts files | ||
234 | with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". | ||
235 | ".srpm", ".deb" and ".bz2" extensions as well as various combinations | ||
236 | of tarball extensions. | ||
237 | </para> | ||
238 | |||
239 | <para> | ||
240 | As mentioned, the Git fetcher has its own unpack method that | ||
241 | is optimized to work with Git trees. | ||
242 | Basically, this method works by cloning the tree into the final | ||
243 | directory. | ||
244 | The process is completed using references so that there is | ||
245 | only one central copy of the Git metadata needed. | ||
246 | </para> | ||
247 | </section> | ||
248 | |||
249 | <section id='bb-fetchers'> | ||
250 | <title>Fetchers</title> | ||
251 | |||
252 | <para> | ||
253 | As mentioned earlier, the URL prefix determines which | ||
254 | fetcher submodule BitBake uses. | ||
255 | Each submodule can support different URL parameters, | ||
256 | which are described in the following sections. | ||
257 | </para> | ||
258 | |||
259 | <section id='local-file-fetcher'> | ||
260 | <title>Local file fetcher (<filename>file://</filename>)</title> | ||
261 | |||
262 | <para> | ||
263 | This submodule handles URLs that begin with | ||
264 | <filename>file://</filename>. | ||
265 | The filename you specify within the URL can be | ||
266 | either an absolute or relative path to a file. | ||
267 | If the filename is relative, the contents of the | ||
268 | <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link> | ||
269 | variable is used in the same way | ||
270 | <filename>PATH</filename> is used to find executables. | ||
271 | Failing that, | ||
272 | <link linkend='var-FILESDIR'><filename>FILESDIR</filename></link> | ||
273 | is used to find the appropriate relative file. | ||
274 | <note> | ||
275 | <filename>FILESDIR</filename> is deprecated and can | ||
276 | be replaced with <filename>FILESPATH</filename>. | ||
277 | Because <filename>FILESDIR</filename> is likely to be | ||
278 | removed, you should not use this variable in any new code. | ||
279 | </note> | ||
280 | If the file cannot be found, it is assumed that it is available in | ||
281 | <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link> | ||
282 | by the time the <filename>download()</filename> method is called. | ||
283 | </para> | ||
284 | |||
285 | <para> | ||
286 | If you specify a directory, the entire directory is | ||
287 | unpacked. | ||
288 | </para> | ||
289 | |||
290 | <para> | ||
291 | Here are a couple of example URLs, the first relative and | ||
292 | the second absolute: | ||
293 | <literallayout class='monospaced'> | ||
294 | SRC_URI = "file://relativefile.patch" | ||
295 | SRC_URI = "file:///Users/ich/very_important_software" | ||
296 | </literallayout> | ||
297 | </para> | ||
298 | </section> | ||
299 | |||
300 | <section id='http-ftp-fetcher'> | ||
301 | <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title> | ||
302 | |||
303 | <para> | ||
304 | This fetcher obtains files from web and FTP servers. | ||
305 | Internally, the fetcher uses the wget utility. | ||
306 | </para> | ||
307 | |||
308 | <para> | ||
309 | The executable and parameters used are specified by the | ||
310 | <filename>FETCHCMD_wget</filename> variable, which defaults | ||
311 | to sensible values. | ||
312 | The fetcher supports a parameter "downloadfilename" that | ||
313 | allows the name of the downloaded file to be specified. | ||
314 | Specifying the name of the downloaded file is useful | ||
315 | for avoiding collisions in | ||
316 | <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link> | ||
317 | when dealing with multiple files that have the same name. | ||
318 | </para> | ||
319 | |||
320 | <para> | ||
321 | Some example URLs are as follows: | ||
322 | <literallayout class='monospaced'> | ||
323 | SRC_URI = "http://oe.handhelds.org/not_there.aac" | ||
324 | SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac" | ||
325 | SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan" | ||
326 | </literallayout> | ||
327 | </para> | ||
328 | </section> | ||
329 | |||
330 | <section id='cvs-fetcher'> | ||
331 | <title>CVS fetcher (<filename>(cvs://</filename>)</title> | ||
332 | |||
333 | <para> | ||
334 | This submodule handles checking out files from the | ||
335 | CVS version control system. | ||
336 | You can configure it using a number of different variables: | ||
337 | <itemizedlist> | ||
338 | <listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis> | ||
339 | The name of the executable to use when running | ||
340 | the <filename>cvs</filename> command. | ||
341 | This name is usually "cvs". | ||
342 | </para></listitem> | ||
343 | <listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis> | ||
344 | The date to use when fetching the CVS source code. | ||
345 | A special value of "now" causes the checkout to | ||
346 | be updated on every build. | ||
347 | </para></listitem> | ||
348 | <listitem><para><emphasis><filename>CVSDIR</filename>:</emphasis> | ||
349 | Specifies where a temporary checkout is saved. | ||
350 | The location is often <filename>DL_DIR/cvs</filename>. | ||
351 | </para></listitem> | ||
352 | <listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis> | ||
353 | The name to use as a "proxy=" parameter to the | ||
354 | <filename>cvs</filename> command. | ||
355 | </para></listitem> | ||
356 | <listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis> | ||
357 | The port number to use as a "proxyport=" parameter to | ||
358 | the <filename>cvs</filename> command. | ||
359 | </para></listitem> | ||
360 | </itemizedlist> | ||
361 | As well as the standard username and password URL syntax, | ||
362 | you can also configure the fetcher with various URL parameters: | ||
363 | </para> | ||
364 | |||
365 | <para> | ||
366 | The supported parameters are as follows: | ||
367 | <itemizedlist> | ||
368 | <listitem><para><emphasis>"method":</emphasis> | ||
369 | The protocol over which to communicate with the CVS server. | ||
370 | By default, this protocol is "pserver". | ||
371 | If "method" is set to "ext", BitBake examines the | ||
372 | "rsh" parameter and sets <filename>CVS_RSH</filename>. | ||
373 | You can use "dir" for local directories. | ||
374 | </para></listitem> | ||
375 | <listitem><para><emphasis>"module":</emphasis> | ||
376 | Specifies the module to check out. | ||
377 | You must supply this parameter. | ||
378 | </para></listitem> | ||
379 | <listitem><para><emphasis>"tag":</emphasis> | ||
380 | Describes which CVS TAG should be used for | ||
381 | the checkout. | ||
382 | By default, the TAG is empty. | ||
383 | </para></listitem> | ||
384 | <listitem><para><emphasis>"date":</emphasis> | ||
385 | Specifies a date. | ||
386 | If no "date" is specified, the | ||
387 | <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link> | ||
388 | of the configuration is used to checkout a specific date. | ||
389 | The special value of "now" causes the checkout to be | ||
390 | updated on every build. | ||
391 | </para></listitem> | ||
392 | <listitem><para><emphasis>"localdir":</emphasis> | ||
393 | Used to rename the module. | ||
394 | Effectively, you are renaming the output directory | ||
395 | to which the module is unpacked. | ||
396 | You are forcing the module into a special | ||
397 | directory relative to <filename>CVSDIR</filename>. | ||
398 | </para></listitem> | ||
399 | <listitem><para><emphasis>"rsh"</emphasis> | ||
400 | Used in conjunction with the "method" parameter. | ||
401 | </para></listitem> | ||
402 | <listitem><para><emphasis>"scmdata":</emphasis> | ||
403 | Causes the CVS metadata to be maintained in the tarball | ||
404 | the fetcher creates when set to "keep". | ||
405 | The tarball is expanded into the work directory. | ||
406 | By default, the CVS metadata is removed. | ||
407 | </para></listitem> | ||
408 | <listitem><para><emphasis>"fullpath":</emphasis> | ||
409 | Controls whether the resulting checkout is at the | ||
410 | module level, which is the default, or is at deeper | ||
411 | paths. | ||
412 | </para></listitem> | ||
413 | <listitem><para><emphasis>"norecurse":</emphasis> | ||
414 | Causes the fetcher to only checkout the specified | ||
415 | directory with no recurse into any subdirectories. | ||
416 | </para></listitem> | ||
417 | <listitem><para><emphasis>"port":</emphasis> | ||
418 | The port to which the CVS server connects. | ||
419 | </para></listitem> | ||
420 | </itemizedlist> | ||
421 | Some example URLs are as follows: | ||
422 | <literallayout class='monospaced'> | ||
423 | SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" | ||
424 | SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" | ||
425 | </literallayout> | ||
426 | </para> | ||
427 | </section> | ||
428 | |||
429 | <section id='svn-fetcher'> | ||
430 | <title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title> | ||
431 | |||
432 | <para> | ||
433 | This fetcher submodule fetches code from the | ||
434 | Subversion source control system. | ||
435 | The executable used is specified by | ||
436 | <filename>FETCHCMD_svn</filename>, which defaults | ||
437 | to "svn". | ||
438 | The fetcher's temporary working directory is set | ||
439 | by <filename>SVNDIR</filename>, which is usually | ||
440 | <filename>DL_DIR/svn</filename>. | ||
441 | </para> | ||
442 | |||
443 | <para> | ||
444 | The supported parameters are as follows: | ||
445 | <itemizedlist> | ||
446 | <listitem><para><emphasis>"module":</emphasis> | ||
447 | The name of the svn module to checkout. | ||
448 | You must provide this parameter. | ||
449 | You can think of this parameter as the top-level | ||
450 | directory of the repository data you want. | ||
451 | </para></listitem> | ||
452 | <listitem><para><emphasis>"protocol":</emphasis> | ||
453 | The protocol to use, which defaults to "svn". | ||
454 | Other options are "svn+ssh" and "rsh". | ||
455 | For "rsh", the "rsh" parameter is also used. | ||
456 | </para></listitem> | ||
457 | <listitem><para><emphasis>"rev":</emphasis> | ||
458 | The revision of the source code to checkout. | ||
459 | </para></listitem> | ||
460 | <listitem><para><emphasis>"date":</emphasis> | ||
461 | The date of the source code to checkout. | ||
462 | Specific revisions are generally much safer to checkout | ||
463 | rather than by date as they do not involve timezones | ||
464 | (e.g. they are much more deterministic). | ||
465 | </para></listitem> | ||
466 | <listitem><para><emphasis>"scmdata":</emphasis> | ||
467 | Causes the “.svn” directories to be available during | ||
468 | compile-time when set to "keep". | ||
469 | By default, these directories are removed. | ||
470 | </para></listitem> | ||
471 | <listitem><para><emphasis>"transportuser":</emphasis> | ||
472 | When required, sets the username for the transport. | ||
473 | By default, this parameter is empty. | ||
474 | The transport username is different than the username | ||
475 | used in the main URL, which is passed to the subversion | ||
476 | command. | ||
477 | </para></listitem> | ||
478 | </itemizedlist> | ||
479 | Following are two examples using svn: | ||
480 | <literallayout class='monospaced'> | ||
481 | SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667" | ||
482 | SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126" | ||
483 | </literallayout> | ||
484 | </para> | ||
485 | </section> | ||
486 | |||
487 | <section id='git-fetcher'> | ||
488 | <title>Git Fetcher (<filename>git://</filename>)</title> | ||
489 | |||
490 | <para> | ||
491 | This fetcher submodule fetches code from the Git | ||
492 | source control system. | ||
493 | The fetcher works by creating a bare clone of the | ||
494 | remote into <filename>GITDIR</filename>, which is | ||
495 | usually <filename>DL_DIR/git2</filename>. | ||
496 | This bare clone is then cloned into the work directory during the | ||
497 | unpack stage when a specific tree is checked out. | ||
498 | This is done using alternates and by reference to | ||
499 | minimize the amount of duplicate data on the disk and | ||
500 | make the unpack process fast. | ||
501 | The executable used can be set with | ||
502 | <filename>FETCHCMD_git</filename>. | ||
503 | </para> | ||
504 | |||
505 | <para> | ||
506 | This fetcher supports the following parameters: | ||
507 | <itemizedlist> | ||
508 | <listitem><para><emphasis>"protocol":</emphasis> | ||
509 | The protocol used to fetch the files. | ||
510 | The default is "git" when a hostname is set. | ||
511 | If a hostname is not set, the Git protocol is "file". | ||
512 | You can also use "http", "https", "ssh" and "rsync". | ||
513 | </para></listitem> | ||
514 | <listitem><para><emphasis>"nocheckout":</emphasis> | ||
515 | Tells the fetcher to not checkout source code when | ||
516 | unpacking when set to "1". | ||
517 | Set this option for the URL where there is a custom | ||
518 | routine to checkout code. | ||
519 | The default is "0". | ||
520 | </para></listitem> | ||
521 | <listitem><para><emphasis>"rebaseable":</emphasis> | ||
522 | Indicates that the upstream Git repository can be rebased. | ||
523 | You should set this parameter to "1" if | ||
524 | revisions can become detached from branches. | ||
525 | In this case, the source mirror tarball is done per | ||
526 | revision, which has a loss of efficiency. | ||
527 | Rebasing the upstream Git repository could cause the | ||
528 | current revision to disappear from the upstream repository. | ||
529 | This option reminds the fetcher to preserve the local cache | ||
530 | carefully for future use. | ||
531 | The default value for this parameter is "0". | ||
532 | </para></listitem> | ||
533 | <listitem><para><emphasis>"nobranch":</emphasis> | ||
534 | Tells the fetcher to not check the SHA validation | ||
535 | for the branch when set to "1". | ||
536 | The default is "0". | ||
537 | Set this option for the recipe that refers to | ||
538 | the commit that is valid for a tag instead of | ||
539 | the branch. | ||
540 | </para></listitem> | ||
541 | <listitem><para><emphasis>"bareclone":</emphasis> | ||
542 | Tells the fetcher to clone a bare clone into the | ||
543 | destination directory without checking out a working tree. | ||
544 | Only the raw Git metadata is provided. | ||
545 | This parameter implies the "nocheckout" parameter as well. | ||
546 | </para></listitem> | ||
547 | <listitem><para><emphasis>"branch":</emphasis> | ||
548 | The branch(es) of the Git tree to clone. | ||
549 | If unset, this is assumed to be "master". | ||
550 | The number of branch parameters much match the number of | ||
551 | name parameters. | ||
552 | </para></listitem> | ||
553 | <listitem><para><emphasis>"rev":</emphasis> | ||
554 | The revision to use for the checkout. | ||
555 | The default is "master". | ||
556 | </para></listitem> | ||
557 | <listitem><para><emphasis>"tag":</emphasis> | ||
558 | Specifies a tag to use for the checkout. | ||
559 | To correctly resolve tags, BitBake must access the | ||
560 | network. | ||
561 | For that reason, tags are often not used. | ||
562 | As far as Git is concerned, the "tag" parameter behaves | ||
563 | effectively the same as the "rev" parameter. | ||
564 | </para></listitem> | ||
565 | <listitem><para><emphasis>"subpath":</emphasis> | ||
566 | Limits the checkout to a specific subpath of the tree. | ||
567 | By default, the whole tree is checked out. | ||
568 | </para></listitem> | ||
569 | <listitem><para><emphasis>"destsuffix":</emphasis> | ||
570 | The name of the path in which to place the checkout. | ||
571 | By default, the path is <filename>git/</filename>. | ||
572 | </para></listitem> | ||
573 | </itemizedlist> | ||
574 | Here are some example URLs: | ||
575 | <literallayout class='monospaced'> | ||
576 | SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" | ||
577 | SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http" | ||
578 | </literallayout> | ||
579 | </para> | ||
580 | </section> | ||
581 | |||
582 | <section id='gitsm-fetcher'> | ||
583 | <title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title> | ||
584 | |||
585 | <para> | ||
586 | This fetcher submodule inherits from the | ||
587 | <link linkend='git-fetcher'>Git fetcher</link> and extends | ||
588 | that fetcher's behavior by fetching a repository's submodules. | ||
589 | <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link> | ||
590 | is passed to the Git fetcher as described in the | ||
591 | "<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>" | ||
592 | section. | ||
593 | <note> | ||
594 | <title>Notes and Warnings</title> | ||
595 | <para> | ||
596 | You must clean a recipe when switching between | ||
597 | '<filename>git://</filename>' and | ||
598 | '<filename>gitsm://</filename>' URLs. | ||
599 | </para> | ||
600 | |||
601 | <para> | ||
602 | The Git Submodules fetcher is not a complete fetcher | ||
603 | implementation. | ||
604 | The fetcher has known issues where it does not use the | ||
605 | normal source mirroring infrastructure properly. | ||
606 | </para> | ||
607 | </note> | ||
608 | </para> | ||
609 | </section> | ||
610 | |||
611 | <section id='clearcase-fetcher'> | ||
612 | <title>ClearCase Fetcher (<filename>ccrc://</filename>)</title> | ||
613 | |||
614 | <para> | ||
615 | This fetcher submodule fetches code from a | ||
616 | <ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink> | ||
617 | repository. | ||
618 | </para> | ||
619 | |||
620 | <para> | ||
621 | To use this fetcher, make sure your recipe has proper | ||
622 | <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>, | ||
623 | <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and | ||
624 | <link linkend='var-PV'><filename>PV</filename></link> settings. | ||
625 | Here is an example: | ||
626 | <literallayout class='monospaced'> | ||
627 | SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" | ||
628 | SRCREV = "EXAMPLE_CLEARCASE_TAG" | ||
629 | PV = "${@d.getVar("SRCREV").replace("/", "+")}" | ||
630 | </literallayout> | ||
631 | The fetcher uses the <filename>rcleartool</filename> or | ||
632 | <filename>cleartool</filename> remote client, depending on | ||
633 | which one is available. | ||
634 | </para> | ||
635 | |||
636 | <para> | ||
637 | Following are options for the <filename>SRC_URI</filename> | ||
638 | statement: | ||
639 | <itemizedlist> | ||
640 | <listitem><para><emphasis><filename>vob</filename></emphasis>: | ||
641 | The name, which must include the | ||
642 | prepending "/" character, of the ClearCase VOB. | ||
643 | This option is required. | ||
644 | </para></listitem> | ||
645 | <listitem><para><emphasis><filename>module</filename></emphasis>: | ||
646 | The module, which must include the | ||
647 | prepending "/" character, in the selected VOB | ||
648 | The <filename>module</filename> and <filename>vob</filename> | ||
649 | options are combined to create the following load rule in | ||
650 | the view config spec: | ||
651 | <literallayout class='monospaced'> | ||
652 | load <vob><module> | ||
653 | </literallayout> | ||
654 | </para></listitem> | ||
655 | <listitem><para><emphasis><filename>proto</filename></emphasis>: | ||
656 | The protocol, which can be either <filename>http</filename> or | ||
657 | <filename>https</filename>. | ||
658 | </para></listitem> | ||
659 | </itemizedlist> | ||
660 | </para> | ||
661 | |||
662 | <para> | ||
663 | By default, the fetcher creates a configuration specification. | ||
664 | If you want this specification written to an area other than the default, | ||
665 | use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable | ||
666 | in your recipe to define where the specification is written. | ||
667 | <note> | ||
668 | the <filename>SRCREV</filename> loses its functionality if you | ||
669 | specify this variable. | ||
670 | However, <filename>SRCREV</filename> is still used to label the | ||
671 | archive after a fetch even though it does not define what is | ||
672 | fetched. | ||
673 | </note> | ||
674 | </para> | ||
675 | |||
676 | <para> | ||
677 | Here are a couple of other behaviors worth mentioning: | ||
678 | <itemizedlist> | ||
679 | <listitem><para> | ||
680 | When using <filename>cleartool</filename>, the login of | ||
681 | <filename>cleartool</filename> is handled by the system. | ||
682 | The login require no special steps. | ||
683 | </para></listitem> | ||
684 | <listitem><para> | ||
685 | In order to use <filename>rcleartool</filename> with authenticated | ||
686 | users, an "rcleartool login" is necessary before using the fetcher. | ||
687 | </para></listitem> | ||
688 | </itemizedlist> | ||
689 | </para> | ||
690 | </section> | ||
691 | |||
692 | <section id='other-fetchers'> | ||
693 | <title>Other Fetchers</title> | ||
694 | |||
695 | <para> | ||
696 | Fetch submodules also exist for the following: | ||
697 | <itemizedlist> | ||
698 | <listitem><para> | ||
699 | Bazaar (<filename>bzr://</filename>) | ||
700 | </para></listitem> | ||
701 | <listitem><para> | ||
702 | Perforce (<filename>p4://</filename>) | ||
703 | </para></listitem> | ||
704 | <listitem><para> | ||
705 | Trees using Git Annex (<filename>gitannex://</filename>) | ||
706 | </para></listitem> | ||
707 | <listitem><para> | ||
708 | Secure FTP (<filename>sftp://</filename>) | ||
709 | </para></listitem> | ||
710 | <listitem><para> | ||
711 | Secure Shell (<filename>ssh://</filename>) | ||
712 | </para></listitem> | ||
713 | <listitem><para> | ||
714 | Repo (<filename>repo://</filename>) | ||
715 | </para></listitem> | ||
716 | <listitem><para> | ||
717 | OSC (<filename>osc://</filename>) | ||
718 | </para></listitem> | ||
719 | <listitem><para> | ||
720 | Mercurial (<filename>hg://</filename>) | ||
721 | </para></listitem> | ||
722 | </itemizedlist> | ||
723 | No documentation currently exists for these lesser used | ||
724 | fetcher submodules. | ||
725 | However, you might find the code helpful and readable. | ||
726 | </para> | ||
727 | </section> | ||
728 | </section> | ||
729 | |||
730 | <section id='auto-revisions'> | ||
731 | <title>Auto Revisions</title> | ||
732 | |||
733 | <para> | ||
734 | We need to document <filename>AUTOREV</filename> and | ||
735 | <filename>SRCREV_FORMAT</filename> here. | ||
736 | </para> | ||
737 | </section> | ||
738 | </chapter> | ||