diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml | 868 |
1 files changed, 0 insertions, 868 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml deleted file mode 100644 index d1bfc23362..0000000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml +++ /dev/null | |||
@@ -1,868 +0,0 @@ | |||
1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> | ||
3 | |||
4 | <chapter> | ||
5 | <title>File Download Support</title> | ||
6 | |||
7 | <para> | ||
8 | BitBake's fetch module is a standalone piece of library code | ||
9 | that deals with the intricacies of downloading source code | ||
10 | and files from remote systems. | ||
11 | Fetching source code is one of the cornerstones of building software. | ||
12 | As such, this module forms an important part of BitBake. | ||
13 | </para> | ||
14 | |||
15 | <para> | ||
16 | The current fetch module is called "fetch2" and refers to the | ||
17 | fact that it is the second major version of the API. | ||
18 | The original version is obsolete and has been removed from the codebase. | ||
19 | Thus, in all cases, "fetch" refers to "fetch2" in this | ||
20 | manual. | ||
21 | </para> | ||
22 | |||
23 | <section id='the-download-fetch'> | ||
24 | <title>The Download (Fetch)</title> | ||
25 | |||
26 | <para> | ||
27 | BitBake takes several steps when fetching source code or files. | ||
28 | The fetcher codebase deals with two distinct processes in order: | ||
29 | obtaining the files from somewhere (cached or otherwise) | ||
30 | and then unpacking those files into a specific location and | ||
31 | perhaps in a specific way. | ||
32 | Getting and unpacking the files is often optionally followed | ||
33 | by patching. | ||
34 | Patching, however, is not covered by this module. | ||
35 | </para> | ||
36 | |||
37 | <para> | ||
38 | The code to execute the first part of this process, a fetch, | ||
39 | looks something like the following: | ||
40 | <literallayout class='monospaced'> | ||
41 | src_uri = (d.getVar('SRC_URI') or "").split() | ||
42 | fetcher = bb.fetch2.Fetch(src_uri, d) | ||
43 | fetcher.download() | ||
44 | </literallayout> | ||
45 | This code sets up an instance of the fetch class. | ||
46 | The instance uses a space-separated list of URLs from the | ||
47 | <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> | ||
48 | variable and then calls the <filename>download</filename> | ||
49 | method to download the files. | ||
50 | </para> | ||
51 | |||
52 | <para> | ||
53 | The instantiation of the fetch class is usually followed by: | ||
54 | <literallayout class='monospaced'> | ||
55 | rootdir = l.getVar('WORKDIR') | ||
56 | fetcher.unpack(rootdir) | ||
57 | </literallayout> | ||
58 | This code unpacks the downloaded files to the | ||
59 | specified by <filename>WORKDIR</filename>. | ||
60 | <note> | ||
61 | For convenience, the naming in these examples matches | ||
62 | the variables used by OpenEmbedded. | ||
63 | If you want to see the above code in action, examine | ||
64 | the OpenEmbedded class file <filename>base.bbclass</filename>. | ||
65 | </note> | ||
66 | The <filename>SRC_URI</filename> and <filename>WORKDIR</filename> | ||
67 | variables are not hardcoded into the fetcher, since those fetcher | ||
68 | methods can be (and are) called with different variable names. | ||
69 | In OpenEmbedded for example, the shared state (sstate) code uses | ||
70 | the fetch module to fetch the sstate files. | ||
71 | </para> | ||
72 | |||
73 | <para> | ||
74 | When the <filename>download()</filename> method is called, | ||
75 | BitBake tries to resolve the URLs by looking for source files | ||
76 | in a specific search order: | ||
77 | <itemizedlist> | ||
78 | <listitem><para><emphasis>Pre-mirror Sites:</emphasis> | ||
79 | BitBake first uses pre-mirrors to try and find source files. | ||
80 | These locations are defined using the | ||
81 | <link linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link> | ||
82 | variable. | ||
83 | </para></listitem> | ||
84 | <listitem><para><emphasis>Source URI:</emphasis> | ||
85 | If pre-mirrors fail, BitBake uses the original URL (e.g from | ||
86 | <filename>SRC_URI</filename>). | ||
87 | </para></listitem> | ||
88 | <listitem><para><emphasis>Mirror Sites:</emphasis> | ||
89 | If fetch failures occur, BitBake next uses mirror locations as | ||
90 | defined by the | ||
91 | <link linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> | ||
92 | variable. | ||
93 | </para></listitem> | ||
94 | </itemizedlist> | ||
95 | </para> | ||
96 | |||
97 | <para> | ||
98 | For each URL passed to the fetcher, the fetcher | ||
99 | calls the submodule that handles that particular URL type. | ||
100 | This behavior can be the source of some confusion when you | ||
101 | are providing URLs for the <filename>SRC_URI</filename> | ||
102 | variable. | ||
103 | Consider the following two URLs: | ||
104 | <literallayout class='monospaced'> | ||
105 | http://git.yoctoproject.org/git/poky;protocol=git | ||
106 | git://git.yoctoproject.org/git/poky;protocol=http | ||
107 | </literallayout> | ||
108 | In the former case, the URL is passed to the | ||
109 | <filename>wget</filename> fetcher, which does not | ||
110 | understand "git". | ||
111 | Therefore, the latter case is the correct form since the | ||
112 | Git fetcher does know how to use HTTP as a transport. | ||
113 | </para> | ||
114 | |||
115 | <para> | ||
116 | Here are some examples that show commonly used mirror | ||
117 | definitions: | ||
118 | <literallayout class='monospaced'> | ||
119 | PREMIRRORS ?= "\ | ||
120 | bzr://.*/.* http://somemirror.org/sources/ \n \ | ||
121 | cvs://.*/.* http://somemirror.org/sources/ \n \ | ||
122 | git://.*/.* http://somemirror.org/sources/ \n \ | ||
123 | hg://.*/.* http://somemirror.org/sources/ \n \ | ||
124 | osc://.*/.* http://somemirror.org/sources/ \n \ | ||
125 | p4://.*/.* http://somemirror.org/sources/ \n \ | ||
126 | svn://.*/.* http://somemirror.org/sources/ \n" | ||
127 | |||
128 | MIRRORS =+ "\ | ||
129 | ftp://.*/.* http://somemirror.org/sources/ \n \ | ||
130 | http://.*/.* http://somemirror.org/sources/ \n \ | ||
131 | https://.*/.* http://somemirror.org/sources/ \n" | ||
132 | </literallayout> | ||
133 | It is useful to note that BitBake supports | ||
134 | cross-URLs. | ||
135 | It is possible to mirror a Git repository on an HTTP | ||
136 | server as a tarball. | ||
137 | This is what the <filename>git://</filename> mapping in | ||
138 | the previous example does. | ||
139 | </para> | ||
140 | |||
141 | <para> | ||
142 | Since network accesses are slow, BitBake maintains a | ||
143 | cache of files downloaded from the network. | ||
144 | Any source files that are not local (i.e. | ||
145 | downloaded from the Internet) are placed into the download | ||
146 | directory, which is specified by the | ||
147 | <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> | ||
148 | variable. | ||
149 | </para> | ||
150 | |||
151 | <para> | ||
152 | File integrity is of key importance for reproducing builds. | ||
153 | For non-local archive downloads, the fetcher code can verify | ||
154 | SHA-256 and MD5 checksums to ensure the archives have been | ||
155 | downloaded correctly. | ||
156 | You can specify these checksums by using the | ||
157 | <filename>SRC_URI</filename> variable with the appropriate | ||
158 | varflags as follows: | ||
159 | <literallayout class='monospaced'> | ||
160 | SRC_URI[md5sum] = "<replaceable>value</replaceable>" | ||
161 | SRC_URI[sha256sum] = "<replaceable>value</replaceable>" | ||
162 | </literallayout> | ||
163 | You can also specify the checksums as parameters on the | ||
164 | <filename>SRC_URI</filename> as shown below: | ||
165 | <literallayout class='monospaced'> | ||
166 | SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" | ||
167 | </literallayout> | ||
168 | If multiple URIs exist, you can specify the checksums either | ||
169 | directly as in the previous example, or you can name the URLs. | ||
170 | The following syntax shows how you name the URIs: | ||
171 | <literallayout class='monospaced'> | ||
172 | SRC_URI = "http://example.com/foobar.tar.bz2;name=foo" | ||
173 | SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d | ||
174 | </literallayout> | ||
175 | After a file has been downloaded and has had its checksum checked, | ||
176 | a ".done" stamp is placed in <filename>DL_DIR</filename>. | ||
177 | BitBake uses this stamp during subsequent builds to avoid | ||
178 | downloading or comparing a checksum for the file again. | ||
179 | <note> | ||
180 | It is assumed that local storage is safe from data corruption. | ||
181 | If this were not the case, there would be bigger issues to worry about. | ||
182 | </note> | ||
183 | </para> | ||
184 | |||
185 | <para> | ||
186 | If | ||
187 | <link linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link> | ||
188 | is set, any download without a checksum triggers an | ||
189 | error message. | ||
190 | The | ||
191 | <link linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link> | ||
192 | variable can be used to make any attempted network access a fatal | ||
193 | error, which is useful for checking that mirrors are complete | ||
194 | as well as other things. | ||
195 | </para> | ||
196 | </section> | ||
197 | |||
198 | <section id='bb-the-unpack'> | ||
199 | <title>The Unpack</title> | ||
200 | |||
201 | <para> | ||
202 | The unpack process usually immediately follows the download. | ||
203 | For all URLs except Git URLs, BitBake uses the common | ||
204 | <filename>unpack</filename> method. | ||
205 | </para> | ||
206 | |||
207 | <para> | ||
208 | A number of parameters exist that you can specify within the | ||
209 | URL to govern the behavior of the unpack stage: | ||
210 | <itemizedlist> | ||
211 | <listitem><para><emphasis>unpack:</emphasis> | ||
212 | Controls whether the URL components are unpacked. | ||
213 | If set to "1", which is the default, the components | ||
214 | are unpacked. | ||
215 | If set to "0", the unpack stage leaves the file alone. | ||
216 | This parameter is useful when you want an archive to be | ||
217 | copied in and not be unpacked. | ||
218 | </para></listitem> | ||
219 | <listitem><para><emphasis>dos:</emphasis> | ||
220 | Applies to <filename>.zip</filename> and | ||
221 | <filename>.jar</filename> files and specifies whether to | ||
222 | use DOS line ending conversion on text files. | ||
223 | </para></listitem> | ||
224 | <listitem><para><emphasis>basepath:</emphasis> | ||
225 | Instructs the unpack stage to strip the specified | ||
226 | directories from the source path when unpacking. | ||
227 | </para></listitem> | ||
228 | <listitem><para><emphasis>subdir:</emphasis> | ||
229 | Unpacks the specific URL to the specified subdirectory | ||
230 | within the root directory. | ||
231 | </para></listitem> | ||
232 | </itemizedlist> | ||
233 | The unpack call automatically decompresses and extracts files | ||
234 | with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". | ||
235 | ".srpm", ".deb" and ".bz2" extensions as well as various combinations | ||
236 | of tarball extensions. | ||
237 | </para> | ||
238 | |||
239 | <para> | ||
240 | As mentioned, the Git fetcher has its own unpack method that | ||
241 | is optimized to work with Git trees. | ||
242 | Basically, this method works by cloning the tree into the final | ||
243 | directory. | ||
244 | The process is completed using references so that there is | ||
245 | only one central copy of the Git metadata needed. | ||
246 | </para> | ||
247 | </section> | ||
248 | |||
249 | <section id='bb-fetchers'> | ||
250 | <title>Fetchers</title> | ||
251 | |||
252 | <para> | ||
253 | As mentioned earlier, the URL prefix determines which | ||
254 | fetcher submodule BitBake uses. | ||
255 | Each submodule can support different URL parameters, | ||
256 | which are described in the following sections. | ||
257 | </para> | ||
258 | |||
259 | <section id='local-file-fetcher'> | ||
260 | <title>Local file fetcher (<filename>file://</filename>)</title> | ||
261 | |||
262 | <para> | ||
263 | This submodule handles URLs that begin with | ||
264 | <filename>file://</filename>. | ||
265 | The filename you specify within the URL can be | ||
266 | either an absolute or relative path to a file. | ||
267 | If the filename is relative, the contents of the | ||
268 | <link linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link> | ||
269 | variable is used in the same way | ||
270 | <filename>PATH</filename> is used to find executables. | ||
271 | If the file cannot be found, it is assumed that it is available in | ||
272 | <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> | ||
273 | by the time the <filename>download()</filename> method is called. | ||
274 | </para> | ||
275 | |||
276 | <para> | ||
277 | If you specify a directory, the entire directory is | ||
278 | unpacked. | ||
279 | </para> | ||
280 | |||
281 | <para> | ||
282 | Here are a couple of example URLs, the first relative and | ||
283 | the second absolute: | ||
284 | <literallayout class='monospaced'> | ||
285 | SRC_URI = "file://relativefile.patch" | ||
286 | SRC_URI = "file:///Users/ich/very_important_software" | ||
287 | </literallayout> | ||
288 | </para> | ||
289 | </section> | ||
290 | |||
291 | <section id='http-ftp-fetcher'> | ||
292 | <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title> | ||
293 | |||
294 | <para> | ||
295 | This fetcher obtains files from web and FTP servers. | ||
296 | Internally, the fetcher uses the wget utility. | ||
297 | </para> | ||
298 | |||
299 | <para> | ||
300 | The executable and parameters used are specified by the | ||
301 | <filename>FETCHCMD_wget</filename> variable, which defaults | ||
302 | to sensible values. | ||
303 | The fetcher supports a parameter "downloadfilename" that | ||
304 | allows the name of the downloaded file to be specified. | ||
305 | Specifying the name of the downloaded file is useful | ||
306 | for avoiding collisions in | ||
307 | <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> | ||
308 | when dealing with multiple files that have the same name. | ||
309 | </para> | ||
310 | |||
311 | <para> | ||
312 | Some example URLs are as follows: | ||
313 | <literallayout class='monospaced'> | ||
314 | SRC_URI = "http://oe.handhelds.org/not_there.aac" | ||
315 | SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac" | ||
316 | SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan" | ||
317 | </literallayout> | ||
318 | </para> | ||
319 | <note> | ||
320 | Because URL parameters are delimited by semi-colons, this can | ||
321 | introduce ambiguity when parsing URLs that also contain semi-colons, | ||
322 | for example: | ||
323 | <literallayout class='monospaced'> | ||
324 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" | ||
325 | </literallayout> | ||
326 | Such URLs should should be modified by replacing semi-colons with '&' characters: | ||
327 | <literallayout class='monospaced'> | ||
328 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" | ||
329 | </literallayout> | ||
330 | In most cases this should work. Treating semi-colons and '&' in queries | ||
331 | identically is recommended by the World Wide Web Consortium (W3C). | ||
332 | Note that due to the nature of the URL, you may have to specify the name | ||
333 | of the downloaded file as well: | ||
334 | <literallayout class='monospaced'> | ||
335 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" | ||
336 | </literallayout> | ||
337 | </note> | ||
338 | </section> | ||
339 | |||
340 | <section id='cvs-fetcher'> | ||
341 | <title>CVS fetcher (<filename>(cvs://</filename>)</title> | ||
342 | |||
343 | <para> | ||
344 | This submodule handles checking out files from the | ||
345 | CVS version control system. | ||
346 | You can configure it using a number of different variables: | ||
347 | <itemizedlist> | ||
348 | <listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis> | ||
349 | The name of the executable to use when running | ||
350 | the <filename>cvs</filename> command. | ||
351 | This name is usually "cvs". | ||
352 | </para></listitem> | ||
353 | <listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis> | ||
354 | The date to use when fetching the CVS source code. | ||
355 | A special value of "now" causes the checkout to | ||
356 | be updated on every build. | ||
357 | </para></listitem> | ||
358 | <listitem><para><emphasis><link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis> | ||
359 | Specifies where a temporary checkout is saved. | ||
360 | The location is often <filename>DL_DIR/cvs</filename>. | ||
361 | </para></listitem> | ||
362 | <listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis> | ||
363 | The name to use as a "proxy=" parameter to the | ||
364 | <filename>cvs</filename> command. | ||
365 | </para></listitem> | ||
366 | <listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis> | ||
367 | The port number to use as a "proxyport=" parameter to | ||
368 | the <filename>cvs</filename> command. | ||
369 | </para></listitem> | ||
370 | </itemizedlist> | ||
371 | As well as the standard username and password URL syntax, | ||
372 | you can also configure the fetcher with various URL parameters: | ||
373 | </para> | ||
374 | |||
375 | <para> | ||
376 | The supported parameters are as follows: | ||
377 | <itemizedlist> | ||
378 | <listitem><para><emphasis>"method":</emphasis> | ||
379 | The protocol over which to communicate with the CVS | ||
380 | server. | ||
381 | By default, this protocol is "pserver". | ||
382 | If "method" is set to "ext", BitBake examines the | ||
383 | "rsh" parameter and sets <filename>CVS_RSH</filename>. | ||
384 | You can use "dir" for local directories. | ||
385 | </para></listitem> | ||
386 | <listitem><para><emphasis>"module":</emphasis> | ||
387 | Specifies the module to check out. | ||
388 | You must supply this parameter. | ||
389 | </para></listitem> | ||
390 | <listitem><para><emphasis>"tag":</emphasis> | ||
391 | Describes which CVS TAG should be used for | ||
392 | the checkout. | ||
393 | By default, the TAG is empty. | ||
394 | </para></listitem> | ||
395 | <listitem><para><emphasis>"date":</emphasis> | ||
396 | Specifies a date. | ||
397 | If no "date" is specified, the | ||
398 | <link linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link> | ||
399 | of the configuration is used to checkout a specific date. | ||
400 | The special value of "now" causes the checkout to be | ||
401 | updated on every build. | ||
402 | </para></listitem> | ||
403 | <listitem><para><emphasis>"localdir":</emphasis> | ||
404 | Used to rename the module. | ||
405 | Effectively, you are renaming the output directory | ||
406 | to which the module is unpacked. | ||
407 | You are forcing the module into a special | ||
408 | directory relative to | ||
409 | <link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>. | ||
410 | </para></listitem> | ||
411 | <listitem><para><emphasis>"rsh"</emphasis> | ||
412 | Used in conjunction with the "method" parameter. | ||
413 | </para></listitem> | ||
414 | <listitem><para><emphasis>"scmdata":</emphasis> | ||
415 | Causes the CVS metadata to be maintained in the tarball | ||
416 | the fetcher creates when set to "keep". | ||
417 | The tarball is expanded into the work directory. | ||
418 | By default, the CVS metadata is removed. | ||
419 | </para></listitem> | ||
420 | <listitem><para><emphasis>"fullpath":</emphasis> | ||
421 | Controls whether the resulting checkout is at the | ||
422 | module level, which is the default, or is at deeper | ||
423 | paths. | ||
424 | </para></listitem> | ||
425 | <listitem><para><emphasis>"norecurse":</emphasis> | ||
426 | Causes the fetcher to only checkout the specified | ||
427 | directory with no recurse into any subdirectories. | ||
428 | </para></listitem> | ||
429 | <listitem><para><emphasis>"port":</emphasis> | ||
430 | The port to which the CVS server connects. | ||
431 | </para></listitem> | ||
432 | </itemizedlist> | ||
433 | Some example URLs are as follows: | ||
434 | <literallayout class='monospaced'> | ||
435 | SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" | ||
436 | SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" | ||
437 | </literallayout> | ||
438 | </para> | ||
439 | </section> | ||
440 | |||
441 | <section id='svn-fetcher'> | ||
442 | <title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title> | ||
443 | |||
444 | <para> | ||
445 | This fetcher submodule fetches code from the | ||
446 | Subversion source control system. | ||
447 | The executable used is specified by | ||
448 | <filename>FETCHCMD_svn</filename>, which defaults | ||
449 | to "svn". | ||
450 | The fetcher's temporary working directory is set by | ||
451 | <link linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>, | ||
452 | which is usually <filename>DL_DIR/svn</filename>. | ||
453 | </para> | ||
454 | |||
455 | <para> | ||
456 | The supported parameters are as follows: | ||
457 | <itemizedlist> | ||
458 | <listitem><para><emphasis>"module":</emphasis> | ||
459 | The name of the svn module to checkout. | ||
460 | You must provide this parameter. | ||
461 | You can think of this parameter as the top-level | ||
462 | directory of the repository data you want. | ||
463 | </para></listitem> | ||
464 | <listitem><para><emphasis>"path_spec":</emphasis> | ||
465 | A specific directory in which to checkout the | ||
466 | specified svn module. | ||
467 | </para></listitem> | ||
468 | <listitem><para><emphasis>"protocol":</emphasis> | ||
469 | The protocol to use, which defaults to "svn". | ||
470 | If "protocol" is set to "svn+ssh", the "ssh" | ||
471 | parameter is also used. | ||
472 | </para></listitem> | ||
473 | <listitem><para><emphasis>"rev":</emphasis> | ||
474 | The revision of the source code to checkout. | ||
475 | </para></listitem> | ||
476 | <listitem><para><emphasis>"scmdata":</emphasis> | ||
477 | Causes the “.svn” directories to be available during | ||
478 | compile-time when set to "keep". | ||
479 | By default, these directories are removed. | ||
480 | </para></listitem> | ||
481 | <listitem><para><emphasis>"ssh":</emphasis> | ||
482 | An optional parameter used when "protocol" is set | ||
483 | to "svn+ssh". | ||
484 | You can use this parameter to specify the ssh | ||
485 | program used by svn. | ||
486 | </para></listitem> | ||
487 | <listitem><para><emphasis>"transportuser":</emphasis> | ||
488 | When required, sets the username for the transport. | ||
489 | By default, this parameter is empty. | ||
490 | The transport username is different than the username | ||
491 | used in the main URL, which is passed to the subversion | ||
492 | command. | ||
493 | </para></listitem> | ||
494 | </itemizedlist> | ||
495 | Following are three examples using svn: | ||
496 | <literallayout class='monospaced'> | ||
497 | SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667" | ||
498 | SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh" | ||
499 | SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1" | ||
500 | </literallayout> | ||
501 | </para> | ||
502 | </section> | ||
503 | |||
504 | <section id='git-fetcher'> | ||
505 | <title>Git Fetcher (<filename>git://</filename>)</title> | ||
506 | |||
507 | <para> | ||
508 | This fetcher submodule fetches code from the Git | ||
509 | source control system. | ||
510 | The fetcher works by creating a bare clone of the | ||
511 | remote into | ||
512 | <link linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>, | ||
513 | which is usually <filename>DL_DIR/git2</filename>. | ||
514 | This bare clone is then cloned into the work directory during the | ||
515 | unpack stage when a specific tree is checked out. | ||
516 | This is done using alternates and by reference to | ||
517 | minimize the amount of duplicate data on the disk and | ||
518 | make the unpack process fast. | ||
519 | The executable used can be set with | ||
520 | <filename>FETCHCMD_git</filename>. | ||
521 | </para> | ||
522 | |||
523 | <para> | ||
524 | This fetcher supports the following parameters: | ||
525 | <itemizedlist> | ||
526 | <listitem><para><emphasis>"protocol":</emphasis> | ||
527 | The protocol used to fetch the files. | ||
528 | The default is "git" when a hostname is set. | ||
529 | If a hostname is not set, the Git protocol is "file". | ||
530 | You can also use "http", "https", "ssh" and "rsync". | ||
531 | </para></listitem> | ||
532 | <listitem><para><emphasis>"nocheckout":</emphasis> | ||
533 | Tells the fetcher to not checkout source code when | ||
534 | unpacking when set to "1". | ||
535 | Set this option for the URL where there is a custom | ||
536 | routine to checkout code. | ||
537 | The default is "0". | ||
538 | </para></listitem> | ||
539 | <listitem><para><emphasis>"rebaseable":</emphasis> | ||
540 | Indicates that the upstream Git repository can be rebased. | ||
541 | You should set this parameter to "1" if | ||
542 | revisions can become detached from branches. | ||
543 | In this case, the source mirror tarball is done per | ||
544 | revision, which has a loss of efficiency. | ||
545 | Rebasing the upstream Git repository could cause the | ||
546 | current revision to disappear from the upstream repository. | ||
547 | This option reminds the fetcher to preserve the local cache | ||
548 | carefully for future use. | ||
549 | The default value for this parameter is "0". | ||
550 | </para></listitem> | ||
551 | <listitem><para><emphasis>"nobranch":</emphasis> | ||
552 | Tells the fetcher to not check the SHA validation | ||
553 | for the branch when set to "1". | ||
554 | The default is "0". | ||
555 | Set this option for the recipe that refers to | ||
556 | the commit that is valid for a tag instead of | ||
557 | the branch. | ||
558 | </para></listitem> | ||
559 | <listitem><para><emphasis>"bareclone":</emphasis> | ||
560 | Tells the fetcher to clone a bare clone into the | ||
561 | destination directory without checking out a working tree. | ||
562 | Only the raw Git metadata is provided. | ||
563 | This parameter implies the "nocheckout" parameter as well. | ||
564 | </para></listitem> | ||
565 | <listitem><para><emphasis>"branch":</emphasis> | ||
566 | The branch(es) of the Git tree to clone. | ||
567 | If unset, this is assumed to be "master". | ||
568 | The number of branch parameters much match the number of | ||
569 | name parameters. | ||
570 | </para></listitem> | ||
571 | <listitem><para><emphasis>"rev":</emphasis> | ||
572 | The revision to use for the checkout. | ||
573 | The default is "master". | ||
574 | </para></listitem> | ||
575 | <listitem><para><emphasis>"tag":</emphasis> | ||
576 | Specifies a tag to use for the checkout. | ||
577 | To correctly resolve tags, BitBake must access the | ||
578 | network. | ||
579 | For that reason, tags are often not used. | ||
580 | As far as Git is concerned, the "tag" parameter behaves | ||
581 | effectively the same as the "rev" parameter. | ||
582 | </para></listitem> | ||
583 | <listitem><para><emphasis>"subpath":</emphasis> | ||
584 | Limits the checkout to a specific subpath of the tree. | ||
585 | By default, the whole tree is checked out. | ||
586 | </para></listitem> | ||
587 | <listitem><para><emphasis>"destsuffix":</emphasis> | ||
588 | The name of the path in which to place the checkout. | ||
589 | By default, the path is <filename>git/</filename>. | ||
590 | </para></listitem> | ||
591 | <listitem><para><emphasis>"usehead":</emphasis> | ||
592 | Enables local <filename>git://</filename> URLs to use the | ||
593 | current branch HEAD as the revision for use with | ||
594 | <filename>AUTOREV</filename>. | ||
595 | The "usehead" parameter implies no branch and only works | ||
596 | when the transfer protocol is | ||
597 | <filename>file://</filename>. | ||
598 | </para></listitem> | ||
599 | </itemizedlist> | ||
600 | Here are some example URLs: | ||
601 | <literallayout class='monospaced'> | ||
602 | SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" | ||
603 | SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http" | ||
604 | </literallayout> | ||
605 | </para> | ||
606 | </section> | ||
607 | |||
608 | <section id='gitsm-fetcher'> | ||
609 | <title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title> | ||
610 | |||
611 | <para> | ||
612 | This fetcher submodule inherits from the | ||
613 | <link linkend='git-fetcher'>Git fetcher</link> and extends | ||
614 | that fetcher's behavior by fetching a repository's submodules. | ||
615 | <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> | ||
616 | is passed to the Git fetcher as described in the | ||
617 | "<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>" | ||
618 | section. | ||
619 | <note> | ||
620 | <title>Notes and Warnings</title> | ||
621 | <para> | ||
622 | You must clean a recipe when switching between | ||
623 | '<filename>git://</filename>' and | ||
624 | '<filename>gitsm://</filename>' URLs. | ||
625 | </para> | ||
626 | |||
627 | <para> | ||
628 | The Git Submodules fetcher is not a complete fetcher | ||
629 | implementation. | ||
630 | The fetcher has known issues where it does not use the | ||
631 | normal source mirroring infrastructure properly. Further, | ||
632 | the submodule sources it fetches are not visible to the | ||
633 | licensing and source archiving infrastructures. | ||
634 | </para> | ||
635 | </note> | ||
636 | </para> | ||
637 | </section> | ||
638 | |||
639 | <section id='clearcase-fetcher'> | ||
640 | <title>ClearCase Fetcher (<filename>ccrc://</filename>)</title> | ||
641 | |||
642 | <para> | ||
643 | This fetcher submodule fetches code from a | ||
644 | <ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink> | ||
645 | repository. | ||
646 | </para> | ||
647 | |||
648 | <para> | ||
649 | To use this fetcher, make sure your recipe has proper | ||
650 | <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>, | ||
651 | <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and | ||
652 | <link linkend='var-bb-PV'><filename>PV</filename></link> settings. | ||
653 | Here is an example: | ||
654 | <literallayout class='monospaced'> | ||
655 | SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" | ||
656 | SRCREV = "EXAMPLE_CLEARCASE_TAG" | ||
657 | PV = "${@d.getVar("SRCREV", False).replace("/", "+")}" | ||
658 | </literallayout> | ||
659 | The fetcher uses the <filename>rcleartool</filename> or | ||
660 | <filename>cleartool</filename> remote client, depending on | ||
661 | which one is available. | ||
662 | </para> | ||
663 | |||
664 | <para> | ||
665 | Following are options for the <filename>SRC_URI</filename> | ||
666 | statement: | ||
667 | <itemizedlist> | ||
668 | <listitem><para><emphasis><filename>vob</filename></emphasis>: | ||
669 | The name, which must include the | ||
670 | prepending "/" character, of the ClearCase VOB. | ||
671 | This option is required. | ||
672 | </para></listitem> | ||
673 | <listitem><para><emphasis><filename>module</filename></emphasis>: | ||
674 | The module, which must include the | ||
675 | prepending "/" character, in the selected VOB. | ||
676 | <note> | ||
677 | The <filename>module</filename> and <filename>vob</filename> | ||
678 | options are combined to create the <filename>load</filename> rule in | ||
679 | the view config spec. | ||
680 | As an example, consider the <filename>vob</filename> and | ||
681 | <filename>module</filename> values from the | ||
682 | <filename>SRC_URI</filename> statement at the start of this section. | ||
683 | Combining those values results in the following: | ||
684 | <literallayout class='monospaced'> | ||
685 | load /example_vob/example_module | ||
686 | </literallayout> | ||
687 | </note> | ||
688 | </para></listitem> | ||
689 | <listitem><para><emphasis><filename>proto</filename></emphasis>: | ||
690 | The protocol, which can be either <filename>http</filename> or | ||
691 | <filename>https</filename>. | ||
692 | </para></listitem> | ||
693 | </itemizedlist> | ||
694 | </para> | ||
695 | |||
696 | <para> | ||
697 | By default, the fetcher creates a configuration specification. | ||
698 | If you want this specification written to an area other than the default, | ||
699 | use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable | ||
700 | in your recipe to define where the specification is written. | ||
701 | <note> | ||
702 | the <filename>SRCREV</filename> loses its functionality if you | ||
703 | specify this variable. | ||
704 | However, <filename>SRCREV</filename> is still used to label the | ||
705 | archive after a fetch even though it does not define what is | ||
706 | fetched. | ||
707 | </note> | ||
708 | </para> | ||
709 | |||
710 | <para> | ||
711 | Here are a couple of other behaviors worth mentioning: | ||
712 | <itemizedlist> | ||
713 | <listitem><para> | ||
714 | When using <filename>cleartool</filename>, the login of | ||
715 | <filename>cleartool</filename> is handled by the system. | ||
716 | The login require no special steps. | ||
717 | </para></listitem> | ||
718 | <listitem><para> | ||
719 | In order to use <filename>rcleartool</filename> with authenticated | ||
720 | users, an "rcleartool login" is necessary before using the fetcher. | ||
721 | </para></listitem> | ||
722 | </itemizedlist> | ||
723 | </para> | ||
724 | </section> | ||
725 | |||
726 | <section id='perforce-fetcher'> | ||
727 | <title>Perforce Fetcher (<filename>p4://</filename>)</title> | ||
728 | |||
729 | <para> | ||
730 | This fetcher submodule fetches code from the | ||
731 | <ulink url='https://www.perforce.com/'>Perforce</ulink> | ||
732 | source control system. | ||
733 | The executable used is specified by | ||
734 | <filename>FETCHCMD_p4</filename>, which defaults | ||
735 | to "p4". | ||
736 | The fetcher's temporary working directory is set by | ||
737 | <link linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>, | ||
738 | which defaults to "DL_DIR/p4". | ||
739 | </para> | ||
740 | |||
741 | <para> | ||
742 | To use this fetcher, make sure your recipe has proper | ||
743 | <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>, | ||
744 | <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and | ||
745 | <link linkend='var-bb-PV'><filename>PV</filename></link> values. | ||
746 | The p4 executable is able to use the config file defined by your | ||
747 | system's <filename>P4CONFIG</filename> environment variable in | ||
748 | order to define the Perforce server URL and port, username, and | ||
749 | password if you do not wish to keep those values in a recipe | ||
750 | itself. | ||
751 | If you choose not to use <filename>P4CONFIG</filename>, | ||
752 | or to explicitly set variables that <filename>P4CONFIG</filename> | ||
753 | can contain, you can specify the <filename>P4PORT</filename> value, | ||
754 | which is the server's URL and port number, and you can | ||
755 | specify a username and password directly in your recipe within | ||
756 | <filename>SRC_URI</filename>. | ||
757 | </para> | ||
758 | |||
759 | <para> | ||
760 | Here is an example that relies on <filename>P4CONFIG</filename> | ||
761 | to specify the server URL and port, username, and password, and | ||
762 | fetches the Head Revision: | ||
763 | <literallayout class='monospaced'> | ||
764 | SRC_URI = "p4://example-depot/main/source/..." | ||
765 | SRCREV = "${AUTOREV}" | ||
766 | PV = "p4-${SRCPV}" | ||
767 | S = "${WORKDIR}/p4" | ||
768 | </literallayout> | ||
769 | </para> | ||
770 | |||
771 | <para> | ||
772 | Here is an example that specifies the server URL and port, | ||
773 | username, and password, and fetches a Revision based on a Label: | ||
774 | <literallayout class='monospaced'> | ||
775 | P4PORT = "tcp:p4server.example.net:1666" | ||
776 | SRC_URI = "p4://user:passwd@example-depot/main/source/..." | ||
777 | SRCREV = "release-1.0" | ||
778 | PV = "p4-${SRCPV}" | ||
779 | S = "${WORKDIR}/p4" | ||
780 | </literallayout> | ||
781 | <note> | ||
782 | You should always set <filename>S</filename> | ||
783 | to <filename>"${WORKDIR}/p4"</filename> in your recipe. | ||
784 | </note> | ||
785 | </para> | ||
786 | </section> | ||
787 | |||
788 | <section id='repo-fetcher'> | ||
789 | <title>Repo Fetcher (<filename>repo://</filename>)</title> | ||
790 | |||
791 | <para> | ||
792 | This fetcher submodule fetches code from | ||
793 | <filename>google-repo</filename> source control system. | ||
794 | The fetcher works by initiating and syncing sources of the | ||
795 | repository into | ||
796 | <link linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>, | ||
797 | which is usually | ||
798 | <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>. | ||
799 | </para> | ||
800 | |||
801 | <para> | ||
802 | This fetcher supports the following parameters: | ||
803 | <itemizedlist> | ||
804 | <listitem><para> | ||
805 | <emphasis>"protocol":</emphasis> | ||
806 | Protocol to fetch the repository manifest (default: git). | ||
807 | </para></listitem> | ||
808 | <listitem><para> | ||
809 | <emphasis>"branch":</emphasis> | ||
810 | Branch or tag of repository to get (default: master). | ||
811 | </para></listitem> | ||
812 | <listitem><para> | ||
813 | <emphasis>"manifest":</emphasis> | ||
814 | Name of the manifest file (default: <filename>default.xml</filename>). | ||
815 | </para></listitem> | ||
816 | </itemizedlist> | ||
817 | Here are some example URLs: | ||
818 | <literallayout class='monospaced'> | ||
819 | SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml" | ||
820 | SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml" | ||
821 | </literallayout> | ||
822 | </para> | ||
823 | </section> | ||
824 | |||
825 | <section id='other-fetchers'> | ||
826 | <title>Other Fetchers</title> | ||
827 | |||
828 | <para> | ||
829 | Fetch submodules also exist for the following: | ||
830 | <itemizedlist> | ||
831 | <listitem><para> | ||
832 | Bazaar (<filename>bzr://</filename>) | ||
833 | </para></listitem> | ||
834 | <listitem><para> | ||
835 | Mercurial (<filename>hg://</filename>) | ||
836 | </para></listitem> | ||
837 | <listitem><para> | ||
838 | npm (<filename>npm://</filename>) | ||
839 | </para></listitem> | ||
840 | <listitem><para> | ||
841 | OSC (<filename>osc://</filename>) | ||
842 | </para></listitem> | ||
843 | <listitem><para> | ||
844 | Secure FTP (<filename>sftp://</filename>) | ||
845 | </para></listitem> | ||
846 | <listitem><para> | ||
847 | Secure Shell (<filename>ssh://</filename>) | ||
848 | </para></listitem> | ||
849 | <listitem><para> | ||
850 | Trees using Git Annex (<filename>gitannex://</filename>) | ||
851 | </para></listitem> | ||
852 | </itemizedlist> | ||
853 | No documentation currently exists for these lesser used | ||
854 | fetcher submodules. | ||
855 | However, you might find the code helpful and readable. | ||
856 | </para> | ||
857 | </section> | ||
858 | </section> | ||
859 | |||
860 | <section id='auto-revisions'> | ||
861 | <title>Auto Revisions</title> | ||
862 | |||
863 | <para> | ||
864 | We need to document <filename>AUTOREV</filename> and | ||
865 | <filename>SRCREV_FORMAT</filename> here. | ||
866 | </para> | ||
867 | </section> | ||
868 | </chapter> | ||