diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 629 |
1 files changed, 629 insertions, 0 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst new file mode 100644 index 0000000000..4285a73afe --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | |||
@@ -0,0 +1,629 @@ | |||
1 | ===================== | ||
2 | File Download Support | ||
3 | ===================== | ||
4 | |||
5 | BitBake's fetch module is a standalone piece of library code that deals | ||
6 | with the intricacies of downloading source code and files from remote | ||
7 | systems. Fetching source code is one of the cornerstones of building | ||
8 | software. As such, this module forms an important part of BitBake. | ||
9 | |||
10 | The current fetch module is called "fetch2" and refers to the fact that | ||
11 | it is the second major version of the API. The original version is | ||
12 | obsolete and has been removed from the codebase. Thus, in all cases, | ||
13 | "fetch" refers to "fetch2" in this manual. | ||
14 | |||
15 | The Download (Fetch) | ||
16 | ==================== | ||
17 | |||
18 | BitBake takes several steps when fetching source code or files. The | ||
19 | fetcher codebase deals with two distinct processes in order: obtaining | ||
20 | the files from somewhere (cached or otherwise) and then unpacking those | ||
21 | files into a specific location and perhaps in a specific way. Getting | ||
22 | and unpacking the files is often optionally followed by patching. | ||
23 | Patching, however, is not covered by this module. | ||
24 | |||
25 | The code to execute the first part of this process, a fetch, looks | ||
26 | something like the following: src_uri = (d.getVar('SRC_URI') or | ||
27 | "").split() fetcher = bb.fetch2.Fetch(src_uri, d) fetcher.download() | ||
28 | This code sets up an instance of the fetch class. The instance uses a | ||
29 | space-separated list of URLs from the ```SRC_URI`` <#var-bb-SRC_URI>`__ | ||
30 | variable and then calls the ``download`` method to download the files. | ||
31 | |||
32 | The instantiation of the fetch class is usually followed by: rootdir = | ||
33 | l.getVar('WORKDIR') fetcher.unpack(rootdir) This code unpacks the | ||
34 | downloaded files to the specified by ``WORKDIR``. | ||
35 | |||
36 | .. note:: | ||
37 | |||
38 | For convenience, the naming in these examples matches the variables | ||
39 | used by OpenEmbedded. If you want to see the above code in action, | ||
40 | examine the OpenEmbedded class file | ||
41 | base.bbclass | ||
42 | . | ||
43 | |||
44 | The ``SRC_URI`` and ``WORKDIR`` variables are not hardcoded into the | ||
45 | fetcher, since those fetcher methods can be (and are) called with | ||
46 | different variable names. In OpenEmbedded for example, the shared state | ||
47 | (sstate) code uses the fetch module to fetch the sstate files. | ||
48 | |||
49 | When the ``download()`` method is called, BitBake tries to resolve the | ||
50 | URLs by looking for source files in a specific search order: | ||
51 | |||
52 | - *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find | ||
53 | source files. These locations are defined using the | ||
54 | ```PREMIRRORS`` <#var-bb-PREMIRRORS>`__ variable. | ||
55 | |||
56 | - *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g | ||
57 | from ``SRC_URI``). | ||
58 | |||
59 | - *Mirror Sites:* If fetch failures occur, BitBake next uses mirror | ||
60 | locations as defined by the ```MIRRORS`` <#var-bb-MIRRORS>`__ | ||
61 | variable. | ||
62 | |||
63 | For each URL passed to the fetcher, the fetcher calls the submodule that | ||
64 | handles that particular URL type. This behavior can be the source of | ||
65 | some confusion when you are providing URLs for the ``SRC_URI`` variable. | ||
66 | Consider the following two URLs: | ||
67 | http://git.yoctoproject.org/git/poky;protocol=git | ||
68 | git://git.yoctoproject.org/git/poky;protocol=http In the former case, | ||
69 | the URL is passed to the ``wget`` fetcher, which does not understand | ||
70 | "git". Therefore, the latter case is the correct form since the Git | ||
71 | fetcher does know how to use HTTP as a transport. | ||
72 | |||
73 | Here are some examples that show commonly used mirror definitions: | ||
74 | PREMIRRORS ?= "\\ bzr://.*/.\* http://somemirror.org/sources/ \\n \\ | ||
75 | cvs://.*/.\* http://somemirror.org/sources/ \\n \\ git://.*/.\* | ||
76 | http://somemirror.org/sources/ \\n \\ hg://.*/.\* | ||
77 | http://somemirror.org/sources/ \\n \\ osc://.*/.\* | ||
78 | http://somemirror.org/sources/ \\n \\ p4://.*/.\* | ||
79 | http://somemirror.org/sources/ \\n \\ svn://.*/.\* | ||
80 | http://somemirror.org/sources/ \\n" MIRRORS =+ "\\ ftp://.*/.\* | ||
81 | http://somemirror.org/sources/ \\n \\ http://.*/.\* | ||
82 | http://somemirror.org/sources/ \\n \\ https://.*/.\* | ||
83 | http://somemirror.org/sources/ \\n" It is useful to note that BitBake | ||
84 | supports cross-URLs. It is possible to mirror a Git repository on an | ||
85 | HTTP server as a tarball. This is what the ``git://`` mapping in the | ||
86 | previous example does. | ||
87 | |||
88 | Since network accesses are slow, BitBake maintains a cache of files | ||
89 | downloaded from the network. Any source files that are not local (i.e. | ||
90 | downloaded from the Internet) are placed into the download directory, | ||
91 | which is specified by the ```DL_DIR`` <#var-bb-DL_DIR>`__ variable. | ||
92 | |||
93 | File integrity is of key importance for reproducing builds. For | ||
94 | non-local archive downloads, the fetcher code can verify SHA-256 and MD5 | ||
95 | checksums to ensure the archives have been downloaded correctly. You can | ||
96 | specify these checksums by using the ``SRC_URI`` variable with the | ||
97 | appropriate varflags as follows: SRC_URI[md5sum] = "value" | ||
98 | SRC_URI[sha256sum] = "value" You can also specify the checksums as | ||
99 | parameters on the ``SRC_URI`` as shown below: SRC_URI = | ||
100 | "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" | ||
101 | If multiple URIs exist, you can specify the checksums either directly as | ||
102 | in the previous example, or you can name the URLs. The following syntax | ||
103 | shows how you name the URIs: SRC_URI = | ||
104 | "http://example.com/foobar.tar.bz2;name=foo" SRC_URI[foo.md5sum] = | ||
105 | 4a8e0f237e961fd7785d19d07fdb994d After a file has been downloaded and | ||
106 | has had its checksum checked, a ".done" stamp is placed in ``DL_DIR``. | ||
107 | BitBake uses this stamp during subsequent builds to avoid downloading or | ||
108 | comparing a checksum for the file again. | ||
109 | |||
110 | .. note:: | ||
111 | |||
112 | It is assumed that local storage is safe from data corruption. If | ||
113 | this were not the case, there would be bigger issues to worry about. | ||
114 | |||
115 | If ```BB_STRICT_CHECKSUM`` <#var-bb-BB_STRICT_CHECKSUM>`__ is set, any | ||
116 | download without a checksum triggers an error message. The | ||
117 | ```BB_NO_NETWORK`` <#var-bb-BB_NO_NETWORK>`__ variable can be used to | ||
118 | make any attempted network access a fatal error, which is useful for | ||
119 | checking that mirrors are complete as well as other things. | ||
120 | |||
121 | .. _bb-the-unpack: | ||
122 | |||
123 | The Unpack | ||
124 | ========== | ||
125 | |||
126 | The unpack process usually immediately follows the download. For all | ||
127 | URLs except Git URLs, BitBake uses the common ``unpack`` method. | ||
128 | |||
129 | A number of parameters exist that you can specify within the URL to | ||
130 | govern the behavior of the unpack stage: | ||
131 | |||
132 | - *unpack:* Controls whether the URL components are unpacked. If set to | ||
133 | "1", which is the default, the components are unpacked. If set to | ||
134 | "0", the unpack stage leaves the file alone. This parameter is useful | ||
135 | when you want an archive to be copied in and not be unpacked. | ||
136 | |||
137 | - *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether | ||
138 | to use DOS line ending conversion on text files. | ||
139 | |||
140 | - *basepath:* Instructs the unpack stage to strip the specified | ||
141 | directories from the source path when unpacking. | ||
142 | |||
143 | - *subdir:* Unpacks the specific URL to the specified subdirectory | ||
144 | within the root directory. | ||
145 | |||
146 | The unpack call automatically decompresses and extracts files with ".Z", | ||
147 | ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and | ||
148 | ".bz2" extensions as well as various combinations of tarball extensions. | ||
149 | |||
150 | As mentioned, the Git fetcher has its own unpack method that is | ||
151 | optimized to work with Git trees. Basically, this method works by | ||
152 | cloning the tree into the final directory. The process is completed | ||
153 | using references so that there is only one central copy of the Git | ||
154 | metadata needed. | ||
155 | |||
156 | .. _bb-fetchers: | ||
157 | |||
158 | Fetchers | ||
159 | ======== | ||
160 | |||
161 | As mentioned earlier, the URL prefix determines which fetcher submodule | ||
162 | BitBake uses. Each submodule can support different URL parameters, which | ||
163 | are described in the following sections. | ||
164 | |||
165 | .. _local-file-fetcher: | ||
166 | |||
167 | Local file fetcher (``file://``) | ||
168 | -------------------------------- | ||
169 | |||
170 | This submodule handles URLs that begin with ``file://``. The filename | ||
171 | you specify within the URL can be either an absolute or relative path to | ||
172 | a file. If the filename is relative, the contents of the | ||
173 | ```FILESPATH`` <#var-bb-FILESPATH>`__ variable is used in the same way | ||
174 | ``PATH`` is used to find executables. If the file cannot be found, it is | ||
175 | assumed that it is available in ```DL_DIR`` <#var-bb-DL_DIR>`__ by the | ||
176 | time the ``download()`` method is called. | ||
177 | |||
178 | If you specify a directory, the entire directory is unpacked. | ||
179 | |||
180 | Here are a couple of example URLs, the first relative and the second | ||
181 | absolute: SRC_URI = "file://relativefile.patch" SRC_URI = | ||
182 | "file:///Users/ich/very_important_software" | ||
183 | |||
184 | .. _http-ftp-fetcher: | ||
185 | |||
186 | HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``) | ||
187 | ------------------------------------------------------------- | ||
188 | |||
189 | This fetcher obtains files from web and FTP servers. Internally, the | ||
190 | fetcher uses the wget utility. | ||
191 | |||
192 | The executable and parameters used are specified by the | ||
193 | ``FETCHCMD_wget`` variable, which defaults to sensible values. The | ||
194 | fetcher supports a parameter "downloadfilename" that allows the name of | ||
195 | the downloaded file to be specified. Specifying the name of the | ||
196 | downloaded file is useful for avoiding collisions in | ||
197 | ```DL_DIR`` <#var-bb-DL_DIR>`__ when dealing with multiple files that | ||
198 | have the same name. | ||
199 | |||
200 | Some example URLs are as follows: SRC_URI = | ||
201 | "http://oe.handhelds.org/not_there.aac" SRC_URI = | ||
202 | "ftp://oe.handhelds.org/not_there_as_well.aac" SRC_URI = | ||
203 | "ftp://you@oe.handhelds.org/home/you/secret.plan" | ||
204 | |||
205 | .. note:: | ||
206 | |||
207 | Because URL parameters are delimited by semi-colons, this can | ||
208 | introduce ambiguity when parsing URLs that also contain semi-colons, | ||
209 | for example: | ||
210 | :: | ||
211 | |||
212 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" | ||
213 | |||
214 | |||
215 | Such URLs should should be modified by replacing semi-colons with '&' | ||
216 | characters: | ||
217 | :: | ||
218 | |||
219 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" | ||
220 | |||
221 | |||
222 | In most cases this should work. Treating semi-colons and '&' in | ||
223 | queries identically is recommended by the World Wide Web Consortium | ||
224 | (W3C). Note that due to the nature of the URL, you may have to | ||
225 | specify the name of the downloaded file as well: | ||
226 | :: | ||
227 | |||
228 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" | ||
229 | |||
230 | |||
231 | .. _cvs-fetcher: | ||
232 | |||
233 | CVS fetcher (``(cvs://``) | ||
234 | ------------------------- | ||
235 | |||
236 | This submodule handles checking out files from the CVS version control | ||
237 | system. You can configure it using a number of different variables: | ||
238 | |||
239 | - *``FETCHCMD_cvs``:* The name of the executable to use when running | ||
240 | the ``cvs`` command. This name is usually "cvs". | ||
241 | |||
242 | - *``SRCDATE``:* The date to use when fetching the CVS source code. A | ||
243 | special value of "now" causes the checkout to be updated on every | ||
244 | build. | ||
245 | |||
246 | - ```CVSDIR`` <#var-bb-CVSDIR>`__\ *:* Specifies where a temporary | ||
247 | checkout is saved. The location is often ``DL_DIR/cvs``. | ||
248 | |||
249 | - *``CVS_PROXY_HOST``:* The name to use as a "proxy=" parameter to the | ||
250 | ``cvs`` command. | ||
251 | |||
252 | - *``CVS_PROXY_PORT``:* The port number to use as a "proxyport=" | ||
253 | parameter to the ``cvs`` command. | ||
254 | |||
255 | As well as the standard username and password URL syntax, you can also | ||
256 | configure the fetcher with various URL parameters: | ||
257 | |||
258 | The supported parameters are as follows: | ||
259 | |||
260 | - *"method":* The protocol over which to communicate with the CVS | ||
261 | server. By default, this protocol is "pserver". If "method" is set to | ||
262 | "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You | ||
263 | can use "dir" for local directories. | ||
264 | |||
265 | - *"module":* Specifies the module to check out. You must supply this | ||
266 | parameter. | ||
267 | |||
268 | - *"tag":* Describes which CVS TAG should be used for the checkout. By | ||
269 | default, the TAG is empty. | ||
270 | |||
271 | - *"date":* Specifies a date. If no "date" is specified, the | ||
272 | ```SRCDATE`` <#var-bb-SRCDATE>`__ of the configuration is used to | ||
273 | checkout a specific date. The special value of "now" causes the | ||
274 | checkout to be updated on every build. | ||
275 | |||
276 | - *"localdir":* Used to rename the module. Effectively, you are | ||
277 | renaming the output directory to which the module is unpacked. You | ||
278 | are forcing the module into a special directory relative to | ||
279 | ```CVSDIR`` <#var-bb-CVSDIR>`__. | ||
280 | |||
281 | - *"rsh"* Used in conjunction with the "method" parameter. | ||
282 | |||
283 | - *"scmdata":* Causes the CVS metadata to be maintained in the tarball | ||
284 | the fetcher creates when set to "keep". The tarball is expanded into | ||
285 | the work directory. By default, the CVS metadata is removed. | ||
286 | |||
287 | - *"fullpath":* Controls whether the resulting checkout is at the | ||
288 | module level, which is the default, or is at deeper paths. | ||
289 | |||
290 | - *"norecurse":* Causes the fetcher to only checkout the specified | ||
291 | directory with no recurse into any subdirectories. | ||
292 | |||
293 | - *"port":* The port to which the CVS server connects. | ||
294 | |||
295 | Some example URLs are as follows: SRC_URI = | ||
296 | "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" SRC_URI = | ||
297 | "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" | ||
298 | |||
299 | .. _svn-fetcher: | ||
300 | |||
301 | Subversion (SVN) Fetcher (``svn://``) | ||
302 | ------------------------------------- | ||
303 | |||
304 | This fetcher submodule fetches code from the Subversion source control | ||
305 | system. The executable used is specified by ``FETCHCMD_svn``, which | ||
306 | defaults to "svn". The fetcher's temporary working directory is set by | ||
307 | ```SVNDIR`` <#var-bb-SVNDIR>`__, which is usually ``DL_DIR/svn``. | ||
308 | |||
309 | The supported parameters are as follows: | ||
310 | |||
311 | - *"module":* The name of the svn module to checkout. You must provide | ||
312 | this parameter. You can think of this parameter as the top-level | ||
313 | directory of the repository data you want. | ||
314 | |||
315 | - *"path_spec":* A specific directory in which to checkout the | ||
316 | specified svn module. | ||
317 | |||
318 | - *"protocol":* The protocol to use, which defaults to "svn". If | ||
319 | "protocol" is set to "svn+ssh", the "ssh" parameter is also used. | ||
320 | |||
321 | - *"rev":* The revision of the source code to checkout. | ||
322 | |||
323 | - *"scmdata":* Causes the “.svn” directories to be available during | ||
324 | compile-time when set to "keep". By default, these directories are | ||
325 | removed. | ||
326 | |||
327 | - *"ssh":* An optional parameter used when "protocol" is set to | ||
328 | "svn+ssh". You can use this parameter to specify the ssh program used | ||
329 | by svn. | ||
330 | |||
331 | - *"transportuser":* When required, sets the username for the | ||
332 | transport. By default, this parameter is empty. The transport | ||
333 | username is different than the username used in the main URL, which | ||
334 | is passed to the subversion command. | ||
335 | |||
336 | Following are three examples using svn: SRC_URI = | ||
337 | "svn://myrepos/proj1;module=vip;protocol=http;rev=667" SRC_URI = | ||
338 | "svn://myrepos/proj1;module=opie;protocol=svn+ssh" SRC_URI = | ||
339 | "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1" | ||
340 | |||
341 | .. _git-fetcher: | ||
342 | |||
343 | Git Fetcher (``git://``) | ||
344 | ------------------------ | ||
345 | |||
346 | This fetcher submodule fetches code from the Git source control system. | ||
347 | The fetcher works by creating a bare clone of the remote into | ||
348 | ```GITDIR`` <#var-bb-GITDIR>`__, which is usually ``DL_DIR/git2``. This | ||
349 | bare clone is then cloned into the work directory during the unpack | ||
350 | stage when a specific tree is checked out. This is done using alternates | ||
351 | and by reference to minimize the amount of duplicate data on the disk | ||
352 | and make the unpack process fast. The executable used can be set with | ||
353 | ``FETCHCMD_git``. | ||
354 | |||
355 | This fetcher supports the following parameters: | ||
356 | |||
357 | - *"protocol":* The protocol used to fetch the files. The default is | ||
358 | "git" when a hostname is set. If a hostname is not set, the Git | ||
359 | protocol is "file". You can also use "http", "https", "ssh" and | ||
360 | "rsync". | ||
361 | |||
362 | - *"nocheckout":* Tells the fetcher to not checkout source code when | ||
363 | unpacking when set to "1". Set this option for the URL where there is | ||
364 | a custom routine to checkout code. The default is "0". | ||
365 | |||
366 | - *"rebaseable":* Indicates that the upstream Git repository can be | ||
367 | rebased. You should set this parameter to "1" if revisions can become | ||
368 | detached from branches. In this case, the source mirror tarball is | ||
369 | done per revision, which has a loss of efficiency. Rebasing the | ||
370 | upstream Git repository could cause the current revision to disappear | ||
371 | from the upstream repository. This option reminds the fetcher to | ||
372 | preserve the local cache carefully for future use. The default value | ||
373 | for this parameter is "0". | ||
374 | |||
375 | - *"nobranch":* Tells the fetcher to not check the SHA validation for | ||
376 | the branch when set to "1". The default is "0". Set this option for | ||
377 | the recipe that refers to the commit that is valid for a tag instead | ||
378 | of the branch. | ||
379 | |||
380 | - *"bareclone":* Tells the fetcher to clone a bare clone into the | ||
381 | destination directory without checking out a working tree. Only the | ||
382 | raw Git metadata is provided. This parameter implies the "nocheckout" | ||
383 | parameter as well. | ||
384 | |||
385 | - *"branch":* The branch(es) of the Git tree to clone. If unset, this | ||
386 | is assumed to be "master". The number of branch parameters much match | ||
387 | the number of name parameters. | ||
388 | |||
389 | - *"rev":* The revision to use for the checkout. The default is | ||
390 | "master". | ||
391 | |||
392 | - *"tag":* Specifies a tag to use for the checkout. To correctly | ||
393 | resolve tags, BitBake must access the network. For that reason, tags | ||
394 | are often not used. As far as Git is concerned, the "tag" parameter | ||
395 | behaves effectively the same as the "rev" parameter. | ||
396 | |||
397 | - *"subpath":* Limits the checkout to a specific subpath of the tree. | ||
398 | By default, the whole tree is checked out. | ||
399 | |||
400 | - *"destsuffix":* The name of the path in which to place the checkout. | ||
401 | By default, the path is ``git/``. | ||
402 | |||
403 | - *"usehead":* Enables local ``git://`` URLs to use the current branch | ||
404 | HEAD as the revision for use with ``AUTOREV``. The "usehead" | ||
405 | parameter implies no branch and only works when the transfer protocol | ||
406 | is ``file://``. | ||
407 | |||
408 | Here are some example URLs: SRC_URI = | ||
409 | "git://git.oe.handhelds.org/git/vip.git;tag=version-1" SRC_URI = | ||
410 | "git://git.oe.handhelds.org/git/vip.git;protocol=http" | ||
411 | |||
412 | .. _gitsm-fetcher: | ||
413 | |||
414 | Git Submodule Fetcher (``gitsm://``) | ||
415 | ------------------------------------ | ||
416 | |||
417 | This fetcher submodule inherits from the `Git fetcher <#git-fetcher>`__ | ||
418 | and extends that fetcher's behavior by fetching a repository's | ||
419 | submodules. ```SRC_URI`` <#var-bb-SRC_URI>`__ is passed to the Git | ||
420 | fetcher as described in the "`Git Fetcher | ||
421 | (``git://``) <#git-fetcher>`__" section. | ||
422 | |||
423 | .. note:: | ||
424 | |||
425 | You must clean a recipe when switching between '``git://``' and | ||
426 | '``gitsm://``' URLs. | ||
427 | |||
428 | The Git Submodules fetcher is not a complete fetcher implementation. | ||
429 | The fetcher has known issues where it does not use the normal source | ||
430 | mirroring infrastructure properly. Further, the submodule sources it | ||
431 | fetches are not visible to the licensing and source archiving | ||
432 | infrastructures. | ||
433 | |||
434 | .. _clearcase-fetcher: | ||
435 | |||
436 | ClearCase Fetcher (``ccrc://``) | ||
437 | ------------------------------- | ||
438 | |||
439 | This fetcher submodule fetches code from a | ||
440 | `ClearCase <http://en.wikipedia.org/wiki/Rational_ClearCase>`__ | ||
441 | repository. | ||
442 | |||
443 | To use this fetcher, make sure your recipe has proper | ||
444 | ```SRC_URI`` <#var-bb-SRC_URI>`__, ```SRCREV`` <#var-bb-SRCREV>`__, and | ||
445 | ```PV`` <#var-bb-PV>`__ settings. Here is an example: SRC_URI = | ||
446 | "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" | ||
447 | SRCREV = "EXAMPLE_CLEARCASE_TAG" PV = "${@d.getVar("SRCREV", | ||
448 | False).replace("/", "+")}" The fetcher uses the ``rcleartool`` or | ||
449 | ``cleartool`` remote client, depending on which one is available. | ||
450 | |||
451 | Following are options for the ``SRC_URI`` statement: | ||
452 | |||
453 | - *``vob``*: The name, which must include the prepending "/" character, | ||
454 | of the ClearCase VOB. This option is required. | ||
455 | |||
456 | - *``module``*: The module, which must include the prepending "/" | ||
457 | character, in the selected VOB. | ||
458 | |||
459 | .. note:: | ||
460 | |||
461 | The | ||
462 | module | ||
463 | and | ||
464 | vob | ||
465 | options are combined to create the | ||
466 | load | ||
467 | rule in the view config spec. As an example, consider the | ||
468 | vob | ||
469 | and | ||
470 | module | ||
471 | values from the | ||
472 | SRC_URI | ||
473 | statement at the start of this section. Combining those values | ||
474 | results in the following: | ||
475 | :: | ||
476 | |||
477 | load /example_vob/example_module | ||
478 | |||
479 | |||
480 | - *``proto``*: The protocol, which can be either ``http`` or ``https``. | ||
481 | |||
482 | By default, the fetcher creates a configuration specification. If you | ||
483 | want this specification written to an area other than the default, use | ||
484 | the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where | ||
485 | the specification is written. | ||
486 | |||
487 | .. note:: | ||
488 | |||
489 | the | ||
490 | SRCREV | ||
491 | loses its functionality if you specify this variable. However, | ||
492 | SRCREV | ||
493 | is still used to label the archive after a fetch even though it does | ||
494 | not define what is fetched. | ||
495 | |||
496 | Here are a couple of other behaviors worth mentioning: | ||
497 | |||
498 | - When using ``cleartool``, the login of ``cleartool`` is handled by | ||
499 | the system. The login require no special steps. | ||
500 | |||
501 | - In order to use ``rcleartool`` with authenticated users, an | ||
502 | "rcleartool login" is necessary before using the fetcher. | ||
503 | |||
504 | .. _perforce-fetcher: | ||
505 | |||
506 | Perforce Fetcher (``p4://``) | ||
507 | ---------------------------- | ||
508 | |||
509 | This fetcher submodule fetches code from the | ||
510 | `Perforce <https://www.perforce.com/>`__ source control system. The | ||
511 | executable used is specified by ``FETCHCMD_p4``, which defaults to "p4". | ||
512 | The fetcher's temporary working directory is set by | ||
513 | ```P4DIR`` <#var-bb-P4DIR>`__, which defaults to "DL_DIR/p4". | ||
514 | The fetcher does not make use of a perforce client, instead it | ||
515 | relies on ``p4 files`` to retrieve a list of | ||
516 | files and ``p4 print`` to transfer the content | ||
517 | of those files locally. | ||
518 | |||
519 | To use this fetcher, make sure your recipe has proper | ||
520 | ```SRC_URI`` <#var-bb-SRC_URI>`__, ```SRCREV`` <#var-bb-SRCREV>`__, and | ||
521 | ```PV`` <#var-bb-PV>`__ values. The p4 executable is able to use the | ||
522 | config file defined by your system's ``P4CONFIG`` environment variable | ||
523 | in order to define the Perforce server URL and port, username, and | ||
524 | password if you do not wish to keep those values in a recipe itself. If | ||
525 | you choose not to use ``P4CONFIG``, or to explicitly set variables that | ||
526 | ``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is | ||
527 | the server's URL and port number, and you can specify a username and | ||
528 | password directly in your recipe within ``SRC_URI``. | ||
529 | |||
530 | Here is an example that relies on ``P4CONFIG`` to specify the server URL | ||
531 | and port, username, and password, and fetches the Head Revision: SRC_URI | ||
532 | = "p4://example-depot/main/source/..." SRCREV = "${AUTOREV}" PV = | ||
533 | "p4-${SRCPV}" S = "${WORKDIR}/p4" | ||
534 | |||
535 | Here is an example that specifies the server URL and port, username, and | ||
536 | password, and fetches a Revision based on a Label: P4PORT = | ||
537 | "tcp:p4server.example.net:1666" SRC_URI = | ||
538 | "p4://user:passwd@example-depot/main/source/..." SRCREV = "release-1.0" | ||
539 | PV = "p4-${SRCPV}" S = "${WORKDIR}/p4" | ||
540 | |||
541 | .. note:: | ||
542 | |||
543 | You should always set | ||
544 | S | ||
545 | to | ||
546 | "${WORKDIR}/p4" | ||
547 | in your recipe. | ||
548 | |||
549 | By default, the fetcher strips the depot location from the local file paths. In | ||
550 | the above example, the content of ``example-depot/main/source/`` will be placed | ||
551 | in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot | ||
552 | paths locally is desirable, the fetcher supports two parameters: | ||
553 | |||
554 | - **"module":** | ||
555 | The top-level depot location or directory to fetch. The value of this | ||
556 | parameter can also point to a single file within the depot, in which case | ||
557 | the local file path will include the module path. | ||
558 | - **"remotepath":** | ||
559 | When used with the value "``keep``", the fetcher will mirror the full depot | ||
560 | paths locally for the specified location, even in combination with the | ||
561 | ``module`` parameter. | ||
562 | |||
563 | Here is an example use of the the ``module`` parameter: :: | ||
564 | |||
565 | SRC_URI = "p4://user:passwd@example-depot/main;module=source/..." | ||
566 | |||
567 | In this case, the content of the top-level directory ``source/`` will be fetched | ||
568 | to ``${P4DIR}``, including the directory itself. The top-level directory will | ||
569 | be accesible at ``${P4DIR}/source/``. | ||
570 | |||
571 | Here is an example use of the the ``remotepath`` parameter: :: | ||
572 | |||
573 | SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep" | ||
574 | |||
575 | In this case, the content of the top-level directory ``source/`` will be fetched | ||
576 | to ``${P4DIR}``, but the complete depot paths will be mirrored locally. The | ||
577 | top-level directory will be accessible at | ||
578 | ``${P4DIR}/example-depot/main/source/``. | ||
579 | |||
580 | .. _repo-fetcher: | ||
581 | |||
582 | Repo Fetcher (``repo://``) | ||
583 | -------------------------- | ||
584 | |||
585 | This fetcher submodule fetches code from ``google-repo`` source control | ||
586 | system. The fetcher works by initiating and syncing sources of the | ||
587 | repository into ```REPODIR`` <#var-bb-REPODIR>`__, which is usually | ||
588 | ```DL_DIR`` <#var-bb-DL_DIR>`__\ ``/repo``. | ||
589 | |||
590 | This fetcher supports the following parameters: | ||
591 | |||
592 | - *"protocol":* Protocol to fetch the repository manifest (default: | ||
593 | git). | ||
594 | |||
595 | - *"branch":* Branch or tag of repository to get (default: master). | ||
596 | |||
597 | - *"manifest":* Name of the manifest file (default: ``default.xml``). | ||
598 | |||
599 | Here are some example URLs: SRC_URI = | ||
600 | "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml" | ||
601 | SRC_URI = | ||
602 | "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml" | ||
603 | |||
604 | Other Fetchers | ||
605 | -------------- | ||
606 | |||
607 | Fetch submodules also exist for the following: | ||
608 | |||
609 | - Bazaar (``bzr://``) | ||
610 | |||
611 | - Mercurial (``hg://``) | ||
612 | |||
613 | - npm (``npm://``) | ||
614 | |||
615 | - OSC (``osc://``) | ||
616 | |||
617 | - Secure FTP (``sftp://``) | ||
618 | |||
619 | - Secure Shell (``ssh://``) | ||
620 | |||
621 | - Trees using Git Annex (``gitannex://``) | ||
622 | |||
623 | No documentation currently exists for these lesser used fetcher | ||
624 | submodules. However, you might find the code helpful and readable. | ||
625 | |||
626 | Auto Revisions | ||
627 | ============== | ||
628 | |||
629 | We need to document ``AUTOREV`` and ``SRCREV_FORMAT`` here. | ||