diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst')
-rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 652 |
1 files changed, 652 insertions, 0 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst new file mode 100644 index 0000000000..6760b10828 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | |||
@@ -0,0 +1,652 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.5 | ||
2 | |||
3 | ===================== | ||
4 | File Download Support | ||
5 | ===================== | ||
6 | |||
7 | | | ||
8 | |||
9 | BitBake's fetch module is a standalone piece of library code that deals | ||
10 | with the intricacies of downloading source code and files from remote | ||
11 | systems. Fetching source code is one of the cornerstones of building | ||
12 | software. As such, this module forms an important part of BitBake. | ||
13 | |||
14 | The current fetch module is called "fetch2" and refers to the fact that | ||
15 | it is the second major version of the API. The original version is | ||
16 | obsolete and has been removed from the codebase. Thus, in all cases, | ||
17 | "fetch" refers to "fetch2" in this manual. | ||
18 | |||
19 | The Download (Fetch) | ||
20 | ==================== | ||
21 | |||
22 | BitBake takes several steps when fetching source code or files. The | ||
23 | fetcher codebase deals with two distinct processes in order: obtaining | ||
24 | the files from somewhere (cached or otherwise) and then unpacking those | ||
25 | files into a specific location and perhaps in a specific way. Getting | ||
26 | and unpacking the files is often optionally followed by patching. | ||
27 | Patching, however, is not covered by this module. | ||
28 | |||
29 | The code to execute the first part of this process, a fetch, looks | ||
30 | something like the following: :: | ||
31 | |||
32 | src_uri = (d.getVar('SRC_URI') or "").split() | ||
33 | fetcher = bb.fetch2.Fetch(src_uri, d) | ||
34 | fetcher.download() | ||
35 | |||
36 | This code sets up an instance of the fetch class. The instance uses a | ||
37 | space-separated list of URLs from the :term:`SRC_URI` | ||
38 | variable and then calls the ``download`` method to download the files. | ||
39 | |||
40 | The instantiation of the fetch class is usually followed by: :: | ||
41 | |||
42 | rootdir = l.getVar('WORKDIR') | ||
43 | fetcher.unpack(rootdir) | ||
44 | |||
45 | This code unpacks the downloaded files to the specified by ``WORKDIR``. | ||
46 | |||
47 | .. note:: | ||
48 | |||
49 | For convenience, the naming in these examples matches the variables | ||
50 | used by OpenEmbedded. If you want to see the above code in action, | ||
51 | examine the OpenEmbedded class file ``base.bbclass`` | ||
52 | . | ||
53 | |||
54 | The ``SRC_URI`` and ``WORKDIR`` variables are not hardcoded into the | ||
55 | fetcher, since those fetcher methods can be (and are) called with | ||
56 | different variable names. In OpenEmbedded for example, the shared state | ||
57 | (sstate) code uses the fetch module to fetch the sstate files. | ||
58 | |||
59 | When the ``download()`` method is called, BitBake tries to resolve the | ||
60 | URLs by looking for source files in a specific search order: | ||
61 | |||
62 | - *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find | ||
63 | source files. These locations are defined using the | ||
64 | :term:`PREMIRRORS` variable. | ||
65 | |||
66 | - *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g | ||
67 | from ``SRC_URI``). | ||
68 | |||
69 | - *Mirror Sites:* If fetch failures occur, BitBake next uses mirror | ||
70 | locations as defined by the :term:`MIRRORS` variable. | ||
71 | |||
72 | For each URL passed to the fetcher, the fetcher calls the submodule that | ||
73 | handles that particular URL type. This behavior can be the source of | ||
74 | some confusion when you are providing URLs for the ``SRC_URI`` variable. | ||
75 | Consider the following two URLs: :: | ||
76 | |||
77 | http://git.yoctoproject.org/git/poky;protocol=git | ||
78 | git://git.yoctoproject.org/git/poky;protocol=http | ||
79 | |||
80 | In the former case, the URL is passed to the ``wget`` fetcher, which does not | ||
81 | understand "git". Therefore, the latter case is the correct form since the Git | ||
82 | fetcher does know how to use HTTP as a transport. | ||
83 | |||
84 | Here are some examples that show commonly used mirror definitions: :: | ||
85 | |||
86 | PREMIRRORS ?= "\ | ||
87 | bzr://.*/.\* http://somemirror.org/sources/ \\n \ | ||
88 | cvs://.*/.\* http://somemirror.org/sources/ \\n \ | ||
89 | git://.*/.\* http://somemirror.org/sources/ \\n \ | ||
90 | hg://.*/.\* http://somemirror.org/sources/ \\n \ | ||
91 | osc://.*/.\* http://somemirror.org/sources/ \\n \ | ||
92 | p4://.*/.\* http://somemirror.org/sources/ \\n \ | ||
93 | svn://.*/.\* http://somemirror.org/sources/ \\n" | ||
94 | |||
95 | MIRRORS =+ "\ | ||
96 | ftp://.*/.\* http://somemirror.org/sources/ \\n \ | ||
97 | http://.*/.\* http://somemirror.org/sources/ \\n \ | ||
98 | https://.*/.\* http://somemirror.org/sources/ \\n" | ||
99 | |||
100 | It is useful to note that BitBake | ||
101 | supports cross-URLs. It is possible to mirror a Git repository on an | ||
102 | HTTP server as a tarball. This is what the ``git://`` mapping in the | ||
103 | previous example does. | ||
104 | |||
105 | Since network accesses are slow, BitBake maintains a cache of files | ||
106 | downloaded from the network. Any source files that are not local (i.e. | ||
107 | downloaded from the Internet) are placed into the download directory, | ||
108 | which is specified by the :term:`DL_DIR` variable. | ||
109 | |||
110 | File integrity is of key importance for reproducing builds. For | ||
111 | non-local archive downloads, the fetcher code can verify SHA-256 and MD5 | ||
112 | checksums to ensure the archives have been downloaded correctly. You can | ||
113 | specify these checksums by using the ``SRC_URI`` variable with the | ||
114 | appropriate varflags as follows: :: | ||
115 | |||
116 | SRC_URI[md5sum] = "value" | ||
117 | SRC_URI[sha256sum] = "value" | ||
118 | |||
119 | You can also specify the checksums as | ||
120 | parameters on the ``SRC_URI`` as shown below: :: | ||
121 | |||
122 | SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" | ||
123 | |||
124 | If multiple URIs exist, you can specify the checksums either directly as | ||
125 | in the previous example, or you can name the URLs. The following syntax | ||
126 | shows how you name the URIs: :: | ||
127 | |||
128 | SRC_URI = "http://example.com/foobar.tar.bz2;name=foo" | ||
129 | SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d | ||
130 | |||
131 | After a file has been downloaded and | ||
132 | has had its checksum checked, a ".done" stamp is placed in ``DL_DIR``. | ||
133 | BitBake uses this stamp during subsequent builds to avoid downloading or | ||
134 | comparing a checksum for the file again. | ||
135 | |||
136 | .. note:: | ||
137 | |||
138 | It is assumed that local storage is safe from data corruption. If | ||
139 | this were not the case, there would be bigger issues to worry about. | ||
140 | |||
141 | If :term:`BB_STRICT_CHECKSUM` is set, any | ||
142 | download without a checksum triggers an error message. The | ||
143 | :term:`BB_NO_NETWORK` variable can be used to | ||
144 | make any attempted network access a fatal error, which is useful for | ||
145 | checking that mirrors are complete as well as other things. | ||
146 | |||
147 | .. _bb-the-unpack: | ||
148 | |||
149 | The Unpack | ||
150 | ========== | ||
151 | |||
152 | The unpack process usually immediately follows the download. For all | ||
153 | URLs except Git URLs, BitBake uses the common ``unpack`` method. | ||
154 | |||
155 | A number of parameters exist that you can specify within the URL to | ||
156 | govern the behavior of the unpack stage: | ||
157 | |||
158 | - *unpack:* Controls whether the URL components are unpacked. If set to | ||
159 | "1", which is the default, the components are unpacked. If set to | ||
160 | "0", the unpack stage leaves the file alone. This parameter is useful | ||
161 | when you want an archive to be copied in and not be unpacked. | ||
162 | |||
163 | - *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether | ||
164 | to use DOS line ending conversion on text files. | ||
165 | |||
166 | - *basepath:* Instructs the unpack stage to strip the specified | ||
167 | directories from the source path when unpacking. | ||
168 | |||
169 | - *subdir:* Unpacks the specific URL to the specified subdirectory | ||
170 | within the root directory. | ||
171 | |||
172 | The unpack call automatically decompresses and extracts files with ".Z", | ||
173 | ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and | ||
174 | ".bz2" extensions as well as various combinations of tarball extensions. | ||
175 | |||
176 | As mentioned, the Git fetcher has its own unpack method that is | ||
177 | optimized to work with Git trees. Basically, this method works by | ||
178 | cloning the tree into the final directory. The process is completed | ||
179 | using references so that there is only one central copy of the Git | ||
180 | metadata needed. | ||
181 | |||
182 | .. _bb-fetchers: | ||
183 | |||
184 | Fetchers | ||
185 | ======== | ||
186 | |||
187 | As mentioned earlier, the URL prefix determines which fetcher submodule | ||
188 | BitBake uses. Each submodule can support different URL parameters, which | ||
189 | are described in the following sections. | ||
190 | |||
191 | .. _local-file-fetcher: | ||
192 | |||
193 | Local file fetcher (``file://``) | ||
194 | -------------------------------- | ||
195 | |||
196 | This submodule handles URLs that begin with ``file://``. The filename | ||
197 | you specify within the URL can be either an absolute or relative path to | ||
198 | a file. If the filename is relative, the contents of the | ||
199 | :term:`FILESPATH` variable is used in the same way | ||
200 | ``PATH`` is used to find executables. If the file cannot be found, it is | ||
201 | assumed that it is available in :term:`DL_DIR` by the | ||
202 | time the ``download()`` method is called. | ||
203 | |||
204 | If you specify a directory, the entire directory is unpacked. | ||
205 | |||
206 | Here are a couple of example URLs, the first relative and the second | ||
207 | absolute: :: | ||
208 | |||
209 | SRC_URI = "file://relativefile.patch" | ||
210 | SRC_URI = "file:///Users/ich/very_important_software" | ||
211 | |||
212 | .. _http-ftp-fetcher: | ||
213 | |||
214 | HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``) | ||
215 | ------------------------------------------------------------- | ||
216 | |||
217 | This fetcher obtains files from web and FTP servers. Internally, the | ||
218 | fetcher uses the wget utility. | ||
219 | |||
220 | The executable and parameters used are specified by the | ||
221 | ``FETCHCMD_wget`` variable, which defaults to sensible values. The | ||
222 | fetcher supports a parameter "downloadfilename" that allows the name of | ||
223 | the downloaded file to be specified. Specifying the name of the | ||
224 | downloaded file is useful for avoiding collisions in | ||
225 | :term:`DL_DIR` when dealing with multiple files that | ||
226 | have the same name. | ||
227 | |||
228 | Some example URLs are as follows: :: | ||
229 | |||
230 | SRC_URI = "http://oe.handhelds.org/not_there.aac" | ||
231 | SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac" | ||
232 | SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan" | ||
233 | |||
234 | .. note:: | ||
235 | |||
236 | Because URL parameters are delimited by semi-colons, this can | ||
237 | introduce ambiguity when parsing URLs that also contain semi-colons, | ||
238 | for example: | ||
239 | :: | ||
240 | |||
241 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" | ||
242 | |||
243 | |||
244 | Such URLs should should be modified by replacing semi-colons with '&' | ||
245 | characters: | ||
246 | :: | ||
247 | |||
248 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" | ||
249 | |||
250 | |||
251 | In most cases this should work. Treating semi-colons and '&' in | ||
252 | queries identically is recommended by the World Wide Web Consortium | ||
253 | (W3C). Note that due to the nature of the URL, you may have to | ||
254 | specify the name of the downloaded file as well: | ||
255 | :: | ||
256 | |||
257 | SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" | ||
258 | |||
259 | |||
260 | .. _cvs-fetcher: | ||
261 | |||
262 | CVS fetcher (``(cvs://``) | ||
263 | ------------------------- | ||
264 | |||
265 | This submodule handles checking out files from the CVS version control | ||
266 | system. You can configure it using a number of different variables: | ||
267 | |||
268 | - :term:`FETCHCMD_cvs <FETCHCMD>`: The name of the executable to use when running | ||
269 | the ``cvs`` command. This name is usually "cvs". | ||
270 | |||
271 | - :term:`SRCDATE`: The date to use when fetching the CVS source code. A | ||
272 | special value of "now" causes the checkout to be updated on every | ||
273 | build. | ||
274 | |||
275 | - :term:`CVSDIR`: Specifies where a temporary | ||
276 | checkout is saved. The location is often ``DL_DIR/cvs``. | ||
277 | |||
278 | - CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the | ||
279 | ``cvs`` command. | ||
280 | |||
281 | - CVS_PROXY_PORT: The port number to use as a "proxyport=" | ||
282 | parameter to the ``cvs`` command. | ||
283 | |||
284 | As well as the standard username and password URL syntax, you can also | ||
285 | configure the fetcher with various URL parameters: | ||
286 | |||
287 | The supported parameters are as follows: | ||
288 | |||
289 | - *"method":* The protocol over which to communicate with the CVS | ||
290 | server. By default, this protocol is "pserver". If "method" is set to | ||
291 | "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You | ||
292 | can use "dir" for local directories. | ||
293 | |||
294 | - *"module":* Specifies the module to check out. You must supply this | ||
295 | parameter. | ||
296 | |||
297 | - *"tag":* Describes which CVS TAG should be used for the checkout. By | ||
298 | default, the TAG is empty. | ||
299 | |||
300 | - *"date":* Specifies a date. If no "date" is specified, the | ||
301 | :term:`SRCDATE` of the configuration is used to | ||
302 | checkout a specific date. The special value of "now" causes the | ||
303 | checkout to be updated on every build. | ||
304 | |||
305 | - *"localdir":* Used to rename the module. Effectively, you are | ||
306 | renaming the output directory to which the module is unpacked. You | ||
307 | are forcing the module into a special directory relative to | ||
308 | :term:`CVSDIR`. | ||
309 | |||
310 | - *"rsh":* Used in conjunction with the "method" parameter. | ||
311 | |||
312 | - *"scmdata":* Causes the CVS metadata to be maintained in the tarball | ||
313 | the fetcher creates when set to "keep". The tarball is expanded into | ||
314 | the work directory. By default, the CVS metadata is removed. | ||
315 | |||
316 | - *"fullpath":* Controls whether the resulting checkout is at the | ||
317 | module level, which is the default, or is at deeper paths. | ||
318 | |||
319 | - *"norecurse":* Causes the fetcher to only checkout the specified | ||
320 | directory with no recurse into any subdirectories. | ||
321 | |||
322 | - *"port":* The port to which the CVS server connects. | ||
323 | |||
324 | Some example URLs are as follows: :: | ||
325 | |||
326 | SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" | ||
327 | SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" | ||
328 | |||
329 | .. _svn-fetcher: | ||
330 | |||
331 | Subversion (SVN) Fetcher (``svn://``) | ||
332 | ------------------------------------- | ||
333 | |||
334 | This fetcher submodule fetches code from the Subversion source control | ||
335 | system. The executable used is specified by ``FETCHCMD_svn``, which | ||
336 | defaults to "svn". The fetcher's temporary working directory is set by | ||
337 | :term:`SVNDIR`, which is usually ``DL_DIR/svn``. | ||
338 | |||
339 | The supported parameters are as follows: | ||
340 | |||
341 | - *"module":* The name of the svn module to checkout. You must provide | ||
342 | this parameter. You can think of this parameter as the top-level | ||
343 | directory of the repository data you want. | ||
344 | |||
345 | - *"path_spec":* A specific directory in which to checkout the | ||
346 | specified svn module. | ||
347 | |||
348 | - *"protocol":* The protocol to use, which defaults to "svn". If | ||
349 | "protocol" is set to "svn+ssh", the "ssh" parameter is also used. | ||
350 | |||
351 | - *"rev":* The revision of the source code to checkout. | ||
352 | |||
353 | - *"scmdata":* Causes the ".svn" directories to be available during | ||
354 | compile-time when set to "keep". By default, these directories are | ||
355 | removed. | ||
356 | |||
357 | - *"ssh":* An optional parameter used when "protocol" is set to | ||
358 | "svn+ssh". You can use this parameter to specify the ssh program used | ||
359 | by svn. | ||
360 | |||
361 | - *"transportuser":* When required, sets the username for the | ||
362 | transport. By default, this parameter is empty. The transport | ||
363 | username is different than the username used in the main URL, which | ||
364 | is passed to the subversion command. | ||
365 | |||
366 | Following are three examples using svn: :: | ||
367 | |||
368 | SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667" | ||
369 | SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh" | ||
370 | SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1" | ||
371 | |||
372 | .. _git-fetcher: | ||
373 | |||
374 | Git Fetcher (``git://``) | ||
375 | ------------------------ | ||
376 | |||
377 | This fetcher submodule fetches code from the Git source control system. | ||
378 | The fetcher works by creating a bare clone of the remote into | ||
379 | :term:`GITDIR`, which is usually ``DL_DIR/git2``. This | ||
380 | bare clone is then cloned into the work directory during the unpack | ||
381 | stage when a specific tree is checked out. This is done using alternates | ||
382 | and by reference to minimize the amount of duplicate data on the disk | ||
383 | and make the unpack process fast. The executable used can be set with | ||
384 | ``FETCHCMD_git``. | ||
385 | |||
386 | This fetcher supports the following parameters: | ||
387 | |||
388 | - *"protocol":* The protocol used to fetch the files. The default is | ||
389 | "git" when a hostname is set. If a hostname is not set, the Git | ||
390 | protocol is "file". You can also use "http", "https", "ssh" and | ||
391 | "rsync". | ||
392 | |||
393 | - *"nocheckout":* Tells the fetcher to not checkout source code when | ||
394 | unpacking when set to "1". Set this option for the URL where there is | ||
395 | a custom routine to checkout code. The default is "0". | ||
396 | |||
397 | - *"rebaseable":* Indicates that the upstream Git repository can be | ||
398 | rebased. You should set this parameter to "1" if revisions can become | ||
399 | detached from branches. In this case, the source mirror tarball is | ||
400 | done per revision, which has a loss of efficiency. Rebasing the | ||
401 | upstream Git repository could cause the current revision to disappear | ||
402 | from the upstream repository. This option reminds the fetcher to | ||
403 | preserve the local cache carefully for future use. The default value | ||
404 | for this parameter is "0". | ||
405 | |||
406 | - *"nobranch":* Tells the fetcher to not check the SHA validation for | ||
407 | the branch when set to "1". The default is "0". Set this option for | ||
408 | the recipe that refers to the commit that is valid for a tag instead | ||
409 | of the branch. | ||
410 | |||
411 | - *"bareclone":* Tells the fetcher to clone a bare clone into the | ||
412 | destination directory without checking out a working tree. Only the | ||
413 | raw Git metadata is provided. This parameter implies the "nocheckout" | ||
414 | parameter as well. | ||
415 | |||
416 | - *"branch":* The branch(es) of the Git tree to clone. If unset, this | ||
417 | is assumed to be "master". The number of branch parameters much match | ||
418 | the number of name parameters. | ||
419 | |||
420 | - *"rev":* The revision to use for the checkout. The default is | ||
421 | "master". | ||
422 | |||
423 | - *"tag":* Specifies a tag to use for the checkout. To correctly | ||
424 | resolve tags, BitBake must access the network. For that reason, tags | ||
425 | are often not used. As far as Git is concerned, the "tag" parameter | ||
426 | behaves effectively the same as the "rev" parameter. | ||
427 | |||
428 | - *"subpath":* Limits the checkout to a specific subpath of the tree. | ||
429 | By default, the whole tree is checked out. | ||
430 | |||
431 | - *"destsuffix":* The name of the path in which to place the checkout. | ||
432 | By default, the path is ``git/``. | ||
433 | |||
434 | - *"usehead":* Enables local ``git://`` URLs to use the current branch | ||
435 | HEAD as the revision for use with ``AUTOREV``. The "usehead" | ||
436 | parameter implies no branch and only works when the transfer protocol | ||
437 | is ``file://``. | ||
438 | |||
439 | Here are some example URLs: :: | ||
440 | |||
441 | SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" | ||
442 | SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http" | ||
443 | |||
444 | .. _gitsm-fetcher: | ||
445 | |||
446 | Git Submodule Fetcher (``gitsm://``) | ||
447 | ------------------------------------ | ||
448 | |||
449 | This fetcher submodule inherits from the :ref:`Git | ||
450 | fetcher<bitbake-user-manual/bitbake-user-manual-fetching:git fetcher | ||
451 | (\`\`git://\`\`)>` and extends that fetcher's behavior by fetching a | ||
452 | repository's submodules. :term:`SRC_URI` is passed to the Git fetcher as | ||
453 | described in the :ref:`bitbake-user-manual/bitbake-user-manual-fetching:git | ||
454 | fetcher (\`\`git://\`\`)` section. | ||
455 | |||
456 | .. note:: | ||
457 | |||
458 | You must clean a recipe when switching between '``git://``' and | ||
459 | '``gitsm://``' URLs. | ||
460 | |||
461 | The Git Submodules fetcher is not a complete fetcher implementation. | ||
462 | The fetcher has known issues where it does not use the normal source | ||
463 | mirroring infrastructure properly. Further, the submodule sources it | ||
464 | fetches are not visible to the licensing and source archiving | ||
465 | infrastructures. | ||
466 | |||
467 | .. _clearcase-fetcher: | ||
468 | |||
469 | ClearCase Fetcher (``ccrc://``) | ||
470 | ------------------------------- | ||
471 | |||
472 | This fetcher submodule fetches code from a | ||
473 | `ClearCase <http://en.wikipedia.org/wiki/Rational_ClearCase>`__ | ||
474 | repository. | ||
475 | |||
476 | To use this fetcher, make sure your recipe has proper | ||
477 | :term:`SRC_URI`, :term:`SRCREV`, and | ||
478 | :term:`PV` settings. Here is an example: :: | ||
479 | |||
480 | SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" | ||
481 | SRCREV = "EXAMPLE_CLEARCASE_TAG" | ||
482 | PV = "${@d.getVar("SRCREV", False).replace("/", "+")}" | ||
483 | |||
484 | The fetcher uses the ``rcleartool`` or | ||
485 | ``cleartool`` remote client, depending on which one is available. | ||
486 | |||
487 | Following are options for the ``SRC_URI`` statement: | ||
488 | |||
489 | - *vob*: The name, which must include the prepending "/" character, | ||
490 | of the ClearCase VOB. This option is required. | ||
491 | |||
492 | - *module*: The module, which must include the prepending "/" | ||
493 | character, in the selected VOB. | ||
494 | |||
495 | .. note:: | ||
496 | |||
497 | The module and vob options are combined to create the load rule in the | ||
498 | view config spec. As an example, consider the vob and module values from | ||
499 | the SRC_URI statement at the start of this section. Combining those values | ||
500 | results in the following: :: | ||
501 | |||
502 | load /example_vob/example_module | ||
503 | |||
504 | - *proto*: The protocol, which can be either ``http`` or ``https``. | ||
505 | |||
506 | By default, the fetcher creates a configuration specification. If you | ||
507 | want this specification written to an area other than the default, use | ||
508 | the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where | ||
509 | the specification is written. | ||
510 | |||
511 | .. note:: | ||
512 | |||
513 | the SRCREV loses its functionality if you specify this variable. However, | ||
514 | SRCREV is still used to label the archive after a fetch even though it does | ||
515 | not define what is fetched. | ||
516 | |||
517 | Here are a couple of other behaviors worth mentioning: | ||
518 | |||
519 | - When using ``cleartool``, the login of ``cleartool`` is handled by | ||
520 | the system. The login require no special steps. | ||
521 | |||
522 | - In order to use ``rcleartool`` with authenticated users, an | ||
523 | "rcleartool login" is necessary before using the fetcher. | ||
524 | |||
525 | .. _perforce-fetcher: | ||
526 | |||
527 | Perforce Fetcher (``p4://``) | ||
528 | ---------------------------- | ||
529 | |||
530 | This fetcher submodule fetches code from the | ||
531 | `Perforce <https://www.perforce.com/>`__ source control system. The | ||
532 | executable used is specified by ``FETCHCMD_p4``, which defaults to "p4". | ||
533 | The fetcher's temporary working directory is set by | ||
534 | :term:`P4DIR`, which defaults to "DL_DIR/p4". | ||
535 | The fetcher does not make use of a perforce client, instead it | ||
536 | relies on ``p4 files`` to retrieve a list of | ||
537 | files and ``p4 print`` to transfer the content | ||
538 | of those files locally. | ||
539 | |||
540 | To use this fetcher, make sure your recipe has proper | ||
541 | :term:`SRC_URI`, :term:`SRCREV`, and | ||
542 | :term:`PV` values. The p4 executable is able to use the | ||
543 | config file defined by your system's ``P4CONFIG`` environment variable | ||
544 | in order to define the Perforce server URL and port, username, and | ||
545 | password if you do not wish to keep those values in a recipe itself. If | ||
546 | you choose not to use ``P4CONFIG``, or to explicitly set variables that | ||
547 | ``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is | ||
548 | the server's URL and port number, and you can specify a username and | ||
549 | password directly in your recipe within ``SRC_URI``. | ||
550 | |||
551 | Here is an example that relies on ``P4CONFIG`` to specify the server URL | ||
552 | and port, username, and password, and fetches the Head Revision: :: | ||
553 | |||
554 | SRC_URI = "p4://example-depot/main/source/..." | ||
555 | SRCREV = "${AUTOREV}" | ||
556 | PV = "p4-${SRCPV}" | ||
557 | S = "${WORKDIR}/p4" | ||
558 | |||
559 | Here is an example that specifies the server URL and port, username, and | ||
560 | password, and fetches a Revision based on a Label: :: | ||
561 | |||
562 | P4PORT = "tcp:p4server.example.net:1666" | ||
563 | SRC_URI = "p4://user:passwd@example-depot/main/source/..." | ||
564 | SRCREV = "release-1.0" | ||
565 | PV = "p4-${SRCPV}" | ||
566 | S = "${WORKDIR}/p4" | ||
567 | |||
568 | .. note:: | ||
569 | |||
570 | You should always set S to "${WORKDIR}/p4" in your recipe. | ||
571 | |||
572 | By default, the fetcher strips the depot location from the local file paths. In | ||
573 | the above example, the content of ``example-depot/main/source/`` will be placed | ||
574 | in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot | ||
575 | paths locally is desirable, the fetcher supports two parameters: | ||
576 | |||
577 | - *"module":* | ||
578 | The top-level depot location or directory to fetch. The value of this | ||
579 | parameter can also point to a single file within the depot, in which case | ||
580 | the local file path will include the module path. | ||
581 | - *"remotepath":* | ||
582 | When used with the value "``keep``", the fetcher will mirror the full depot | ||
583 | paths locally for the specified location, even in combination with the | ||
584 | ``module`` parameter. | ||
585 | |||
586 | Here is an example use of the the ``module`` parameter: :: | ||
587 | |||
588 | SRC_URI = "p4://user:passwd@example-depot/main;module=source/..." | ||
589 | |||
590 | In this case, the content of the top-level directory ``source/`` will be fetched | ||
591 | to ``${P4DIR}``, including the directory itself. The top-level directory will | ||
592 | be accesible at ``${P4DIR}/source/``. | ||
593 | |||
594 | Here is an example use of the the ``remotepath`` parameter: :: | ||
595 | |||
596 | SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep" | ||
597 | |||
598 | In this case, the content of the top-level directory ``source/`` will be fetched | ||
599 | to ``${P4DIR}``, but the complete depot paths will be mirrored locally. The | ||
600 | top-level directory will be accessible at | ||
601 | ``${P4DIR}/example-depot/main/source/``. | ||
602 | |||
603 | .. _repo-fetcher: | ||
604 | |||
605 | Repo Fetcher (``repo://``) | ||
606 | -------------------------- | ||
607 | |||
608 | This fetcher submodule fetches code from ``google-repo`` source control | ||
609 | system. The fetcher works by initiating and syncing sources of the | ||
610 | repository into :term:`REPODIR`, which is usually | ||
611 | ``${DL_DIR}/repo``. | ||
612 | |||
613 | This fetcher supports the following parameters: | ||
614 | |||
615 | - *"protocol":* Protocol to fetch the repository manifest (default: | ||
616 | git). | ||
617 | |||
618 | - *"branch":* Branch or tag of repository to get (default: master). | ||
619 | |||
620 | - *"manifest":* Name of the manifest file (default: ``default.xml``). | ||
621 | |||
622 | Here are some example URLs: :: | ||
623 | |||
624 | SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml" | ||
625 | SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml" | ||
626 | |||
627 | Other Fetchers | ||
628 | -------------- | ||
629 | |||
630 | Fetch submodules also exist for the following: | ||
631 | |||
632 | - Bazaar (``bzr://``) | ||
633 | |||
634 | - Mercurial (``hg://``) | ||
635 | |||
636 | - npm (``npm://``) | ||
637 | |||
638 | - OSC (``osc://``) | ||
639 | |||
640 | - Secure FTP (``sftp://``) | ||
641 | |||
642 | - Secure Shell (``ssh://``) | ||
643 | |||
644 | - Trees using Git Annex (``gitannex://``) | ||
645 | |||
646 | No documentation currently exists for these lesser used fetcher | ||
647 | submodules. However, you might find the code helpful and readable. | ||
648 | |||
649 | Auto Revisions | ||
650 | ============== | ||
651 | |||
652 | We need to document ``AUTOREV`` and ``SRCREV_FORMAT`` here. | ||