linux/poky.git/bitbake/lib/bb/fetch2, branch yocto-5.0.2

bitbake: fetch2/wget: Fix failure path for files that are empty or don't exist

2024-06-24T13:46:52+00:00

When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: 8714a02e13477a9d97858b3642e05f28247454b5) Signed-off-by: Richard Purdie (cherry picked from commit c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Steve Sakoman

bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

2024-06-24T13:46:52+00:00

Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Rudolf J Streif Signed-off-by: Richard Purdie (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman

bitbake: fetch2/gcp: Add missing runfetchcmd import

2024-05-29T14:50:00+00:00

This adds the missing import. This bug was introduced with 1ab1d36c. (Bitbake rev: b5159c0373e2e7d403aed16e096ad655f38b1fa7) Signed-off-by: Sven Schwermer Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftps

2024-04-11T07:47:07+00:00

Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley Signed-off-by: Richard Purdie

bitbake: fetch2/git: Install Git LFS in local repository config

2024-03-22T16:26:46+00:00

Git uses a lock file to prevent concurrent modifications to the global config, so if unpack tasks for different recipes try to run "git lfs install" simultaneously the operation can fail: error: could not lock config file /home/build/.gitconfig: File exists exit status 255 Run `git lfs install --force` to reset Git configuration. Adding "--local" sets the smudge and clean filters in the local repository's config instead of modifying the user's global config. (Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7) Signed-off-by: Derek Erdmann Signed-off-by: Richard Purdie

bitbake: fetch2: Fix misleading "no output" msg

2024-03-22T16:26:45+00:00

When a command is run with a non-null log, errors are only output to the log and are not returned in the exception. In that case direct users to that logfile instead of telling the command had no output. (Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a) Signed-off-by: Viswanath Kraleti Signed-off-by: Richard Purdie

bitbake: fetch2: handle URIs with single-valued query parameters

2024-03-22T16:26:45+00:00

Whilst typically the URI query is a list of key-value pairs, that's not actually required by the URI specification. For example: http://example.com/foo?bar is a valid query, but this will result in the fetcher raising an exception: File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__ self.query = self._param_str_split(urlp.query, "&") File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]: ValueError: not enough values to unpack (expected 2, got 1) In this case the query is just "bar", but the fetcher is trying to split it into a key-value pair. The URI object exposes the parsed query explicitly as a dictionary of key-value pairs, so we have to be a little creative here: if a value is None then it isn't a key-value pair, but a bare key. Fix this by handling elements without the deliminator in _param_str_split() (by assigning the value to None), and handle a None value when formatting the query in _param_str_join(). (Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: bitbake: fetch2/git: Escape parentheses in git src name

2024-03-22T16:26:45+00:00

FIXES [YOCTO #15404] When using git fetcher on a repo with parentheses in its URL, the invocation of the git clone command will fail. The clone directory is not quoted thus the shell will return an error and won't execute the command. (Bitbake rev: b5624ee5643d881afa004571a096a189ab5389b5) Cc: Philippe Rivest (Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6) Signed-off-by: Philippe Rivest Signed-off-by: Richard Purdie

bitbake: fetch/git: Avoid clean upon failure

2024-03-03T16:26:17+00:00

Currently when git fetches fail, it destroys all the existing local clone data. For large repositories this can introduce long build delays when for example, you just typo'd the git revision hash. The git fetcher should be able to recover most directories so when the fetch is for a git repo, avoid removing things unless clean is explicitly called (e.g. a -c cleanall task). (Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443) Signed-off-by: Richard Purdie

bitbake: fetch2: Ensure that git LFS objects are available

2024-02-23T14:34:05+00:00

The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie