linux/poky.git/bitbake/lib/bb/fetch2, branch yocto-4.0.24

bitbake: fetch/wget: Increase timeout to 100s from 30s

2024-12-02T14:23:20+00:00

Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: 3f88b005244a0afb5d5c7260e54a94a453ec9b3e) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: fetch2: use persist_data context managers

2024-12-02T14:23:19+00:00

Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s. See https://docs.python.org/3/whatsnew/3.13.html#sqlite3 The previous commit fixed persist_data's context manager to close the connection, but we were never actually using `with` in the first place. This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: 9789c55ecc90ba074596061fa16e90d3e8accb02) Signed-off-by: Chris Laplante (cherry picked from commit 61f803c7d92a012b62837b0cdae4789a394b260e) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: gitsm: Remove downloads/tmpdir when failed

2024-11-02T13:06:09+00:00

The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 15582daed9a18330bcf1ad316a57d46571bbf7c6) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Steve Sakoman

bitbake: gitsm: Add call_process_submodules() to remove duplicated code

2024-11-02T13:06:09+00:00

There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: ff2dfda55258d8034ea748d87222e51124a03f02) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Steve Sakoman

bitbake: fetch2/git: Use quote from shlex, not pipes

2024-11-02T13:06:09+00:00

The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 53264dc14554890b3a2afc83cb1749cf10d86854) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

2024-10-10T19:01:06+00:00

Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 07081a94997142746f7d345c27bc6805231d025d) Signed-off-by: Rudolf J Streif Signed-off-by: Richard Purdie (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman (cherry picked from commit 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Steve Sakoman

bitbake: fetch/wget: Move files into place atomically

2024-10-10T19:01:06+00:00

(Bitbake rev: aaf9c1b8673f5b181255a08eee4e9a342863e896) Signed-off-by: Richard Purdie (cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Steve Sakoman

bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftps

2024-10-10T19:01:06+00:00

Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: 477f90f9515cb7cd093920cee5518f662cef2e05) Signed-off-by: Rob Woolley Signed-off-by: Richard Purdie (cherry picked from commit f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Steve Sakoman

bitbake: fetch2: Ensure that git LFS objects are available

2024-03-01T18:00:58+00:00

The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (cherry picked from commit cfae1556bf671acec119a6c8bbc4b667a856b9ae) (Bitbake rev: 40fd5f4eef7460ca67f32cfce8e229e67e1ff607) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie Signed-off-by: Philip Lorenz Signed-off-by: Steve Sakoman

bitbake: fetch/git: Fix local clone url to make it work with repo

2023-04-11T10:31:52+00:00

The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie