linux/poky.git/bitbake/lib/bb/fetch2, branch scarthgap-5.0.12

bitbake: gcp.py: remove slow calls to gsutil stat

2025-06-11T15:17:34+00:00

The changes of 1ab1d36c0af6fc58a974106b61ff4d37da6cb229 added calls to "gsutil stat" to avoid unhandled exceptions, however: - in the case of checkstatus() this is redundant with the call to self.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False and does not throw an exception in case the file does not exist. - Also the call to gsutil stat is much slower than using the python client to call exists() so we should not replace the call to exists() with a call to gsutil stat. - I think the intent of calling check_network_access in checkstatus() was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable. - add a try/except block in download() instead of the extra call to gsutil [RP: Tweak to avoid import until needed so google module isn't required for everyone] (Bitbake rev: 59df5390381792aba4f3f5185000adf5109267fb) Signed-off-by: Etienne Cordonnier Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: fetch2: Avoid deprecation warning

2025-06-05T15:41:15+00:00

>From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated. In future Python versions they will be keyword-only parameters." Avoid the warning. (Bitbake rev: 65b744bec9756ee2f43adbfa33c14899638e2b9f) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: fetch2: use persist_data context managers

2024-12-06T13:50:24+00:00

Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s. See https://docs.python.org/3/whatsnew/3.13.html#sqlite3 The previous commit fixed persist_data's context manager to close the connection, but we were never actually using `with` in the first place. This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: 6c2641f7a9e92c1b82e306f59ddd3c1249c52cbf) Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: fetch/wget: Increase timeout to 100s from 30s

2024-12-06T13:50:24+00:00

Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: c7f282cd27edfd78830b61db586ed669808893a5) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: fetch2/git: Use quote from shlex, not pipes

2024-11-28T13:36:00+00:00

The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 70bd343fb273ad174e56d08c8b80c5594501e030) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: gitsm: Remove downloads/tmpdir when failed

2024-11-02T13:12:00+00:00

The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 6894f68204130713d3651fceacadc7f8061174a8) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Steve Sakoman

bitbake: gitsm: Add call_process_submodules() to remove duplicated code

2024-11-02T13:12:00+00:00

There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: 76de94d46ce270c4e485f8178725973adf0f42eb) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Steve Sakoman

bitbake: fetch2/wget: Fix failure path for files that are empty or don't exist

2024-06-24T13:46:52+00:00

When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: 8714a02e13477a9d97858b3642e05f28247454b5) Signed-off-by: Richard Purdie (cherry picked from commit c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Steve Sakoman

bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

2024-06-24T13:46:52+00:00

Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Rudolf J Streif Signed-off-by: Richard Purdie (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman

bitbake: fetch2/gcp: Add missing runfetchcmd import

2024-05-29T14:50:00+00:00

This adds the missing import. This bug was introduced with 1ab1d36c. (Bitbake rev: b5159c0373e2e7d403aed16e096ad655f38b1fa7) Signed-off-by: Sven Schwermer Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman