linux/poky.git/bitbake/lib/bb/fetch2/wget.py, branch yocto-4.0.19

bitbake: fetch2: Abstract fetcher environment to a function

2022-02-20T16:45:25+00:00

The changing of the environment inside the wget fetcher can race if threading is used, such as with sstate in OE-Core. Abstract the function so the environment can be correct before the function is called, removing the race since the enviroment is then no longer changed. (Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23) Signed-off-by: Richard Purdie

bitbake: wget: Fix grammar "can happen"

2022-02-20T16:45:25+00:00

(Bitbake rev: 52630eefb5174e4ca357ac57085093a7f5767bd8) Signed-off-by: Zygmunt Krynicki Signed-off-by: Richard Purdie

bitbake: fetch2/wget: move loop-invariant load of BB_ORIGENV

2022-02-20T16:45:24+00:00

BB_ORIGENV is used as a fallback environment block. It is repeatedly accessed inside the loop. Since it is a loop invariant, move it out of the loop. (Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a) Signed-off-by: Zygmunt Krynicki Signed-off-by: Richard Purdie

bitbake: fetch2/wget: do not hardcode tarball compressors in version check

2021-12-08T20:22:53+00:00

(Bitbake rev: ba3aa8591327d43935f000c6884637997438ecb2) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: fetch2/wget: add redirectauth parameter

2021-12-08T20:22:53+00:00

Add a parameter that limits sending Basic authentication in the Authorization header to only the first host and not any that we're redirected to. Ignoring potential security concerns, temporary AWS URLs will reject any request that includes authentication details in both the query parameters (from the redirect) and in the Authorization header. Temporary AWS URLs are now being used for release assets from private Github repositories. According to the previous discussion linked below, they're also in use by bitbucket. See also: https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/ (Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0) Signed-off-by: Justin Bronder Signed-off-by: Richard Purdie

bitbake: fetch/wget: Add timeout for checkstatus calls (30s)

2021-11-10T19:25:38+00:00

We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 15s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] (Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3) Signed-off-by: Richard Purdie

bitbake: fetch2/wget: Enable ftps

2021-09-16T08:51:14+00:00

The fetcher would fail with: Could not find a fetcher which supports the URL: ftps://... (Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4) Signed-off-by: Daniel Ammann Signed-off-by: Richard Purdie

bitbake: fetch2/wget: fix 'no_proxy' handling

2021-08-23T07:30:54+00:00

The urllib.request.ProxyHandler constructor only reads the $http_proxy + $https_proxy environment variables. $no_proxy is evaluated later when the url is opened. It is therefore not sufficient to just construct the proxy handler in the | with bb.utils.environment(**newenv): context, but the 'opener.open(r)' call must also be made there. (Bitbake rev: 076baf4fbd328d247508fd399866a397eb34f67e) Signed-off-by: Enrico Scholz Signed-off-by: Richard Purdie

bitbake: fetch2/wget: fetch securely by default

2021-08-12T05:28:01+00:00

The days of broken certificates are behind us now, so instead of always passing --no-check-certificate to wget, don't pass it by default and instead only pass it BB_CHECK_SSL_CERTS = "0". [ YOCTO #14108 ] (Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch2/wget: ensure all variables are set when calling urllib

2021-08-12T05:28:01+00:00

Instead of just exporting the proxy variables when calling into urllib, use bb.utils.environment() to export all of the known variables that are needed for proper connectivity. Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can find the certificates in buildtools environments (Bitbake rev: 116637b0e9aabae7f680b102dbf3577b8a58f049) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie