linux/poky.git/bitbake/lib/bb/fetch2/wget.py, branch 5.2_M3

bitbake: fetch2: remove unnecessary unquote

2025-02-11T21:18:39+00:00

The URI path is already unquoted. Remove the unnecessary unquote function calls for URI path values. (Bitbake rev: 3de12bbc28b5a4189b849720735cf3e268d3941d) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: fetch2: remove unnecessary expand function calls

2025-02-11T21:18:39+00:00

The fetch data class already expands the type, host, path, user, pswd and parm variables. The fetcher classes already expand the localfile variable. The getVar function expands the returned string per default. Remove unnecessary expand function calls to simplify the code. (Bitbake rev: 1b1eb037b861fbf20491ac17e519e9eaf232b858) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: fetch2/wget: correctly construct the target URL in checkstatus()

2024-12-20T16:25:28+00:00

ud.path has been unescaped (eg %20 is space) but as we're reconstructing a URL we should re-escape it. For example, unzip has a SRC_URI containing "UnZip%206.x%20%28latest%29/UnZip%206.0/unzip60.tar.gz" which then throws exceptions if the unescaped string " (latest)" is used. Also, this code uses the extracted ud.host and ud.path variables. These are unescaped but potentially stale as eg the cargo fetcher subclasses Wget() and reassigns ud.url on construction. Simplify the code by reconstructing a URL from ud.url directly instead of bouncing through intermediate variables that may be wrong or unescaped. (Bitbake rev: c9906bb289dcfd9ae41f10bd5399ccc17a4cc437) Signed-off-by: Ross Burton Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie

bitbake: fetch2/wget: handle HTTP 308 Permanent Redirect

2024-12-20T16:25:28+00:00

urllib2.HTTPRedirectHandler.redirect_request doesn't handle HTTP reponse code 308 (Permanent Redirect). This was fixed in c379bc5 but can't be worked around without copying the entire redirect_request() method. When we can depend on Python 3.13, FixedHTTPRedirectHandler can be removed. (Bitbake rev: 365829a2803b954ee6cb0364749551a91d806075) Signed-off-by: Ross Burton Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie

bitbake: fetch/wget: Increase timeout to 100s from 30s

2024-11-21T12:16:28+00:00

Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3) Signed-off-by: Richard Purdie

bitbake: fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()

2024-10-22T10:16:32+00:00

This eliminates the last usage of 'fake mozilla' in bitbake, and it's then truthful everywhere about presenting itself, or wget (when that is used). I understand this will make people nervous so I want to provide an extended decription. 1. How was this tested? - bitbake-selftest -k FetchCheckStatusTest (tests a few hardcoded URIs, all passed) - bitbake -k -c checkuri world (runs checkstatus() over all recipes in oe-core, and all passed again - this hopefully goes a long way to reassure everyone that hosts around the world and various CDNs typically do not have a problem with user-agent strings they haven't seen before or bitbake user-agent specifically) 2. What about that removed cloudflare comment? I digged into git history, and I think it is not fully accurate. First, 'fake mozilla' agent is used only for checkstatus() - in actual fetching with wget it is not. And that has not been a problem for anyone. Second, here's how the comment occured. Usage of 'fake mozilla' was introduced here: https://git.yoctoproject.org/poky/commit/?h=master&id=ab26fdae9e5ae56bb84196698d3fa4fd568fe903 At that point it did not have to be specifically 'mozilla', the commit message indicates that any User-Agent would have been ok. Mozilla was simply copied from upstream version check for convenience. Later on, the string was updated to a more recent Mozilla: https://git.yoctoproject.org/poky/commit/?h=master&id=9f123238261a68e37cec634782e9320633cac5d4 The claim in the added comment become something else: that User-Agent *must* a browser, without evidence or tests. Even though it demonstrably doesn't have to be - wget is ok. 3. What if someone has a server that is ok with wget agent, but not ok with bitbake agent? Please see point one. It's not impossible but I think it's highly unlikely. I do think we should rather tell servers the truth, and learn where the actual issues are. Then we can consider options - whether that would be pretending to be wget, or allowing user-agent to be configured. We should also add such servers to bitbake-selftest so we know what they are. (Bitbake rev: 234f9e810494394527f59fdf22eb86435d046d53) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: fetch2/wget.py: use default wget user-agent in upstream version checks

2024-10-22T10:16:32+00:00

This has been set to various mozilla versions from this point onwards without explanation or specific examples: https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6 However, sourceforge has recently banned the current 'fake mozilla' version we've been supplying, resulting in broken version checks: alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk ... --2024-10-12 11:53:23-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. HTTP request sent, awaiting response... 403 Forbidden 2024-10-12 11:53:24 ERROR 403: Forbidden. gptfdisk 1.0.10 UNKNOWN_BROKEN Alexander Kanavin On the other hand, using wget's default user-agent works just fine: --2024-10-12 12:16:06-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] (Bitbake rev: 138117dfc05248dcb48707c838c8c1cc09f12d07) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: fetch/wget: checkstatus: drop shared connecton when catch Timeout error

2024-07-04T21:56:41+00:00

* to avoid wrong http response in checkstatus function: in wget checkstatus() we are using 'HTTPConnectionCache' to share connections 1. state_file1(exists on http server) use shared connection to send request 2. http_server recieved request of state_file1, but delayed by some reason to sent respone 3. state_file1 checkstatus() failed by timeout and drop shared connection 4. state_file2(not exists on http server) get shared connection and send request 5. http_server finally send 200 response for state_file1 6. state_file2 recived 200 response and thought it was exists on http_server (Bitbake rev: bf6d0282ab88b4edc4b9e58184cd76cce965abbd) Signed-off-by: y75zhang Signed-off-by: Richard Purdie

bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

2024-06-07T12:45:43+00:00

Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Rudolf J Streif Signed-off-by: Richard Purdie

bitbake: fetch2/wget: Fix failure path for files that are empty or don't exist

2024-05-31T15:56:25+00:00

When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Richard Purdie