linux/poky.git/bitbake/lib/bb/fetch2/wget.py, branch 5.3_M2

bitbake: fetch2/wget: use long arguments for clarity

2025-03-27T13:40:31+00:00

It's best practise to use long-form arguments in scripts where the conciseness of short arguments is less useful than in an interactive terminal. (Bitbake rev: 54039bc9d169871af6d36578df4c21bff296f6de) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch2/wget: don't conditionalise --continue

2025-03-27T13:40:31+00:00

If the target file exists, we pass --continue so that wget will try to resume the download instead of starting from the beginning. However if the file doesn't exist --continue has no effect, so there's no need to conditionalise the use of the argument. (Bitbake rev: a5ee50d4d2b2e6c00abe1551382afd9799345dbf) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch2/wget: consider downloadfilename when checking for upstream

2025-03-27T13:40:31+00:00

latest_versionstring() currently looks at just the end of the URI when guessing what the filename to look for is, but this doesn't work if the URL filename is not simple. For example, miniupnpd has a SRC_URI of: http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz;downloadfilename=${BP}.tar.gz The filename component of this is "download.php", which causes the heuristics in latest_versionstring() to exit early. Instead, if the downloadfilename is set then use that, as it's often the actual filename that we're after. (Bitbake rev: 2d5f135e997d13fabda0ad266fd5c928ee33f487) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch2: remove unnecessary unquote

2025-02-11T21:18:39+00:00

The URI path is already unquoted. Remove the unnecessary unquote function calls for URI path values. (Bitbake rev: 3de12bbc28b5a4189b849720735cf3e268d3941d) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: fetch2: remove unnecessary expand function calls

2025-02-11T21:18:39+00:00

The fetch data class already expands the type, host, path, user, pswd and parm variables. The fetcher classes already expand the localfile variable. The getVar function expands the returned string per default. Remove unnecessary expand function calls to simplify the code. (Bitbake rev: 1b1eb037b861fbf20491ac17e519e9eaf232b858) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: fetch2/wget: correctly construct the target URL in checkstatus()

2024-12-20T16:25:28+00:00

ud.path has been unescaped (eg %20 is space) but as we're reconstructing a URL we should re-escape it. For example, unzip has a SRC_URI containing "UnZip%206.x%20%28latest%29/UnZip%206.0/unzip60.tar.gz" which then throws exceptions if the unescaped string " (latest)" is used. Also, this code uses the extracted ud.host and ud.path variables. These are unescaped but potentially stale as eg the cargo fetcher subclasses Wget() and reassigns ud.url on construction. Simplify the code by reconstructing a URL from ud.url directly instead of bouncing through intermediate variables that may be wrong or unescaped. (Bitbake rev: c9906bb289dcfd9ae41f10bd5399ccc17a4cc437) Signed-off-by: Ross Burton Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie

bitbake: fetch2/wget: handle HTTP 308 Permanent Redirect

2024-12-20T16:25:28+00:00

urllib2.HTTPRedirectHandler.redirect_request doesn't handle HTTP reponse code 308 (Permanent Redirect). This was fixed in c379bc5 but can't be worked around without copying the entire redirect_request() method. When we can depend on Python 3.13, FixedHTTPRedirectHandler can be removed. (Bitbake rev: 365829a2803b954ee6cb0364749551a91d806075) Signed-off-by: Ross Burton Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie

bitbake: fetch/wget: Increase timeout to 100s from 30s

2024-11-21T12:16:28+00:00

Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3) Signed-off-by: Richard Purdie

bitbake: fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()

2024-10-22T10:16:32+00:00

This eliminates the last usage of 'fake mozilla' in bitbake, and it's then truthful everywhere about presenting itself, or wget (when that is used). I understand this will make people nervous so I want to provide an extended decription. 1. How was this tested? - bitbake-selftest -k FetchCheckStatusTest (tests a few hardcoded URIs, all passed) - bitbake -k -c checkuri world (runs checkstatus() over all recipes in oe-core, and all passed again - this hopefully goes a long way to reassure everyone that hosts around the world and various CDNs typically do not have a problem with user-agent strings they haven't seen before or bitbake user-agent specifically) 2. What about that removed cloudflare comment? I digged into git history, and I think it is not fully accurate. First, 'fake mozilla' agent is used only for checkstatus() - in actual fetching with wget it is not. And that has not been a problem for anyone. Second, here's how the comment occured. Usage of 'fake mozilla' was introduced here: https://git.yoctoproject.org/poky/commit/?h=master&id=ab26fdae9e5ae56bb84196698d3fa4fd568fe903 At that point it did not have to be specifically 'mozilla', the commit message indicates that any User-Agent would have been ok. Mozilla was simply copied from upstream version check for convenience. Later on, the string was updated to a more recent Mozilla: https://git.yoctoproject.org/poky/commit/?h=master&id=9f123238261a68e37cec634782e9320633cac5d4 The claim in the added comment become something else: that User-Agent *must* a browser, without evidence or tests. Even though it demonstrably doesn't have to be - wget is ok. 3. What if someone has a server that is ok with wget agent, but not ok with bitbake agent? Please see point one. It's not impossible but I think it's highly unlikely. I do think we should rather tell servers the truth, and learn where the actual issues are. Then we can consider options - whether that would be pretending to be wget, or allowing user-agent to be configured. We should also add such servers to bitbake-selftest so we know what they are. (Bitbake rev: 234f9e810494394527f59fdf22eb86435d046d53) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: fetch2/wget.py: use default wget user-agent in upstream version checks

2024-10-22T10:16:32+00:00

This has been set to various mozilla versions from this point onwards without explanation or specific examples: https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6 However, sourceforge has recently banned the current 'fake mozilla' version we've been supplying, resulting in broken version checks: alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk ... --2024-10-12 11:53:23-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. HTTP request sent, awaiting response... 403 Forbidden 2024-10-12 11:53:24 ERROR 403: Forbidden. gptfdisk 1.0.10 UNKNOWN_BROKEN Alexander Kanavin On the other hand, using wget's default user-agent works just fine: --2024-10-12 12:16:06-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] (Bitbake rev: 138117dfc05248dcb48707c838c8c1cc09f12d07) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie