linux/poky.git/bitbake/lib/bb/fetch2/wget.py, branch styhead-5.1.4

bitbake: fetch/wget: Increase timeout to 100s from 30s

2024-11-30T13:41:59+00:00

Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: 70b09f62bc5622fa64bb668cdad610bd48bf7bb9) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: fetch/wget: checkstatus: drop shared connecton when catch Timeout error

2024-07-04T21:56:41+00:00

* to avoid wrong http response in checkstatus function: in wget checkstatus() we are using 'HTTPConnectionCache' to share connections 1. state_file1(exists on http server) use shared connection to send request 2. http_server recieved request of state_file1, but delayed by some reason to sent respone 3. state_file1 checkstatus() failed by timeout and drop shared connection 4. state_file2(not exists on http server) get shared connection and send request 5. http_server finally send 200 response for state_file1 6. state_file2 recived 200 response and thought it was exists on http_server (Bitbake rev: bf6d0282ab88b4edc4b9e58184cd76cce965abbd) Signed-off-by: y75zhang Signed-off-by: Richard Purdie

bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

2024-06-07T12:45:43+00:00

Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Rudolf J Streif Signed-off-by: Richard Purdie

bitbake: fetch2/wget: Fix failure path for files that are empty or don't exist

2024-05-31T15:56:25+00:00

When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Richard Purdie

bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftps

2024-04-11T07:47:07+00:00

Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley Signed-off-by: Richard Purdie

bitbake: Revert "bitbake: wget.py: always use the custom user agent"

2024-02-20T12:56:40+00:00

This reverts commit 987ab2a446aab235d6e86e97c05f25fb800d7acc. There's been a report that this breaks downloads from Jfrog Artifactory as self.user_agent is set to 'Mozilla Firefox', and when Artifactory sees that, it sends a response tailored for showing in an interactive browser (which in my opinion it has every right to). If we're using wget, we should say so via wget's default; handling uncooperative servers should be done on per-recipe basis, and ideally with tickets to admins of those servers. (Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: wget.py: always use the custom user agent

2024-02-10T14:13:51+00:00

Add the "--user-agent" paramater in the wget base command to perform all wget commands with this parameter, because a few HTTP servers block requests with the default wget user agent. For example, "hg.openjdk.org" never send a response to requests have been sent with wget: wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73 Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected. HTTP request sent, awaiting response... (Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c) Signed-off-by: Thomas Perrot Signed-off-by: Richard Purdie

bitbake: fetch/wget/checkstatus(): include the URL in debugging output about status check failure

2024-01-10T13:55:33+00:00

Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: wget.py: Combine urlopener exceptions

2023-03-11T20:49:22+00:00

No reason to have three identical exception handles, refactor to catch any of the exceptions with the same block of code. (Bitbake rev: b29f6e04091b6bfe697dc41c76880de466736fc3) Signed-off-by: Mark Hatle Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: wget.py: Add catch TimeoutError exception

2023-03-11T20:49:22+00:00

We've observed TimeoutError exceptions during the sstate-cache mirror fetch, it appears that due to the number of (invalid) files requested the remote side is eventually dropping the connection (not closing it) which can result in a TimeoutError exception being sent, while rate it is different from the urllib.error.URLError or ConnectionResetError. (Bitbake rev: 6041b34740deee09ea65d705702555456a5e05d8) Signed-off-by: Mark Hatle Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie