linux/poky.git/bitbake/lib/bb/fetch2/wget.py, branch uninative-2.1

bitbake: fetch/wget: use with to ensure the response is closed

2017-11-21T13:06:45+00:00

(Bitbake rev: 8c487176d311557031cedba76185f14f0e7a14cd) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch/wget: improve proxy object

2017-11-21T13:06:45+00:00

The connection cache class uses a dummy file object but it doesn't have a closed attribute, so we can't use it in a context manager. (Bitbake rev: 7b072ef91d16331eae11bd60f229ce1f0c175995) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: wget: fix FusionForge workaround

2017-07-31T14:13:53+00:00

My previous assertion about FusionForge appears to have been wrong, or FusionForge has changed behaviour, or both. FusionForge now mandates that downloads have the Accept header set, despite that header being optional, and returns a 406 Not Acceptable error if it isn't set. As we were pretending that 406 was actually 405 (Moved) and tried to handle it as a redirect this results in an infinite loop until Python kills the recursion. Delete the handling of 406 as 405, and pass Accept: */* in the headers. (Bitbake rev: bb70ae0c9aac5ec688026d23a64ac0cac1947187) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch/wget: mitigate a wget race condition when listing FTP directories

2017-07-25T14:53:19+00:00

When wget is fetching a listing for a directory over FTP it writes to a temporary file called .listing in the current directory. If there are many such operations happening in parallel - for example during 'bitbake world -c checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to write to, read, and delete the same file. This results in various failures such as the file disappearing before wget has processed it or the file changing contents, which causes checkpkg to randomly fail. Mitigate the race condition by creating a temporary directory to run wget in when doing directory listings. [ YOCTO #11828 ] (Bitbake rev: 91d4ca93df092cf86ab84faaa94cc66ff9f43057) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch2/wget.py: improve error handling during sstate check

2017-07-21T08:26:37+00:00

When the sstate is accessed via HTTP, the existence check can fail due to network issues, in which case bitbake silently continues without sstate. One such network issue is an HTTP server like Python's own SimpleHTTP which closes the TCP connection despite an explicit "Keep-Alive" in the HTTP request header. The server does that without a "close" in the HTTP response header, so the socket remains in the connection cache, leading to "urlopen failed: " (only visible in "bitbake -D -D" output) when trying to use the cached connection again. The connection might also get closed for other reasons (proxy, timeouts, etc.), so this is something that the client should be able to handle. This is achieved by checking for the error, removing the bad connection, and letting the check_status() method try again with a new connection. It is necessary to let the second attempt fail permanently, because bad proxy setups have been observed to also lead to such broken connections. In that case, we need to abort for real after trying twice, otherwise a build would just hang forever. [YOCTO #11782] (Bitbake rev: 6fa07752bbd3ac345cd8617da49a70e0b2dd565f) Signed-off-by: Patrick Ohly Signed-off-by: Richard Purdie

bitbake: wget: Fix handling of urls with user/password

2017-03-31T09:08:36+00:00

URL decoding was improved in the core a while ago and this looks like a leftover from those times which caused urls needing a user/password to fail. Use the parameters from the core instead of the broken split implementation. [YOCTO #11262] (Bitbake rev: 6a917ec99d659e684b15fa8af94c325172676062) Signed-off-by: Richard Purdie

bitbake: fetch2: don't use deprecated bb.data APIs

2017-03-01T11:16:07+00:00

Cleanup some more usage of bb.data APIs in the fetchers. (Bitbake rev: 9752fd1c10b8fcc819822fa6eabc2c1050fcc03b) Signed-off-by: Andre McCurdy Signed-off-by: Richard Purdie

bitbake: wget.py: match .lz tarballs as well when determining latest version

2017-03-01T11:16:07+00:00

(Bitbake rev: c55e09fbd2b64a2b909d9992350b6b2e26e5e86d) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: fetch2/wget: fixup case with no useful netrc data

2016-12-21T08:17:01+00:00

Commit 873e33d0479e977520106b65d149ff1799195bf6 [fetch2/wget: add Basic Auth from netrc to checkstatus()] causes "Fetcher failure for URL: 'https://www.example.com/'. URL https://www.example.com/ doesn't work." on new builds when a user has a .netrc file but there is no default and no matching host. The call to netrc.authenticators() will return None in these cases and the attempted assignment to the 3-tuple will raise a TypeError exception. Add the TypeError to the exceptions caught to get around this issue. (Bitbake rev: c0c0af40ebddaf9dc99353c580a65d4c04295613) Signed-off-by: Mark Asselstine Signed-off-by: Richard Purdie

bitbake: fetch2/wget: add Basic Auth from netrc to checkstatus()

2016-12-20T15:22:51+00:00

fetch2/wget uses urllib to check the status of the mirrors, wget will use netrc to pass login and password information however checkstatus will skip that. This adds netrc login and password to checkstatus so both will work the same. (Bitbake rev: 873e33d0479e977520106b65d149ff1799195bf6) Signed-off-by: Matthew McClintock Signed-off-by: Richard Purdie