summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2/wget.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-04-111-1/+4
| | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "bitbake: wget.py: always use the custom user agent"Alexander Kanavin2024-02-201-4/+3
| | | | | | | | | | | | | | | | | | This reverts commit 987ab2a446aab235d6e86e97c05f25fb800d7acc. There's been a report that this breaks downloads from Jfrog Artifactory as self.user_agent is set to 'Mozilla Firefox', and when Artifactory sees that, it sends a response tailored for showing in an interactive browser (which in my opinion it has every right to). If we're using wget, we should say so via wget's default; handling uncooperative servers should be done on per-recipe basis, and ideally with tickets to admins of those servers. (Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: always use the custom user agentThomas Perrot2024-02-101-3/+4
| | | | | | | | | | | | | | | | | | | Add the "--user-agent" paramater in the wget base command to perform all wget commands with this parameter, because a few HTTP servers block requests with the default wget user agent. For example, "hg.openjdk.org" never send a response to requests have been sent with wget: wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73 Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected. HTTP request sent, awaiting response... (Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c) Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget/checkstatus(): include the URL in debugging output about ↵Alexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | status check failure Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Combine urlopener exceptionsMark Hatle2023-03-111-17/+1
| | | | | | | | | | | No reason to have three identical exception handles, refactor to catch any of the exceptions with the same block of code. (Bitbake rev: b29f6e04091b6bfe697dc41c76880de466736fc3) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Add catch TimeoutError exceptionMark Hatle2023-03-111-0/+8
| | | | | | | | | | | | | | We've observed TimeoutError exceptions during the sstate-cache mirror fetch, it appears that due to the number of (invalid) files requested the remote side is eventually dropping the connection (not closing it) which can result in a TimeoutError exception being sent, while rate it is different from the urllib.error.URLError or ConnectionResetError. (Bitbake rev: 6041b34740deee09ea65d705702555456a5e05d8) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Drop unused importRichard Purdie2023-02-261-1/+0
| | | | | | | | This import is no longer used anywhere so can be removed. (Bitbake rev: 956128e394581855bf0d03b32a975dc91c2a7e0c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: clean up netrc usageRoss Burton2023-02-171-4/+5
| | | | | | | | | | | Assigning a return value which is potentially None to a tuple and catching TypeError is pretty ugly. Rewrite the code to explicitly check the value for clarity. (Bitbake rev: f4ebb27616ac2df27c29a6052b1526a4c48db607) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: correctly match versioned directoriesAlexander Kanavin2022-12-111-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | When obtaining latest upstream versions, the code needs to check if the existing tarball is in a versioned directory (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and if it is, it needs to first obtain the list of all such versioned directories and then check all of them by going one step up in the directory hierarchy. Existing code was returning a correct match when the component name did not have numbers, e.g. a check on 'source/epiphany/43/' would return 43, but was stopping too soon when the component name itself had numbers ('source/libxml2/2.10/' would return libxml2). This change ensures the last match is taken instead of the first. Also, adjust the fetcher tests to check that versioned directories are correctly traversed in this case (e.g. the step to go one level up is taken and a new tarball is discovered in a different versioned directory). (Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: handle username/password in uriKasper Revsbech2022-12-091-1/+2
| | | | | | | | | | | | | | | | | In checkstatus() opener.open() is used to check if an artifact is available. The check fails if the uri contains username password in the format: "username:password@hostname..". Moreover, the checkstatus function already uses the username from the "ud" object to craft a header, is username and password is provided. This fix ensure the uri in the Requests object used does not contain username as password. (Bitbake rev: 88350002d45e0aa85ecd5356da2c8d71e450641e) Signed-off-by: Kasper Revsbech <kasper.revsbech.ext@siemensgamesa.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Avoid bad checksum race issuesRichard Purdie2022-09-051-0/+5
| | | | | | | | | | | | | | | | | If two recipes have conflicting checksums for a file, the code will currently remove the existing file when a mismatch is downloaded, even if another task successfully fetched it. This changes the code to verify the checksum (if possible) before replacing the file. This removes a potential race window and stops builds failing everywhere from one incorrect checksum. To make this work, we need to be able to override localpath and avoid NoChecksum errors being logged. (Bitbake rev: 4b8de2e7d12667d69d86ffe6e9f85a7932c4c9a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Move files into place atomicallyRichard Purdie2022-06-081-4/+7
| | | | | | (Bitbake rev: cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Abstract fetcher environment to a functionRichard Purdie2022-02-201-8/+1
| | | | | | | | | | | The changing of the environment inside the wget fetcher can race if threading is used, such as with sstate in OE-Core. Abstract the function so the environment can be correct before the function is called, removing the race since the enviroment is then no longer changed. (Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Fix grammar "can happen"Zygmunt Krynicki2022-02-201-1/+1
| | | | | | | (Bitbake rev: 52630eefb5174e4ca357ac57085093a7f5767bd8) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: move loop-invariant load of BB_ORIGENVZygmunt Krynicki2022-02-201-4/+3
| | | | | | | | | | | BB_ORIGENV is used as a fallback environment block. It is repeatedly accessed inside the loop. Since it is a loop invariant, move it out of the loop. (Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: do not hardcode tarball compressors in version checkAlexander Kanavin2021-12-081-1/+1
| | | | | | | (Bitbake rev: ba3aa8591327d43935f000c6884637997438ecb2) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: add redirectauth parameterJustin Bronder2021-12-081-1/+11
| | | | | | | | | | | | | | | | | | | | Add a parameter that limits sending Basic authentication in the Authorization header to only the first host and not any that we're redirected to. Ignoring potential security concerns, temporary AWS URLs will reject any request that includes authentication details in both the query parameters (from the redirect) and in the Authorization header. Temporary AWS URLs are now being used for release assets from private Github repositories. According to the previous discussion linked below, they're also in use by bitbucket. See also: https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/ (Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Add timeout for checkstatus calls (30s)Richard Purdie2021-11-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 15s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] (Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Enable ftpsDaniel Ammann2021-09-161-1/+1
| | | | | | | | | | The fetcher would fail with: Could not find a fetcher which supports the URL: ftps://... (Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fix 'no_proxy' handlingEnrico Scholz2021-08-231-43/+44
| | | | | | | | | | | | | | | | | | | The urllib.request.ProxyHandler constructor only reads the $http_proxy + $https_proxy environment variables. $no_proxy is evaluated later when the url is opened. It is therefore not sufficient to just construct the proxy handler in the | with bb.utils.environment(**newenv): context, but the 'opener.open(r)' call must also be made there. (Bitbake rev: 076baf4fbd328d247508fd399866a397eb34f67e) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fetch securely by defaultRoss Burton2021-08-121-3/+16
| | | | | | | | | | | | | The days of broken certificates are behind us now, so instead of always passing --no-check-certificate to wget, don't pass it by default and instead only pass it BB_CHECK_SSL_CERTS = "0". [ YOCTO #14108 ] (Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: ensure all variables are set when calling urllibRoss Burton2021-08-121-13/+34
| | | | | | | | | | | | | | Instead of just exporting the proxy variables when calling into urllib, use bb.utils.environment() to export all of the known variables that are needed for proper connectivity. Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can find the certificates in buildtools environments (Bitbake rev: 116637b0e9aabae7f680b102dbf3577b8a58f049) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: when checking latest versions, consider all numerical ↵Alexander Kanavin2021-05-141-1/+1
| | | | | | | | | | | | | | directories Previously the regex was maching x.y, but wasn't matching x, which is a problem e.g. here: https://download.gnome.org/sources/epiphany/ (the new gnome version scheme adds 40-series at the end). (Bitbake rev: c03101576f447263ea38e8464210d3a3a2c27226) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatusAlejandro Hernandez Samaniego2021-02-261-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | If an exception is raised when running host python code, the fetcher immediately crashes, this might be temporary depending on the servers reliability. Catch the exception when the connection was reset and try once again to fetch the data. File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto 0665: if self._timeout_occurred: 0666: raise OSError("cannot read from timed out object") 0667: while True: 0668: try: *** 0669: return self._sock.recv_into(b) 0670: except timeout: 0671: self._timeout_occurred = True 0672: raise 0673: except error as e: Exception: ConnectionResetError: [Errno 104] Connection reset by peer (Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-3/+3
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Update user-agentChristophe Priouzeau2021-01-231-3/+8
| | | | | | | | | | | | With the usage of enterprise proxy, the user-agent defined are too old and refused by proxy configuration. Updating to something more modern is desirable. (Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400) Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com> Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Remove buffering parameterRichard Purdie2020-08-241-4/+1
| | | | | | | | | | The buffering parameter was removed in python 3.1 and made default so we can clean up the code. This removes weird looking double exceptions when connections fail. (Bitbake rev: 06b7bafbd18a47c8db2f7b943dc535c65df176bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Set User-Agent when checking status of a URLPaul Barker2020-04-241-0/+1
| | | | | | | | | | | | | | | | | | When a website is behind a CDN like Cloudflare there may be a "Browser Integrity Check" or other test applied to requests before they are allowed through to the server. Downloading via wget passes these tests as headers are set appropriately, however the Python urllib module may fail these tests unless additional headers are set. This causes Wget.checkstatus() to fail where Wget.download() would actually succeed. For Cloudflare in particular a valid User-Agent is needed, it's easy to add this to the headers in Wget.checkstatus(). The user agent string is copied from Wget._fetch_index(). (Bitbake rev: 4679d3cdb9cdf23f3962aa61c599ad7474591f9f) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fix downloadfilename parameterJean-Marie LEMETAYER2020-01-271-3/+4
| | | | | | | | | | | When using a download filename with characters which can be interpreted by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the filename fixes the issue. (Bitbake rev: ed652dce5200161068eccdbfaaaefde33136eb09) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: remove unused importsFrazer Clews2020-01-191-3/+0
| | | | | | | | | | removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: avoid 'maximum recursion depth' RuntimeErrors when ↵Chris Laplante via bitbake-devel2019-07-271-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | handling 403 codes The code says that some servers respond with 403 codes when they really mean 405 codes. But we still need to account for legitimate 403 codes. Before this change, I noticed that sstate mirror checking was taking a very long time when I purposely entered incorrect credentials into my .netrc file for our sstate mirror. Instrumenting the code, I discovered tracebacks like the following for every mirror access attempt: File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus fetcher.checkstatus() File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus ret = try_mirrors(self, self.d, ud, mirrors, True) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors ret = try_mirror_url(fetch, origud, uds[index], ld, check) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url found = ud.method.checkstatus(fetch, ud, ld) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus opener.open(r) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) ... (repeats until recursion depth is reached) Solution is to make sure we only attempt the GET request once when handling 403/405 error codes. (Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-041-2/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-041-13/+0
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+2
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Remove pointless lambda functionRichard Purdie2019-03-191-1/+1
| | | | | | (Bitbake rev: 329986c3f2d98c4cd1a43b725194003575e718d5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Clean up whitespace/commentsRichard Purdie2019-03-191-19/+13
| | | | | | | | Clean up the whitespace and comments style, keep pylint happy. (Bitbake rev: ee59fdaae68543ade03cacfdbbf14fdc7e469412) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix typo in proxy handlingRichard Purdie2019-03-191-1/+1
| | | | | | | | | Fix a typo in a variable which would affect proxy handling spotted by pylint. (Bitbake rev: 19f4ca0119e33df64da629253d5002fcdb385f4f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Cleanup module importsRichard Purdie2019-03-191-6/+5
| | | | | | (Bitbake rev: ef062a3251af474fdeb480149856be3887b8c1c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix authentication in checkstatus() of the wget fetcherStefan Klug2019-03-191-2/+2
| | | | | | | | | I wonder how this used to work for anybody. (Bitbake rev: 9481e9cd688f635a7fe2ac60d5826ac26f0cb9ba) Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Fix Deprecated warnings from regexsRichard Purdie2019-01-161-12/+12
| | | | | | | | | Fix handling of escape characters in regexs and hence fix python Deprecation warnings which will be problematic in python 3.8. (Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix typo in exception nameRichard Purdie2018-12-081-1/+1
| | | | | | | | | This could clearly never have worked since the python3 migration but as its in an error path, it doesn't get tested/used much. (Bitbake rev: 704f27dc28d50a6dc02c8b64274ee4ecb3058c4a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: use with to ensure the response is closedRoss Burton2017-11-211-1/+2
| | | | | | | (Bitbake rev: 8c487176d311557031cedba76185f14f0e7a14cd) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: improve proxy objectRoss Burton2017-11-211-0/+1
| | | | | | | | | | The connection cache class uses a dummy file object but it doesn't have a closed attribute, so we can't use it in a context manager. (Bitbake rev: 7b072ef91d16331eae11bd60f229ce1f0c175995) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: fix FusionForge workaroundRoss Burton2017-07-311-6/+3
| | | | | | | | | | | | | | | | | My previous assertion about FusionForge appears to have been wrong, or FusionForge has changed behaviour, or both. FusionForge now mandates that downloads have the Accept header set, despite that header being optional, and returns a 406 Not Acceptable error if it isn't set. As we were pretending that 406 was actually 405 (Moved) and tried to handle it as a redirect this results in an infinite loop until Python kills the recursion. Delete the handling of 406 as 405, and pass Accept: */* in the headers. (Bitbake rev: bb70ae0c9aac5ec688026d23a64ac0cac1947187) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: mitigate a wget race condition when listing FTP directoriesRoss Burton2017-07-251-12/+11
| | | | | | | | | | | | | | | | | | | | | | When wget is fetching a listing for a directory over FTP it writes to a temporary file called .listing in the current directory. If there are many such operations happening in parallel - for example during 'bitbake world -c checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to write to, read, and delete the same file. This results in various failures such as the file disappearing before wget has processed it or the file changing contents, which causes checkpkg to randomly fail. Mitigate the race condition by creating a temporary directory to run wget in when doing directory listings. [ YOCTO #11828 ] (Bitbake rev: 91d4ca93df092cf86ab84faaa94cc66ff9f43057) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: improve error handling during sstate checkPatrick Ohly2017-07-211-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the sstate is accessed via HTTP, the existence check can fail due to network issues, in which case bitbake silently continues without sstate. One such network issue is an HTTP server like Python's own SimpleHTTP which closes the TCP connection despite an explicit "Keep-Alive" in the HTTP request header. The server does that without a "close" in the HTTP response header, so the socket remains in the connection cache, leading to "urlopen failed: <urlopen error [Errno 9] Bad file descriptor>" (only visible in "bitbake -D -D" output) when trying to use the cached connection again. The connection might also get closed for other reasons (proxy, timeouts, etc.), so this is something that the client should be able to handle. This is achieved by checking for the error, removing the bad connection, and letting the check_status() method try again with a new connection. It is necessary to let the second attempt fail permanently, because bad proxy setups have been observed to also lead to such broken connections. In that case, we need to abort for real after trying twice, otherwise a build would just hang forever. [YOCTO #11782] (Bitbake rev: 6fa07752bbd3ac345cd8617da49a70e0b2dd565f) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Fix handling of urls with user/passwordRichard Purdie2017-03-311-3/+2
| | | | | | | | | | | | | URL decoding was improved in the core a while ago and this looks like a leftover from those times which caused urls needing a user/password to fail. Use the parameters from the core instead of the broken split implementation. [YOCTO #11262] (Bitbake rev: 6a917ec99d659e684b15fa8af94c325172676062) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: don't use deprecated bb.data APIsAndre McCurdy2017-03-011-3/+2
| | | | | | | | | Cleanup some more usage of bb.data APIs in the fetchers. (Bitbake rev: 9752fd1c10b8fcc819822fa6eabc2c1050fcc03b) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: match .lz tarballs as well when determining latest versionAlexander Kanavin2017-03-011-1/+1
| | | | | | | (Bitbake rev: c55e09fbd2b64a2b909d9992350b6b2e26e5e86d) Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fixup case with no useful netrc dataMark Asselstine2016-12-211-1/+1
| | | | | | | | | | | | | | | | Commit 873e33d0479e977520106b65d149ff1799195bf6 [fetch2/wget: add Basic Auth from netrc to checkstatus()] causes "Fetcher failure for URL: 'https://www.example.com/'. URL https://www.example.com/ doesn't work." on new builds when a user has a .netrc file but there is no default and no matching host. The call to netrc.authenticators() will return None in these cases and the attempted assignment to the 3-tuple will raise a TypeError exception. Add the TypeError to the exceptions caught to get around this issue. (Bitbake rev: c0c0af40ebddaf9dc99353c580a65d4c04295613) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-13 03:48:22 +0000