summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: fetch/git: fix AttributeError in shallow extraction logicChristopher Larson2018-11-221-5/+8
| | | | | | | | | | | | | | | This code checks to see if shallow is either disabled or the tarball is missing, but the else block tries to print the tarball filename, and this attribute doesn't exist at all when shallow is disabled. Handle the two cases separately to give sane errors for both cases without the exception: Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow' (Bitbake rev: bdbb558342ebb4e64384c9838d2485d9299d91a6) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm.py: Allow shrinkwrap resolved relative URL which ↵Parthiban Nallathambi2018-11-211-1/+1
| | | | | | | | | | | | | | | | | startswith 'http' (e.g http-proxy) shrinkwrap resolved relative URL can start with http. For example, "resolved: http-proxy/-/http-proxy-${PV}.tgz" is still relative URL to npm registry, but starts with http. Current if statement compares the startswith 'resolved' to 'http', which makes impossible to use npm download. Condtional comparison now strictly checks for "http://" and "https://" (Bitbake rev: f76075aa1a5159fd4d62949cb588346888b9fe60) Signed-off-by: Parthiban Nallathambi <pn@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Also export GIT_SSH environment variableMike Crowe2018-11-071-0/+1
| | | | | | | | | | | Support users who want to specify a custom ssh to Git using an environment variable. (This lets build systems work without requiring users to configure Git explicitly.) (Bitbake rev: db46fb2755a4b1033ab60051ce511cc9dd7e34fb) Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm.py: Fix the references when the module and path are ↵Mark Hatle2018-11-071-35/+34
| | | | | | | | | | | | | | | | | | | | | | | different Git does not require the module and target path to be the same in the .gitmodules file. This incorrect assumption was being made previously causing various unpack failures. An example .gitmodule showing this issue: [submodule "plugins/WaveShaper/Libs/inih"] path = plugins/wolf-shaper/Libs/inih url = https://github.com/pdesaulniers/inih.git The unpack function also needed to work in a loop on the overall submodules_queue. Before it could have missed items that were not in the primary repository. (Bitbake rev: 5a7009c204f4d2254e3b2d83ad9319ac23f1cf4d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm.py: Disable branch checking on submodulesMark Hatle2018-11-071-1/+1
| | | | | | | | | | | | Submodules by definition refer to a specific commit, not branch. If we don't ignore the branch, then any commits on a submodule on a branch different then the original module will trigger a failure that the commit is not on the branch. (Bitbake rev: fdc1dbf96f153b496de52acd8263366a1ff303ad) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid incorrect getVarFlag callRichard Purdie2018-10-181-1/+3
| | | | | | | | | Calling getVarFlag with flag=None makes no sense, don't do it. Bitbake used to silently ignore this, it now warns so avoid the warning. (Bitbake rev: a68de8ace62eaba23856bfb301efbbe1824322aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: provide information about missing sourcesUrs Fässler2018-10-181-8/+21
| | | | | | | | | | | Provide more information in the case the sources are not found in the unpack step. (Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: explicitly show the decision logic to select the source ↵Urs Fässler2018-10-181-3/+4
| | | | | | | | | | in code (Bitbake rev: 78d8fb3a7899e1404a513be1c2b2b4440da8b12d) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: prevent access to non-existing clonedirUrs Fässler2018-10-181-1/+3
| | | | | | | | | | | | A user friendly error is throw when neither the clonedir nor fullshallow exist. Without the check, a difficult to interpret error is throw from within the fetch command. (Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: extract the function which ensures that a valid symlink existsUrs Fässler2018-10-181-10/+17
| | | | | | | | | | | For better readability and future use, we extract the function which ensures that a given symlink exists. (Bitbake rev: 5e69ca56533666a097bb23d09ab673e5c862051c) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: avoid circular recursion with SRCPV in PRPeter Marko2018-10-181-1/+3
| | | | | | | | | | | Some recent changes broke SRCPV and workaround was introduced to avoid circular dependency if SRCPV is in PV. However there is still the same error if SRCPV is in PR. (Bitbake rev: 05ee4845f925b8528a7ce9cffb4bae425b8fa1e9) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Allow relative URIs when fetchingMark Hatle2018-10-041-0/+5
| | | | | | | | | | | Some repositories may specify a relative submodule path. If this happens, it is our responsibility to use the parents URL (ud) and handle any relative processing ourselves. (Bitbake rev: fd9893c338df7828b2c01f1d548aa86dfcf7c50a) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: checkout submodules recursivelyLaurent Bonnans2018-10-041-30/+38
| | | | | | | | | The new fetcher did not run 'git submodule update' recursively. (Bitbake rev: 377ed943357b6d47d41d84edbf971741f44affa9) Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: handle submodules nested inside subdirsLaurent Bonnans2018-10-041-6/+9
| | | | | | | | | | | | | | The new gitsm fetcher assumed that submodules were living directly in the 'modules' directory, whereas they can be arbitrarily nested inside subdirectories. Solve it by first creating the parent of the destination directory for the symlink and copy steps. (Bitbake rev: 3dbc84e9c572f43209b79f3656d7dc35a6d982ba) Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: fix a typoLaurent Bonnans2018-10-041-1/+1
| | | | | | | | | 'qbareclone' in place of 'bareclone' (Bitbake rev: 90a3181f1397ae05862f4e89a9bbac606e74504e) Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: use correct check to decide if the shallow tarball ↵Urs Fässler2018-10-041-1/+1
| | | | | | | | | | | | | should be unpacked The shallow_tarball check is never true due a check on the caller side. The tarball check is not related to the code on the caller side. (Bitbake rev: 086eddcf8c7520ff5c52ce2a11ca9bf5b5fe5d7e) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: prevent access to non-existing fullshallow tarballUrs Fässler2018-10-041-1/+1
| | | | | | | | (Bitbake rev: b7f00a8c11672a2ee0408e210fb174cda3384e3f) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: improve readability of method need_updateUrs Fässler2018-10-041-4/+9
| | | | | | | | | | | To improve the readability we extract the different scenarios of why the clonedir needs an update. (Bitbake rev: 9038e029f4f0ab413727de76c74248cbb3cdc9ea) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm.py: Rework the git submodule fetcherMark Hatle2018-09-261-123/+156
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The prior fetcher did not know how to work with MIRRORS, and did not honor BB_NO_NETWORK and similar. The new fetcher approach recursively calls 'gitsm' download on each submodule detected. This ensures that it will go throug the standard download process. Each downloaded submodule is then 'attached' to the original download in the 'modules' directory. This mimics the behavior of: git submodule init but there is no chance it will contact the network without permission. It then corrects upstream reference URIs. The unpack steps simply copies the items from the downloads to the destdir. Once copied the submodules are connected and we then run: git submodule update According to the git documentation, git submodule init can and will modify the project configuration and may connect to the network. Doing the work manually prevents this. (This manual process is allowed based on my reading of the documentation.) See: https://git-scm.com/book/en/v2/Git-Tools-Submodules The small change to the existing test is due to this new code always assuming the code is from a remote system, and not a 'local' repository. If this assumption proves to be incorrect -- code will need to be added to deal with local repositories without an upstream URI. (Bitbake rev: 9c6b39adf9781fa6745f48913a97c859fa37eb5b) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetcher: Fixed remote removal not throwing exception.Paulo Neves2018-08-281-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this fix it is assumed that the removal of the remote can only fail because there is not remote to remove. This is a false assumption. Example error which would be ignored: git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output: Note: A branch outside the refs/remotes/ hierarchy was not removed; to delete it, use: git branch -d master error: could not lock config file config error: Could not remove config section 'remote.origin' Due to the masking of this error a stranger error will be presented to the user, because this time we do not mask the exception: git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output: fatal: remote origin already exists. The most likely reason that the remote cannot be removed nor modified is that the DL_DIR/git2 does not have permissions compatible with the user running bitbake. This commit fixes: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728 (Bitbake rev: 9c86c582a10c9b23abad7d34b6cbf12f7086294d) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Use correct check in parameter handling in uri_replace()Nikolai Merinov2018-06-211-2/+2
| | | | | | | | | | If uri_find contain parameters then original URI parameters should be checked against parameters from uri_find instead of parameters from uri_replace. (Bitbake rev: 8efa7826a61501589afa33eb698c0ab3a622bf2e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: unify the way fetchers determine DL_DIR and FETCHCMDAndre McCurdy2018-06-158-21/+23
| | | | | | | | | | | | | | | Currently there is quite some variation between the fetchers in terms of how they determine the subdirectory within DL_DIR and the base fetch command to run. Some rely on variables being set externally (e.g. from bitbake.conf in oe-core), some respect these external variables but provide fallback defaults and some use only hardcoded internal values. Try to unify the approach used across the various fetchers. (Bitbake rev: efd5e35af4b08501c67e8b30f30d9457f6fdf610) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/gitsm: avoid live submodule fetching during unpack()Matt Hoosier2018-06-071-8/+75
| | | | | | | | | | | | | | | | | | Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task. Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules. [YOCTO #12739] (Bitbake rev: 11b6a5d5c1b1bb0ce0c5bb3983610d13a3e8f84a) Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix import error for Python 3.6.5Tzu Hsiang Lin2018-05-292-3/+1
| | | | | | | | | | | | When running bitbake command with Python 3.6.5 always result in import error causing by the change of distutils module. This patch replaces the method to search executable in PATH by "/usr/bin/env <command>". (Bitbake rev: bd9a1b063633af2936ba1dd87b19202424900151) Signed-off-by: Tzu Hsiang Lin <t9360341@ntut.org.tw> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/__init__.py: Fixed handling of uris with empty pathJakub Dębski2018-05-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | For mirrors or premirrors defined like: "http://.*/.* http://somewhere.org" fetching ends with errors because function fetch2/__init__.py:encodeurl() creates url like "http://somewhere.orgsomefile.tar.gz". It happens because function fetch2/__init__.py:decodeurl() for url "http://somewhere.org" returns ['http', 'somewhere.org', '', '', '', {}] and then in function fetch2/__init__.py:uri_replace() variable result_decode will be ['http', 'somewhere.org', 'somefile.tar.gz', '', '', {}] (because of line: result_decoded[loc] = os.path.join(result_decoded[loc], basename)) for which encodeurl returns "http://somewhere.orgsomefile.tar.gz". In addition for mirror "http://.*/.* http://somewhere.org/" everything works fine. (Bitbake rev: d822ae24ef5485e550804cbd9130ebd73b2aa48e) Signed-off-by: Jakub Dębski <jdebski@enigma.com.pl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: make fewer calls to _contains_ref() from download()Andre McCurdy2018-05-221-6/+6
| | | | | | | | | | | | | | | Updating a local git repo clone currently results in multiple calls to self._contains_ref(), some of which appear to be redundant and can be eliminated by minor tweaks to the logic in download(). Also drop redundant calls to os.path.exists(ud.clonedir) before self.need_update(), since need_update() includes its own built-in check for the existance of ud.clonedir. (Bitbake rev: 61b0df5523afc8f805043f3adc9c106690e6f133) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: log exception if ls-remote failsRoss Burton2018-03-281-1/+2
| | | | | | | (Bitbake rev: 8212446de11c0e370c55f88cde86334b760cd939) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Handle missing donestamp file when content is validNathan Rossi2018-02-141-11/+10
| | | | | | | | | | | | | | | | | | | | | | | | In order to allow users to manually populate the download directory with valid content change the assumption that missing the donestamp file means unfetched content. This allows users to populate the download dir without needing to create dummy .done files such that a user does not need a PREMIRROR when using BB_NO_NETWORK to provide valid content files in the download directory. To ensure the correct result this change also fails first if the localpath does not exist. This prevents further parts of the function attempting to calculating the checksum on non-existent files. This also fixes some edge conditions around where if the donestamp exists but the localpath does not it returns, and did not remove the donestamp. Also added test cases to cover this use case and additional use cases where for example the fetcher does not support checksums. (Bitbake rev: a335dbbb65d5b56e71d98cf3e4fa9bfbec1dcde6) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm.py: Fix inverted condition to prevent infinite loopBöszörményi Zoltán2018-02-141-3/+5
| | | | | | | | | | | | | | | At least the cli-color node module has dependencies that have cyclic dependency among themselves. npm.py is prepared to deal with such a case but the condition is handled only for downloading or not a dependency again, but then it goes checking the its dependency which causes an infinite loop in _getdependencies(). Make this function simply return when a dependency is already downloaded and only download and check its dependencies when not. (Bitbake rev: 545540420112992e53f4a83104af10452df168d0) Signed-off-by: Zoltán Böszörményi <zboszor@pr.hu> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Use 'git-make-shallow' from bin directoryNathan Rossi2018-02-141-1/+4
| | | | | | | | | | | | | | Move the code that existed in tests/fetch.py for determining the path to 'git-make-shallow' into the git module and reference it. This ensures that 'git-make-shallow' is always available and the desired version regardless of the path variable or whether git exposes the command. (Bitbake rev: 6b508ab8fd5aa796c1c00c970e81e5e93f84d35d) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Add pack-refs command to mirror handling codeRichard Purdie2018-01-291-0/+1
| | | | | | | | | | | | | We've noticed issues on our infrastucture iterating over the many tag/branch/head reference files that some git repositories may contain. By issuing the pack-refs command, we move these all to a single file which speeds up operations with the mirror repos in the downloads directory in general. (Bitbake rev: f8126aaf774186a6eaf0bd4067b89c074594886c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/__init__: Disable pseudo in runfetchcmd()Peter Kjellerstedt2018-01-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a fetcher, e.g., git, is run when pseudo is active it will think it is running as root. If it in turn uses ssh (as git does), ssh too will think it is running as root. This will cause it to try to read root's ssh configuration from /root/.ssh which will fail. If ssh then needs to ask for credentials it will hang indefinitely as there is nowhere for it to ask the user for them (and even if there was it would not access the correct private keys). The solution to the above is to temporarily disable pseudo while executing any fetcher commands. There should be no reason for them to be executed under pseudo anyway so this should not be a problem. RP Ammendum: We finally did get more information about how to reproduce this problem, something needs to trigger bb.fetch2.get_srcrev() in a pseudo context, for example when AUTOREV is in use or the recipe doesn't have a defined SRCREV. That SRC_URI needs to be using protocol=ssh. This would trigger an ls-remote of the remote repo and if that happens under pseudo, the wrong ssh credentials may be attempted which can hang. [YOCTO #12464] (Bitbake rev: ceaca281cafa662aa2385b95641bce309dce843d) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix unpack of .txz filesAndré Draszik2017-11-291-1/+1
| | | | | | | | | | .txz is the same as .tar.xz, and can be found in the wild. (Bitbake rev: 2ba8a6b25ccc12e7b543e8450121e5311c7a701d) Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: use with to ensure the response is closedRoss Burton2017-11-211-1/+2
| | | | | | | (Bitbake rev: 8c487176d311557031cedba76185f14f0e7a14cd) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: improve proxy objectRoss Burton2017-11-211-0/+1
| | | | | | | | | | The connection cache class uses a dummy file object but it doesn't have a closed attribute, so we can't use it in a context manager. (Bitbake rev: 7b072ef91d16331eae11bd60f229ce1f0c175995) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix missing logger import in repo fetcherOleksandr Andrushchenko2017-11-071-0/+1
| | | | | | | | | | | | | After cleaning deprecated API usage repo fetcher is missing logger as it was indirectly imported via deprecated bb.data. Fix this by importing logger directly. Fixes: 9752fd1c10b8 ("fetch2: don't use deprecated bb.data APIs") (Bitbake rev: f8e027d26603db2f1fe757dca767ea35d95174c7) Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add stub latest_versionstring to FetchMethodRoss Burton2017-09-111-0/+8
| | | | | | | (Bitbake rev: c45453d1f6bc7bcecd84f58e2f7d93d6bd1e8499) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Replace deprecated git branch parameter "--set-upstream"Andre Rosa2017-09-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | Since 2017-08-17 (git version 2.14.1.473.g3ec7d702a) using deprecated git branch parameter "--set-upstream" causes a fetcher error. Replace it by "--set-upstream-to". https://git.kernel.org/pub/scm/git/git.git/commit/?id=52668846ea2d41ffbd87cda7cb8e492dea9f2c4d says, it's deprecated since 2012-08-30 so hopefully all still supported host distributions have new enough git to support "--set-upstream-to". ERROR: PACKAGE do_unpack: Fetcher failure: ...; git -c core.fsyncobjectfiles=0 branch --set-upstream master origin/master failed with exit code 128, output: fatal: the '--set-upstream' option is no longer supported. Please use '--track' or '--set-upstream-to' instead. ERROR: PACKAGE do_unpack: Function failed: base_do_unpack (Bitbake rev: 2ab50074c1a6c56a8a178755de108447d7b7acaf) Signed-off-by: Andre Rosa <andre.rosa@lge.com> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: add noverify parameter to skip lockdown/shrinkwrapPaul Eggleton2017-08-311-20/+21
| | | | | | | | | | | | | | | | When fetching source for the first time within scripts such as OpenEmbedded's recipetool, we don't want to be showing warnings about NPM_SHRINKWRAP or NPM_LOCKDOWN not being set since there's no way we could have set them in advance. Previously we were using ud.ignore_checksums to suppress these but since we are now using a more standard task-based path to fetch the source, we need to disable these through the metadata. Look for a "noverify" parameter set on the npm URL and skip the checks if it is set to "1". (Bitbake rev: 8c4b35d1e4d31bae9fddd129d5ba230acb72c3bb) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: don't mandate path element in encodeurl()Paul Eggleton2017-08-231-6/+5
| | | | | | | | | | | | | URLs do not have to have a path; currently our npm URLs don't, so encodeurl() needs to handle if the path element isn't specified. This fixes errors using OpenEmbedded's devtool add / recipetool create on an npm URL after OE-Core revision ecca596b75cfda2f798a0bdde75f4f774e23a95b that uses decodeurl() and encodeurl() to change URL parameter values. (Bitbake rev: d5cab2dbf5682d2fd08e58316a3bf39a10f63df2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: fix FusionForge workaroundRoss Burton2017-07-311-6/+3
| | | | | | | | | | | | | | | | | My previous assertion about FusionForge appears to have been wrong, or FusionForge has changed behaviour, or both. FusionForge now mandates that downloads have the Accept header set, despite that header being optional, and returns a 406 Not Acceptable error if it isn't set. As we were pretending that 406 was actually 405 (Moved) and tried to handle it as a redirect this results in an infinite loop until Python kills the recursion. Delete the handling of 406 as 405, and pass Accept: */* in the headers. (Bitbake rev: bb70ae0c9aac5ec688026d23a64ac0cac1947187) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix checkstatus fallback to MIRRORSRoss Burton2017-07-311-3/+2
| | | | | | | | | | The checkstatus() code was expecting checkstatus to throw exceptions if it failed, but in general it should return False. (Bitbake rev: 57be5cc6228518e60f564570a39cebbeb6cf564e) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/fetch2/__init__.py: Fix unpack commentMark Hatle2017-07-271-1/+1
| | | | | | | (Bitbake rev: b8f477de204ab5d0680b2b7c42370d13395be46c) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/__init__.py: replace stray logger.warn() with logger.warning()Andre McCurdy2017-07-251-1/+1
| | | | | | | | | | | | | | Update stray usage of deprecated logger.warn(), which was introduced to fetch2/__init__.py after all other instances had been replaced by logger.warning(): http://git.openembedded.org/bitbake/commit/?id=5a53e7d7b017769a6eb0f0a6335735a1fe51a5ec http://git.openembedded.org/bitbake/commit/?id=676a5f592e8507e81b8f748d58acfea7572f8796 (Bitbake rev: 1b14f115a9e929e29e91e8ac70826a3fe7259961) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: mitigate a wget race condition when listing FTP directoriesRoss Burton2017-07-251-12/+11
| | | | | | | | | | | | | | | | | | | | | | When wget is fetching a listing for a directory over FTP it writes to a temporary file called .listing in the current directory. If there are many such operations happening in parallel - for example during 'bitbake world -c checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to write to, read, and delete the same file. This results in various failures such as the file disappearing before wget has processed it or the file changing contents, which causes checkpkg to randomly fail. Mitigate the race condition by creating a temporary directory to run wget in when doing directory listings. [ YOCTO #11828 ] (Bitbake rev: 91d4ca93df092cf86ab84faaa94cc66ff9f43057) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: improve error handling during sstate checkPatrick Ohly2017-07-211-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the sstate is accessed via HTTP, the existence check can fail due to network issues, in which case bitbake silently continues without sstate. One such network issue is an HTTP server like Python's own SimpleHTTP which closes the TCP connection despite an explicit "Keep-Alive" in the HTTP request header. The server does that without a "close" in the HTTP response header, so the socket remains in the connection cache, leading to "urlopen failed: <urlopen error [Errno 9] Bad file descriptor>" (only visible in "bitbake -D -D" output) when trying to use the cached connection again. The connection might also get closed for other reasons (proxy, timeouts, etc.), so this is something that the client should be able to handle. This is achieved by checking for the error, removing the bad connection, and letting the check_status() method try again with a new connection. It is necessary to let the second attempt fail permanently, because bad proxy setups have been observed to also lead to such broken connections. In that case, we need to abort for real after trying twice, otherwise a build would just hang forever. [YOCTO #11782] (Bitbake rev: 6fa07752bbd3ac345cd8617da49a70e0b2dd565f) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fire an event when there are missing checksumsPaul Eggleton2017-07-211-0/+10
| | | | | | | | | | | If BB_STRICT_CHECKSUMS is set to anything other than "1" i.e. we're not going to raise an error, then fire an event so that scripts can listen for it and get the checksums. (Bitbake rev: 8b2ccb4b865f2df118ef668847df682a83f9c500) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow hiding checksum warningPaul Eggleton2017-07-211-0/+6
| | | | | | | | | | | | | If BB_STRICT_CHECKSUMS is set to "ignore" then don't display a warning if no checksums are specified in the recipe. This is not intended to be used from recipes - it is needed when we move to using more standard code paths to fetch new files from scripts i.e. where we don't know what the checksums are in advance. (Bitbake rev: f15ca7339de8a448a93a14cf6130b3925178a920) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix indentationPaul Eggleton2017-07-181-47/+47
| | | | | | | (Bitbake rev: 6683338598ce97278f188fbcd780c3e3754e5b9a) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npm fetcher: fix unknown variable name.Ismo Puustinen2017-06-291-1/+1
| | | | | | | | | 'mirrortarball' is supposed to be a local variable to the function. (Bitbake rev: a457cbfb1f20a47db3978290921d0708cd96bd70) Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-23 17:26:17 +0000