summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2/__init__.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-19/+19
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle empty elements in _param_str_splitRoss Burton2021-01-231-1/+1
| | | | | | | | | | | | | _param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws an exception if a lone semicolon is used as the value doesn't split into two items. Fix by checking that the result of the first split has content. (Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch2: use relative symlinks for anything pulled from ↵Richard Purdie2020-12-091-5/+2
| | | | | | | | | | | | | PREMIRRORS" This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb. Unfortunately this caused problems where incorrect paths were found in paths and files were not being found. This was reported on the mailing list but no invetigation or resolution was found. Revert until we can figure out what was wrong. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: use relative symlinks for anything pulled from PREMIRRORSHombourger, Cedric2020-11-241-2/+5
| | | | | | | | | | | | | | | | | | try_mirror_url() creates a symlink named as the original file to make everything look like files specified in SRC_URI were downloaded from their original location. The link is however created as an absolute reference, this makes DL_DIR non-relocatable. This also causes issues with the Isar project since it bind mounts DL_DIR to /downloads to perform some of its build tasks in a chrooted environment (rendering all symbolic links from DL_DIR invalid). Modify ensure_symlink() to take an optional "relative" argument and have that function use os.path.relpath() to produce a relative symlink. (Bitbake rev: 481e66ea8fc2fc91903127d66b0f1b0fe86baedb) Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> Cc: Chris Larson <Chris_Larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/fetch2/__init__.py: drop _PYTHON_SYSCONFIGDATA_NAME unsettingAlexander Kanavin2020-11-201-5/+0
| | | | | | | | | | With introduction of python3targetconfig class in core this is no longer needed. (Bitbake rev: 47b64cfacd7c498ef9ed5486d117f2d69a39f225) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add zstd support to unpackMert Kirpici2020-11-031-0/+4
| | | | | | | | | | This commit adds unpack support to fetched files with extensions: '.tar.zst', '.tzst' and '.zst' (Bitbake rev: 4348743a7e36f3a495e2dc48f83989d9481060f2) Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix handling of `\` in file:// SRC_URILeif Middelschulte2020-09-251-1/+1
| | | | | | | | | | | | | Using backslashes in file:// URIs was broken. Either the resolver would fail or the subsequent `cp` command. Try to avoid this by putting the filenames into quotes. Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=8161 (Bitbake rev: aa857fa2e9cf3b0e43a9049b04ec4b0b3c779b11) Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Drop globbing supprt in file:// SRC_URIsRichard Purdie2020-08-261-12/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | Globbing in file:// urls is terminally broken. Currently when its used, the file checksum code is basically bypassed. This means changes to the source files don't change the task checksum, the task doesn't rebuild when the inputs change and things generally break. To make globbing work generically, we'd have to scan the file system for all possible matches to the glob and log whether they exist or not. We can't simply log the files which exist, we have to also know which files could later exist and influence the choice of file so we know when to reparse. For a simple file://xxx/*, this could be done but for bigger patterns, it becomes much more problemtic. We already support file://xxx/ in urls. So, lets decide we'll not support globs in file://urls. Worse case users can put files in a directory and reference that, moving files into place if needed. Remove all the glob special cases (see the comments if anyone doesn't believe this is terminally broken) and error to the user if they have such urls. (Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add the ability to list expanded URL dataPaul Barker2020-06-041-0/+25
| | | | | | | | | | | | | | | | | | Some fetchers may download additional sources along with those explicitly listed in SRC_URI. These "implicit URLs" will be needed by the archiver to ensure that all sources can be archived. We can't just return a list of URL strings since each URL may need its own SRCREV data so we return a list of FetchData objects. Each fetcher can override the implicit_urldata() function to provide the additional FetchData objects. For now this is just needed in the gitsm fetcher to walk git submodules recursively. (Bitbake rev: 1350f241b7d991bd191ce9e44f6662e4376c6e24) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix urldata_cache key issuesRichard Purdie2020-03-301-4/+7
| | | | | | | | | Upon inspection its clear the way the keys for this cache were being handled would break it and cause the cache to never be used. Fix this. (Bitbake rev: 9a5dd1be63395c76d3fac2c3c7ba6557fe47b442) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY ↵Khem Raj2020-03-241-14/+4
| | | | | | | | | | | | | | is "cache"" As per mailing list discussion, the cache policy was behaving correctly before and wouldn't expect to update after the initial fetch even for AUTOREV. This reverts commit ba093a38539960e645e994a66ed7872a604c00a9. (Bitbake rev: 51f827911b7202de3e855e683fdbd732d7a84e09) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Limit shown checksums to sha256Richard Purdie2020-03-021-1/+4
| | | | | | | | | | | | | | | | Currently bitbake will list many checksums for a recipe when none are present, encouraging users to add them all to a recipe. We don't need/want them all. We used to show md5 and sha256 but given the concerns about md5, switch to showing just sha256 going forward which seems like the sensible one to standardise upon. There will be no change to existing recipe functionality. (Bitbake rev: 47f0c849ed13ba554d9523b926d92405e8251702) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY is "cache"Peter Kjellerstedt2020-02-191-4/+14
| | | | | | | | | | | | Mark any keys used to cache the srcrevs for a recipe as "dontcache" if BB_DONT_CACHE is set for the recipe. Remove any such keys upon the next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will make sure the srcrev is updated as expected if ${AUTOREV} is used. (Bitbake rev: ba093a38539960e645e994a66ed7872a604c00a9) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Make fetcher_compare_revisions() workPeter Kjellerstedt2020-02-191-19/+14
| | | | | | | | | | This seems to have been broken for a very long time. Now it also works regardless of BB_SRCREV_POLICY. (Bitbake rev: ffd663a8e07e2e39e8ca2d2493f4f98037c5f9e4) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add the npmsw fetcherJean-Marie LEMETAYER2020-01-271-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | This commit adds a new npmsw fetcher that fetches every npm dependencies described in a npm shrinkwrap file: https://docs.npmjs.com/files/shrinkwrap.json.html The main package must be fetched separately: SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \ npmsw://${THISDIR}/npm-shrinkwrap.json" Since a separation has been created between the package and its dependencies, the package can also be fetched with a non npm fetcher without impacting the general behavior: SRC_URI = "git://github.com/foo/bar.git;protocol=https \ npmsw://${THISDIR}/npm-shrinkwrap.json" (Bitbake rev: f5223be54450bf20e0bfbd53b372a7748a44b475) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow fetchers to forward the done conditionJean-Marie LEMETAYER2020-01-271-10/+20
| | | | | | | | | | | | | This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the done condition to a proxy fetcher. (Bitbake rev: ee3a2545e99e6e99559a72bcda64797ae674ec71) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow fetchers to forward the mirrors managementJean-Marie LEMETAYER2020-01-271-4/+10
| | | | | | | | | | | | | This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "try_mirrors" functions to a proxy fetcher. (Bitbake rev: 462c9a2b368a1720da276310b1d5d0423b7cefea) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow fetchers to forward the donestamp managementJean-Marie LEMETAYER2020-01-271-5/+17
| | | | | | | | | | | | | This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "verify_donestamp" and "update_stamp" functions to a proxy fetcher. (Bitbake rev: f7612c0704b4252bba5157ce9a94d8888c6d0760) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add more hash functions for checksum verificationJean-Marie LEMETAYER2020-01-271-1/+1
| | | | | | | | | | | | | | | | | | This commit enables the "sha1", "sha384" and "sha512" hash functions in the supported checksum list. This allows to use more SRC_URI checksums functions for a url: SRC_URI[sha1sum] = "..." SRC_URI[sha384sum] = "..." SRC_URI[sha512sum] = "..." The npm fetcher needs this to support subresource integrity: https://www.w3.org/TR/SRI/ (Bitbake rev: cd80a646aa841b71e68282bb8d11194abb5df0e4) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: refactor checksum verificationJean-Marie LEMETAYER2020-01-271-71/+85
| | | | | | | | | | | | This commit refactors the way checksums are verified to be more generic. The support of new hash functions is now limited to the update of the CHECKSUM_LIST variable. (Bitbake rev: debd9eeaf5638755d8956b2d65b904fe02826966) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: do not suffix srcrev cache key with PNChris Laplante2020-01-191-2/+1
| | | | | | | | | | | | | | | | | | Prior to this change, two different recipes pulling from the same exact repo could get a different SRCREV during a single parse session. This was originally observed using git. For git at least, it still allows recipes to pull from the same repo, but with different branches or tags, since the form of the srcrev cache key for git is: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Where the 'unresolvedrev' part is the branch or tag name. (Bitbake rev: 6c938e6fd29beebe09b32be839dae008fe6491d2) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: amend code to use proper singleton comparisons where possibleFrazer Clews2020-01-191-2/+2
| | | | | | | | | | | amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. (Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on ↵Aníbal Limón2019-12-301-2/+2
| | | | | | | | | | | | | | | directories The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory. Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn" (Bitbake rev: 923aff060d8aba8456979c35b16d300ba7c13ff9) Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure cached url data is matched to a datastoreRichard Purdie2019-11-051-2/+2
| | | | | | | | | | | | | | | | | | | | There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 0540abd338ed2ffd822edbd2947cab2f18873422) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: show warning when renaming the archive with bad checksum failedMartin Jansa2019-09-031-1/+2
| | | | | | | | | | | | | | | | | | | | | * noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: 9a1bf4ba9ec00c2a222d820f8f83d1f056b021d6) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: runfetchcmd(): unset _PYTHON_SYSCONFIGDATA_NAMEMatthias Schiffer2019-05-301-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME; unfortunately, this also affects Python scripts run as fetch commands like git-make-shallow, breaking it with a message like Failed to import the site module Traceback (most recent call last): File "/usr/lib/python3.6/site.py", line 570, in <module> main() File "/usr/lib/python3.6/site.py", line 556, in main known_paths = addusersitepackages(known_paths) File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages user_site = getusersitepackages() File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages user_base = getuserbase() # this will also set USER_BASE File "/usr/lib/python3.6/site.py", line 254, in getuserbase USER_BASE = get_config_var('userbase') File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var return get_config_vars().get(name) File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars _init_posix(_CONFIG_VARS) File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0) ModuleNotFoundError: No module named '_sysconfigdata' on an Ubuntu 18.04 system (and likely others) when building with BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS. Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this. (Bitbake rev: d94ccd506d04aff182ab48f501f6f366d5dd14f5) Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-041-2/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-041-13/+0
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+2
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Use bb.utils.to_boolean() for BB_NO_NETWORKRobert Yang2019-03-261-2/+2
| | | | | | | | | Make it consistent with BB_FETCH_PREMIRRORONLY. (Bitbake rev: 4c6013f47ef36b03f590c909d7c9a2f50b698620) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLYRobert Yang2019-03-261-1/+1
| | | | | | | | | | | | | | The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous, and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case, so unify it to the previous one. (As BB_NO_NETWORK does). And also use bb.utils.to_boolean() to make them consistent. (Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Print SCMs list when SRCREV_FORMAT is not setRobert Yang2019-03-241-1/+2
| | | | | | | | | | This makes it easier to debug, especially when multipe SCMs like gitsm, otherwise we don't know why there are multiple SCMs. (Bitbake rev: 313fe5e86b254eadfdead706be4bd7b274d5e3c0) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: runfetchcmd(): Print workdir in debug messageRobert Yang2019-03-241-1/+4
| | | | | | | | | | | | | | | The workdir is an important message when run git command, for exmample, before the patch: Running git -c core.fsyncobjectfiles=0 branch --contains [snip] We don't know where it is running, now it is: Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git Which is easier to debug. (Bitbake rev: f25c05b73218d0e8edf2dd63b36bc09e547c5455) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix undefined variable issuesRichard Purdie2019-03-191-5/+5
| | | | | | | | | Various refactors have left hanging variables, often in debug messages, hence why they haven't been spotted. Fix them (pylint spotted the issues). (Bitbake rev: e1f252408982a62485bbf874a01fd30a02aeed32) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix os.errno referencesRichard Purdie2018-12-081-2/+2
| | | | | | | | | | os.errno used to happen to work but is invalid. Correct to use errno. [YOCTO #13068] (Bitbake rev: 03aae96a8859409b1ce7e3d5c1197371fa96bd14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid warning about incorrect character escaping in regexRichard Purdie2018-11-271-1/+1
| | | | | | | | | | | Fixes: lib/bb/fetch2/__init__.py:259: DeprecationWarning: invalid escape sequence \w re.compile("^\w+:(?!//)").match(uri): (Bitbake rev: 2f0b6cea2be7e89ef01181b10899b3e7f11e8f20) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: don't use shell=True when listing ar filesRoss Burton2018-11-231-1/+1
| | | | | | | (Bitbake rev: 7414b3537e8adfb41a9581d70bf8296c4f7d38c0) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Also export GIT_SSH environment variableMike Crowe2018-11-071-0/+1
| | | | | | | | | | | Support users who want to specify a custom ssh to Git using an environment variable. (This lets build systems work without requiring users to configure Git explicitly.) (Bitbake rev: db46fb2755a4b1033ab60051ce511cc9dd7e34fb) Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid incorrect getVarFlag callRichard Purdie2018-10-181-1/+3
| | | | | | | | | Calling getVarFlag with flag=None makes no sense, don't do it. Bitbake used to silently ignore this, it now warns so avoid the warning. (Bitbake rev: a68de8ace62eaba23856bfb301efbbe1824322aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: extract the function which ensures that a valid symlink existsUrs Fässler2018-10-181-10/+17
| | | | | | | | | | | For better readability and future use, we extract the function which ensures that a given symlink exists. (Bitbake rev: 5e69ca56533666a097bb23d09ab673e5c862051c) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: avoid circular recursion with SRCPV in PRPeter Marko2018-10-181-1/+3
| | | | | | | | | | | Some recent changes broke SRCPV and workaround was introduced to avoid circular dependency if SRCPV is in PV. However there is still the same error if SRCPV is in PR. (Bitbake rev: 05ee4845f925b8528a7ce9cffb4bae425b8fa1e9) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Use correct check in parameter handling in uri_replace()Nikolai Merinov2018-06-211-2/+2
| | | | | | | | | | If uri_find contain parameters then original URI parameters should be checked against parameters from uri_find instead of parameters from uri_replace. (Bitbake rev: 8efa7826a61501589afa33eb698c0ab3a622bf2e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/__init__.py: Fixed handling of uris with empty pathJakub Dębski2018-05-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | For mirrors or premirrors defined like: "http://.*/.* http://somewhere.org" fetching ends with errors because function fetch2/__init__.py:encodeurl() creates url like "http://somewhere.orgsomefile.tar.gz". It happens because function fetch2/__init__.py:decodeurl() for url "http://somewhere.org" returns ['http', 'somewhere.org', '', '', '', {}] and then in function fetch2/__init__.py:uri_replace() variable result_decode will be ['http', 'somewhere.org', 'somefile.tar.gz', '', '', {}] (because of line: result_decoded[loc] = os.path.join(result_decoded[loc], basename)) for which encodeurl returns "http://somewhere.orgsomefile.tar.gz". In addition for mirror "http://.*/.* http://somewhere.org/" everything works fine. (Bitbake rev: d822ae24ef5485e550804cbd9130ebd73b2aa48e) Signed-off-by: Jakub Dębski <jdebski@enigma.com.pl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Handle missing donestamp file when content is validNathan Rossi2018-02-141-11/+10
| | | | | | | | | | | | | | | | | | | | | | | | In order to allow users to manually populate the download directory with valid content change the assumption that missing the donestamp file means unfetched content. This allows users to populate the download dir without needing to create dummy .done files such that a user does not need a PREMIRROR when using BB_NO_NETWORK to provide valid content files in the download directory. To ensure the correct result this change also fails first if the localpath does not exist. This prevents further parts of the function attempting to calculating the checksum on non-existent files. This also fixes some edge conditions around where if the donestamp exists but the localpath does not it returns, and did not remove the donestamp. Also added test cases to cover this use case and additional use cases where for example the fetcher does not support checksums. (Bitbake rev: a335dbbb65d5b56e71d98cf3e4fa9bfbec1dcde6) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/__init__: Disable pseudo in runfetchcmd()Peter Kjellerstedt2018-01-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a fetcher, e.g., git, is run when pseudo is active it will think it is running as root. If it in turn uses ssh (as git does), ssh too will think it is running as root. This will cause it to try to read root's ssh configuration from /root/.ssh which will fail. If ssh then needs to ask for credentials it will hang indefinitely as there is nowhere for it to ask the user for them (and even if there was it would not access the correct private keys). The solution to the above is to temporarily disable pseudo while executing any fetcher commands. There should be no reason for them to be executed under pseudo anyway so this should not be a problem. RP Ammendum: We finally did get more information about how to reproduce this problem, something needs to trigger bb.fetch2.get_srcrev() in a pseudo context, for example when AUTOREV is in use or the recipe doesn't have a defined SRCREV. That SRC_URI needs to be using protocol=ssh. This would trigger an ls-remote of the remote repo and if that happens under pseudo, the wrong ssh credentials may be attempted which can hang. [YOCTO #12464] (Bitbake rev: ceaca281cafa662aa2385b95641bce309dce843d) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix unpack of .txz filesAndré Draszik2017-11-291-1/+1
| | | | | | | | | | .txz is the same as .tar.xz, and can be found in the wild. (Bitbake rev: 2ba8a6b25ccc12e7b543e8450121e5311c7a701d) Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add stub latest_versionstring to FetchMethodRoss Burton2017-09-111-0/+8
| | | | | | | (Bitbake rev: c45453d1f6bc7bcecd84f58e2f7d93d6bd1e8499) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: don't mandate path element in encodeurl()Paul Eggleton2017-08-231-6/+5
| | | | | | | | | | | | | URLs do not have to have a path; currently our npm URLs don't, so encodeurl() needs to handle if the path element isn't specified. This fixes errors using OpenEmbedded's devtool add / recipetool create on an npm URL after OE-Core revision ecca596b75cfda2f798a0bdde75f4f774e23a95b that uses decodeurl() and encodeurl() to change URL parameter values. (Bitbake rev: d5cab2dbf5682d2fd08e58316a3bf39a10f63df2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix checkstatus fallback to MIRRORSRoss Burton2017-07-311-3/+2
| | | | | | | | | | The checkstatus() code was expecting checkstatus to throw exceptions if it failed, but in general it should return False. (Bitbake rev: 57be5cc6228518e60f564570a39cebbeb6cf564e) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/fetch2/__init__.py: Fix unpack commentMark Hatle2017-07-271-1/+1
| | | | | | | (Bitbake rev: b8f477de204ab5d0680b2b7c42370d13395be46c) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-09 20:00:48 +0000