summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2: Change git fetcher not to destroy old referencesRichard Purdie2020-09-081-1/+1
| | | | | | | | | | | | | | | | It looks like we're about to see a lot of changes in branch names in repos. If we have the prune option here, those old names are lost, the changes propagate to our source mirrors and our old releases break. We have the force option so any replaced references should be replaced, its only orphaned branches which will now be preserved. I believe this behaviour will cause us fewer problems given the changes that look likely to happen. (Bitbake rev: e2fc4147bbe436ac79de187d92d3bc80a8a95349) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: _revision_key: collapse adjacent slashesChris Laplante2020-02-111-1/+3
| | | | | | | | | | | | | | | | >From a SRCREV caching point of view, there is no reason to treat the following upstreams as different: SRC_URI = "git://github.com/file/file.git" SRC_URI = "git://github.com//file/file.git" (Bitbake rev: 716cdf737bc536f84ed1254d464c9f286e0d5a9a) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 425e21c14955dd38868c6e97637df3bbe0f89fac) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: do not suffix srcrev cache key with PNChris Laplante2020-02-111-2/+1
| | | | | | | | | | | | | | | | | | | | | Prior to this change, two different recipes pulling from the same exact repo could get a different SRCREV during a single parse session. This was originally observed using git. For git at least, it still allows recipes to pull from the same repo, but with different branches or tags, since the form of the srcrev cache key for git is: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Where the 'unresolvedrev' part is the branch or tag name. (Bitbake rev: 63556259f2e34003ad0618a8fe19e1cfe8aef5c9) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6c938e6fd29beebe09b32be839dae008fe6491d2) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on ↵Aníbal Limón2020-01-111-2/+2
| | | | | | | | | | | | | | | | | | directories The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory. Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn" (Bitbake rev: 4c7d689341f471efdf8ab7c2a7c6a19d2d370f5c) Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 923aff060d8aba8456979c35b16d300ba7c13ff9) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/clearcase: Fix warnings from python 3.8Richard Purdie2019-11-251-3/+3
| | | | | | | | | | | | | | | | | | bitbake/lib/bb/fetch2/clearcase.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="? if command is 'mkview': bitbake/lib/bb/fetch2/clearcase.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'rmview': bitbake/lib/bb/fetch2/clearcase.py:159: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'setcs': Python 3.8 is quite correct and we so mean "==" here, fix it to avoid the warnings. (Bitbake rev: 2cccc14304855cb55f339e465f6ba6ed0c69a7ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/hg: Fix various runtime issuesVolker Vogelhuber2019-11-251-10/+17
| | | | | | | | | | | | Fix mercurial fetching after breakage from changes to the core fetcher. Fix username and password usage and setting moddir needed by setup_revisions. (Bitbake rev: c61c8356cce4d7307f74147dcf2b2cf103db84a8) Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: fetch shallow revs when neededChristopher Larson2019-11-251-7/+16
| | | | | | | | | | | | | | | | When bitbake determines if a git clone needs updating, it only checks for the needed srcrevs, not the revs listed in BB_GIT_SHALLOW_REVS, which will fail if using shallow and the needed rev was added to the upstream git repo after a previous fetch. Ensure that we also check for shallow revs. [YOCTO #13586] (Bitbake rev: 24e3c7189e7d41bcbb46078a41c3a9daf391202a) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure cached url data is matched to a datastoreRichard Purdie2019-11-131-2/+2
| | | | | | | | | | | | | | | | | | | | There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 1a79651c518abc35b99005c137ab7e82a99c75b0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: refactor check for git-lfs commandRoss Burton2019-09-271-8/+9
| | | | | | | | | | Refactor the git-lfs checking: this means both clearer code in the download() function and allows unit testing to monkeypatch the functionality. (Bitbake rev: 33cf9172ded50a869f7201ba463ab9ecc69b8252) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: add git-lfs toggle optionRoss Burton2019-09-191-7/+11
| | | | | | | | | | | | | | | Add a new 'lfs' option to the git fetcher so that we can optionally not fetch git-lfs content, for repositories that contain LFS data that we don't actually need for building. By default lfs is set to 1, so if the repository has LFS content then git-lfs is required. Setting lfs to 0 will mean that git-lfs won't be required to fetch, and some files will be missing. (Bitbake rev: be0b78ccfc5ede98041bc0545a15092494b12b26) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: svn fetcher: allow "svn propget svn:externals" to failMikko Rapeli2019-09-181-1/+1
| | | | | | | | | | | | | | Not all servers and repositories have this property set which results in failures like this when actual svn checkout command succeeded: svn: warning: W200017: Property 'svn:externals' not found on '' svn: E200000: A problem occurred; see other errors for details (Bitbake rev: 238636f033cbf18e5741f0ea0e64db40e84f5838) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: show warning when renaming the archive with bad checksum failedMartin Jansa2019-09-031-1/+2
| | | | | | | | | | | | | | | | | | | | | * noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: 9a1bf4ba9ec00c2a222d820f8f83d1f056b021d6) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: avoid 'maximum recursion depth' RuntimeErrors when ↵Chris Laplante via bitbake-devel2019-07-271-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | handling 403 codes The code says that some servers respond with 403 codes when they really mean 405 codes. But we still need to account for legitimate 403 codes. Before this change, I noticed that sstate mirror checking was taking a very long time when I purposely entered incorrect credentials into my .netrc file for our sstate mirror. Instrumenting the code, I discovered tracebacks like the following for every mirror access attempt: File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus fetcher.checkstatus() File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus ret = try_mirrors(self, self.d, ud, mirrors, True) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors ret = try_mirror_url(fetch, origud, uds[index], ld, check) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url found = ud.method.checkstatus(fetch, ud, ld) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus opener.open(r) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) ... (repeats until recursion depth is reached) Solution is to make sure we only attempt the GET request once when handling 403/405 error codes. (Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: Use npm pack to download node modules instead of wgetMads Andreasen2019-07-271-22/+23
| | | | | | | | | | Using npm pack to download the main node module and its dependencies allow for the use of private npm modules and access to them via .npmrc (Bitbake rev: e5eda3871893e4eadeb311aeb997e183675598f4) Signed-off-by: Mads Andreasen <mads@andreasen.cc> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/clearcase: Fix class import errorsCHerzig@Gauselmann.de2019-07-161-1/+3
| | | | | | | (Bitbake rev: 9a5152fa4613a1164cbf2a0248460e75207b2624) Signed-off-by: Christian Herzig <cherzig@gauselmann.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: runfetchcmd(): unset _PYTHON_SYSCONFIGDATA_NAMEMatthias Schiffer2019-05-301-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME; unfortunately, this also affects Python scripts run as fetch commands like git-make-shallow, breaking it with a message like Failed to import the site module Traceback (most recent call last): File "/usr/lib/python3.6/site.py", line 570, in <module> main() File "/usr/lib/python3.6/site.py", line 556, in main known_paths = addusersitepackages(known_paths) File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages user_site = getusersitepackages() File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages user_base = getuserbase() # this will also set USER_BASE File "/usr/lib/python3.6/site.py", line 254, in getuserbase USER_BASE = get_config_var('userbase') File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var return get_config_vars().get(name) File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars _init_posix(_CONFIG_VARS) File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0) ModuleNotFoundError: No module named '_sysconfigdata' on an Ubuntu 18.04 system (and likely others) when building with BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS. Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this. (Bitbake rev: d94ccd506d04aff182ab48f501f6f366d5dd14f5) Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/npm: fix npw view parsingJean-Marie LEMETAYER2019-05-211-14/+5
| | | | | | | | | | | | | | | | | Fixes [YOCTO #13344] When parsing manually the 'npm view --json' ouput, an extra closing brackets in a JSON string can leads the fetcher to fail with a JSONDecodeError exception. This commit use the JSON parser to extract: - The last object in the returned array if there are multiple results. - The returned object if there is only one result. (Bitbake rev: 3d319c79981811d3cfd4732885057db4fd5afcc2) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: svn.py: Stop SVN from directly pulling from an external layer w/o ↵Mark Hatle2019-05-211-0/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | fetcher Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems. By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user. It is up to the user to determine if these are necessary for the recipe. They may disable the warning by adding "externals=nowarn" to the url. In the future we would like to parse this list and see if the items are already in the SRC_URI for that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble. Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded. (Bitbake rev: bf53f07c3647e57d8452a7743a2b04bcb72c80d6) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix a bug where the wrong path was used for the submodule initMark Hatle2019-05-161-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Because we are trying to avoid network activity and use our own fetcher, the system emulates the behavior of 'git submodule init'. git submodule init uses the .gitmodules file, where typically the module name and path are the same. However, in this case the module name and path (in the tree) were different. i.e.: [submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"] path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared url = https://github.com/Azure/azure-c-shared-utility.git Previously the code assumed the 'path' was both the checkout location under .git/modules, as well as the path to extract the components. This proved to be incorrect as the .git/modules path needs to match the submodule 'name'. This causes the components that were fetched to be initialized in the wrong location, which later caused the 'git submodule update' process to skip not properly initialized modules. A test case was added for this specific case to ensure a regression does not appear in the future. (Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-0417-34/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-0416-204/+4
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-0418-1/+39
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/git: git-lfs checkNaveen Saini2019-04-291-0/+25
| | | | | | | | | | | | Build will fail if repository has lfs contents in absense of git-lfs tool on host. Build will pass if repository may or may not contains lfs content if host has git-lfs installed. [YOCTO #13198] (Bitbake rev: 3f2c2eb2f59707828bdcdd6414db837da8dc3b0e) Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Add need_update method to determine when we are going to a ↵Mark Hatle2019-04-011-0/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | new SRCREV If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly. The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch. To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present. This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work. This issue was discovered by William A. Kennington III <wak@google.com>. The included test case was also written by him, and included unmodified. (Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Use bb.utils.to_boolean() for BB_NO_NETWORKRobert Yang2019-03-261-2/+2
| | | | | | | | | Make it consistent with BB_FETCH_PREMIRRORONLY. (Bitbake rev: 4c6013f47ef36b03f590c909d7c9a2f50b698620) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLYRobert Yang2019-03-263-3/+3
| | | | | | | | | | | | | | The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous, and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case, so unify it to the previous one. (As BB_NO_NETWORK does). And also use bb.utils.to_boolean() to make them consistent. (Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Print SCMs list when SRCREV_FORMAT is not setRobert Yang2019-03-241-1/+2
| | | | | | | | | | This makes it easier to debug, especially when multipe SCMs like gitsm, otherwise we don't know why there are multiple SCMs. (Bitbake rev: 313fe5e86b254eadfdead706be4bd7b274d5e3c0) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Fix clean to remove clonedirRobert Yang2019-03-241-3/+11
| | | | | | | | | | | | | | | | | | | | | | The localpath is a symlink to clonedir when it is cloned from a mirror, for example: $ bitbake systemtap-native -cfetch $ ls downloads/git2 sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git mirror.path.git.sourceware.org.git.systemtap.git There are both sourceware.org.git.systemtap.git and mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink sourceware.org.git.systemtap.git is created by try_mirror_url(), but do_cleanall" only removed the symlink, didn't remove the real dir mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions, for example, I assumed that do_cleanall removed everything, but it didn't, and it would the re-used next time when do_fetch. This patch fixes the problem. (Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: runfetchcmd(): Print workdir in debug messageRobert Yang2019-03-241-1/+4
| | | | | | | | | | | | | | | The workdir is an important message when run git command, for exmample, before the patch: Running git -c core.fsyncobjectfiles=0 branch --contains [snip] We don't know where it is running, now it is: Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git Which is easier to debug. (Bitbake rev: f25c05b73218d0e8edf2dd63b36bc09e547c5455) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix undefined variable issuesRichard Purdie2019-03-191-5/+5
| | | | | | | | | Various refactors have left hanging variables, often in debug messages, hence why they haven't been spotted. Fix them (pylint spotted the issues). (Bitbake rev: e1f252408982a62485bbf874a01fd30a02aeed32) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Remove pointless lambda functionRichard Purdie2019-03-191-1/+1
| | | | | | (Bitbake rev: 329986c3f2d98c4cd1a43b725194003575e718d5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Clean up whitespace/commentsRichard Purdie2019-03-191-19/+13
| | | | | | | | Clean up the whitespace and comments style, keep pylint happy. (Bitbake rev: ee59fdaae68543ade03cacfdbbf14fdc7e469412) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix typo in proxy handlingRichard Purdie2019-03-191-1/+1
| | | | | | | | | Fix a typo in a variable which would affect proxy handling spotted by pylint. (Bitbake rev: 19f4ca0119e33df64da629253d5002fcdb385f4f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Cleanup module importsRichard Purdie2019-03-191-6/+5
| | | | | | (Bitbake rev: ef062a3251af474fdeb480149856be3887b8c1c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix authentication in checkstatus() of the wget fetcherStefan Klug2019-03-191-2/+2
| | | | | | | | | I wonder how this used to work for anybody. (Bitbake rev: 9481e9cd688f635a7fe2ac60d5826ac26f0cb9ba) Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar callsAndré Draszik2019-02-121-1/+1
| | | | | | | | | | | | | | | | | getVar() has been defaulting to expanding by default for a long time (2016), thus remove the True option from getVar() calls with a regex search and replace. Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: The fetcher did not process some recursive submodules properly.Mark Hatle2019-01-281-1/+3
| | | | | | | | | | | | | | Move the submodule processing outside of the if statement to avoid any optimizations that may happen. Update the test cases to include the additional case, and split the other test cases into individual tests to make it easier to figure out what the failure may be. (Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsmy.py: Fix unpack of submodules of submodulesMark Hatle2019-01-241-3/+6
| | | | | | | | | | | | | | If the submodule is in a subdirectory, it needs to have that structure preserved. This means the unpack path needs to be in the 'dirname' of the final path -- since the unpack directory name is specified in the URI. Additional specific test cases were added to ensure this is working properly based on two recent error reports. (Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Fix relative URLsMark Hatle2019-01-241-5/+6
| | | | | | | | | | | Prior code happened to work with relative URLs, when the code was recently restructured it caused all relative urls to no longer work. Restore the prior code flow for relative support and better comment why that code is there. (Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Refactor the functions and simplify the classMark Hatle2019-01-161-144/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The update_submodules and unpack_submodules functions were nearly indentical, so we made a common function where the different behavior could be passed in by the download and unpack users. The new function is process_submodules. Moved the parse_gitmodules function under the new process_submodules, since there are no external callers. Refactor the file relative path processing to the URL translation code. We also add a warning to the translation if a relative ssh URL has been detected. Since this can cause a problem. In the case of a relative URL that does not work after being translated, it should be possible to use the MIRROR functions to manual translate the generated relative URL into one that works properly. Remove 'git config' processing on download contents. It turns out this is not necessary since all of the later components work using the git fetcher. Limit the 'git submodule update' call to only when unpacking a non-bare repository. Submodules are always loaded as bare, so this prevents intermediate unpacks from being attempted. Finally, the test cases were updated and the new commit ids in the test repository were updates as well. (Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Rework the shallow fetcher and test caseMark Hatle2019-01-161-62/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | A custom shallow submodule is no longer necessary, as the regular git fetcher is used and shallow handling works with the same code. The only general difference between the regular change is simply declaring a clone as shallow, when appropriate. This also removes a potential race condition in copying repositories vs cloning them. The gitsm shallow fetcher test was revised to verify that the submodule is shallow cloned along with the primary repository. The first step of this change was to be sure to clean the gitsubmodule download directory, as was previously done with the may gitsource directory. Additional test components were added to verify commit counts, and an obsolete (and likely incorrect) test for the .git/modules directory to be empty was also removed. (Bitbake rev: f9cc4684dcf4281acc557cda8cb35602354ac3d6) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: revise unpackMark Hatle2019-01-161-38/+78
| | | | | | | | | | | | | | | | | Greatly simply the unpack rule by copying the general functionality of update_submodules as unpack_submodules. This will recursively construct a set of urls and unpack them using the standard system behaviors. The overall code may be slightly bigger, but this ensures that all of the standard locks are inplace, ensuring the code doesn't change out from under the unpack function. (This could have happened before due to using 'cp' instead of further unpacks on submodules. This may still happen in shallow clones.) (Bitbake rev: 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Optimize code and attempt to resolve locking issueMark Hatle2019-01-161-14/+0
| | | | | | | | | | | | | | | | | | | | | | | It was reported that a race condition on a shared download directory could occur with the gitsm fetcher, the result happened with a call to git config that occured within the update_submodules. Since the fetch is locked by the upper level, it was probably the prior need_update(...) function causing this because of some old code. The gitsm class inherits the git class. The need_update was overridding the version in gitsm, so that it forceably checked the submodules. It's clear we can optimize the code by only updating if the primary repository needs updating. Since we don't care if the submodule repository has changed because if the primary hasn't, references to the submodule won't change. (Bitbake rev: 346338667edca1f58ace769ad417548da2b8d981) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Add support for alternative URL formats from submodule filesMark Hatle2019-01-161-2/+15
| | | | | | | | | | | | | | | | | | | | | | The following appear to be the git supported formats: proto://user:pass@host/path (URI format) user@host:path (SSH format) /path or ./path or ../path (local file format) We adjust the parsing to find out if we have a URI format or not. When we are NOT in URI format, we do our best to determine SSH or file format by looking for a ':' in the overall string. If we find a ':' we assume SSH format and adjust accordingly. Note, in SSH format we simply replace the ':' with a '/' when constructing the URL. However, if the original path was ":/...", we don't want '//' so we deal with this corner case as well. (Bitbake rev: dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Fix when a submodule is defined, but not initializedMark Hatle2019-01-161-4/+28
| | | | | | | | | | | | | | | | | | | | | | It is possible for a submodule to be defined in the .gitmodules file, but never initialized in the repository itself. This shows itself when searching for the defined module hash you will get back a empty value. Similarly we need to identify and skip defined but not initialized submodules during the unpack stages as well. Thanks to raphael.lisicki@siemens.com for their help is figuring out how to resolve this issue. Additionally a problem was found where, while unlikely, it may be possible for the wrong revision to have been searched using ls-tree. This has been resolved in the update_submodules function by keeping the correct revision along with the submodule path. (Bitbake rev: 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Fix Deprecated warnings from regexsRichard Purdie2019-01-162-14/+14
| | | | | | | | | Fix handling of escape characters in regexs and hence fix python Deprecation warnings which will be problematic in python 3.8. (Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix os.errno referencesRichard Purdie2018-12-081-2/+2
| | | | | | | | | | os.errno used to happen to work but is invalid. Correct to use errno. [YOCTO #13068] (Bitbake rev: 03aae96a8859409b1ce7e3d5c1197371fa96bd14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix typo in exception nameRichard Purdie2018-12-081-1/+1
| | | | | | | | | This could clearly never have worked since the python3 migration but as its in an error path, it doesn't get tested/used much. (Bitbake rev: 704f27dc28d50a6dc02c8b64274ee4ecb3058c4a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Fix SVN repository concurrent update raceUlf Magnusson2018-12-011-28/+36
| | | | | | | | | | | | | | | | | | | | | | | | | The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN repositories from which tarballs are generated. These repositories were protected from concurrent update with a lock on the tarballs. However, the tarballs are specific to the SRCREV and module checked out (many tarballs can come from the same repository), meaning a repository could be modified concurrently if two recipes checked out two different SRCREVs or modules from it in parallel. This caused errors like the following: ERROR: Fetcher failure: Fetch command failed with exit code 1, output: svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details) svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked. svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked. Fix it by adding a per-repository lock that's independent of the module and SRCREV. (Bitbake rev: 3f1f183a17bf3580da8a4ffd6dab30b62c2654a8) Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de> Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid warning about incorrect character escaping in regexRichard Purdie2018-11-271-1/+1
| | | | | | | | | | | Fixes: lib/bb/fetch2/__init__.py:259: DeprecationWarning: invalid escape sequence \w re.compile("^\w+:(?!//)").match(uri): (Bitbake rev: 2f0b6cea2be7e89ef01181b10899b3e7f11e8f20) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-01-17 22:54:32 +0000