summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2: Change git fetcher not to destroy old referencesRichard Purdie2020-09-081-1/+1
| | | | | | | | | | | | | | | | It looks like we're about to see a lot of changes in branch names in repos. If we have the prune option here, those old names are lost, the changes propagate to our source mirrors and our old releases break. We have the force option so any replaced references should be replaced, its only orphaned branches which will now be preserved. I believe this behaviour will cause us fewer problems given the changes that look likely to happen. (Bitbake rev: 12d8cc3fecd550c4aadf0519e80711d755ee75ba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure cached url data is matched to a datastoreRichard Purdie2019-11-131-2/+2
| | | | | | | | | | | | | | | | | | | | There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 97067634b1f149b56844b10e3a5e8d0d980b6e34) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: show warning when renaming the archive with bad checksum failedMartin Jansa2019-09-061-1/+2
| | | | | | | | | | | | | | | | | | | | | * noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: d36438759344caa447d9a0bf30749a0aa31d1fba) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Add need_update method to determine when we are going to a ↵Mark Hatle2019-06-181-0/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | new SRCREV If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly. The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch. To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present. This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work. This issue was discovered by William A. Kennington III <wak@google.com>. The included test case was also written by him, and included unmodified. (Bitbake rev: 4ce92f43eeac6a4bfd06e8567fa6891614b5b3b0) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix a bug where the wrong path was used for the submodule initMark Hatle2019-06-181-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Because we are trying to avoid network activity and use our own fetcher, the system emulates the behavior of 'git submodule init'. git submodule init uses the .gitmodules file, where typically the module name and path are the same. However, in this case the module name and path (in the tree) were different. i.e.: [submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"] path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared url = https://github.com/Azure/azure-c-shared-utility.git Previously the code assumed the 'path' was both the checkout location under .git/modules, as well as the path to extract the components. This proved to be incorrect as the .git/modules path needs to match the submodule 'name'. This causes the components that were fetched to be initialized in the wrong location, which later caused the 'git submodule update' process to skip not properly initialized modules. A test case was added for this specific case to ensure a regression does not appear in the future. (Bitbake rev: ffd7ed530a17d22df576d986ac78428a6979e79c) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: The fetcher did not process some recursive submodules properly.Mark Hatle2019-03-241-1/+3
| | | | | | | | | | | | | | | | | | Move the submodule processing outside of the if statement to avoid any optimizations that may happen. Update the test cases to include the additional case, and split the other test cases into individual tests to make it easier to figure out what the failure may be. (Bitbake rev: 7c1eb51d1e8a4c5f39bf9dddf05fb0b3598da72b) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0ec98c01ae50f95c9c74acf53013ac59e0e72b08) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsmy.py: Fix unpack of submodules of submodulesMark Hatle2019-03-241-3/+6
| | | | | | | | | | | | | | | | | | If the submodule is in a subdirectory, it needs to have that structure preserved. This means the unpack path needs to be in the 'dirname' of the final path -- since the unpack directory name is specified in the URI. Additional specific test cases were added to ensure this is working properly based on two recent error reports. (Bitbake rev: acca06d060e49b2441562b4dc94416af9ab8187e) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8c8ecec2a722bc2885e2648d41ac8df07bdf660d) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Fix relative URLsMark Hatle2019-03-241-5/+6
| | | | | | | | | | | | | | | Prior code happened to work with relative URLs, when the code was recently restructured it caused all relative urls to no longer work. Restore the prior code flow for relative support and better comment why that code is there. (Bitbake rev: 39afa6ead1d72813b2a294f065d759bad08fb53d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 14bb383af0ca98e0e04ec217e537b6a899f3658e) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Refactor the functions and simplify the classMark Hatle2019-03-241-144/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The update_submodules and unpack_submodules functions were nearly indentical, so we made a common function where the different behavior could be passed in by the download and unpack users. The new function is process_submodules. Moved the parse_gitmodules function under the new process_submodules, since there are no external callers. Refactor the file relative path processing to the URL translation code. We also add a warning to the translation if a relative ssh URL has been detected. Since this can cause a problem. In the case of a relative URL that does not work after being translated, it should be possible to use the MIRROR functions to manual translate the generated relative URL into one that works properly. Remove 'git config' processing on download contents. It turns out this is not necessary since all of the later components work using the git fetcher. Limit the 'git submodule update' call to only when unpacking a non-bare repository. Submodules are always loaded as bare, so this prevents intermediate unpacks from being attempted. Finally, the test cases were updated and the new commit ids in the test repository were updates as well. (Bitbake rev: 8b02eb8e5d8e3a09e19ed96a4ccdf5f755e97a74) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 610dbee5634677f5055e2b36a3043cd197fb8c51) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Rework the shallow fetcher and test caseMark Hatle2019-03-241-62/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A custom shallow submodule is no longer necessary, as the regular git fetcher is used and shallow handling works with the same code. The only general difference between the regular change is simply declaring a clone as shallow, when appropriate. This also removes a potential race condition in copying repositories vs cloning them. The gitsm shallow fetcher test was revised to verify that the submodule is shallow cloned along with the primary repository. The first step of this change was to be sure to clean the gitsubmodule download directory, as was previously done with the may gitsource directory. Additional test components were added to verify commit counts, and an obsolete (and likely incorrect) test for the .git/modules directory to be empty was also removed. (Bitbake rev: 85dc1c65b661f9712ae98587d4d0d868146c8cff) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f9cc4684dcf4281acc557cda8cb35602354ac3d6) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: revise unpackMark Hatle2019-03-241-38/+78
| | | | | | | | | | | | | | | | | | | | | Greatly simply the unpack rule by copying the general functionality of update_submodules as unpack_submodules. This will recursively construct a set of urls and unpack them using the standard system behaviors. The overall code may be slightly bigger, but this ensures that all of the standard locks are inplace, ensuring the code doesn't change out from under the unpack function. (This could have happened before due to using 'cp' instead of further unpacks on submodules. This may still happen in shallow clones.) (Bitbake rev: 02ce6783bd3cfc117c77ca0380a87b0e1c60d8db) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Optimize code and attempt to resolve locking issueMark Hatle2019-03-241-14/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | It was reported that a race condition on a shared download directory could occur with the gitsm fetcher, the result happened with a call to git config that occured within the update_submodules. Since the fetch is locked by the upper level, it was probably the prior need_update(...) function causing this because of some old code. The gitsm class inherits the git class. The need_update was overridding the version in gitsm, so that it forceably checked the submodules. It's clear we can optimize the code by only updating if the primary repository needs updating. Since we don't care if the submodule repository has changed because if the primary hasn't, references to the submodule won't change. (Bitbake rev: 4660933f83e528766d71eab662cc79dcf17b4be7) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 346338667edca1f58ace769ad417548da2b8d981) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Add support for alternative URL formats from submodule filesMark Hatle2019-03-241-2/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | The following appear to be the git supported formats: proto://user:pass@host/path (URI format) user@host:path (SSH format) /path or ./path or ../path (local file format) We adjust the parsing to find out if we have a URI format or not. When we are NOT in URI format, we do our best to determine SSH or file format by looking for a ':' in the overall string. If we find a ':' we assume SSH format and adjust accordingly. Note, in SSH format we simply replace the ':' with a '/' when constructing the URL. However, if the original path was ":/...", we don't want '//' so we deal with this corner case as well. (Bitbake rev: a21b2598531d52123933a0ac1c4deeecfdd1697d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Fix when a submodule is defined, but not initializedMark Hatle2019-03-241-4/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | It is possible for a submodule to be defined in the .gitmodules file, but never initialized in the repository itself. This shows itself when searching for the defined module hash you will get back a empty value. Similarly we need to identify and skip defined but not initialized submodules during the unpack stages as well. Thanks to raphael.lisicki@siemens.com for their help is figuring out how to resolve this issue. Additionally a problem was found where, while unlikely, it may be possible for the wrong revision to have been searched using ls-tree. This has been resolved in the update_submodules function by keeping the correct revision along with the submodule path. (Bitbake rev: e9c965bd816c8e4e5cc8e45db8e0dab597ce5a6f) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: don't use shell=True when listing ar filesRoss Burton2019-01-081-1/+1
| | | | | | | (Bitbake rev: ff2e5d435046886791551587a0988f21ddfe6e2a) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Fix SVN repository concurrent update raceUlf Magnusson2018-12-061-28/+36
| | | | | | | | | | | | | | | | | | | | | | | | | The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN repositories from which tarballs are generated. These repositories were protected from concurrent update with a lock on the tarballs. However, the tarballs are specific to the SRCREV and module checked out (many tarballs can come from the same repository), meaning a repository could be modified concurrently if two recipes checked out two different SRCREVs or modules from it in parallel. This caused errors like the following: ERROR: Fetcher failure: Fetch command failed with exit code 1, output: svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details) svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked. svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked. Fix it by adding a per-repository lock that's independent of the module and SRCREV. (Bitbake rev: cddd27fb66bf82465cf8eb5012cb8e3c088625a5) Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de> Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid warning about incorrect character escaping in regexRichard Purdie2018-12-061-1/+1
| | | | | | | | | | | Fixes: lib/bb/fetch2/__init__.py:259: DeprecationWarning: invalid escape sequence \w re.compile("^\w+:(?!//)").match(uri): (Bitbake rev: 9c619f1eb1583582fafda6acabaf08a5761215ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: fix AttributeError in shallow extraction logicChristopher Larson2018-11-261-5/+8
| | | | | | | | | | | | | | | This code checks to see if shallow is either disabled or the tarball is missing, but the else block tries to print the tarball filename, and this attribute doesn't exist at all when shallow is disabled. Handle the two cases separately to give sane errors for both cases without the exception: Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow' (Bitbake rev: b0ad961064ef6ecd77fab3435e5652c4ae284426) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Also export GIT_SSH environment variableMike Crowe2018-11-071-0/+1
| | | | | | | | | | | Support users who want to specify a custom ssh to Git using an environment variable. (This lets build systems work without requiring users to configure Git explicitly.) (Bitbake rev: db46fb2755a4b1033ab60051ce511cc9dd7e34fb) Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm.py: Fix the references when the module and path are ↵Mark Hatle2018-11-071-35/+34
| | | | | | | | | | | | | | | | | | | | | | | different Git does not require the module and target path to be the same in the .gitmodules file. This incorrect assumption was being made previously causing various unpack failures. An example .gitmodule showing this issue: [submodule "plugins/WaveShaper/Libs/inih"] path = plugins/wolf-shaper/Libs/inih url = https://github.com/pdesaulniers/inih.git The unpack function also needed to work in a loop on the overall submodules_queue. Before it could have missed items that were not in the primary repository. (Bitbake rev: 5a7009c204f4d2254e3b2d83ad9319ac23f1cf4d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm.py: Disable branch checking on submodulesMark Hatle2018-11-071-1/+1
| | | | | | | | | | | | Submodules by definition refer to a specific commit, not branch. If we don't ignore the branch, then any commits on a submodule on a branch different then the original module will trigger a failure that the commit is not on the branch. (Bitbake rev: fdc1dbf96f153b496de52acd8263366a1ff303ad) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid incorrect getVarFlag callRichard Purdie2018-10-181-1/+3
| | | | | | | | | Calling getVarFlag with flag=None makes no sense, don't do it. Bitbake used to silently ignore this, it now warns so avoid the warning. (Bitbake rev: a68de8ace62eaba23856bfb301efbbe1824322aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: provide information about missing sourcesUrs Fässler2018-10-181-8/+21
| | | | | | | | | | | Provide more information in the case the sources are not found in the unpack step. (Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: explicitly show the decision logic to select the source ↵Urs Fässler2018-10-181-3/+4
| | | | | | | | | | in code (Bitbake rev: 78d8fb3a7899e1404a513be1c2b2b4440da8b12d) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: prevent access to non-existing clonedirUrs Fässler2018-10-181-1/+3
| | | | | | | | | | | | A user friendly error is throw when neither the clonedir nor fullshallow exist. Without the check, a difficult to interpret error is throw from within the fetch command. (Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: extract the function which ensures that a valid symlink existsUrs Fässler2018-10-181-10/+17
| | | | | | | | | | | For better readability and future use, we extract the function which ensures that a given symlink exists. (Bitbake rev: 5e69ca56533666a097bb23d09ab673e5c862051c) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: avoid circular recursion with SRCPV in PRPeter Marko2018-10-181-1/+3
| | | | | | | | | | | Some recent changes broke SRCPV and workaround was introduced to avoid circular dependency if SRCPV is in PV. However there is still the same error if SRCPV is in PR. (Bitbake rev: 05ee4845f925b8528a7ce9cffb4bae425b8fa1e9) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Allow relative URIs when fetchingMark Hatle2018-10-041-0/+5
| | | | | | | | | | | Some repositories may specify a relative submodule path. If this happens, it is our responsibility to use the parents URL (ud) and handle any relative processing ourselves. (Bitbake rev: fd9893c338df7828b2c01f1d548aa86dfcf7c50a) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: checkout submodules recursivelyLaurent Bonnans2018-10-041-30/+38
| | | | | | | | | The new fetcher did not run 'git submodule update' recursively. (Bitbake rev: 377ed943357b6d47d41d84edbf971741f44affa9) Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: handle submodules nested inside subdirsLaurent Bonnans2018-10-041-6/+9
| | | | | | | | | | | | | | The new gitsm fetcher assumed that submodules were living directly in the 'modules' directory, whereas they can be arbitrarily nested inside subdirectories. Solve it by first creating the parent of the destination directory for the symlink and copy steps. (Bitbake rev: 3dbc84e9c572f43209b79f3656d7dc35a6d982ba) Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: fix a typoLaurent Bonnans2018-10-041-1/+1
| | | | | | | | | 'qbareclone' in place of 'bareclone' (Bitbake rev: 90a3181f1397ae05862f4e89a9bbac606e74504e) Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: use correct check to decide if the shallow tarball ↵Urs Fässler2018-10-041-1/+1
| | | | | | | | | | | | | should be unpacked The shallow_tarball check is never true due a check on the caller side. The tarball check is not related to the code on the caller side. (Bitbake rev: 086eddcf8c7520ff5c52ce2a11ca9bf5b5fe5d7e) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: prevent access to non-existing fullshallow tarballUrs Fässler2018-10-041-1/+1
| | | | | | | | (Bitbake rev: b7f00a8c11672a2ee0408e210fb174cda3384e3f) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: improve readability of method need_updateUrs Fässler2018-10-041-4/+9
| | | | | | | | | | | To improve the readability we extract the different scenarios of why the clonedir needs an update. (Bitbake rev: 9038e029f4f0ab413727de76c74248cbb3cdc9ea) Signed-off-by: Urs Fässler <urs.fassler@bbv.ch> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm.py: Rework the git submodule fetcherMark Hatle2018-09-261-123/+156
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The prior fetcher did not know how to work with MIRRORS, and did not honor BB_NO_NETWORK and similar. The new fetcher approach recursively calls 'gitsm' download on each submodule detected. This ensures that it will go throug the standard download process. Each downloaded submodule is then 'attached' to the original download in the 'modules' directory. This mimics the behavior of: git submodule init but there is no chance it will contact the network without permission. It then corrects upstream reference URIs. The unpack steps simply copies the items from the downloads to the destdir. Once copied the submodules are connected and we then run: git submodule update According to the git documentation, git submodule init can and will modify the project configuration and may connect to the network. Doing the work manually prevents this. (This manual process is allowed based on my reading of the documentation.) See: https://git-scm.com/book/en/v2/Git-Tools-Submodules The small change to the existing test is due to this new code always assuming the code is from a remote system, and not a 'local' repository. If this assumption proves to be incorrect -- code will need to be added to deal with local repositories without an upstream URI. (Bitbake rev: 9c6b39adf9781fa6745f48913a97c859fa37eb5b) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetcher: Fixed remote removal not throwing exception.Paulo Neves2018-08-281-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this fix it is assumed that the removal of the remote can only fail because there is not remote to remove. This is a false assumption. Example error which would be ignored: git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output: Note: A branch outside the refs/remotes/ hierarchy was not removed; to delete it, use: git branch -d master error: could not lock config file config error: Could not remove config section 'remote.origin' Due to the masking of this error a stranger error will be presented to the user, because this time we do not mask the exception: git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output: fatal: remote origin already exists. The most likely reason that the remote cannot be removed nor modified is that the DL_DIR/git2 does not have permissions compatible with the user running bitbake. This commit fixes: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728 (Bitbake rev: 9c86c582a10c9b23abad7d34b6cbf12f7086294d) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Use correct check in parameter handling in uri_replace()Nikolai Merinov2018-06-211-2/+2
| | | | | | | | | | If uri_find contain parameters then original URI parameters should be checked against parameters from uri_find instead of parameters from uri_replace. (Bitbake rev: 8efa7826a61501589afa33eb698c0ab3a622bf2e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: unify the way fetchers determine DL_DIR and FETCHCMDAndre McCurdy2018-06-158-21/+23
| | | | | | | | | | | | | | | Currently there is quite some variation between the fetchers in terms of how they determine the subdirectory within DL_DIR and the base fetch command to run. Some rely on variables being set externally (e.g. from bitbake.conf in oe-core), some respect these external variables but provide fallback defaults and some use only hardcoded internal values. Try to unify the approach used across the various fetchers. (Bitbake rev: efd5e35af4b08501c67e8b30f30d9457f6fdf610) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/gitsm: avoid live submodule fetching during unpack()Matt Hoosier2018-06-071-8/+75
| | | | | | | | | | | | | | | | | | Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task. Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules. [YOCTO #12739] (Bitbake rev: 11b6a5d5c1b1bb0ce0c5bb3983610d13a3e8f84a) Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix import error for Python 3.6.5Tzu Hsiang Lin2018-05-292-3/+1
| | | | | | | | | | | | When running bitbake command with Python 3.6.5 always result in import error causing by the change of distutils module. This patch replaces the method to search executable in PATH by "/usr/bin/env <command>". (Bitbake rev: bd9a1b063633af2936ba1dd87b19202424900151) Signed-off-by: Tzu Hsiang Lin <t9360341@ntut.org.tw> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/__init__.py: Fixed handling of uris with empty pathJakub Dębski2018-05-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | For mirrors or premirrors defined like: "http://.*/.* http://somewhere.org" fetching ends with errors because function fetch2/__init__.py:encodeurl() creates url like "http://somewhere.orgsomefile.tar.gz". It happens because function fetch2/__init__.py:decodeurl() for url "http://somewhere.org" returns ['http', 'somewhere.org', '', '', '', {}] and then in function fetch2/__init__.py:uri_replace() variable result_decode will be ['http', 'somewhere.org', 'somefile.tar.gz', '', '', {}] (because of line: result_decoded[loc] = os.path.join(result_decoded[loc], basename)) for which encodeurl returns "http://somewhere.orgsomefile.tar.gz". In addition for mirror "http://.*/.* http://somewhere.org/" everything works fine. (Bitbake rev: d822ae24ef5485e550804cbd9130ebd73b2aa48e) Signed-off-by: Jakub Dębski <jdebski@enigma.com.pl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: make fewer calls to _contains_ref() from download()Andre McCurdy2018-05-221-6/+6
| | | | | | | | | | | | | | | Updating a local git repo clone currently results in multiple calls to self._contains_ref(), some of which appear to be redundant and can be eliminated by minor tweaks to the logic in download(). Also drop redundant calls to os.path.exists(ud.clonedir) before self.need_update(), since need_update() includes its own built-in check for the existance of ud.clonedir. (Bitbake rev: 61b0df5523afc8f805043f3adc9c106690e6f133) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: log exception if ls-remote failsRoss Burton2018-03-281-1/+2
| | | | | | | (Bitbake rev: 8212446de11c0e370c55f88cde86334b760cd939) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Handle missing donestamp file when content is validNathan Rossi2018-02-141-11/+10
| | | | | | | | | | | | | | | | | | | | | | | | In order to allow users to manually populate the download directory with valid content change the assumption that missing the donestamp file means unfetched content. This allows users to populate the download dir without needing to create dummy .done files such that a user does not need a PREMIRROR when using BB_NO_NETWORK to provide valid content files in the download directory. To ensure the correct result this change also fails first if the localpath does not exist. This prevents further parts of the function attempting to calculating the checksum on non-existent files. This also fixes some edge conditions around where if the donestamp exists but the localpath does not it returns, and did not remove the donestamp. Also added test cases to cover this use case and additional use cases where for example the fetcher does not support checksums. (Bitbake rev: a335dbbb65d5b56e71d98cf3e4fa9bfbec1dcde6) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm.py: Fix inverted condition to prevent infinite loopBöszörményi Zoltán2018-02-141-3/+5
| | | | | | | | | | | | | | | At least the cli-color node module has dependencies that have cyclic dependency among themselves. npm.py is prepared to deal with such a case but the condition is handled only for downloading or not a dependency again, but then it goes checking the its dependency which causes an infinite loop in _getdependencies(). Make this function simply return when a dependency is already downloaded and only download and check its dependencies when not. (Bitbake rev: 545540420112992e53f4a83104af10452df168d0) Signed-off-by: Zoltán Böszörményi <zboszor@pr.hu> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Use 'git-make-shallow' from bin directoryNathan Rossi2018-02-141-1/+4
| | | | | | | | | | | | | | Move the code that existed in tests/fetch.py for determining the path to 'git-make-shallow' into the git module and reference it. This ensures that 'git-make-shallow' is always available and the desired version regardless of the path variable or whether git exposes the command. (Bitbake rev: 6b508ab8fd5aa796c1c00c970e81e5e93f84d35d) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Add pack-refs command to mirror handling codeRichard Purdie2018-01-291-0/+1
| | | | | | | | | | | | | We've noticed issues on our infrastucture iterating over the many tag/branch/head reference files that some git repositories may contain. By issuing the pack-refs command, we move these all to a single file which speeds up operations with the mirror repos in the downloads directory in general. (Bitbake rev: f8126aaf774186a6eaf0bd4067b89c074594886c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/__init__: Disable pseudo in runfetchcmd()Peter Kjellerstedt2018-01-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a fetcher, e.g., git, is run when pseudo is active it will think it is running as root. If it in turn uses ssh (as git does), ssh too will think it is running as root. This will cause it to try to read root's ssh configuration from /root/.ssh which will fail. If ssh then needs to ask for credentials it will hang indefinitely as there is nowhere for it to ask the user for them (and even if there was it would not access the correct private keys). The solution to the above is to temporarily disable pseudo while executing any fetcher commands. There should be no reason for them to be executed under pseudo anyway so this should not be a problem. RP Ammendum: We finally did get more information about how to reproduce this problem, something needs to trigger bb.fetch2.get_srcrev() in a pseudo context, for example when AUTOREV is in use or the recipe doesn't have a defined SRCREV. That SRC_URI needs to be using protocol=ssh. This would trigger an ls-remote of the remote repo and if that happens under pseudo, the wrong ssh credentials may be attempted which can hang. [YOCTO #12464] (Bitbake rev: ceaca281cafa662aa2385b95641bce309dce843d) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix unpack of .txz filesAndré Draszik2017-11-291-1/+1
| | | | | | | | | | .txz is the same as .tar.xz, and can be found in the wild. (Bitbake rev: 2ba8a6b25ccc12e7b543e8450121e5311c7a701d) Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: use with to ensure the response is closedRoss Burton2017-11-211-1/+2
| | | | | | | (Bitbake rev: 8c487176d311557031cedba76185f14f0e7a14cd) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-21 11:30:20 +0000