| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch adds an API to bb.fetch2 to enable users to plug in an unpack
tracer that can trace each source file back to its corresponding
upstream source url, even when multiple upstream sources are combined
together in the same unpack directory. This may be required for software
composition analysis, license compliance, and detailed SBoM generation.
This patch provides only the needed hooks in bb.fetch2 code and a dummy
abstract class defining the API; users may load their own unpack tracer
class by setting the BB_UNPACK_TRACER_CLASS config parameter.
(Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f)
Signed-off-by: Alberto Pianon <alberto@pianon.eu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Explicitly pass down the lfs parameter from the source repo URI to
submodules. Add smudge skip to final checkout phase to make sure we
don't access the network here. Everything should have been fetched and
setup from the lfs logic in the git fetcher at this point.
(Bitbake rev: 1f8f21fe024b391d3a6670c64b839db0eee083c3)
Signed-off-by: Randolph Sapp <rs@ti.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
[YOCTO #13550]
(Bitbake rev: 5e45b8eab60d651c98a950533043a4c96b9c8b01)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The submodule repository URI contains a path to something not
necessarily on the local filesystem. This means that we can't use
realpath to normalise it without risking getting bad results if the path
happens to match something on the local filesystem. This situation can
cause very confusing errors if that matching local path happens to be a
symlink to somewhere else.
Using normpath rather than realpath means that the path simplification
follows simple rules on the string rather than looking at the local
filesystem and avoids problems.
(Bitbake rev: 47b271e6c8d96960ebe70f80e58f30cc4cbf42e1)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Co-authored-by: Dave Craig
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Just like download_submodule() does, fixed warings when run
bb.fetch2.Fetch([url], d) in process_submodules' function:
WARNING: grpc-native-1.50.0-r0 do_fetch: URL: gitsm://github.com/abseil/abseil-cpp.git;protocol=https;name=third_party/abseil-cpp;subpath=third_party/abseil-cpp does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls.
(Bitbake rev: 0ed7c75eb0508a1f699f47d7f22d559501865f61)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression
in submodules path parsing. As the result gitsm fetcher fails on each
submodule which name begins from the name of the parent repo which is
totally valid usecase [Yocto #14045] [1]
Fix the code to error out only if submodule's name is equal to parent
name but not if it's part of it.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4
(Bitbake rev: 3ad27272c18f2bb9edd441f840167a3dabd5407b)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If submodule refers to specific revision of the parent repository it
causes deadlock in bitbake locking mechanism (lock is acquired to fetch
the parent and cannot be released before all submodules are fetched).
raise FetchError in such situation to prevent deadlocking.
[Yocto 14045]
(Bitbake rev: 0361ecf7eb82c386a9842cf1f3cb706c0a112e77)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
As specified by git submodule manual relative urls can start either
with '..' or './', second case was incorrectly managed leading to an
interpretation of urls starting with './' as absoulte urls.
(Bitbake rev: 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b)
Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.
A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball. If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org. edk2-firmware will then unpack and build
successfully.
I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before. The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.
However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.
The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases. Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.
(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the bitbake.srcrev nugget is not present for the commit we're
interested in we should not just bail out and say that an update is
needed. Instead we can recursively walk through the submodules and check
for the presence of the required commits.
(Bitbake rev: cfc78316309556bec487ef0a5a9205e41f1be86f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
This prevents multiple bitbake.srcrev entries being created when a
repository has more than one submodule. It also ensures that the
bitbake.srcrev entry is not added unless all submodules are correctly
fetched.
(Bitbake rev: 860ff1193fe53f04696d41635a720c2d1f29fa7f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some fetchers may download additional sources along with those
explicitly listed in SRC_URI. These "implicit URLs" will be needed by
the archiver to ensure that all sources can be archived.
We can't just return a list of URL strings since each URL may need its
own SRCREV data so we return a list of FetchData objects.
Each fetcher can override the implicit_urldata() function to provide the
additional FetchData objects. For now this is just needed in the gitsm
fetcher to walk git submodules recursively.
(Bitbake rev: 1350f241b7d991bd191ce9e44f6662e4376c6e24)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked temporarily so that the .gitmodules file can be examined.
(Bitbake rev: 3987db953e414255ce278bc25a5f6cec0f2a30c7)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Because we are trying to avoid network activity and use our own fetcher,
the system emulates the behavior of 'git submodule init'.
git submodule init uses the .gitmodules file, where typically the module
name and path are the same. However, in this case the module name and
path (in the tree) were different. i.e.:
[submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"]
path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared
url = https://github.com/Azure/azure-c-shared-utility.git
Previously the code assumed the 'path' was both the checkout location
under .git/modules, as well as the path to extract the components. This
proved to be incorrect as the .git/modules path needs to match the submodule
'name'. This causes the components that were fetched to be initialized in
the wrong location, which later caused the 'git submodule update' process to
skip not properly initialized modules.
A test case was added for this specific case to ensure a regression does
not appear in the future.
(Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
new SRCREV
If the system had previously fetched a source repository for use by gitsm,
and then the SRCREV was updated and the new commit already existed, the system
would not re-evaluate the submodules and update them accordingly.
The cause of this issue was that need_update was being used, unmodified, from
the base git fetcher. It did not have any knowledge, nor did it care if we
were moving commits and needed to re-evaluate what was happening due to this
switch.
To fix the issue, during the download process we add all processed (by
gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to
use a new need_update function that not only checks if the git commit is
present, but if we have previously processed this commit to ensure all of the
submodule components are also present.
This approach is used, instead of iterating over the submodules in need_update
to avoid a potential race condition that has affected us in the past. The
need_update is called only with the parent locking. Any time we need to dive
into the submodules, we need to lock, and unlock them, at each stage. This
opens the possibility of errors in either the code, or unintended race
conditions with rm_work.
This issue was discovered by William A. Kennington III <wak@google.com>. The
included test case was also written by him, and included unmodified.
(Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move the submodule processing outside of the if statement to avoid any
optimizations that may happen.
Update the test cases to include the additional case, and split the other
test cases into individual tests to make it easier to figure out what
the failure may be.
(Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the submodule is in a subdirectory, it needs to have that structure
preserved. This means the unpack path needs to be in the 'dirname' of the
final path -- since the unpack directory name is specified in the URI.
Additional specific test cases were added to ensure this is working properly
based on two recent error reports.
(Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Prior code happened to work with relative URLs, when the code was recently
restructured it caused all relative urls to no longer work. Restore the prior
code flow for relative support and better comment why that code is there.
(Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The update_submodules and unpack_submodules functions were nearly indentical,
so we made a common function where the different behavior could be passed
in by the download and unpack users. The new function is process_submodules.
Moved the parse_gitmodules function under the new process_submodules, since
there are no external callers.
Refactor the file relative path processing to the URL translation code.
We also add a warning to the translation if a relative ssh URL has been
detected. Since this can cause a problem.
In the case of a relative URL that does not work after being translated,
it should be possible to use the MIRROR functions to manual translate the
generated relative URL into one that works properly.
Remove 'git config' processing on download contents. It turns out this is not
necessary since all of the later components work using the git fetcher.
Limit the 'git submodule update' call to only when unpacking a non-bare
repository. Submodules are always loaded as bare, so this prevents
intermediate unpacks from being attempted.
Finally, the test cases were updated and the new commit ids in the test
repository were updates as well.
(Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A custom shallow submodule is no longer necessary, as the regular git
fetcher is used and shallow handling works with the same code.
The only general difference between the regular change is simply declaring a
clone as shallow, when appropriate.
This also removes a potential race condition in copying repositories
vs cloning them.
The gitsm shallow fetcher test was revised to verify that the submodule
is shallow cloned along with the primary repository.
The first step of this change was to be sure to clean the gitsubmodule download
directory, as was previously done with the may gitsource directory.
Additional test components were added to verify commit counts, and an
obsolete (and likely incorrect) test for the .git/modules directory to be
empty was also removed.
(Bitbake rev: f9cc4684dcf4281acc557cda8cb35602354ac3d6)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Greatly simply the unpack rule by copying the general functionality of
update_submodules as unpack_submodules. This will recursively construct
a set of urls and unpack them using the standard system behaviors.
The overall code may be slightly bigger, but this ensures that all of the
standard locks are inplace, ensuring the code doesn't change out from
under the unpack function. (This could have happened before due to using
'cp' instead of further unpacks on submodules. This may still happen in
shallow clones.)
(Bitbake rev: 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was reported that a race condition on a shared download directory could
occur with the gitsm fetcher, the result happened with a call to
git config
that occured within the update_submodules. Since the fetch is locked by the
upper level, it was probably the prior need_update(...) function causing this
because of some old code.
The gitsm class inherits the git class. The need_update was overridding the
version in gitsm, so that it forceably checked the submodules.
It's clear we can optimize the code by only updating if the primary repository
needs updating. Since we don't care if the submodule repository has changed
because if the primary hasn't, references to the submodule won't change.
(Bitbake rev: 346338667edca1f58ace769ad417548da2b8d981)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The following appear to be the git supported formats:
proto://user:pass@host/path (URI format)
user@host:path (SSH format)
/path or ./path or ../path (local file format)
We adjust the parsing to find out if we have a URI format or not.
When we are NOT in URI format, we do our best to determine SSH or
file format by looking for a ':' in the overall string. If we find
a ':' we assume SSH format and adjust accordingly.
Note, in SSH format we simply replace the ':' with a '/' when constructing
the URL. However, if the original path was ":/...", we don't want '//' so
we deal with this corner case as well.
(Bitbake rev: dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is possible for a submodule to be defined in the .gitmodules file, but
never initialized in the repository itself. This shows itself when searching
for the defined module hash you will get back a empty value.
Similarly we need to identify and skip defined but not initialized submodules
during the unpack stages as well.
Thanks to raphael.lisicki@siemens.com for their help is figuring out how
to resolve this issue.
Additionally a problem was found where, while unlikely, it may be possible
for the wrong revision to have been searched using ls-tree. This has been
resolved in the update_submodules function by keeping the correct revision
along with the submodule path.
(Bitbake rev: 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
different
Git does not require the module and target path to be the same in the
.gitmodules file. This incorrect assumption was being made previously
causing various unpack failures.
An example .gitmodule showing this issue:
[submodule "plugins/WaveShaper/Libs/inih"]
path = plugins/wolf-shaper/Libs/inih
url = https://github.com/pdesaulniers/inih.git
The unpack function also needed to work in a loop on the overall
submodules_queue. Before it could have missed items that were not in the
primary repository.
(Bitbake rev: 5a7009c204f4d2254e3b2d83ad9319ac23f1cf4d)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Submodules by definition refer to a specific commit, not branch. If we don't
ignore the branch, then any commits on a submodule on a branch different then
the original module will trigger a failure that the commit is not on the
branch.
(Bitbake rev: fdc1dbf96f153b496de52acd8263366a1ff303ad)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Some repositories may specify a relative submodule path. If this happens,
it is our responsibility to use the parents URL (ud) and handle any relative
processing ourselves.
(Bitbake rev: fd9893c338df7828b2c01f1d548aa86dfcf7c50a)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
The new fetcher did not run 'git submodule update' recursively.
(Bitbake rev: 377ed943357b6d47d41d84edbf971741f44affa9)
Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The new gitsm fetcher assumed that submodules were living directly in
the 'modules' directory, whereas they can be arbitrarily nested inside
subdirectories.
Solve it by first creating the parent of the destination directory for
the symlink and copy steps.
(Bitbake rev: 3dbc84e9c572f43209b79f3656d7dc35a6d982ba)
Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
'qbareclone' in place of 'bareclone'
(Bitbake rev: 90a3181f1397ae05862f4e89a9bbac606e74504e)
Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The prior fetcher did not know how to work with MIRRORS, and did not
honor BB_NO_NETWORK and similar.
The new fetcher approach recursively calls 'gitsm' download on each
submodule detected. This ensures that it will go throug the
standard download process.
Each downloaded submodule is then 'attached' to the original download in
the 'modules' directory. This mimics the behavior of:
git submodule init
but there is no chance it will contact the network without permission.
It then corrects upstream reference URIs.
The unpack steps simply copies the items from the downloads to the destdir.
Once copied the submodules are connected and we then run:
git submodule update
According to the git documentation, git submodule init can and will modify
the project configuration and may connect to the network. Doing the
work manually prevents this. (This manual process is allowed based
on my reading of the documentation.)
See: https://git-scm.com/book/en/v2/Git-Tools-Submodules
The small change to the existing test is due to this new code always assuming
the code is from a remote system, and not a 'local' repository. If this
assumption proves to be incorrect -- code will need to be added to deal
with local repositories without an upstream URI.
(Bitbake rev: 9c6b39adf9781fa6745f48913a97c859fa37eb5b)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.
Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.
[YOCTO #12739]
(Bitbake rev: 11b6a5d5c1b1bb0ce0c5bb3983610d13a3e8f84a)
Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When we're building from a shallow mirror tarball, we don't want to do
anything with ud.clonedir, as it's not being used when we unpack. As such,
disable updating the submodules in that case. Also include the repositories in
.git/modules in the shallow tarball. It does not actually make the submodule
repositories shallow at this time.
(Bitbake rev: 6c0613f1f2f9d4f009545f82a9173e80396f9d34)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Cleanup some more usage of bb.data APIs in the fetchers.
(Bitbake rev: 9752fd1c10b8fcc819822fa6eabc2c1050fcc03b)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
When the gitsm fetcher is used with a repo that includes a .gitattributes
file that makes git modify files on cloning (e.g. line break characters),
the subsequent checkout performed in the update_submodules function fails.
This is fixed by adding the force flag (-f) to the checkout command.
(Bitbake rev: c05e1396625b14e66d795408ea2ae4cd2afc3209)
Signed-off-by: Ola Redell <ola.redell@retotech.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Fix the methods in all fetchers so they don't change
the current working directory of the calling process, which
could lead to "changed cwd" warnings from bitbake.
(Bitbake rev: 6aa78bf3bd1f75728209e2d01faef31cb8887333)
Signed-off-by: Matt Madison <matt@madison.systems>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
nested submodules
This fixes a problem when the repository contains multiple levels of submodules via a resursive submodule init.
(Bitbake rev: dbafbe229360ffe5908b106a9c10e274712b9b17)
Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fix a problem when checking out a commit that changes the submodules
previously checkout.
Example:
Recipe uses branch A and then it updates to use branch B, but branch B has
different submodules dependencies then what branch A previously had.
(Bitbake rev: 54a3864246f2be0b62761f639a1d5c9407aded4f)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is nessary when specified branch with submodules is different then
default (master) branch.
[YOCTO #7771]
(Bitbake rev: f7b0b5e33e00f3ce0744322eee93835ee76bf184)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: ef2bf63e56b87e19d1a9059dd2d81a9a1a537254)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Git versions older than 1.7.10 put absolute paths in configuration files
for the submodule repositories, leading to errors when the repository
checkout is moved. We move the repository as a matter of course in the
gitsm fetcher; the failure occurs in do_unpack). Change the absolute
paths to be relative during processing to fix this.
(At the time of writing, Ubuntu 12.04.4 LTS ships Git version 1.7.9.5,
hence the desire to fix this rather than just mandating a newer Git
version.)
Fixes [YOCTO #5525].
(Bitbake rev: e700d5a41deed4ee837465af526ed30c8a579933)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
This allows FETCHCMD_git to override the fetcher command as the git fetcher does.
[YOCTO #5717]
(Bitbake rev: 23ab943be3a33077d6ad8be68bba53cd1e2270b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is no good reason to keep passing around the url parameter when
its contained within urldata (ud). This is left around due to
legacy reasons, some functions take it, some don't and its time
to cleanup.
This is fetcher internal API, there are a tiny number of external users
of the internal API (buildhistory and distrodata) which can be fixed up
after this change.
(Bitbake rev: 6a48474de9505a3700863f31839a7c53c5e18a8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds very basic git submodule support to the fetcher. It can be
used by replacing a git:// url prefix with a gitsm:// prefix, otherwise
behaviour is the same as the git fetcher. Whilst this code should be
functional, its not as efficient as the usual git fetcher due to the
need to checkout the tree to fetch/update the submodule information. git
doesn't support submodule operations on the bare clones the standard git
fetcher uses which is also problematic.
This code does however give a starting point to people wanting to use
submodules.
(Bitbake rev: 25e0b0bc50114f1fbf955de23cc0c96f5f7a41e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|