linux/poky.git/bitbake/lib/bb/fetch2/gitsm.py, branch yocto-5.0

bitbake: fetch2: Add API for upstream source tracing

2023-10-15T08:12:43+00:00

This patch adds an API to bb.fetch2 to enable users to plug in an unpack tracer that can trace each source file back to its corresponding upstream source url, even when multiple upstream sources are combined together in the same unpack directory. This may be required for software composition analysis, license compliance, and detailed SBoM generation. This patch provides only the needed hooks in bb.fetch2 code and a dummy abstract class defining the API; users may load their own unpack tracer class by setting the BB_UNPACK_TRACER_CLASS config parameter. (Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f) Signed-off-by: Alberto Pianon Signed-off-by: Richard Purdie

bitbake: gitsm: tolerate git-lfs in submodules

2023-08-30T09:41:54+00:00

Explicitly pass down the lfs parameter from the source repo URI to submodules. Add smudge skip to final checkout phase to make sure we don't access the network here. Everything should have been fetched and setup from the lfs logic in the git fetcher at this point. (Bitbake rev: 1f8f21fe024b391d3a6670c64b839db0eee083c3) Signed-off-by: Randolph Sapp Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2/gitsm: Document that we won't support propagating user parameter

2023-07-29T12:26:19+00:00

[YOCTO #13550] (Bitbake rev: 5e45b8eab60d651c98a950533043a4c96b9c8b01) Signed-off-by: Yoann Congal Signed-off-by: Richard Purdie

bitbake: gitsm: Fix path construction for relative submodule URI

2023-02-26T11:49:41+00:00

The submodule repository URI contains a path to something not necessarily on the local filesystem. This means that we can't use realpath to normalise it without risking getting bad results if the path happens to match something on the local filesystem. This situation can cause very confusing errors if that matching local path happens to be a symlink to somewhere else. Using normpath rather than realpath means that the path simplification follows simple rules on the string rather than looking at the local filesystem and avoids problems. (Bitbake rev: 47b271e6c8d96960ebe70f80e58f30cc4cbf42e1) Signed-off-by: Mike Crowe Co-authored-by: Dave Craig Signed-off-by: Richard Purdie

bitbake: gitsm.py: process_submodules(): Set nobranch=1 for url

2022-11-14T16:22:00+00:00

Just like download_submodule() does, fixed warings when run bb.fetch2.Fetch([url], d) in process_submodules' function: WARNING: grpc-native-1.50.0-r0 do_fetch: URL: gitsm://github.com/abseil/abseil-cpp.git;protocol=https;name=third_party/abseil-cpp;subpath=third_party/abseil-cpp does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls. (Bitbake rev: 0ed7c75eb0508a1f699f47d7f22d559501865f61) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

bitbake: gitsm: Fix regression in gitsm submodule path parsing

2022-11-14T16:22:00+00:00

Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression in submodules path parsing. As the result gitsm fetcher fails on each submodule which name begins from the name of the parent repo which is totally valid usecase [Yocto #14045] [1] Fix the code to error out only if submodule's name is equal to parent name but not if it's part of it. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4 (Bitbake rev: 3ad27272c18f2bb9edd441f840167a3dabd5407b) Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie

bitbake: gitsm: Error out if submodule refers to parent repo

2022-09-01T09:07:00+00:00

If submodule refers to specific revision of the parent repository it causes deadlock in bitbake locking mechanism (lock is acquired to fetch the parent and cannot be released before all submodules are fetched). raise FetchError in such situation to prevent deadlocking. [Yocto 14045] (Bitbake rev: 0361ecf7eb82c386a9842cf1f3cb706c0a112e77) Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie

bitbake: fetch2: gitsm: fix incorrect handling of git submodule relative urls

2022-07-01T11:20:18+00:00

As specified by git submodule manual relative urls can start either with '..' or './', second case was incorrectly managed leading to an interpretation of urls starting with './' as absoulte urls. (Bitbake rev: 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b) Signed-off-by: Gennaro Iorio Signed-off-by: Richard Purdie

bitbake: lib/bb: Clean up use of len()

2021-11-03T10:12:42+00:00

(Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d) Signed-off-by: Richard Purdie

bitbake: fetch2/gitsm: remove the 'nugget' SRCREV caching

2021-10-08T15:45:05+00:00

The cached revisions which are used to decide if a repository doesn't need to be updated are misleading when used in conjunction with mirror tarballs and can cause partial fetches to happen, resulting in unpack errors as repositories were not fetched. A concrete example: edk2-firmware in meta-arm is at version 202102 (ef91b0). This is built on the autobuilder so the source mirror contains the repository as a mirror tarball. If I build edk2-firmware 202102 the gitsm fetcher will initially download the top-level repository and then iterate into the submodules to also fetch those repositories, including cmocka from cryptomilk.org. edk2-firmware will then unpack and build successfully. I then update edk2-firmware to 202105 (e1999b) and build it. Gitsm.needs_update() starts by calling Git.needs_update() which returns False, as the mirror tarball contains this revision. It then looks at the "nuggets" which are SRCREVs it has fetched before. The mirror tarball itself contains the nugget for e1999b as this has been built on the autobuilder, so needs_update return False, no more fetching is done, and the build proceeds to unpack. However, as part of the 202105 upgrade the URL of the cmocka submodule changed, and this new repository was never fetched. This means that unpack fails as one of the required git repositories isn't available. The nugget codepaths appear to be an attempt at optimising the fetch process, but have demonstratable failure cases. Just removing them entirely solves the edk2-firmware example, and all of the fetcher test cases still pass. (Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie