linux/poky.git/bitbake/lib/bb/fetch2, branch yocto-4.0.19

bitbake: fetch2: Ensure that git LFS objects are available

2024-03-01T18:00:58+00:00

The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (cherry picked from commit cfae1556bf671acec119a6c8bbc4b667a856b9ae) (Bitbake rev: 40fd5f4eef7460ca67f32cfce8e229e67e1ff607) Signed-off-by: Philip Lorenz Signed-off-by: Richard Purdie Signed-off-by: Philip Lorenz Signed-off-by: Steve Sakoman

bitbake: fetch/git: Fix local clone url to make it work with repo

2023-04-11T10:31:52+00:00

The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie (cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch2/git: show SRCREV and git repo in error message about fixed SRCREV

2023-02-23T22:23:07+00:00

* in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to be one character shorter and because fetcher uses: if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]): to decide which SRCREV values are fixed SRCREVs this one was considered as tag or branch name, because it was only 39 chars long The original error message wasn't very helpful as it doesn't show which repo or which SRCREV was considered missing: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) with SRCPV included in PV as error recomments it's a bit better: bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16 The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T with this change the first error will read: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) (Bitbake rev: 9bbdedc0ba7ca819b898e2a29a151d6a2014ca11) Signed-off-by: Martin Jansa Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch2/git: Clarify the meaning of namespace

2023-02-04T10:34:05+00:00

Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: 86f2fa5261da959cda706c794a0047e5e89d4d6b) Signed-off-by: Marek Vasut Signed-off-by: Richard Purdie (cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

2023-02-04T10:34:05+00:00

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt (Bitbake rev: c17fc1468ab84663b919e2809606b1b8ea2bebd9) Signed-off-by: Marek Vasut Signed-off-by: Alexandre Belloni (cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: gitsm: Fix regression in gitsm submodule path parsing

2022-12-09T13:02:50+00:00

Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression in submodules path parsing. As the result gitsm fetcher fails on each submodule which name begins from the name of the parent repo which is totally valid usecase [Yocto #14045] [1] Fix the code to error out only if submodule's name is equal to parent name but not if it's part of it. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4 (Bitbake rev: 7e268c107bb0240d583d2c34e24a71e373382509) Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie (cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: gitsm: Error out if submodule refers to parent repo

2022-10-11T20:57:28+00:00

If submodule refers to specific revision of the parent repository it causes deadlock in bitbake locking mechanism (lock is acquired to fetch the parent and cannot be released before all submodules are fetched). raise FetchError in such situation to prevent deadlocking. [Yocto 14045] (Bitbake rev: 987712c4c8fefd86a1f5116c11ee86e296e852ee) Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie (cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: Fix npm to use https rather than http

2022-10-11T20:57:28+00:00

Hit this error while building nlf-native recently: { "error": { "summary": "URI malformed", "detail": "" } } Some poking about led me to discover that: 1) The npm.py tool replaces npm:// with http://, not https:// 2) Some versions of the npm tool don't handle 301 redirects properly, choosing to display the above error instead when using the default nodejs registry It would be good to go fix npm to handle the redirect properly, but it seems like it would also be good to assume secure http when contacting a registry, hence, this patch (Bitbake rev: bb5c43220f5f1c3d82334c65aff1ce13008db8d9) Signed-off-by: Neil Horman Signed-off-by: Richard Purdie (cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bitbake: Add copyright headers where missing

2022-10-11T20:57:28+00:00

Where copyright headers were not present, add them to make things clear. (Bitbake rev: e591325b2bd901c381003deb96a7b32a7148e93e) Signed-off-by: Richard Purdie (cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch2: Ensure directory exists before creating symlink

2022-09-16T16:53:22+00:00

If the mirrors code is trying to create a symlink and the parent directory doesn't exist, as might be the case for sstate mirrors where the fetch is into a subdir, it can silently fail. Ensure the directory exists in this case to avoid issues. (Bitbake rev: ff3afb1c1bb236c4a52c62a74f2917071e0af55b) Signed-off-by: Richard Purdie (cherry picked from commit eff16e474ee7dc49ae433420a4c8d15d3314a618) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie