linux/poky.git/bitbake/lib/bb/fetch2/gitsm.py, branch honister-3.4.2

bitbake: fetch2/gitsm: remove the 'nugget' SRCREV caching

2021-10-08T15:45:05+00:00

The cached revisions which are used to decide if a repository doesn't need to be updated are misleading when used in conjunction with mirror tarballs and can cause partial fetches to happen, resulting in unpack errors as repositories were not fetched. A concrete example: edk2-firmware in meta-arm is at version 202102 (ef91b0). This is built on the autobuilder so the source mirror contains the repository as a mirror tarball. If I build edk2-firmware 202102 the gitsm fetcher will initially download the top-level repository and then iterate into the submodules to also fetch those repositories, including cmocka from cryptomilk.org. edk2-firmware will then unpack and build successfully. I then update edk2-firmware to 202105 (e1999b) and build it. Gitsm.needs_update() starts by calling Git.needs_update() which returns False, as the mirror tarball contains this revision. It then looks at the "nuggets" which are SRCREVs it has fetched before. The mirror tarball itself contains the nugget for e1999b as this has been built on the autobuilder, so needs_update return False, no more fetching is done, and the build proceeds to unpack. However, as part of the 202105 upgrade the URL of the cmocka submodule changed, and this new repository was never fetched. This means that unpack fails as one of the required git repositories isn't available. The nugget codepaths appear to be an attempt at optimising the fetch process, but have demonstratable failure cases. Just removing them entirely solves the edk2-firmware example, and all of the fetcher test cases still pass. (Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: logging: Make bitbake logger compatible with python logger

2021-02-10T23:48:16+00:00

The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: fetch2/gitsm: Make need_update() process submodules

2020-06-15T13:55:25+00:00

If the bitbake.srcrev nugget is not present for the commit we're interested in we should not just bail out and say that an update is needed. Instead we can recursively walk through the submodules and check for the presence of the required commits. (Bitbake rev: cfc78316309556bec487ef0a5a9205e41f1be86f) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: fetch2/gitsm: Mark srcrev as fetched once all submodules are processed

2020-06-15T13:55:25+00:00

This prevents multiple bitbake.srcrev entries being created when a repository has more than one submodule. It also ensures that the bitbake.srcrev entry is not added unless all submodules are correctly fetched. (Bitbake rev: 860ff1193fe53f04696d41635a720c2d1f29fa7f) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: fetch2: Add the ability to list expanded URL data

2020-06-04T12:28:16+00:00

Some fetchers may download additional sources along with those explicitly listed in SRC_URI. These "implicit URLs" will be needed by the archiver to ensure that all sources can be archived. We can't just return a list of URL strings since each URL may need its own SRCREV data so we return a list of FetchData objects. Each fetcher can override the implicit_urldata() function to provide the additional FetchData objects. For now this is just needed in the gitsm fetcher to walk git submodules recursively. (Bitbake rev: 1350f241b7d991bd191ce9e44f6662e4376c6e24) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: fetch2/gitsm: Unpack shallow mirror tarballs

2020-03-11T14:43:25+00:00

When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked temporarily so that the .gitmodules file can be examined. (Bitbake rev: 3987db953e414255ce278bc25a5f6cec0f2a30c7) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: lib: remove unused imports

2020-01-19T13:31:05+00:00

removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews Signed-off-by: Richard Purdie

bitbake: gitsm: Fix a bug where the wrong path was used for the submodule init

2019-05-16T08:17:15+00:00

Because we are trying to avoid network activity and use our own fetcher, the system emulates the behavior of 'git submodule init'. git submodule init uses the .gitmodules file, where typically the module name and path are the same. However, in this case the module name and path (in the tree) were different. i.e.: [submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"] path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared url = https://github.com/Azure/azure-c-shared-utility.git Previously the code assumed the 'path' was both the checkout location under .git/modules, as well as the path to extract the components. This proved to be incorrect as the .git/modules path needs to match the submodule 'name'. This causes the components that were fetched to be initialized in the wrong location, which later caused the 'git submodule update' process to skip not properly initialized modules. A test case was added for this specific case to ensure a regression does not appear in the future. (Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie

bitbake: bitbake: Strip old editor directives from file headers

2019-05-04T09:44:10+00:00

There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie

bitbake: bitbake: Drop duplicate license boilerplace text

2019-05-04T09:44:10+00:00

With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie