linux/poky.git/bitbake/lib/bb/fetch2, branch thud

bitbake: fetch/git: Handle github dropping git:// support

2021-11-03T11:31:58+00:00

github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. (Bitbake rev: 964958b8b11dc69fb289fc6c97c1dbc8d76ad0f8) Signed-off-by: Richard Purdie

bitbake: fetch2: Fix os.errno references

2021-01-31T11:28:39+00:00

os.errno used to happen to work but is invalid. Correct to use errno. [YOCTO #13068] (Bitbake rev: b3fc65289d33274cd5dace4d4ffe55be11c991f4) Signed-off-by: Richard Purdie

bitbake: fetch2: Change git fetcher not to destroy old references

2020-09-08T13:46:13+00:00

It looks like we're about to see a lot of changes in branch names in repos. If we have the prune option here, those old names are lost, the changes propagate to our source mirrors and our old releases break. We have the force option so any replaced references should be replaced, its only orphaned branches which will now be preserved. I believe this behaviour will cause us fewer problems given the changes that look likely to happen. (Bitbake rev: 12d8cc3fecd550c4aadf0519e80711d755ee75ba) Signed-off-by: Richard Purdie

bitbake: fetch2: Ensure cached url data is matched to a datastore

2019-11-13T22:00:33+00:00

There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 97067634b1f149b56844b10e3a5e8d0d980b6e34) Signed-off-by: Richard Purdie

bitbake: fetch2: show warning when renaming the archive with bad checksum failed

2019-09-06T10:58:56+00:00

* noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: d36438759344caa447d9a0bf30749a0aa31d1fba) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: gitsm: Add need_update method to determine when we are going to a new SRCREV

2019-06-18T10:31:10+00:00

If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly. The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch. To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present. This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work. This issue was discovered by William A. Kennington III . The included test case was also written by him, and included unmodified. (Bitbake rev: 4ce92f43eeac6a4bfd06e8567fa6891614b5b3b0) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie

bitbake: gitsm: Fix a bug where the wrong path was used for the submodule init

2019-06-18T10:31:10+00:00

Because we are trying to avoid network activity and use our own fetcher, the system emulates the behavior of 'git submodule init'. git submodule init uses the .gitmodules file, where typically the module name and path are the same. However, in this case the module name and path (in the tree) were different. i.e.: [submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"] path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared url = https://github.com/Azure/azure-c-shared-utility.git Previously the code assumed the 'path' was both the checkout location under .git/modules, as well as the path to extract the components. This proved to be incorrect as the .git/modules path needs to match the submodule 'name'. This causes the components that were fetched to be initialized in the wrong location, which later caused the 'git submodule update' process to skip not properly initialized modules. A test case was added for this specific case to ensure a regression does not appear in the future. (Bitbake rev: ffd7ed530a17d22df576d986ac78428a6979e79c) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie

bitbake: gitsm: The fetcher did not process some recursive submodules properly.

2019-03-24T16:42:22+00:00

Move the submodule processing outside of the if statement to avoid any optimizations that may happen. Update the test cases to include the additional case, and split the other test cases into individual tests to make it easier to figure out what the failure may be. (Bitbake rev: 7c1eb51d1e8a4c5f39bf9dddf05fb0b3598da72b) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 0ec98c01ae50f95c9c74acf53013ac59e0e72b08) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsmy.py: Fix unpack of submodules of submodules

2019-03-24T16:42:22+00:00

If the submodule is in a subdirectory, it needs to have that structure preserved. This means the unpack path needs to be in the 'dirname' of the final path -- since the unpack directory name is specified in the URI. Additional specific test cases were added to ensure this is working properly based on two recent error reports. (Bitbake rev: acca06d060e49b2441562b4dc94416af9ab8187e) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 8c8ecec2a722bc2885e2648d41ac8df07bdf660d) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: Fix relative URLs

2019-03-24T16:42:22+00:00

Prior code happened to work with relative URLs, when the code was recently restructured it caused all relative urls to no longer work. Restore the prior code flow for relative support and better comment why that code is there. (Bitbake rev: 39afa6ead1d72813b2a294f065d759bad08fb53d) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 14bb383af0ca98e0e04ec217e537b6a899f3658e) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie