linux/poky.git/bitbake, branch yocto-2.6.2

bitbake: gitsm: The fetcher did not process some recursive submodules properly.

2019-03-24T16:42:22+00:00

Move the submodule processing outside of the if statement to avoid any optimizations that may happen. Update the test cases to include the additional case, and split the other test cases into individual tests to make it easier to figure out what the failure may be. (Bitbake rev: 7c1eb51d1e8a4c5f39bf9dddf05fb0b3598da72b) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 0ec98c01ae50f95c9c74acf53013ac59e0e72b08) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsmy.py: Fix unpack of submodules of submodules

2019-03-24T16:42:22+00:00

If the submodule is in a subdirectory, it needs to have that structure preserved. This means the unpack path needs to be in the 'dirname' of the final path -- since the unpack directory name is specified in the URI. Additional specific test cases were added to ensure this is working properly based on two recent error reports. (Bitbake rev: acca06d060e49b2441562b4dc94416af9ab8187e) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 8c8ecec2a722bc2885e2648d41ac8df07bdf660d) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: Fix relative URLs

2019-03-24T16:42:22+00:00

Prior code happened to work with relative URLs, when the code was recently restructured it caused all relative urls to no longer work. Restore the prior code flow for relative support and better comment why that code is there. (Bitbake rev: 39afa6ead1d72813b2a294f065d759bad08fb53d) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 14bb383af0ca98e0e04ec217e537b6a899f3658e) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: Refactor the functions and simplify the class

2019-03-24T16:42:22+00:00

The update_submodules and unpack_submodules functions were nearly indentical, so we made a common function where the different behavior could be passed in by the download and unpack users. The new function is process_submodules. Moved the parse_gitmodules function under the new process_submodules, since there are no external callers. Refactor the file relative path processing to the URL translation code. We also add a warning to the translation if a relative ssh URL has been detected. Since this can cause a problem. In the case of a relative URL that does not work after being translated, it should be possible to use the MIRROR functions to manual translate the generated relative URL into one that works properly. Remove 'git config' processing on download contents. It turns out this is not necessary since all of the later components work using the git fetcher. Limit the 'git submodule update' call to only when unpacking a non-bare repository. Submodules are always loaded as bare, so this prevents intermediate unpacks from being attempted. Finally, the test cases were updated and the new commit ids in the test repository were updates as well. (Bitbake rev: 8b02eb8e5d8e3a09e19ed96a4ccdf5f755e97a74) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 610dbee5634677f5055e2b36a3043cd197fb8c51) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: Rework the shallow fetcher and test case

2019-03-24T16:42:22+00:00

A custom shallow submodule is no longer necessary, as the regular git fetcher is used and shallow handling works with the same code. The only general difference between the regular change is simply declaring a clone as shallow, when appropriate. This also removes a potential race condition in copying repositories vs cloning them. The gitsm shallow fetcher test was revised to verify that the submodule is shallow cloned along with the primary repository. The first step of this change was to be sure to clean the gitsubmodule download directory, as was previously done with the may gitsource directory. Additional test components were added to verify commit counts, and an obsolete (and likely incorrect) test for the .git/modules directory to be empty was also removed. (Bitbake rev: 85dc1c65b661f9712ae98587d4d0d868146c8cff) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit f9cc4684dcf4281acc557cda8cb35602354ac3d6) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: revise unpack

2019-03-24T16:42:22+00:00

Greatly simply the unpack rule by copying the general functionality of update_submodules as unpack_submodules. This will recursively construct a set of urls and unpack them using the standard system behaviors. The overall code may be slightly bigger, but this ensures that all of the standard locks are inplace, ensuring the code doesn't change out from under the unpack function. (This could have happened before due to using 'cp' instead of further unpacks on submodules. This may still happen in shallow clones.) (Bitbake rev: 02ce6783bd3cfc117c77ca0380a87b0e1c60d8db) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: Optimize code and attempt to resolve locking issue

2019-03-24T16:42:22+00:00

It was reported that a race condition on a shared download directory could occur with the gitsm fetcher, the result happened with a call to git config that occured within the update_submodules. Since the fetch is locked by the upper level, it was probably the prior need_update(...) function causing this because of some old code. The gitsm class inherits the git class. The need_update was overridding the version in gitsm, so that it forceably checked the submodules. It's clear we can optimize the code by only updating if the primary repository needs updating. Since we don't care if the submodule repository has changed because if the primary hasn't, references to the submodule won't change. (Bitbake rev: 4660933f83e528766d71eab662cc79dcf17b4be7) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 346338667edca1f58ace769ad417548da2b8d981) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: tests/fetch.py: Add alternative gitsm test case

2019-03-24T16:42:22+00:00

In order to test the ssh processing in gitsm, we add an alternative testcase that can be downloaded from git.yoctoproject.org. However, this test case requries (read) access, via ssh, to git.yoctoproject.org. (Bitbake rev: 54802438f2f2d73aba3e7fb8d6cce45aa7cffe77) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit c8554cdc1287620fe8e8960561e614567879a010) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: Add support for alternative URL formats from submodule files

2019-03-24T16:42:22+00:00

The following appear to be the git supported formats: proto://user:pass@host/path (URI format) user@host:path (SSH format) /path or ./path or ../path (local file format) We adjust the parsing to find out if we have a URI format or not. When we are NOT in URI format, we do our best to determine SSH or file format by looking for a ':' in the overall string. If we find a ':' we assume SSH format and adjust accordingly. Note, in SSH format we simply replace the ':' with a '/' when constructing the URL. However, if the original path was ":/...", we don't want '//' so we deal with this corner case as well. (Bitbake rev: a21b2598531d52123933a0ac1c4deeecfdd1697d) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: gitsm.py: Fix when a submodule is defined, but not initialized

2019-03-24T16:42:22+00:00

It is possible for a submodule to be defined in the .gitmodules file, but never initialized in the repository itself. This shows itself when searching for the defined module hash you will get back a empty value. Similarly we need to identify and skip defined but not initialized submodules during the unpack stages as well. Thanks to raphael.lisicki@siemens.com for their help is figuring out how to resolve this issue. Additionally a problem was found where, while unlikely, it may be possible for the wrong revision to have been searched using ls-tree. This has been resolved in the update_submodules function by keeping the correct revision along with the submodule path. (Bitbake rev: e9c965bd816c8e4e5cc8e45db8e0dab597ce5a6f) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie (cherry picked from commit 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3) Signed-off-by: Scott Murray Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie