linux/poky.git/bitbake/lib/bb, branch yocto-3.1.23

bitbake: fetch2/git: Clarify the meaning of namespace

2023-02-13T07:48:16+00:00

Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: c527976e28165de9606497c48d834872fe68372e) Signed-off-by: Marek Vasut Signed-off-by: Richard Purdie (cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

2023-02-13T07:48:16+00:00

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt (Bitbake rev: efb2903e6c94a5c884485ecb91f1fca7e8ee18f1) Signed-off-by: Marek Vasut Signed-off-by: Alexandre Belloni (cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url

2023-02-13T07:48:16+00:00

This commit replaces the instances where escaped double quotes are used to support SRC_URI url containing spaces with the more pythonic shlex.quote(). (Bitbake rev: ecc1dac4ad8c8593810c69a25d674b0e0bed6097) Signed-off-by: Charlie Davies Signed-off-by: Richard Purdie (cherry picked from commit 4f9ba9c794de55bea0343267467bddea99844374) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bb/utils: include SSL certificate paths in export_proxies

2023-02-06T23:42:55+00:00

bb.utils.export_proxies() is a poor-man's alternative for the environment setup code in bb/fetch2, but it's used in several places where recipes want to download manually (such as cve-update-db-native). Notably, export_proxies() doesn't pass on the SSL certificate paths from the original environment, so if SSL_CERT_FILE needs to be set (for example, in a buildtools environment) then proxies work but SSL doesn't. In an ideal world export_proxies and the same logic in fetch2 would merge, but until then we can add the SSL_CERT_ variables and duplicate the basic logic: check the datastore first and then the original environment for variables. [ YOCTO #15000 ] (Bitbake rev: c16d364dbf68d2a500fecaf8d6e6d62b11475d9f) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie (cherry picked from commit c19035e8e71c419c5688a86bfc9c946c96f638e8) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: utils: Fix lockfile path length issues

2022-11-22T23:26:15+00:00

If the path to bitbake.lock is in a deep directory, bitbake will hang. The reason was that the max file length limiting code (to 255 chars) was including the directory name and it should only act on the filename within the directory. Fix it to just use the base filename. [YOCTO #14766] (Bitbake rev: e3db9c2e9eded3c5cb6040714a6054b44f6b3880) Signed-off-by: Richard Purdie (cherry picked from commit 89d70e7b71eecfe06592202f326e566c579ba01d) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: utils: Handle lockfile filenames that are too long for filesystems

2022-11-22T23:26:15+00:00

The fetcher mirror code can go crazy creating lock filenames which exceed the filesystem limits. When this happens, the code will loop/hang. Handle the filename too long exception correctly but also truncate lockfile lengths to under 256 since the worst case situation is lockfile overlap and lack of parallelism. (Bitbake rev: 30d42ef030d03e11322b6b05ea7bbb64ab3d6f21) Signed-off-by: Richard Purdie (cherry picked from commit 63baf3440b16e41ac6601de21ced94a94bdf1509) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: tests/fetch: Allow handling of a file:// url within a submodule

2022-10-27T15:39:14+00:00

CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 8ea8e443005ad92f4ad264d9abd9e90e33fb5c17) Signed-off-by: Richard Purdie

bitbake: utils: Pass lock argument in fileslocked

2022-09-02T10:45:35+00:00

Pass additional arguments in the fileslocked() context manager to the underlying lockfile() function. This allows the context manager to be used for any types of locks (non-blocking, shared, etc.) that the lockfile() function supports. (Bitbake rev: 048d682b031644fb9f0d41a489bacb873aa27bd7) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue: Change pressure file warning to a note

2022-08-24T14:43:28+00:00

The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: 3b719e8e115b7fde869f62ddc180e045c1b51cdf) Signed-off-by: Richard Purdie

bitbake: bitbake: runqueue: add memory pressure regulation

2022-08-23T14:57:11+00:00

Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) (Bitbake rev: 82b683f8c7a559f4fcab68f6a0fa7dc3dc20fa05) Signed-off-by: Aryaman Gupta Signed-off-by: Randy Macleod Signed-off-by: Richard Purdie