linux/poky.git/bitbake, branch yocto-3.1.21

bitbake: utils: Fix lockfile path length issues

2022-11-22T23:26:15+00:00

If the path to bitbake.lock is in a deep directory, bitbake will hang. The reason was that the max file length limiting code (to 255 chars) was including the directory name and it should only act on the filename within the directory. Fix it to just use the base filename. [YOCTO #14766] (Bitbake rev: e3db9c2e9eded3c5cb6040714a6054b44f6b3880) Signed-off-by: Richard Purdie (cherry picked from commit 89d70e7b71eecfe06592202f326e566c579ba01d) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: utils: Handle lockfile filenames that are too long for filesystems

2022-11-22T23:26:15+00:00

The fetcher mirror code can go crazy creating lock filenames which exceed the filesystem limits. When this happens, the code will loop/hang. Handle the filename too long exception correctly but also truncate lockfile lengths to under 256 since the worst case situation is lockfile overlap and lack of parallelism. (Bitbake rev: 30d42ef030d03e11322b6b05ea7bbb64ab3d6f21) Signed-off-by: Richard Purdie (cherry picked from commit 63baf3440b16e41ac6601de21ced94a94bdf1509) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: tests/fetch: Allow handling of a file:// url within a submodule

2022-10-27T15:39:14+00:00

CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 8ea8e443005ad92f4ad264d9abd9e90e33fb5c17) Signed-off-by: Richard Purdie

bitbake: utils: Pass lock argument in fileslocked

2022-09-02T10:45:35+00:00

Pass additional arguments in the fileslocked() context manager to the underlying lockfile() function. This allows the context manager to be used for any types of locks (non-blocking, shared, etc.) that the lockfile() function supports. (Bitbake rev: 048d682b031644fb9f0d41a489bacb873aa27bd7) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue: Change pressure file warning to a note

2022-08-24T14:43:28+00:00

The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: 3b719e8e115b7fde869f62ddc180e045c1b51cdf) Signed-off-by: Richard Purdie

bitbake: bitbake: runqueue: add memory pressure regulation

2022-08-23T14:57:11+00:00

Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) (Bitbake rev: 82b683f8c7a559f4fcab68f6a0fa7dc3dc20fa05) Signed-off-by: Aryaman Gupta Signed-off-by: Randy Macleod Signed-off-by: Richard Purdie

bitbake: bitbake: runqueue: add cpu/io pressure regulation

2022-08-23T14:57:11+00:00

Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. >From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) (Bitbake rev: 66741d216e9d4343e82a94f00cd39751632a5b96) Signed-off-by: Aryaman Gupta Signed-off-by: Randy Macleod Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2/wget: Update user-agent

2022-08-22T14:42:30+00:00

With the usage of enterprise proxy, the user-agent defined are too old and refused by proxy configuration. Updating to something more modern is desirable. (Bitbake rev: 17be38290d1e971cd89785e6bf44caef0a6416f8) Signed-off-by: Christophe Priouzeau Signed-off-by: Ross Burton Signed-off-by: Richard Purdie (cherry picked from commit 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch/wget: Move files into place atomically

2022-07-15T11:41:59+00:00

(Bitbake rev: 7fc4cffebf5dcc1d050416c0b7f7d58c765c1d69) Signed-off-by: Richard Purdie (cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch/git: Fix usehead for non-default names

2022-07-15T11:41:59+00:00

The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: a247f56df680382d62910bb9a174e0fdd29e4ca8) Signed-off-by: Joey Degges Signed-off-by: Richard Purdie (cherry picked from commit 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Paulo Neves Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie