linux/poky.git/bitbake, branch yocto-3.1.20

bitbake: utils: Pass lock argument in fileslocked

2022-09-02T10:45:35+00:00

Pass additional arguments in the fileslocked() context manager to the underlying lockfile() function. This allows the context manager to be used for any types of locks (non-blocking, shared, etc.) that the lockfile() function supports. (Bitbake rev: 048d682b031644fb9f0d41a489bacb873aa27bd7) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue: Change pressure file warning to a note

2022-08-24T14:43:28+00:00

The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: 3b719e8e115b7fde869f62ddc180e045c1b51cdf) Signed-off-by: Richard Purdie

bitbake: bitbake: runqueue: add memory pressure regulation

2022-08-23T14:57:11+00:00

Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) (Bitbake rev: 82b683f8c7a559f4fcab68f6a0fa7dc3dc20fa05) Signed-off-by: Aryaman Gupta Signed-off-by: Randy Macleod Signed-off-by: Richard Purdie

bitbake: bitbake: runqueue: add cpu/io pressure regulation

2022-08-23T14:57:11+00:00

Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. >From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) (Bitbake rev: 66741d216e9d4343e82a94f00cd39751632a5b96) Signed-off-by: Aryaman Gupta Signed-off-by: Randy Macleod Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2/wget: Update user-agent

2022-08-22T14:42:30+00:00

With the usage of enterprise proxy, the user-agent defined are too old and refused by proxy configuration. Updating to something more modern is desirable. (Bitbake rev: 17be38290d1e971cd89785e6bf44caef0a6416f8) Signed-off-by: Christophe Priouzeau Signed-off-by: Ross Burton Signed-off-by: Richard Purdie (cherry picked from commit 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch/wget: Move files into place atomically

2022-07-15T11:41:59+00:00

(Bitbake rev: 7fc4cffebf5dcc1d050416c0b7f7d58c765c1d69) Signed-off-by: Richard Purdie (cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch/git: Fix usehead for non-default names

2022-07-15T11:41:59+00:00

The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: a247f56df680382d62910bb9a174e0fdd29e4ca8) Signed-off-by: Joey Degges Signed-off-by: Richard Purdie (cherry picked from commit 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Paulo Neves Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: knotty: reduce keep-alive timeout from 5000s (83 minutes) to 10 minutes

2022-06-26T11:14:23+00:00

The keep alive timeout is excessively long at 83 minutes (5000 seconds), reduce this to 10 minutes: this should be long enough that it rarely triggers in normal builds, but when it does it has useful information. (Bitbake rev: a496a8952d8542ce814b13f460811d8849d25a3c) Signed-off-by: Ross Burton Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie (cherry picked from commit dcf52157d3635925491783be656c6b76d1efe1a4) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: knotty: display active tasks when printing keepAlive() message

2022-06-26T11:14:22+00:00

In interactive bitbake sessions it is obvious what tasks are running when one of them hangs or otherwise takes a long time. However, in non-interactive sessions (such as automated builds) bitbake just prints a message saying that it is "still alive" with no clues as to what tasks are active still. By simply listing the active tasks when printing the keep alive message, we don't need to parse the bitbake log to identify which of the tasks is still active and has presumably hung. (Bitbake rev: 36fe8bae9fec61547ee0b13bcb721033afd3ac0e) Signed-off-by: Ross Burton Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie (cherry picked from commit 30f6c3f175617beea8e8bb75dcf255611e3fc2fd) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bin/bitbake-getvar: Add a new command to query a variable value (with history)

2022-06-26T11:14:22+00:00

We've talked about having this for long enough. Add a command which queries a single variable value with history. This saves "bitbake -e | grep" and avoids the various pitfalls that has. It also provides a neat example of using tinfoil to make such a query. Parameters to limit the output to just the value, to limit to a variable flag and to not expand the output are provided. [YOCTO #10748] (Bitbake rev: 47ed06d441152f8b6d374cacfac2c668c354423e) Signed-off-by: Richard Purdie (cherry picked from commit 4c1881b620e885f55d7772f8626b8a76c2828333) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie