linux/poky.git/bitbake/lib/bb/runqueue.py, branch dunfell-23.0.23

bitbake: runqueue: Change pressure file warning to a note

2022-08-24T14:43:28+00:00

The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: 3b719e8e115b7fde869f62ddc180e045c1b51cdf) Signed-off-by: Richard Purdie

bitbake: bitbake: runqueue: add memory pressure regulation

2022-08-23T14:57:11+00:00

Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) (Bitbake rev: 82b683f8c7a559f4fcab68f6a0fa7dc3dc20fa05) Signed-off-by: Aryaman Gupta Signed-off-by: Randy Macleod Signed-off-by: Richard Purdie

bitbake: bitbake: runqueue: add cpu/io pressure regulation

2022-08-23T14:57:11+00:00

Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. >From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) (Bitbake rev: 66741d216e9d4343e82a94f00cd39751632a5b96) Signed-off-by: Aryaman Gupta Signed-off-by: Randy Macleod Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: build/msg: Cleanup verbose option handling

2021-10-02T08:55:51+00:00

The levels of indirection to set these verbose logging options is rather crazy. This attempts to turn things into two specific options with much more specific meanings. For now its all still controlled by the commandline verbose option and should funciton as previously, with the addition that the BB_VERBOSE_LOGS option can now be task specific. (Bitbake rev: b4117231bf070703b9375af4411bcd160e07fdae) Signed-off-by: Richard Purdie (cherry picked from commit 423c046f2173aaff3072dc3d0882d01b8a0b0212) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Fix deferred task issues

2021-05-13T12:12:00+00:00

In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: fa068b5a3430b1b580cacfaf9011cdc3324d5844) Signed-off-by: Richard Purdie (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Add setscene task overlap sanity check

2021-03-24T14:36:21+00:00

We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. (Bitbake rev: da306cd4430a6fcf6b62ed0e6e5310432aa4c4c4) Signed-off-by: Richard Purdie (cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Fix task execution corruption issue

2021-03-24T14:36:21+00:00

We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. (Bitbake rev: 8a13955d8da86b976bb136ba1e724a02f7953bca) Signed-off-by: Richard Purdie (cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Avoid unpickle errors in rare cases

2020-07-02T15:11:40+00:00

In rare cases the pickled data from a task contains "" which causes backtrace. This can be reproduced with something like: do_unpack_prepend () { bb.warn("") } There are several solutions but the easiest is to catch this exception and look for the next marker instead as this should be the only way such an unpickle error could occur. This fixes rare exceptions seen on the autobuilder. Also add in other potential exceptions listed in the pickle manual page so that better debug is obtained should there be an error in this code path in future. exitcode doesn't need the same handling since we control what is in that data field and it could never contain (Bitbake rev: 6d780fe3a111adbf3f3d2dda22d5a0787b195b62) Signed-off-by: Richard Purdie (cherry picked from commit 5ada512d6f9cbbdf1172ff7818117c38b12225ca) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Lower setscene complete logging level

2020-03-13T11:29:23+00:00

Lowers the level of the log message when setscene tasks have completed. This message can occur multiple times when hash equivalence is enabled, since the runqueue switches between executing setscene tasks and normal tasks. Since this is primarily of use when debugging hash equivalence, use the hash equivalence logger at VERBOSE level. [YOCTO #13813] (Bitbake rev: 7dd5b3900622008ff34ec70d71c6e994f460a46f) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue/siggen: Lower hash equivalence logging

2020-03-13T11:29:23+00:00

Lowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json) [YOCTO #13813] (Bitbake rev: 2ddb649ea31afe052f08e3969e36abf6fb515bc2) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie