linux/poky.git/bitbake/lib/bb/runqueue.py, branch yocto-5.1.1

bitbake: runqueue: Avoid save_unitaskhashes

2024-06-07T12:45:43+00:00

The save comes with an IO overhead which can slow down the rehash loop in bitbake a lot. We only needed to do this when recipes were doing unihash cache copying. Now they aren't doing that, drop this IO pain point. (Bitbake rev: dfc15ef99302dea22a051c9eb8398ffd5cf1fc20) Signed-off-by: Richard Purdie

bitbake: bb: Use namedtuple for Task data

2024-05-28T22:46:21+00:00

Task dependency data is becoming unwieldy with the number of indices it contains. Convert it to use a named tuple instead, which allows members to be indexed by a named property or an index (which allows it to retain backward compatibility). (Bitbake rev: 26446cca4d22734c3f1b328a205c169dadb7e494) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue: Improve rehash get_unihash parallelism

2024-05-28T22:46:21+00:00

Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. (Bitbake rev: c1949d5350342eaaf6ab988d7bfba99496d55523) Signed-off-by: Richard Purdie

bitbake: runqueue: Process unihashes in parallel at init

2024-05-28T22:46:21+00:00

Improve the runqueue init code to call unihash queries in parallel since this is faster and more efficient, particularly on slower links with longer round trip times. The call to the function from cooker is unneeded since that function calls prepare() and hence this functionality will already have run, so drop that obsolete call. (Bitbake rev: 721c97a115a7a4bf21955be79391bd6e0099f40e) Signed-off-by: Richard Purdie

bitbake: runqueue: Allow rehash loop to exit in case of interrupts

2024-05-28T22:46:21+00:00

The initial hash serve loop exits in the case where interrupts are present but probably checks a bit too often. Tweak that and also allow the slow rehash loop to break on interrupt, improving bitbake Ctrl+C response. (Bitbake rev: 4534365591fd17bcc2b684900863b67bc69519ae) Signed-off-by: Richard Purdie

bitbake: runqueue: Add timing warnings around slow loops

2024-05-28T22:46:21+00:00

With hashserve enabled, there are two slow paths/loops, one at initial runqueue generation and also during the rehash process when new outhashes are found. Add timing information at the hashserve log level for when these loops take longer than 30s or 60s overall. This will leave evidence in the logs when things are running particularly slowly. (Bitbake rev: 6c357ede08e0b2a93bdaad2c1d631994faf2b784) Signed-off-by: Richard Purdie

bitbake: siggen/runqueue: Report which dependencies affect the taskhash

2024-05-08T13:51:43+00:00

Report which task dependencies in BB_TASKDEPDATA are included in the taskhash. This allows tasks to identify which tasks dependencies may change without the task re-running. Knowing this information is important for tasks that want to transfer information from dependencies (such as SPDX) (Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake-worker: allow '=' in environment variable values

2024-03-22T16:26:45+00:00

Limit the split to key & value (2 items) instead of the n items one can get if there are '=' characters in the value. Fixes [YOCTO #15447] (Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb) Signed-off-by: david d zuhn Signed-off-by: Richard Purdie

bitbake: runqueue: Add support for BB_LOADFACTOR_MAX

2024-02-23T14:34:05+00:00

Some ditros don't enable /proc/pressure and it tends to be those which we see bitbake timeout issues on, seemingly as load gets too high and the bitbake processes don't get scheduled in for minutes at a time. Add support for stopping running extra tasks if the system load average goes above a certain threshold by setting BB_LOADFACTOR_MAX. The value used is scaled by CPU number, so a value of 1 would be when the load average equals the number of cpu cores of the system, under one only starts tasks when the load average is below the number of cores. This means you can centrally set a value such as 1.5 which will then scale correctly to different sized machines with differing numbers of CPUs. The pressure regulation is probably more accurate and responsive, however our graphs do show singificant load spikes on some workers and this patch is aimed at trying to avoid those. Pressure regulation is used where available in preference to this load factor regulation when both are set. (Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49) Signed-off-by: Richard Purdie

bitbake: runqueue: Improve setcene performance when encoutering many 'hard' dependencies

2024-02-13T13:52:10+00:00

"bitbake world -n --setscene-only" shows poor performance as the numbers of tasks increases. Analysys shows this is due to the "deferred" hard dependencies being repeatedly processed which doesn't allow much forward progress in overall task execution. To avoid this, mark when it has been done and don't reprocess until dependencies are updated. We have to be careful as we've seen bugs where these dependencies aren't processed at the right time. They also have to be reproceseed during task migration/rehashing so the code has to "self heal" the data structures. (Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f) Signed-off-by: Richard Purdie