linux/poky.git/bitbake/lib/bb/runqueue.py, branch walnascar

bitbake: bitbake: runqueue: Verify mcdepends are valid

2025-03-03T21:38:57+00:00

In order to avoid a potentially confusing backtrace, check that the mcdepend is valid when we add it. Add a test case to ensure invalid configurations are caught and trigger an error. [RP: Reworked test case to simplify and improve code] (Bitbake rev: ff523497270f37b484b44a4445c2194791bcb6ff) Signed-off-by: Mark Hatle Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie

bitbake: lib: Remove double imports

2025-02-27T15:12:55+00:00

* Remove double imports mentioned by pylint (Bitbake rev: 741db6719efca5aa9ef2c15e60cdd624e4aa1a8d) Signed-off-by: Michael Estner Signed-off-by: Richard Purdie

bitbake: runqueue: Fix scenetask processing performance issue

2024-11-29T22:13:25+00:00

Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. (Bitbake rev: 923c19b9713e398d8e66e6d4422dfd4c18a03486) Signed-off-by: Richard Purdie

bitbake: runqueue: Optimise setscene loop processing

2024-11-29T22:13:25+00:00

Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. (Bitbake rev: 00f4d932e3af0eeb333339cbe942010fc76dee0f) Signed-off-by: Richard Purdie

bitbake: runqueue: Fix performance of multiconfigs with large overlap

2024-11-29T22:13:25+00:00

There have been complaints about the performance of large multiconfig builds for a while. The key missing data point was that the builds needed to have large overlaps in sstate objects. This can be simulated by building the same things with just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of deferred tasks. The issue is that the expensive checks in the setscene loop were hit every time through runqueue's execute function before the check on deferred tasks. This leads to task execution starvation as that only happens once per iteration. Move the skip check earlier in the function which speeds things up enormously and should improve performance of such builds for users. (Bitbake rev: 9c6c506757f2b3e28c8b20513b45da6b4659c95f) Signed-off-by: Richard Purdie

bitbake: bitbake-worker/cooker: Increase default pipe size

2024-11-21T12:16:28+00:00

The default pipe size is 64kb on builds, which can be inefficient for larger log files from workers. Increase the pipe size to 512kb since build systems have decent amounts of memory and this is a more efficient way of batching the data. Tweak the default read sizes to match the pipe size for efficiency. Since the contstant is only present in python 3.10 onwards, add some compatibility code. (Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7) Signed-off-by: Richard Purdie

bitbake: runqueue: Avoid dumpsigs idle loop blocking

2024-11-07T13:36:04+00:00

We're seeing some failures on hosts where slow "idle" loop iterations are causing bitbake server timeouts. These seem to happen particularly in the dump_signatures() function within runqueue. That isn't entirely surprising since it creates a pool of threads to execute work an at best can take around 10s to execture and return control backto the main loop. On a slow system, it is understandable this can take longer, particularly as these functions are creating large chunks of IO. Since the work is being done in threads, we can launch them, return to idle and check on the results periodically as they complete. This should hopefully address some of the remaining timeout issues we see on the autobuilder in oe-selftest sstate tests. (Bitbake rev: e66f1b643b4b77404ba31f2704cda5af9bf00a57) Signed-off-by: Richard Purdie

bitbake: runqueue: Avoid save_unitaskhashes

2024-06-07T12:45:43+00:00

The save comes with an IO overhead which can slow down the rehash loop in bitbake a lot. We only needed to do this when recipes were doing unihash cache copying. Now they aren't doing that, drop this IO pain point. (Bitbake rev: dfc15ef99302dea22a051c9eb8398ffd5cf1fc20) Signed-off-by: Richard Purdie

bitbake: bb: Use namedtuple for Task data

2024-05-28T22:46:21+00:00

Task dependency data is becoming unwieldy with the number of indices it contains. Convert it to use a named tuple instead, which allows members to be indexed by a named property or an index (which allows it to retain backward compatibility). (Bitbake rev: 26446cca4d22734c3f1b328a205c169dadb7e494) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue: Improve rehash get_unihash parallelism

2024-05-28T22:46:21+00:00

Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. (Bitbake rev: c1949d5350342eaaf6ab988d7bfba99496d55523) Signed-off-by: Richard Purdie