linux/poky.git/bitbake/lib/bb/runqueue.py, branch kirkstone-4.0.22

bitbake: runqueue: show number of currently running bitbake threads when pressure changes

2023-12-11T16:13:49+00:00

* it might be a bit confusing as it shows number of threads before making the decision to start more tasks and also it can show only a few tasks running, but not because of pressure when there just aren't many tasks left or wait for their dependencies to be finished first * example output: NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads (Bitbake rev: 42a1c9fe698a03feb34c5bba223c6e6e0350925b) Signed-off-by: Martin Jansa Signed-off-by: Steve Sakoman

bitbake: runqueue: show more pressure data

2023-12-11T16:13:49+00:00

* with latest bitbake I'm seeing very low number of bitbake tasks executed in parallel, probably due to pressure regulation show the values this is based on in the note * also simplify a bit by counting the pressure and exceeds signs only once (Bitbake rev: 1050145c3004861ebede4777fd4fbd89d0470716) Signed-off-by: Martin Jansa Signed-off-by: Steve Sakoman

bitbake: runqueue.py: fix PSI check logic

2023-12-11T16:13:49+00:00

The current calculation is not correct because if tdiff is less than 1.0, it's not taken into consideration when calculating the current pressure. Also, make it clear that the 1.0s is the psi accumulation cycle, which might be changed in the future. We have this cycle because it could largely avoid the 0 result issue, that is, if the interval between checks are too small, the result might be 0. With this accumulation logic, which has been there but let's make it clear, this 0 result problem could be mitigated. (Bitbake rev: 76889ff0a8938a3d77603d2af176aa9e264df839) Signed-off-by: Chen Qi Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: runqueue: Add pressure change logging

2023-12-11T16:13:49+00:00

It is currently hard to tell when bitbake is throttling task execution due to system pressure changes. Add notes to the console output to make this clearer, only generating output when the values change. (Bitbake rev: b5f77e8159ad321f31999af8304f082a2c56b537) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: runqueue: fix PSI check calculation

2023-12-11T16:13:49+00:00

The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. (Bitbake rev: 2b2997ab6bdda730e4b638f416311a73e0c42156) Signed-off-by: Chen Qi Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: runqueue: convert deferral messages from bb.note to bb.debug

2023-11-09T14:41:17+00:00

Using multiconfig to target baremetal pieces of the system and building corresponding toolchains for them results in hundreds and hundreds of "Deferring %s after %s" and "Deferred task %s now buildable". To clean up the output and to reduce risk of missing important warnings, convert these notice messages to debug messages. (Bitbake rev: 3505d8d8c02b041946670ab6bc5751e54fe292ff) Signed-off-by: Denys Dmytriyenko Signed-off-by: Denys Dmytriyenko Signed-off-by: Richard Purdie (cherry picked from commit 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318) Signed-off-by: Steve Sakoman

bitbake: bitbake-worker/runqueue: Avoid unnecessary bytes object copies

2023-10-18T15:13:24+00:00

declaring queue=b"" creates an object of types bytes(). bytes() is an immutable object, and therefore doing "self.queue = self.queue + r" creates a new object containing "self.queue" concatenated with "r". On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker, so those copies significantly slow down the initialization of the bitbake-worker. Rather use bytearray() which a mutable type, and use extend() to avoid copies. In my test setup, byterray.extend() is 10.000 times faster than copying the queue, for a queue size of 180MB. (Bitbake rev: 9993a89e5b97dda5f3657e5a7cc3a4fa94ff7111) Signed-off-by: Etienne Cordonnier Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: runqueue: Fix deferred task/multiconfig race issue

2023-07-12T15:11:37+00:00

If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: 907416ee1062f87f5844ab0638b54616abfc1a22) Signed-off-by: Richard Purdie (cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Steve Sakoman

bitbake: runqueue: Fix race issues around hash equivalence and sstate reuse

2022-11-24T15:25:19+00:00

We identified a use case where a native recipe (autoconf-native) was rebuilt with no change in output yet the sstate for do_package tasks wasn't being used. The issue is that do_package tasks have a hard dependency on pseudo-native:do_populate_sysroot. That task was one of the many tasks being rehashed when autoconf-native's hash was changed. If update_tasks processed a recipe before it had processed pseudo-native, that recipe would be marked as not possible from sstate and would run the full tasks. The fix is to split the processing into two passes, first to handle the existing covered/notcovered updates, then in the second pass, check whether there are "harddep" issues. This defers the do_package tasks until after pseudo-native is installed from sstate as expected and everything works well again. (Bitbake rev: 72a3afd99e8b785cb2a2f687e71a58e08cdd9c74) Signed-off-by: Richard Purdie (cherry picked from commit e479d1e418a7d34f0a4663b4a0e22bb11503c8ab) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Drop deadlock breaking force fail

2022-10-11T20:57:28+00:00

I'm 99% certain this failing of a scenequeue task corrupts runqueue and causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted and did weird things so drop this code. We've seen builds where the deadlock triggers and it then tries to run tasks where the SQ task already ran with very confusing failures. It is likely it is this code causing it. (Bitbake rev: f386298fc056ef130c2eb6dabf25eafbd55f55ca) Signed-off-by: Richard Purdie (cherry picked from commit 8efced47fcb47851a370fd6786df6fb377f99963) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie