linux/poky.git/bitbake/lib/bb/runqueue.py, branch halstead/hashclient

bitbake: bitbake-worker: allow '=' in environment variable values

2024-03-22T16:26:45+00:00

Limit the split to key & value (2 items) instead of the n items one can get if there are '=' characters in the value. Fixes [YOCTO #15447] (Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb) Signed-off-by: david d zuhn Signed-off-by: Richard Purdie

bitbake: runqueue: Add support for BB_LOADFACTOR_MAX

2024-02-23T14:34:05+00:00

Some ditros don't enable /proc/pressure and it tends to be those which we see bitbake timeout issues on, seemingly as load gets too high and the bitbake processes don't get scheduled in for minutes at a time. Add support for stopping running extra tasks if the system load average goes above a certain threshold by setting BB_LOADFACTOR_MAX. The value used is scaled by CPU number, so a value of 1 would be when the load average equals the number of cpu cores of the system, under one only starts tasks when the load average is below the number of cores. This means you can centrally set a value such as 1.5 which will then scale correctly to different sized machines with differing numbers of CPUs. The pressure regulation is probably more accurate and responsive, however our graphs do show singificant load spikes on some workers and this patch is aimed at trying to avoid those. Pressure regulation is used where available in preference to this load factor regulation when both are set. (Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49) Signed-off-by: Richard Purdie

bitbake: runqueue: Improve setcene performance when encoutering many 'hard' dependencies

2024-02-13T13:52:10+00:00

"bitbake world -n --setscene-only" shows poor performance as the numbers of tasks increases. Analysys shows this is due to the "deferred" hard dependencies being repeatedly processed which doesn't allow much forward progress in overall task execution. To avoid this, mark when it has been done and don't reprocess until dependencies are updated. We have to be careful as we've seen bugs where these dependencies aren't processed at the right time. They also have to be reproceseed during task migration/rehashing so the code has to "self heal" the data structures. (Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f) Signed-off-by: Richard Purdie

bitbake: runqueue: Optimise taskname lookups in next_buildable_task

2024-02-13T13:52:10+00:00

A quick profile of bitbake world showed 147 million calls to taskname_from_tid(). The next_buildable_task function is performance senstive so move the call inside the if block to reduce the number of calls and speed the code up. (Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9) Signed-off-by: Richard Purdie

bitbake: runqueue: Improve performance for executing tasks

2024-02-13T13:52:10+00:00

Now that runqueue performance profiling works again we can see a lot of time is lost in build_taskdepdata. Whilst we can't compute that in advance, we can compute the individual entries. Therefore put a cache in place to compute those and save overhead in starting up tasks. (Bitbake rev: c4519b542702ba25023e53d77b275a6fa571ec50) Signed-off-by: Richard Purdie

bitbake: runqueue: Fix runall all bug

2024-01-12T11:59:36+00:00

Where chains of RDEPENDS are multiple levels deep, the runall code was not accounting for this and recursing deeply enough to gather all dependencies. Fix this by iterating over the result until no more dependencies are found. Tested-by: Jonas Gorski Reported-by: Jonas Gorski (Bitbake rev: 966f25dfc23a6d17b2b6d3e0100e9ae264f99025) Signed-off-by: Richard Purdie

bitbake: bitbake/runqueue: rework 'bitbake -S printdiff' logic

2024-01-10T16:32:13+00:00

Previously printdiff code would iterate over tasks that were reported as invalid or absent, trying to follow dependency chains that would reach the most basic invalid items in the tree. While this works in tightly controlled local builds, it can lead to bizarre reports against industrial-sized sstate caches, as the code would not consider whether the overall target can be fulfilled from valid sstate objects, and instead report missing sstate signature files that perhaps were never even created due to hash equivalency providing shortcuts in builds. This commit reworks the logic in two ways: - start the iteration over final targets rather than missing objects and try to recursively arrive at the root of the invalid object dependency. A previous version of this patch relied relies on finding the most 'recent' signature in stamps or sstate in a different function later, and recursively comparing that to the current signature, which is unreliable on real world caches. - if a given object can be fulfilled from sstate, recurse only into its setscene dependencies; bitbake wouldn't care if dependencies for the actual task are absent, and neither should printdiff I wrote a recursive function for following dependencies, as doing recursive algorithms non-recursively can result in write-only code, as was the case here. [YOCTO #15289] (Bitbake rev: aadeca63da5d96160ce4d6d71da556e2e033f9b7) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: bitbake/runqueue: prioritize local stamps over sstate signatures in printdiff

2024-01-05T11:59:08+00:00

Even with the reworked printdiff code, sstate which is heavily used in parallel can throw races at the tests: if a new matching, but otherwise unrelated sstate signature appears between writing out local stamps and listing matching sstate files, then that signature will be deemed 'the latest' and the actual local stamp will be discarded. This change ensures the scenario does not happen. It also makes use of the reworked find_siginfo(), particularly the 'sstate' entry in returned results. (Bitbake rev: c8574b796dabb69699c70540dd95a44d8f7388ab) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: siggen: Ensure version of siggen is verified

2024-01-05T11:59:08+00:00

Since we need to change the form of the siggen function, we need to add versioning and some verison checks. This means if a newer bitbake is used with older metadata we can detect it. (Bitbake rev: 721556568413508213d22c29985e305a45a8d68a) Signed-off-by: Richard Purdie

bitbake: bitbake-diffsigs/runqueue: adapt to reworked find_siginfo()

2024-01-05T11:59:08+00:00

In particular having 'time' explicitly used as a sorting key should make it more clear how the entries are being sorted. (Bitbake rev: 5439aca056c84ab4410aaf24bdb68e896191d8e1) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie