linux/poky.git/bitbake/lib, branch styhead

bitbake: event/utils: Avoid deadlock from lock_timeout() and recursive events

2025-03-13T14:21:43+00:00

We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. (Bitbake rev: 82b9f42126983579da03bdbb4e3ebf07346118a7) Signed-off-by: Richard Purdie (cherry picked from commit 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Steve Sakoman

bitbake: utils: Add signal blocking for lock_timeout

2025-03-13T14:21:43+00:00

We never want to exit whilst holding these locks as it deadlocks all python threads. Add signal blocking around the lock critical part so a signal shouldn't cause such an exit. (Bitbake rev: de552cf58fcbdea64b359a0fdcdf3673da3a780d) Signed-off-by: Richard Purdie (cherry picked from commit a097755c671e2b530dea6200a94b39fa9dca246c) Signed-off-by: Steve Sakoman

bitbake: utils: Tweak lock_timeout logic

2025-03-13T14:21:43+00:00

We should really try and take the lock in the try/finally block so that in some rare cases such as badly timed interrupt/signal, we always release the lock. (Bitbake rev: f0f7bed1291afd63a27f1863901445e6d40c4a09) Signed-off-by: Richard Purdie (cherry picked from commit a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2) Signed-off-by: Steve Sakoman

bitbake: utils: Print information about lock issue before exiting

2025-03-13T14:21:43+00:00

(Bitbake rev: bde34a174886161def6f9f2c5194870a4edfefb2) Signed-off-by: Richard Purdie (cherry picked from commit cdf6c51a064f8f335c3262b7f102618996f1a229) Signed-off-by: Steve Sakoman

bitbake: bblayers/query: Fix using "removeprefix" string method

2025-02-21T14:34:58+00:00

The minimum Python version required for Yocto 5.0 is 3.8 which causes failure in poky/bitbake/lib/bblayers/query.py when listing layers by using command "bitbake-layers show-recipes -f --bare --mc MC" for the given multiconfig MC. The reason for that failure is the use of "removeprefix" string method which got introduced in Python 3.9. This patch replaces the "removeprefix" method with an equivalent solution supported by Python 3.8. (Bitbake rev: 58e5c70a0572ff5994dc181694e05cd5d3ddaf66) Signed-off-by: Joerg Schmidt Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: cooker: Make cooker 'skiplist' per-multiconfig/mc

2025-02-12T14:29:33+00:00

Previously, the cooker skiplist was shared across multiconfigs (including default ''). If you had a recipe that was incompatible with several multiconfigs for different reasons, then the displayed reason (i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might vary across invocations of bitbake. This was caused by the random order in which recipes are parsed under different multiconfig contexts, with each skip reason overwriting the previously assigned reason. I hit this specificially when using COMPATIBLE_MACHINE, but COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too. (Bitbake rev: a8578d12dfe5a3c92119bfee224595a392d6b0dd) Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie (cherry picked from commit c51f01a35ed9a928402eab0899598b5c59602eef) Signed-off-by: Steve Sakoman

bitbake: tests/fetch: Fix git shallow test failure with git >= 2.48

2025-01-29T14:20:43+00:00

>From git 2.48 release notes: """ When "git fetch $remote" notices that refs/remotes/$remote/HEAD is missing and discovers what branch the other side points with its HEAD, refs/remotes/$remote/HEAD is updated to point to it. """ This means with git 2.48 onwards, there is a mystery "HEAD" revision appearing in some of our shallow clone tests. We can avoid this by using the same canonicalization as used for the reference revisions. This resolves autobuilder failures on the Fedora 40 workers. (Bitbake rev: f4b2ae97a8dd745b463e5e10fbc8d30d29d223c5) Signed-off-by: Richard Purdie (cherry picked from commit c83444d1210740e27b1744d3aa7c5cad4e28db2f) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: runqueue: Fix scenetask processing performance issue

2024-12-09T14:25:53+00:00

Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. (Bitbake rev: 1452e74cc4ddfaadc6537a049877a66cec253c8d) Signed-off-by: Richard Purdie (cherry picked from commit 923c19b9713e398d8e66e6d4422dfd4c18a03486) Signed-off-by: Steve Sakoman

bitbake: runqueue: Optimise setscene loop processing

2024-12-09T14:25:53+00:00

Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. (Bitbake rev: 5465f9b6e3b4748e563efc53657af96f02a41c7a) Signed-off-by: Richard Purdie (cherry picked from commit 00f4d932e3af0eeb333339cbe942010fc76dee0f) Signed-off-by: Steve Sakoman

bitbake: runqueue: Fix performance of multiconfigs with large overlap

2024-12-09T14:25:53+00:00

There have been complaints about the performance of large multiconfig builds for a while. The key missing data point was that the builds needed to have large overlaps in sstate objects. This can be simulated by building the same things with just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of deferred tasks. The issue is that the expensive checks in the setscene loop were hit every time through runqueue's execute function before the check on deferred tasks. This leads to task execution starvation as that only happens once per iteration. Move the skip check earlier in the function which speeds things up enormously and should improve performance of such builds for users. (Bitbake rev: 5392a2bf6d82626e11eec5578eb4de53ba09f1fe) Signed-off-by: Richard Purdie (cherry picked from commit 9c6c506757f2b3e28c8b20513b45da6b4659c95f) Signed-off-by: Steve Sakoman