linux/poky.git/bitbake/lib/bb/utils.py, branch 5.3_M2

bitbake: utils: Remove multiprocessingpool

2025-07-23T10:34:31+00:00

This API is no longer used and bitbake has moved beyond Python 2.7.3 as the minimum version, so remove it. (Bitbake rev: 0eb7b5dd512ed8d8b77b5779858b9fbd99edb4a4) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: Use a "fork" multiprocessing context

2025-07-23T10:34:31+00:00

Python 3.14 changes the default multiprocessing context from "fork" to "forkserver"; however bitbake heavily relies on "fork" to efficiently pass data to the child processes. As such, make "fork" context in the bb namespace and use it in place of the normal multiprocessing module. Note that multiprocessing contexts were added in Python 3.4, so this should be safe to use even before Python 3.14 [YOCTO #15858] (Bitbake rev: 62be9113d98fccb347c6aa0a10d5c4ee2857f8b6) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: utils: Split profile reports into separate files

2025-07-21T16:49:04+00:00

Use a more logical name for the profile reports and put each report into a separate file since people struggle to discover them currently. (Bitbake rev: a8145c84e0899285a5e6a809f1515118b002b106) Signed-off-by: Richard Purdie

bitbake: utils: Optimise signal/sigmask performance

2025-07-21T16:49:04+00:00

Running "time bitbake -pP idle" with a valid cache shows around 800,000 calls to enum creation from python's signal.py. We don't care about this overhead and it adversely affects cache load time quite badly. Try and use _signal directly, falling back to signal, which avoids this overhead we don't need and makes cache loading much faster. (Bitbake rev: ee5fce67ce35b025c68aa61e2e758903269ee346) Signed-off-by: Richard Purdie

bitbake: cooker/process/utils: Create profiling common function to remove code duplication

2025-07-17T09:45:57+00:00

We have code duplication in the way we handle profiling of code sections. Create a common function in utils which covers this. The main loop and idle loop profile files were also reversed. Fix this and the naming, removing a couple of unused variables containing the profile log names in the process too. (Bitbake rev: b4f6bae97ac9607420fc49fd4c9e957d89c9a5f3) Signed-off-by: Richard Purdie

bitbake: utils: Refactor filemode variable conversion to a function

2025-07-01T07:49:37+00:00

We have other places in the code where we need to take filemode/mask information from a bitbake variable and turn it into a real python number. Turn this internal code into public API in bb.utils and add some tests for it. (Bitbake rev: d89e30fb2fb15b09f2cb95c4e5aa9f749ca257ea) Signed-off-by: Richard Purdie

bitbake: utils: format and improve docstrings

2025-04-24T10:53:00+00:00

Format the docstrings of the utils modules to be automatically documented with the autodoc Sphinx extensions. Sphinx syntax can be used in those for proper formatting. Cross-referencing with :term: is not possible in these. (Bitbake rev: 2fa1c7ad43639c6d25c94b7794bcce5f5ff74e10) Signed-off-by: Antonin Godard Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie

bitbake: utils: lock_timeout_nocheck() ensure `l` is initialized

2025-04-24T10:53:00+00:00

lock_timeout_nocheck() can be interrupted immediately after enterring the try-block and prior to initializing 'l', for example with a ctrl-C, the code in finally will still be run and the 'if l' will fail. Initialize 'l' as False to avoid this possiblity. (Bitbake rev: 4885cd9d275ba2ab60e5c76aed856c34533cd3ae) Signed-off-by: Mark Asselstine Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie

bitbake: event/utils: Avoid deadlock from lock_timeout() and recursive events

2025-03-03T21:38:57+00:00

We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. (Bitbake rev: 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Richard Purdie

bitbake: utils: Add signal blocking for lock_timeout

2025-03-03T21:38:57+00:00

We never want to exit whilst holding these locks as it deadlocks all python threads. Add signal blocking around the lock critical part so a signal shouldn't cause such an exit. (Bitbake rev: a097755c671e2b530dea6200a94b39fa9dca246c) Signed-off-by: Richard Purdie