linux/poky.git/bitbake/lib/bb/utils.py, branch walnascar-5.2

bitbake: event/utils: Avoid deadlock from lock_timeout() and recursive events

2025-03-03T21:38:57+00:00

We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. (Bitbake rev: 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Richard Purdie

bitbake: utils: Add signal blocking for lock_timeout

2025-03-03T21:38:57+00:00

We never want to exit whilst holding these locks as it deadlocks all python threads. Add signal blocking around the lock critical part so a signal shouldn't cause such an exit. (Bitbake rev: a097755c671e2b530dea6200a94b39fa9dca246c) Signed-off-by: Richard Purdie

bitbake: utils: Tweak lock_timeout logic

2025-03-03T21:38:57+00:00

We should really try and take the lock in the try/finally block so that in some rare cases such as badly timed interrupt/signal, we always release the lock. (Bitbake rev: a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2) Signed-off-by: Richard Purdie

bitbake: utils: Print information about lock issue before exiting

2025-03-03T21:38:57+00:00

(Bitbake rev: cdf6c51a064f8f335c3262b7f102618996f1a229) Signed-off-by: Richard Purdie

bitbake: utils: Clean up imports

2025-02-18T23:07:07+00:00

* Remove double imports * Re-roder the imports as mentioned by pylint (Bitbake rev: 6de536bbaee8cf7664c5702a96a5ca18ad09fb9d) Signed-off-by: Michael Estner Signed-off-by: Richard Purdie

bitbake: utils: add Go mod h1 checksum support

2025-01-11T18:35:01+00:00

Add support for the Go mod h1 hash. The hash is based on the Go dirhash package. The package defines hashes over directory trees and is uses for Go mod files and zip archives. (Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: utils: Improve multiple lock handling

2024-10-11T11:17:36+00:00

We should release locks in the opposite order to the order we locked them in to avoid potentially problematic behaviour. This hasn't been observed with our current usage and is just a preventative improvement. (Bitbake rev: 4fbef87adaca2df8c1e877aec5588c2e5466f252) Signed-off-by: Richard Purdie

bitbake: utils: better estimate number of available cpus

2024-03-22T16:26:45+00:00

When running in a cgroup which is limited to a subset of cpus (via cpuset.cpus), cpu_count() should return the number of cpus that can be used instead of the number of cpus the system has. This also aligns the semantics with nproc. (Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118) Signed-off-by: Felix Moessbauer Signed-off-by: Richard Purdie

bitbake: utils: remove BB_ENV_PASSTHROUGH from preserved_envvars()

2024-03-05T12:25:08+00:00

preserved_envvars() is used when the BB_ENV_PASSTHROUGH environment variable is not set. Therefore, its code shouldn't return this variable. (Bitbake rev: 0a33b560233b983456178541603ab96fea22238b) Signed-off-by: Michael Opdenacker Reported-by: Robert P. J. Day Tested-by: Michael Opdenacker Signed-off-by: Richard Purdie

bitbake: utils: Fix mkdir with PosixPath

2023-12-17T19:07:21+00:00

Avoid: Exception: AttributeError: 'PosixPath' object has no attribute 'find' (Bitbake rev: 0b37fe89ba12549109905b6d0e6d07d342162436) Signed-off-by: Richard Purdie