linux/poky.git/bitbake/lib/bb/event.py, branch styhead-5.1.4

bitbake: event/utils: Avoid deadlock from lock_timeout() and recursive events

2025-03-13T14:21:43+00:00

We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. (Bitbake rev: 82b9f42126983579da03bdbb4e3ebf07346118a7) Signed-off-by: Richard Purdie (cherry picked from commit 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Steve Sakoman

bitbake: Remove custom exception backtrace formatting

2024-11-30T13:41:59+00:00

Removes the code in bitbake to show custom backtrace formatting for exceptions. In particular, the bitbake exception code prints function arguments, which while helpful is a security problem when passwords and other secrets can be passed as function arguments. As it turns out, the handling of the custom serialized exception stack frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass formatted exceptions"), which changed the events to pass a preformatted stacktrack list of strings, but the passing of the serialized data was never removed. Change all the code to use the python traceback API to format exceptions instead of the custom code; conveniently traceback.format_exception() also returns a list of stack trace strings, so it can be used as a drop in replacement for bb.exception.format_exception() (Bitbake rev: 1f7889c2f5e192fb108b51883aac8685ccb187f2) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: bitbake: event: Inject empty lines to make code match lineno in filename

2024-01-10T13:55:33+00:00

So that we can get the correct error messages. * In python 3.10.9, the error message was: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 4 # SPDX-License-Identifier: MIT ^^^^^ SyntaxError: invalid syntax This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4. * Now the error message and lineno is correct: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 256 an error line ^^^^^ SyntaxError: invalid syntax And no impact on parsing time: * Before: $ rm -fr cache tmp; time bitbake -p real 0m27.254s * Now: $ rm -fr cache tmp; time bitbake -p real 0m27.200s (Bitbake rev: c212933d9c786806852c87f188250a4f0a14c048) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

bitbake: bitbake: cooker: add a new function to retrieve task signatures

2023-09-28T11:38:26+00:00

adding a new command in cooker to compute and get task signatures this commit also add the associated command and event needed to get the signatures using tinfoil (Bitbake rev: 05c15162de90c41dad67e37a95ec9fdb440a7864) Signed-off-by: Julien Stephan Signed-off-by: Richard Purdie

bitbake: event: add bb.event.ParseError

2023-04-13T11:01:45+00:00

Add bb.event.ParseError to let error-report can catch this kind of error. (Bitbake rev: 316524ab59a5e738c25e062923ee5717d88ae5c7) Signed-off-by: Mingli Yu Signed-off-by: Richard Purdie

bitbake: event/cooker/runqueue: Add ability to interrupt longer running code

2023-02-23T12:12:18+00:00

Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie

bitbake: lib/bb: Update thread/process locks to use a timeout

2023-01-05T11:50:17+00:00

The thread/process locks we use translate to futexes in Linux. If a process dies holding the lock, anything else trying to take the lock will hang indefinitely. An example would be the OOM killer taking out a parser process. To avoid bitbake processes just hanging indefinitely, add a timeout to our lock calls using a context manager. If we can't obtain the lock after waiting 5 minutes, hard exit out using os._exit(1). Use _exit() to avoid locking in any other places trying to write error messages to event handler queues (which also need locks). Whilst a bit harsh, this should mean we stop having lots of long running processes in cases where things are never going to work out and also avoids hanging builds on the autobuilder. (Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b) Signed-off-by: Richard Purdie

bitbake: event: Always use threadlock

2022-12-31T17:05:17+00:00

With the move to a server idle thread, we always need threading. The existing accessor functions could end up turning this off! I was going to hold the lock whilst changing it, check if the value was already set, cache the result and also fix the event code to always release the lock with a try/finally. Instead, disable the existing functions and use a with: block to handle the lock, keeping things much simpler. (Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce) Signed-off-by: Richard Purdie

bitbake: event: Add enable/disable heartbeat code

2022-12-31T17:05:17+00:00

Currently heartbeat events are always generated by the server whilst it is active. Change this so they only appear when builds are running, which is when most code would expect to be executed. This removes a number of races around changes in the datastore which can happen outside of builds. (Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e) Signed-off-by: Richard Purdie

bitbake: event: builtins fix for 'd' deletion

2022-12-29T00:07:06+00:00

I've been seeing event handlers where 'd' seems to disappear half way through event handler execution. This is problematic when multiple threads are active since this code assumes single threading. The easiest fix is to change the handler function calls to contain d as a parameter as we do elsewhere for other functions. This will break any non-text handlers but I was only able to spot one of those in runqueue. It will also break handlers than call functions that assume 'd' is in the global namespace but those failures should be obvious and we can fix those to pass d around. This solution avoids manipulating builtins which was always a horrible thing to do anyway and solves the issue without needing locking, thankfully. (Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e) Signed-off-by: Richard Purdie