linux/poky.git/bitbake/lib/bb/cooker.py, branch 4.2_M3

bitbake: cooker: Fix memory resident cache invalidation issue

2023-02-24T15:38:24+00:00

We've been seeing weird PRServ failures on the autobuilder. These had one failure always followed by a second. Whilst I can't reproduce the first, if I made that test fail, I could reproduce the second with memory resident bitbake. This was with the tests: prservice.BitbakePrTests.test_import_export_replace_db and then prservice.BitbakePrTests.test_pr_service_deb_arch_dep which was giving a strange looking error: NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa) ERROR: No such task: do_package_write_rpm ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1' where the issue is that selftest.inc written by the test framework and containing PACKAGE_CLASSES = "package_deb" was being ignored. The issue is the cached_statements{} within BBHandler() is not being invalidated at the right time. This patch changes the code to ensure base configuration is not parsed until inotify updates have been processed. (Bitbake rev: cada37c6b9e5862ca2c5a54ad6fd1e1f1939cd9c) Signed-off-by: Richard Purdie

bitbake: event/cooker/runqueue: Add ability to interrupt longer running code

2023-02-23T12:12:18+00:00

Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie

bitbake: cooker: Ensure lock is held with changing notifier

2023-02-20T15:18:53+00:00

We've seen a couple of cases which bitbake hangs due to an inotifer exception such as: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) Ensure we don't destory the inotifier when the idle thread is reading is by holding the lock during setup/teardown. (Bitbake rev: 8fc5c50c2e23017833f93bcd514d708a14fa4266) Signed-off-by: Richard Purdie

bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache location

2023-01-26T21:50:31+00:00

Currently the codeparser cache is set from CACHE, which is typically in bitbake.conf which means we can't read/write any cache until it is found/read. We may well have python expressions to parse before that happens. The net result is suboptimal functioning of the codeparser cache since it will often be invalidated by data that is never written. This patch changes the codeparser and filechecksum caches to use BB_CACHE as their setting and defaults it to ${TOPDIR}/cache. The patch doesn't change where the "persistent" data such as prserver and hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing cache resists (still currently CACHE). I've left those for a later patch. The patch does ensure data parsed by the core datastore parsing calls is written back since this is now much more useful after this change. (Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628) Signed-off-by: Richard Purdie

bitbake: cooker: Fix parsing race around cache handling

2023-01-24T21:59:44+00:00

When draining the result queue from the parsing processes, cache objects can be created even if they are then immediately destroyed. The reset in the sync code needs to happen after any objects have been created. Change the ordering to fix this. This ordering has caused various cache errors, particularly when interrupting parsing with Ctrl+C. (Bitbake rev: f45a94e6720dacf7f51ac147c115a6f608769093) Signed-off-by: Richard Purdie

bitbake: cooker: Fix siggen recipe cache race issue

2023-01-24T21:59:44+00:00

We need to reset the cache before the start() call which starts the parsing processs, tweak the code to ensure this is the case. (Bitbake rev: 4712d6237df5d2188238294fbdb9f107b068a3a7) Signed-off-by: Richard Purdie

bitbake: cooker: Fix exit handling issues

2023-01-11T23:14:48+00:00

The shutdown() call should definitely not be waiting for idle, since we expect execution and events to continue even after setting either shutdown state. This was causing the UI to appear to hang at Ctrl+C interrupts. Also ensure that if we reset cooker, we stop any active parser processes since these currently appear to be being left behind. (Bitbake rev: 4f73c2eb12ee074f3b7d4717380649f6ca8f3def) Signed-off-by: Richard Purdie

bitbake: process/cooker/command: Fix currentAsyncCommand locking/races

2023-01-11T10:58:36+00:00

currentAsyncCommand currently doesn't have any locking and we have a conflict in "idle" conditions since the idle functions count needs to be zero *and* there needs to be no active command. Move the changes/checks of currentAsyncCommand to within the lock and then we can add it to the condition for idle, simplifying some of the code. (Bitbake rev: b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3) Signed-off-by: Richard Purdie

bitbake: cooker/command: Drop async command handler indirection via cooker

2023-01-11T10:58:36+00:00

Indirecting the async command handler via cooker is confusing and no longer needed. Drop it to make things slightly clearer. (Bitbake rev: 4a41a7d0594e6a84a67b9de70a505858aebcd84a) Signed-off-by: Richard Purdie

bitbake: cooker: Clean up inotify idle handler

2023-01-06T17:40:01+00:00

We no longer need to abstract the inotify callback handler, remove the abstraction and simplify/clean up the code. (Bitbake rev: af4ccab8acc49e91bf7647f209d69f4858618466) Signed-off-by: Richard Purdie