linux/poky.git/bitbake/lib/bb/server, branch scarthgap-5.0.15

bitbake: Use a "fork" multiprocessing context

2025-09-17T22:36:44+00:00

Python 3.14 changes the default multiprocessing context from "fork" to "forkserver"; however bitbake heavily relies on "fork" to efficiently pass data to the child processes. As such, make "fork" context in the bb namespace and use it in place of the normal multiprocessing module. Note that multiprocessing contexts were added in Python 3.4, so this should be safe to use even before Python 3.14 [YOCTO #15858] (Bitbake rev: 15d7448e04aa78c827d2cef9eb1a62bd6e0dd119) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

bitbake: process/server: Fix typo

2024-02-10T15:25:22+00:00

Ensure the message matches the filenames the code actually uses. (Bitbake rev: deb7db2e2b125c6a6732db4f185f4de5926494fd) Signed-off-by: Richard Purdie

bitbake: process: Add profile logging for main loop

2024-02-10T14:13:51+00:00

When the idle/main loop was added, we didn't include profiling information for it. There is a performance issue in there, add logging for it. (Bitbake rev: d8d5cd43a60560f67e86f4f625113b0f73b944c0) Signed-off-by: Richard Purdie

bitbake: server/process: catch and expand multiprocessing connection exceptions

2024-01-10T14:02:38+00:00

Doing builds on systems with limited resources, or with high demand package builds such as chromium it isn't uncommon for the OOM Killer to be triggered and for bitbake-server to be selected as the process to be killed. When the bitbake-server does terminate unexpectedly due to the OOM Killer or otherwise, this currently results in a generic python traceback with little indication as to what has failed. Here we trap and raise the exceptions while extending the exception text in runCommand() to make it clear that this is most likely caused by the bitbake-server unexpectedly terminating. Callers of runCommand() should be updated to properly handle the BrokenPipeError and EOFError exceptions to avoid printing a python traceback, but even if they don't, the added text in the exceptions should provide some hints as to what might have caused the failure. (Bitbake rev: 5ff62b802f79acc86bbd6a99484f08501ff5dc2d) Signed-off-by: Mark Asselstine Signed-off-by: Richard Purdie

bitbake: bitbake/lib: spawn server/worker using the current Python interpreter

2023-09-26T09:37:17+00:00

The user may have invoked ./bin/bitbake using a different Python interpreter than whatever python3 is on $PATH (for example, explicitly using a different version). However, as the server and workers are spawned directly they'll use the hashbang and thus a different Python. We also ensure that argv[0] is set to sys.executable instead of 'bitbake-server' or 'bitbake-worker', so that sys.executable is set to the right value inside the child. Without this the server won't be able to start any workers. (Bitbake rev: b44d5d2a53d3082c8ce94e09c0cf833e33e25aec) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: server/process: Disable the flush() call in server logging

2023-09-18T10:35:05+00:00

We've been chasing bitbake timeouts for a while and it was unclear where things were blocking on IO. It appears the flush() call in server logging can cause pauses up to minutes long on systems with slow (spinning) disks that are heavily loaded with IO. Since the flush() was added to aid debugging of other timing issues, we shouldn't need it now and it can be disabled. Leave a comment as a reminder of the pain this can cause. (Bitbake rev: afbc169e1490a86d6250969f780062c426eb4682) Signed-off-by: Richard Purdie

bitbake: lib: Drop inotify support and replace with mtime checks

2023-09-18T10:35:05+00:00

With the flush in serverlog() removed and a memory resident bitbake with a 60s timeout, the following could fail in strange ways: rm bitbake-cookerdaemon.log bitbake-layers add-layer ../meta-virtualization/ bitbake-layers add-layer ../meta-openembedded/meta-oe/ bitbake -m specifically that it might error adding meta-oe with an error related to meta-virt. This clearly shows that whilst bblayers.conf was modified, bitbake was not recognising that. This would fit with the random autobuilder issues seen when the serverlog flush() call was removed. The issue appears to be that you have no way to "sync()" the inotify events with the command stream coming over the socket. There is no way to know if there are changes in the IO queue which bitbake needs to wait for before proceeding with the next command. I did experiment with os.sync() and fsync on the inotify fd, however nothing addressed the issue. Since it is extremely important we have accurate cache data, the only realistic thing to do is to switch to stat() calls and check mtime. For bitbake commands, this is straightforward since we can revalidate the cache upon new connections/commands. For tinfoil this is problematic and we need to introduce and explict command "revalidateCaches" that the code can use to force bitbake to re-check it's cache validity. I've exposed this through tinfoil with a new "modified_files" function. So, this patch: a) drops inotify support within bitbake's cooker/server and switch to using mtime b) requires a new function call in tinfoil when metadata has been modified (Bitbake rev: da3ec3801bdb80180b3f1ac24edb27a698415ff7) Signed-off-by: Richard Purdie

bitbake: server/process: Add more timing debug

2023-09-05T07:14:50+00:00

It is helpful to have timestamps on the ping failures so that they can be matched against the bitbake logs. It is also useful to understand how long the server takes for form a reply verses when it is sent. (Bitbake rev: 65969a7a8f5ae22c230431d2db080eb187a27708) Signed-off-by: Richard Purdie

bitbake: server/process: fix sig handle

2023-08-11T06:58:31+00:00

process.signal_received is a list for signum and not iterable, change a suitable method to handle sig. (Bitbake rev: bfc53b190bd2530c2bfcea0690127d7eff620f45) Signed-off-by: Yang Xu Signed-off-by: Richard Purdie

bitbake: server/process: Show command in timeout message

2023-06-30T16:58:47+00:00

To learn more about the server timeout issues, be clear in the error message about which command is showing the timeout. It is currently unclear if this is the original command or a ping to the server. (Bitbake rev: ac3cd866274f67b29eff89e393132bdabf76dbfd) Signed-off-by: Richard Purdie