linux/poky.git/bitbake/lib/bb/server/process.py, branch master-next

bitbake: Use a "fork" multiprocessing context

2025-07-23T10:34:31+00:00

Python 3.14 changes the default multiprocessing context from "fork" to "forkserver"; however bitbake heavily relies on "fork" to efficiently pass data to the child processes. As such, make "fork" context in the bb namespace and use it in place of the normal multiprocessing module. Note that multiprocessing contexts were added in Python 3.4, so this should be safe to use even before Python 3.14 [YOCTO #15858] (Bitbake rev: 62be9113d98fccb347c6aa0a10d5c4ee2857f8b6) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: main: Add an option to specify what to profile

2025-07-17T09:45:57+00:00

Starting with python 3.12, profiling now stays enabled over threads yet you can't extract the profile data in the threads themselves, which makes it difficult to use for our use case. Our main loop starts the idle loop which starts the parsing threads and this means we can't profile in the main loop and the parsing threads or the idle loop at the same time due to this. Add options to the commandline so you can specify which piece of bitbake you want to enable profiling for. This allows some profiling with python 3.12 onwards rather than crashing. (Bitbake rev: 09f29a4968841ee5070f70277ba8c253bb14f017) Signed-off-by: Richard Purdie

bitbake: cooker/process/utils: Create profiling common function to remove code duplication

2025-07-17T09:45:57+00:00

We have code duplication in the way we handle profiling of code sections. Create a common function in utils which covers this. The main loop and idle loop profile files were also reversed. Fix this and the naming, removing a couple of unused variables containing the profile log names in the process too. (Bitbake rev: b4f6bae97ac9607420fc49fd4c9e957d89c9a5f3) Signed-off-by: Richard Purdie

bitbake: server/process: Decrease idle/main loop frequency

2024-11-07T13:36:04+00:00

The idle and main loops have socket select calls to know when to execute. This means we can increase the normal timeout frequency since this is just a fall back and have some small efficiency gains. (Bitbake rev: 8d8e17af8619c976819170c9d5d9a686a666c317) Signed-off-by: Richard Purdie

bitbake: server/process: Don't send heartbeats when no idle functions

2024-11-07T13:36:04+00:00

If there are no idle functions present, don't sent heartbeat events. These are only meant to happen while builds are active. (Bitbake rev: 9a2d5e63b07c3912838781776c61f0f1ac9640e1) Signed-off-by: Richard Purdie

bitbake: server/process: Merge a function to simplfy code

2024-11-07T13:36:04+00:00

Keeping this code separate just makes the code harder to understand, merge them. (Bitbake rev: e5ac26a0e1779df1da3277bf48899c8f7642f1f8) Signed-off-by: Richard Purdie

bitbake: process/server: Fix typo

2024-02-10T15:25:22+00:00

Ensure the message matches the filenames the code actually uses. (Bitbake rev: deb7db2e2b125c6a6732db4f185f4de5926494fd) Signed-off-by: Richard Purdie

bitbake: process: Add profile logging for main loop

2024-02-10T14:13:51+00:00

When the idle/main loop was added, we didn't include profiling information for it. There is a performance issue in there, add logging for it. (Bitbake rev: d8d5cd43a60560f67e86f4f625113b0f73b944c0) Signed-off-by: Richard Purdie

bitbake: server/process: catch and expand multiprocessing connection exceptions

2024-01-10T14:02:38+00:00

Doing builds on systems with limited resources, or with high demand package builds such as chromium it isn't uncommon for the OOM Killer to be triggered and for bitbake-server to be selected as the process to be killed. When the bitbake-server does terminate unexpectedly due to the OOM Killer or otherwise, this currently results in a generic python traceback with little indication as to what has failed. Here we trap and raise the exceptions while extending the exception text in runCommand() to make it clear that this is most likely caused by the bitbake-server unexpectedly terminating. Callers of runCommand() should be updated to properly handle the BrokenPipeError and EOFError exceptions to avoid printing a python traceback, but even if they don't, the added text in the exceptions should provide some hints as to what might have caused the failure. (Bitbake rev: 5ff62b802f79acc86bbd6a99484f08501ff5dc2d) Signed-off-by: Mark Asselstine Signed-off-by: Richard Purdie

bitbake: bitbake/lib: spawn server/worker using the current Python interpreter

2023-09-26T09:37:17+00:00

The user may have invoked ./bin/bitbake using a different Python interpreter than whatever python3 is on $PATH (for example, explicitly using a different version). However, as the server and workers are spawned directly they'll use the hashbang and thus a different Python. We also ensure that argv[0] is set to sys.executable instead of 'bitbake-server' or 'bitbake-worker', so that sys.executable is set to the right value inside the child. Without this the server won't be able to start any workers. (Bitbake rev: b44d5d2a53d3082c8ce94e09c0cf833e33e25aec) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie