linux/poky.git/bitbake/bin/bitbake-worker, branch 5.2_M2

bitbake: bitbake-worker/cooker: Increase default pipe size

2024-11-21T12:16:28+00:00

The default pipe size is 64kb on builds, which can be inefficient for larger log files from workers. Increase the pipe size to 512kb since build systems have decent amounts of memory and this is a more efficient way of batching the data. Tweak the default read sizes to match the pipe size for efficiency. Since the contstant is only present in python 3.10 onwards, add some compatibility code. (Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7) Signed-off-by: Richard Purdie

bitbake: bitbake-worker: Improve bytearray truncation performance

2024-11-21T12:16:28+00:00

If there are large amounts of data being transferred to the cooker from the worker, recreating the bytearray becomes inefficient as it happens for every pipesize block of data, defaulting to 64kb. Instead we can use the deletion API for bytearrays to make this more efficient and avoid the object recreation. We noticed this with a strace ptest image taking days to complete the build after having 6GB of data in the testimage log. Whilst there are other issues there, making this code more efficient doesn't hurt. (Bitbake rev: a4a72b7edb368f352784c856a647236a887010dd) Signed-off-by: Richard Purdie

bitbake: bitbake-worker: Fix silent hang issue caused by unexpected stdout content

2024-03-22T16:26:46+00:00

This patch addresses an issue in bitbake-worker where stdout, reserved for status reporting, is improperly accessed by child processes. The problem occurs during the execution of parseRecipe, which calls anonymous functions. If these functions use print-like operations, they can inadvertently output data to stdout. This unexpected data can cause the runqueue to hang silently, if the stdout buffer is flushed before exec_task is executed. To prevent this, the patch redirects stdout to /dev/null and ensures it is flushed prior to the execution of exec_task. (Bitbake rev: 08f3e677d6af27a41a918aaa9da9c1c9b20a0b95) Signed-off-by: Yang Xu Signed-off-by: Richard Purdie

bitbake: bitbake-worker: allow '=' in environment variable values

2024-03-22T16:26:45+00:00

Limit the split to key & value (2 items) instead of the n items one can get if there are '=' characters in the value. Fixes [YOCTO #15447] (Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb) Signed-off-by: david d zuhn Signed-off-by: Richard Purdie

bitbake: bitbake-worker: Fix bug where umask 0 was not being applied to a task

2024-03-22T16:26:45+00:00

* In the current implementation, "umask" variable is initially set to None and overwritten with user-specified value. However, in the worker implementation, a faulty if clause would only check whether the variable contains a value that evaluates to True, and not whether the variable is defined, so the value of 0 would lead to umask not being changed. This bug makes it impossible to have a task set its umask to value 0, for any possible reason it may want to. Fix this bug by extending the condition checked in the worker implementation. (Bitbake rev: 19f9df6c750c592316a0fa18165b68636281fe3e) Signed-off-by: Sava Jakovljev Signed-off-by: Richard Purdie

bitbake: bitbake-worker: add header with length of message

2023-11-06T16:45:11+00:00

The IPC mechanism between runqueue.py and bitbake-worker is currently not scalable: The data is sent with the format pickled-data, and bitbake-worker has no information about the size of the message. Therefore, the bitbake-worker is calling select() and read() in a loop, and then calling "self.queue.find(b"")" for each chunk received. This does not scale, because queue.find has a linear complexity relative to the size of the queue, and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI. The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup 35 seconds before this fix, and 1.5 seconds after this fix). This commit adds a 4 bytes header after , so that bitbake-worker knows how many bytes need to be received, and does not need to constantly search the whole queue for . (Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5) Signed-off-by: Etienne Cordonnier Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: bitbake-worker/runqueue: Avoid unnecessary bytes object copies

2023-09-22T06:45:47+00:00

declaring queue=b"" creates an object of types bytes(). bytes() is an immutable object, and therefore doing "self.queue = self.queue + r" creates a new object containing "self.queue" concatenated with "r". On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker, so those copies significantly slow down the initialization of the bitbake-worker. Rather use bytearray() which a mutable type, and use extend() to avoid copies. In my test setup, byterray.extend() is 10.000 times faster than copying the queue, for a queue size of 180MB. (Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a) Signed-off-by: Etienne Cordonnier Signed-off-by: Richard Purdie

bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a recipe

2023-05-25T12:16:24+00:00

There are times when it would be useful for code to know which layer (or collection in old bitbake terms) it is contained within. Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe so that it is possible to determine this. To do it, we need to pass data from the cooker into the recipe endpoints, since only the top level cooker information knows about the layer structure which makes the patch a bit painful. The idea is that this would make layer overrides possible: OVERRIDES .= ":layer-${FILE_LAYERNAME}" which then opens possibilities like: WARN_QA:append:layer-core = " patch-fuzz" as an example where OE-Core could enable specific QA tests only for that specific layer. (Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf) Signed-off-by: Richard Purdie

bitbake: data: Evaluate the value of export/unexport/network flags

2023-02-17T18:02:04+00:00

Currently the export/unexport/network flags are acted on only by presence or lack of in the data store. This is deliberate due to performance reasons since a given recipe shell task might export ~80-90 variables and this means expanding flags for every one of them. This does catch users unaware since values which expand to nothing don't work e.g. ${@""} and setting values to "0" wouldn't work either. The downside to this patch is slow down in parsing speed of around 4%. The network flag is easier to change and doesn't affect performance, it was made to operate the same as export/unexport for consitency reasons. (Bitbake rev: 3d96c07f9fd4ba1a84826811d14bb4e98ad58846) Signed-off-by: Richard Purdie

bitbake: lib/bb: Update thread/process locks to use a timeout

2023-01-05T11:50:17+00:00

The thread/process locks we use translate to futexes in Linux. If a process dies holding the lock, anything else trying to take the lock will hang indefinitely. An example would be the OOM killer taking out a parser process. To avoid bitbake processes just hanging indefinitely, add a timeout to our lock calls using a context manager. If we can't obtain the lock after waiting 5 minutes, hard exit out using os._exit(1). Use _exit() to avoid locking in any other places trying to write error messages to event handler queues (which also need locks). Whilst a bit harsh, this should mean we stop having lots of long running processes in cases where things are never going to work out and also avoids hanging builds on the autobuilder. (Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b) Signed-off-by: Richard Purdie