linux/poky.git/bitbake/bin/bitbake-worker, branch nanbield-4.3.2

bitbake: bitbake-worker/runqueue: Avoid unnecessary bytes object copies

2023-09-22T06:45:47+00:00

declaring queue=b"" creates an object of types bytes(). bytes() is an immutable object, and therefore doing "self.queue = self.queue + r" creates a new object containing "self.queue" concatenated with "r". On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker, so those copies significantly slow down the initialization of the bitbake-worker. Rather use bytearray() which a mutable type, and use extend() to avoid copies. In my test setup, byterray.extend() is 10.000 times faster than copying the queue, for a queue size of 180MB. (Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a) Signed-off-by: Etienne Cordonnier Signed-off-by: Richard Purdie

bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a recipe

2023-05-25T12:16:24+00:00

There are times when it would be useful for code to know which layer (or collection in old bitbake terms) it is contained within. Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe so that it is possible to determine this. To do it, we need to pass data from the cooker into the recipe endpoints, since only the top level cooker information knows about the layer structure which makes the patch a bit painful. The idea is that this would make layer overrides possible: OVERRIDES .= ":layer-${FILE_LAYERNAME}" which then opens possibilities like: WARN_QA:append:layer-core = " patch-fuzz" as an example where OE-Core could enable specific QA tests only for that specific layer. (Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf) Signed-off-by: Richard Purdie

bitbake: data: Evaluate the value of export/unexport/network flags

2023-02-17T18:02:04+00:00

Currently the export/unexport/network flags are acted on only by presence or lack of in the data store. This is deliberate due to performance reasons since a given recipe shell task might export ~80-90 variables and this means expanding flags for every one of them. This does catch users unaware since values which expand to nothing don't work e.g. ${@""} and setting values to "0" wouldn't work either. The downside to this patch is slow down in parsing speed of around 4%. The network flag is easier to change and doesn't affect performance, it was made to operate the same as export/unexport for consitency reasons. (Bitbake rev: 3d96c07f9fd4ba1a84826811d14bb4e98ad58846) Signed-off-by: Richard Purdie

bitbake: lib/bb: Update thread/process locks to use a timeout

2023-01-05T11:50:17+00:00

The thread/process locks we use translate to futexes in Linux. If a process dies holding the lock, anything else trying to take the lock will hang indefinitely. An example would be the OOM killer taking out a parser process. To avoid bitbake processes just hanging indefinitely, add a timeout to our lock calls using a context manager. If we can't obtain the lock after waiting 5 minutes, hard exit out using os._exit(1). Use _exit() to avoid locking in any other places trying to write error messages to event handler queues (which also need locks). Whilst a bit harsh, this should mean we stop having lots of long running processes in cases where things are never going to work out and also avoids hanging builds on the autobuilder. (Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b) Signed-off-by: Richard Purdie

bitbake: siggen: Add dummy dataCaches from task context/datastore

2022-12-17T08:52:28+00:00

One of the challenges in maintaining the code is that it sometimes uses a datacaches structure and sometimes a datastore. Rather than continue the current dual API madness, have the worker contexts create a dummy datacaches structure with the entries we need. Whilst this does need to be kept in sync with the real structure, that doesn't change and this allows the code to be simplified. With this new approach, we can unify the stamps dependency code again. (Bitbake rev: c6d325fc9b53e9d588ab273ee3c2a99a70fba42c) Signed-off-by: Richard Purdie

bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the system

2022-12-08T10:49:53+00:00

Get rid of the duplicate code and add extra check that the locale en_US.UTF-8 is available on the system. This new helper method is now located right above the method filter_environment() which sets LC_ALL environment variable to 'en_US.UTF-8'. [YOCTO #10165] (Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c) Signed-off-by: Frank de Brabander Signed-off-by: Richard Purdie

bitbake: worker/runqueue: Reduce initial data transfer in workerdata

2022-11-20T08:31:28+00:00

When setting up the worker we were transfering large amounts of data which aren't needed until task execution time. Defer the fakeroot and taskdeps data until they're needed for a specific task. This will duplicate some information when executing different tasks for a given recipe but as is is spread over the build run, it shouldn't be an issue overall. Also take the opportunity to clean up the silly length argument lists that were being passed around at the expense of extra dictionary keys. (Bitbake rev: 3a82acdcf40bdccd933c4dcef3d7e480f0d7ad3a) Signed-off-by: Richard Purdie

bitbake: cache/cookerdata: Move recipe parsing functions from cache to databuilder

2022-11-20T08:31:28+00:00

When 'NoCache' was written, databuilder/cookerdata didn't exist. It does now and the recipe parsing functionality contained in NoCache clearly belongs there, it isn't a cache function. Move those functions, renaming to match the style in databuilder but otherwise not changing functionality for now. Fix up the callers to match (which make it clear this is the right move). (Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca) Signed-off-by: Richard Purdie

bitbake: BBHandler/cooker: Implement recipe and global classes

2022-08-12T10:49:29+00:00

We have some confusion for users since some classes are meant to work in the configuration space (or "globally") and some are meant to be selected by recipes individually. The cleanest way I could find to clarify this is to create "classes-global" and "classes-recipe" directories which contain the approproate classes and have bitbake switch scope between them at the appropriate point during parsing. The existing "classes" directory is always searched as a fallback. Once a class is moved to a specific directory, it will no longer be found in the incorrect context. A good example from OE is that INHERIT += "testimage" will no longer work but IMAGE_CLASSES += "testimage" will, which makes the global scope cleaner by only including it where it is useful and intended to be used (images). (Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e) Signed-off-by: Richard Purdie

bitbake: bitbake: Add copyright headers where missing

2022-08-12T10:49:29+00:00

Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie