linux/poky.git/bitbake/lib/bb/cooker.py, branch 3.1_M1

bitbake: cooker: Remove a left-over comment about expanded_data

2019-11-14T14:06:42+00:00

This should have been removed together with expanded_data in commit e3694e73 (cooker/command: Drop expanded_data). (Bitbake rev: 33197db8abf82be240d7c1c5c9d2484a08a90849) Signed-off-by: Peter Kjellerstedt Signed-off-by: Richard Purdie

bitbake: hashserv: Don't daemonize server process

2019-09-27T12:02:19+00:00

The hash server process is terminated and waited on with join(), so it should not be a daemon. Daemonizing it cause races with the server cleanup, especially in the selftest because the process may not have terminated and cleanup up its socket before the test cleanup runs and tries to do it. [YOCTO #13542] (Bitbake rev: 7c829675581818f92d57056b57fbd3880829b6bd) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: Rework hash equivalence

2019-09-18T16:52:01+00:00

Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: Revert "bitbake: cooker: Ensure bbappends are found in stable order"

2019-09-16T22:02:45+00:00

This reverts commit 94c0c7f15c7a6244a8576ed948ffc21afb96ba82. This ignores the layer priority, making the issue much worse. E.g. I'm seeing a lot of failures caused by missing users, because base-passwd bbappends applied in unexpected order caused different passwd.master to be found in re-ordered FILESPATH. (Bitbake rev: 2dc862237dba82da37c8ac9289e0a21409b1305c) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: bitbake: cooker: Ensure bbappends are found in stable order

2019-09-07T12:08:54+00:00

Thanks to wildcards in bbappend filenames, it's possible to have multiple bbappends that apply to the same recipe in the same directory. In order to get sstate hits between different workspaces, we want to apply those bbappend files in a consistent order. Since readdir() returns files in a non-deterministic order between workspaces (based on inode number and/or time of creation), we'll need to sort its result in order to have any consistency. (Bitbake rev: 94c0c7f15c7a6244a8576ed948ffc21afb96ba82) Signed-off-by: Wes Lindauer Signed-off-by: Richard Purdie

bitbake: cooker.py: remove generation of recipe-depends.dot

2019-08-28T15:44:09+00:00

The information of recipe-depends.dot is misleading. e.g. $ grep xz recipe-depends.dot | grep bzip2 "bzip2" -> "xz" "xz" -> "bzip2" Users would wonder why they get some circular dependency. The information is derived from removing the task names of task-depends.dot. It's not giving people any additonal information, and it's misleading. So we remove the generation of this file. (Bitbake rev: 4c484cc01e3eee7ab2ab0359fd680b4dbd31dc30) Signed-off-by: Chen Qi Signed-off-by: Richard Purdie

bitbake: cooker: Improve hash server startup code to avoid exit tracebacks

2019-08-14T16:28:23+00:00

At exit the hashserv code was causing tracebacks as join() wasn't being called from the thread that started the process. Ensure that the hashserver is started from the pre_serve hook which is the final thread the cooker runs in. This avoids the traceback at the expense of some horrific poking into data stores which will ultimately need improving through a proper API. (Bitbake rev: 05888700e5f6cba48a26c8a4c447634a28e3baa6) Signed-off-by: Richard Purdie

bitbake: cooker: Cleanup the queue before call process.join()

2019-08-08T09:22:36+00:00

Fixed: $ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p Press *one* Ctrl-C when the parsing process is at about 50%, then the processes are not exited: Keyboard Interrupt, closing down... Timeout while waiting for a reply from the bitbake server It hangs at process.join(), according to: https://docs.python.org/3.7/library/multiprocessing.html Cleanup the queue before call process.join() can fix the problem. (Bitbake rev: 3eddfadd19b2ce4c061861abf0c340e3825b41ff) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

bitbake: hashserv: Switch from threads to multiprocessing

2019-08-06T10:21:32+00:00

There were hard to debug lockups when trying to use threading to start hashserv as a thread. Switch to multiprocessing which doesn't show the same locking problems. (Bitbake rev: be23d887c8e244f1ef961298fbc9214d0fd0968a) Signed-off-by: Richard Purdie

bitbake: cooker/hashserv: Allow autostarting of a local hash server using BB_HASHSERVE

2019-08-06T10:21:31+00:00

Its useful, particularly in the local developer model of usage, for bitbake to start and stop a hash equivalence server on local port, rather than relying on one being started by the user before the build. The new BB_HASHSERVE variable supports this. The database handling is moved internally into the hashserv code so that different threads/processes can be used for the server without errors. (Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587) Signed-off-by: Richard Purdie