linux/poky.git/bitbake/lib/hashserv, branch master-next

bitbake: Use a "fork" multiprocessing context

2025-07-23T10:34:31+00:00

Python 3.14 changes the default multiprocessing context from "fork" to "forkserver"; however bitbake heavily relies on "fork" to efficiently pass data to the child processes. As such, make "fork" context in the bb namespace and use it in place of the normal multiprocessing module. Note that multiprocessing contexts were added in Python 3.4, so this should be safe to use even before Python 3.14 [YOCTO #15858] (Bitbake rev: 62be9113d98fccb347c6aa0a10d5c4ee2857f8b6) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Fix deprecation warning about sqlite adapter

2025-05-19T16:55:50+00:00

The default adapters for sqlite datetime are deprecated as of Python 3.12, so implement our own. [YOCTO #15333] (Bitbake rev: 38a1d715bf58acbc9cb21eed413b3542c81cf15a) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add `gc-mark-stream` command for batch hash marking

2025-03-13T16:52:44+00:00

Implements the `gc-mark-stream` command to allow for marking equivalence entries in batch, by making use of stream mode communication to the server. The aim of this is to improve efficiency by reducing the impact of latency when marking a high volume of hash entries. Example usage of the new `gc-mark-stream` command: ``` $ cat << HASHES | \ ./bin/bitbake-hashclient --address "ws://localhost:8688/ws" gc-mark-stream "alive" unihash f37918cc02eb5a520b1aff86faacbc0a38124646 unihash af36b199320e611fbb16f1f277d3ee1d619ca58b taskhash a1117c1f5a7c9ab2f5a39cc6fe5e6152169d09c0 method oe.sstatesig.OEOuthashBasic HASHES ``` (Bitbake rev: c84715f28cd36666ea07a179d91b8c32ea0df8e7) Signed-off-by: Alexander Marques Signed-off-by: Richard Purdie

bitbake: hashserv: client: Fix changing stream modes

2024-05-31T15:56:25+00:00

When switching from normal mode to stream mode, skip calling self._set_mode() again because this will cause a recursion into the _set_mode() function and causes problems. Also cleanup some of the error checking during this process This bug affected when a client would attempt to switch from one stream mode to another, and meant that the server would get an invalid message from the client. This would cause the server to disconnect the client, and the client would then reconnect in normal mode which was the mode it wanted anyway and thus it would carry on without any errors. This made the bug not visible on the client side, but resulting in a lot of backtrace JSON decoding exceptions in the server logs. (Bitbake rev: 1826bc41ab3369ac40034c5eaf698748b769b881) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: asyncrpc: Remove ClientPool

2024-05-31T15:56:25+00:00

Batching support on the client side has proven to be a much more effective way of dealing with server latency than multiple client connections and is also much nicer on the server, so drop the client pool support from asyncrpc and the hash server (Bitbake rev: 6f80560f1c7010d09fe5448fdde616aef8468102) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: Drop older python version compatibility code

2024-05-31T15:56:25+00:00

cooker: We can call multiprocessing close() unconditionally and tweak a comment give 3.8 is now the minimum version. lib/bb: We can drop the logger addition code only needed before 3.6 asyncrpc/hashserv: Since the minimum version is 3.8, we can drop the conditional code. (Bitbake rev: 16f4386400f88ba50605307961c248bef09895c1) Signed-off-by: Richard Purdie

bitbake: hashserv: server: Add support for SO_REUSEPORT

2024-05-30T06:38:10+00:00

SO_REUSEPORT is a socket option that allows multiple servers to listen on the same TCP port, and the kernel will automatically load balance the connections between them. This is particularly helpful for the hash server since it runs in a single thread. To take advantage of a multi-core server, multiple servers can be started in parallel with this option (up to 1 per CPU) and the kernel will load balance between them. (Bitbake rev: d72d5a7decb489e2af0ebc43cfea0ca3e4353e9b) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: client: Add batch stream API

2024-05-30T06:38:10+00:00

Changes the stream mode to do "batch" processing. This means that the sending and reciving of messages is done simultaneously so that messages can be sent as fast as possible without having to wait for each reply. This allows multiple messages to be in flight at once, reducing the effect of the round trip latency from the server. (Bitbake rev: e768d0f17bdb97f6ff013ec3a41f182fecd47a55) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: client: Fix mode state errors

2024-04-16T06:33:19+00:00

Careful reading of the code can contrive cases where poorly timed ConnectionError's will result in the client mode being incorrectly reset to MODE_NORMAL when it should actual be a stream mode for the current command. Fix this by no longer attempting to restore the mode when the connection is setup. Instead, attempt to set the stream mode inside the send wrapper for the stream data, which means that it should always end up in the correct mode before continuing. Also, factor out the transition to normal mode into a invoke() override so it doesn't need to be specified over and over again. (Bitbake rev: 0cd276fd98eeca463518d4a42675fffb18d6b3de) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: asyncrpc: include parse_address from hashserv

2024-04-14T05:31:45+00:00

Moving the code and related definitions from hashserv/__init__.py to asyncrpc/client.py, allowing this function to be used in other asyncrpc clients. (Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3) Signed-off-by: Michael Opdenacker Suggested-by: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni Signed-off-by: Richard Purdie