linux/poky.git/bitbake/lib/hashserv/client.py, branch walnascar-5.2

bitbake: hashserv: Add `gc-mark-stream` command for batch hash marking

2025-03-13T16:52:44+00:00

Implements the `gc-mark-stream` command to allow for marking equivalence entries in batch, by making use of stream mode communication to the server. The aim of this is to improve efficiency by reducing the impact of latency when marking a high volume of hash entries. Example usage of the new `gc-mark-stream` command: ``` $ cat << HASHES | \ ./bin/bitbake-hashclient --address "ws://localhost:8688/ws" gc-mark-stream "alive" unihash f37918cc02eb5a520b1aff86faacbc0a38124646 unihash af36b199320e611fbb16f1f277d3ee1d619ca58b taskhash a1117c1f5a7c9ab2f5a39cc6fe5e6152169d09c0 method oe.sstatesig.OEOuthashBasic HASHES ``` (Bitbake rev: c84715f28cd36666ea07a179d91b8c32ea0df8e7) Signed-off-by: Alexander Marques Signed-off-by: Richard Purdie

bitbake: hashserv: client: Fix changing stream modes

2024-05-31T15:56:25+00:00

When switching from normal mode to stream mode, skip calling self._set_mode() again because this will cause a recursion into the _set_mode() function and causes problems. Also cleanup some of the error checking during this process This bug affected when a client would attempt to switch from one stream mode to another, and meant that the server would get an invalid message from the client. This would cause the server to disconnect the client, and the client would then reconnect in normal mode which was the mode it wanted anyway and thus it would carry on without any errors. This made the bug not visible on the client side, but resulting in a lot of backtrace JSON decoding exceptions in the server logs. (Bitbake rev: 1826bc41ab3369ac40034c5eaf698748b769b881) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: asyncrpc: Remove ClientPool

2024-05-31T15:56:25+00:00

Batching support on the client side has proven to be a much more effective way of dealing with server latency than multiple client connections and is also much nicer on the server, so drop the client pool support from asyncrpc and the hash server (Bitbake rev: 6f80560f1c7010d09fe5448fdde616aef8468102) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: client: Add batch stream API

2024-05-30T06:38:10+00:00

Changes the stream mode to do "batch" processing. This means that the sending and reciving of messages is done simultaneously so that messages can be sent as fast as possible without having to wait for each reply. This allows multiple messages to be in flight at once, reducing the effect of the round trip latency from the server. (Bitbake rev: e768d0f17bdb97f6ff013ec3a41f182fecd47a55) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: client: Fix mode state errors

2024-04-16T06:33:19+00:00

Careful reading of the code can contrive cases where poorly timed ConnectionError's will result in the client mode being incorrectly reset to MODE_NORMAL when it should actual be a stream mode for the current command. Fix this by no longer attempting to restore the mode when the connection is setup. Instead, attempt to set the stream mode inside the send wrapper for the stream data, which means that it should always end up in the correct mode before continuing. Also, factor out the transition to normal mode into a invoke() override so it doesn't need to be specified over and over again. (Bitbake rev: 0cd276fd98eeca463518d4a42675fffb18d6b3de) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add Client Pool

2024-02-19T11:58:12+00:00

Implements a Client Pool derived from the AsyncRPC client pool that allows querying for multiple equivalent hashes in parallel (Bitbake rev: ba4c764d8061c7b88cd4985ca493d6ea6e317106) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add unihash-exists API

2024-02-19T11:58:12+00:00

Adds API to check if the server is aware of the existence of a given unihash. This can be used as an optimization for sstate where a client can query the hash equivalence server to check if a unihash exists before querying the sstate cache. If the hash server isn't aware of the existence of a unihash, then there is very likely not a matching sstate object, so this should be able to significantly cut down on the number of negative hits on the sstate cache. (Bitbake rev: cfe0ac071cfb998e4a1dd263f8860b140843361a) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add Unihash Garbage Collection

2024-02-19T11:58:12+00:00

Adds support for removing unused unihashes from the database. This is done using a "mark and sweep" style of garbage collection where a collection is started by marking which unihashes should be kept in the database, then performing a sweep to remove any unmarked hashes. (Bitbake rev: 433d4a075a1acfbd2a2913061739353a84bb01ed) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add database column query API

2023-11-09T17:33:03+00:00

Adds an API to retrieve the columns that can be queried on from the database backend. This prevents front end applications from needing to hardcode the query columns (Bitbake rev: abfce2b68bdab02ea2e9a63fbb3b9e270428a0a6) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add db-usage API

2023-11-09T17:33:03+00:00

Adds an API to query the server for the usage of the database (e.g. how many rows are present in each table) (Bitbake rev: c9c1224447e147e0de92953bc85cea75670b898c) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie