linux/poky.git/bitbake/lib/hashserv/client.py, branch 4.2_M1

bitbake: hashserv: Fix diverging report race condition

2021-10-11T10:00:06+00:00

Fixes the hashequivalence server to resolve the diverging report race error. This error occurs when the same task(hash) is run simultaneous on two different builders, and then the results are reported back but the hashes diverge (e.g. have different outhashes), and one outhash is equivalent to a hash and another is not. If taskhash was not originally in the database, the client will fallback to using the taskhash as the suggested unihash and the server will see reports come in like: taskhash: A unihash: A outhash: B taskhash: C unihash: C outhash: B taskhash: C unihash: C outhash: D Note that the second and third reports are the same taskhash, with diverging outhashes. Taskhash C should be equivalent to taskhash (and unihash) A because they share an outhash B, but the server would not do this when tasks were reported in the order shown. It became clear while trying to fix this that single large table to store all reported hashes was going to make these updates difficult since updating the unihash of all entries would be complex and time consuming. Instead, it makes more sense to split apart the database into two tables: One that maps taskhashes to unihashes and one that maps outhashes to taskhashes. This should hopefully improve the parsing query times as well since they only care about the taskhashes to unihashes table, at the cost of more complex INNER JOIN queries on the lesser used API. Note this change does delete existing hash equivlance data and starts a new database table rather than converting existing data. (Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: async: Close sync client event loop

2021-10-11T10:00:06+00:00

Prevents `ResourceWarning: unclosed event loop` warnings when using the synchronous client and python exits (Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserver/client.py: drop unused imports

2021-05-14T06:57:57+00:00

(Bitbake rev: 9dc77266085c605b108641a9d76ac4dbdc064c34) Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: hashserv: Refactor to use asyncrpc

2021-04-27T14:12:57+00:00

The asyncrpc module can now be used to provide the json & asyncio based RPC system used by hashserv. (Bitbake rev: 5afb9586b0a4a23a05efb0e8ff4a97262631ae4a) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: Use generic ConnectionError

2021-04-27T14:12:57+00:00

The Python built-in ConnectionError type can be used instead of a custom HashConnectionError type. This will make code refactoring simpler. (Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: Add get-outhash message

2021-02-10T23:48:16+00:00

The get-outhash message can be sent via the get_outhash client method. This works in a similar way to the get message but looks up a db entry by outhash rather than by taskhash. It is intended to be used as a read-only form of the report message. As both handle_get_outhash and handle_report use the same query string we can factor this out. (Bitbake rev: dc19606ada29a4d8afde4fcecd8ec986b47b867e) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: client: Fix handling of null responses

2021-02-06T09:12:00+00:00

If the server returns an empty response ("null" in json), this cannot be iterated to check for the presence of the "chunk-stream" key. (Bitbake rev: bf75370bcd6d02ed08cd959eec6190196b792515) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: Fix broken AF_UNIX path length limit

2020-12-10T13:39:15+00:00

Fixes the bug were long paths would break Unix domain socket clients (for real this time; the previous attempt was missing os.path.basename). Adds some tests to prevent regressions (Bitbake rev: 77790e3656048eff5cb1a086c727d86d32773b68) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: client: Fix AF_UNIX path length limits

2020-12-09T12:27:25+00:00

Restores a fix for unix domain socket path length limits when using the synchronous hash equivalence client that was accidentally removed when the async client was added. Unfortunately, it's much more difficult to fix the same problem when using the async client directly due to the interaction of chdir() and async code, but this will at least restore the old behavior in the synchronous case. (Bitbake rev: 53e85022a8b1c8f407c9418260c59beffb96f0f9) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: hashserve: Add support for readonly upstream

2020-11-24T15:26:12+00:00

Adds support for an upstream server to be specified. The upstream server will be queried for equivalent hashes whenever a miss is found in the local server. If the server returns a match, it is merged into the local database. In order to keep the get stream queries as fast as possible since they are the critical path when bitbake is preparing the run queue, missing tasks provided by the server are not immediately pulled from the upstream server, but instead are put into a queue to be backfilled by a worker task later. (Bitbake rev: e6d6c0b39393e9bdf378c1eba141f815e26b724b) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie