linux/poky.git/bitbake/lib/hashserv, branch kirkstone-4.0

bitbake: hashserv: Improve behaviour for better determinism/sstate reuse

2021-10-11T10:00:06+00:00

We have a choice of policy with hashequivalence - whether to reduce sstate duplication in the sstate feed to a minimum or have maximal sstate reuse from the user's perspective. The challenge is that non-matching outhashes are generated due to determinism issues, or due to differences in host gcc version, architecture and so on and the question is how to reconcile then. The approach before this patch is that any new match is added and matches can update. This has the side effect that a queried value from the server can change due to the replacement and you may not always get the same value from the server. With the client side caching bitbake has, this can be suboptimal and when using the autobuilder sstate feed, it results in poor artefact reuse. This patch switches to the other possible behaviour, once a hash is assigned, it doesn't change. This means some sstate artefacts may be duplicated but dependency chains aren't invalidated which I suspect may give better overall performance. Update the tests to match the new behaviour. (Bitbake rev: 20d6ac753efa364349100cdc863e5eabec8e5b78) Signed-off-by: Richard Purdie

bitbake: hashserv: Fix diverging report race condition

2021-10-11T10:00:06+00:00

Fixes the hashequivalence server to resolve the diverging report race error. This error occurs when the same task(hash) is run simultaneous on two different builders, and then the results are reported back but the hashes diverge (e.g. have different outhashes), and one outhash is equivalent to a hash and another is not. If taskhash was not originally in the database, the client will fallback to using the taskhash as the suggested unihash and the server will see reports come in like: taskhash: A unihash: A outhash: B taskhash: C unihash: C outhash: B taskhash: C unihash: C outhash: D Note that the second and third reports are the same taskhash, with diverging outhashes. Taskhash C should be equivalent to taskhash (and unihash) A because they share an outhash B, but the server would not do this when tasks were reported in the order shown. It became clear while trying to fix this that single large table to store all reported hashes was going to make these updates difficult since updating the unihash of all entries would be complex and time consuming. Instead, it makes more sense to split apart the database into two tables: One that maps taskhashes to unihashes and one that maps outhashes to taskhashes. This should hopefully improve the parsing query times as well since they only care about the taskhashes to unihashes table, at the cost of more complex INNER JOIN queries on the lesser used API. Note this change does delete existing hash equivlance data and starts a new database table rather than converting existing data. (Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add tests for diverging reports

2021-10-11T10:00:06+00:00

(Bitbake rev: 953c8d622c9d1bc1eb06bcaf1eaa3aa9f85d0bc2) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: async: Close sync client event loop

2021-10-11T10:00:06+00:00

Prevents `ResourceWarning: unclosed event loop` warnings when using the synchronous client and python exits (Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: asyncrpc: Defer all asyncio to child process

2021-08-23T07:30:54+00:00

Reworks the async I/O API so that the async loop is only created in the child process. This requires deferring the creation of the server until the child process and a queue to transfer the bound address back to the parent process (Bitbake rev: 8555869cde39f9e9a9ced5a3e5788209640f6d50) Signed-off-by: Joshua Watt [small loop -> self.loop fix in serv.py] Signed-off-by: Scott Murray Signed-off-by: Richard Purdie

bitbake: bitbake: asyncrpc: Catch early SIGTERM

2021-07-29T22:21:24+00:00

If the SIGTERM signal is sent to an asyncrpc server before it has installed the SIGTERM handler in the main loop, it may miss the signal which will can cause the calling process to wait forever on the join(). To resolve this, the calling process should mask of SIGTERM before forking the server process and the server should unmask the signal only after the handler is installed. To simplify the usage of the server, an new helper function called serve_as_process() is added to do this automatically and correctly. Thanks: Scott Murray for helping debug (Bitbake rev: ef2865efa98ad20823267364f2159d8d8c931400) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserver/client.py: drop unused imports

2021-05-14T06:57:57+00:00

(Bitbake rev: 9dc77266085c605b108641a9d76ac4dbdc064c34) Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: hashserv/server.py: drop unused imports

2021-05-14T06:57:57+00:00

remove unused vars. (Bitbake rev: 3287d28a506f67abd192799e61ef28e74ce7002d) Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: hashserv: Refactor to use asyncrpc

2021-04-27T14:12:57+00:00

The asyncrpc module can now be used to provide the json & asyncio based RPC system used by hashserv. (Bitbake rev: 5afb9586b0a4a23a05efb0e8ff4a97262631ae4a) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: Use generic ConnectionError

2021-04-27T14:12:57+00:00

The Python built-in ConnectionError type can be used instead of a custom HashConnectionError type. This will make code refactoring simpler. (Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie