linux/poky.git/bitbake/lib/hashserv/tests.py, branch 4.2_M1

bitbake: hashserv: Improve behaviour for better determinism/sstate reuse

2021-10-11T10:00:06+00:00

We have a choice of policy with hashequivalence - whether to reduce sstate duplication in the sstate feed to a minimum or have maximal sstate reuse from the user's perspective. The challenge is that non-matching outhashes are generated due to determinism issues, or due to differences in host gcc version, architecture and so on and the question is how to reconcile then. The approach before this patch is that any new match is added and matches can update. This has the side effect that a queried value from the server can change due to the replacement and you may not always get the same value from the server. With the client side caching bitbake has, this can be suboptimal and when using the autobuilder sstate feed, it results in poor artefact reuse. This patch switches to the other possible behaviour, once a hash is assigned, it doesn't change. This means some sstate artefacts may be duplicated but dependency chains aren't invalidated which I suspect may give better overall performance. Update the tests to match the new behaviour. (Bitbake rev: 20d6ac753efa364349100cdc863e5eabec8e5b78) Signed-off-by: Richard Purdie

bitbake: hashserv: Fix diverging report race condition

2021-10-11T10:00:06+00:00

Fixes the hashequivalence server to resolve the diverging report race error. This error occurs when the same task(hash) is run simultaneous on two different builders, and then the results are reported back but the hashes diverge (e.g. have different outhashes), and one outhash is equivalent to a hash and another is not. If taskhash was not originally in the database, the client will fallback to using the taskhash as the suggested unihash and the server will see reports come in like: taskhash: A unihash: A outhash: B taskhash: C unihash: C outhash: B taskhash: C unihash: C outhash: D Note that the second and third reports are the same taskhash, with diverging outhashes. Taskhash C should be equivalent to taskhash (and unihash) A because they share an outhash B, but the server would not do this when tasks were reported in the order shown. It became clear while trying to fix this that single large table to store all reported hashes was going to make these updates difficult since updating the unihash of all entries would be complex and time consuming. Instead, it makes more sense to split apart the database into two tables: One that maps taskhashes to unihashes and one that maps outhashes to taskhashes. This should hopefully improve the parsing query times as well since they only care about the taskhashes to unihashes table, at the cost of more complex INNER JOIN queries on the lesser used API. Note this change does delete existing hash equivlance data and starts a new database table rather than converting existing data. (Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add tests for diverging reports

2021-10-11T10:00:06+00:00

(Bitbake rev: 953c8d622c9d1bc1eb06bcaf1eaa3aa9f85d0bc2) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: asyncrpc: Catch early SIGTERM

2021-07-29T22:21:24+00:00

If the SIGTERM signal is sent to an asyncrpc server before it has installed the SIGTERM handler in the main loop, it may miss the signal which will can cause the calling process to wait forever on the join(). To resolve this, the calling process should mask of SIGTERM before forking the server process and the server should unmask the signal only after the handler is installed. To simplify the usage of the server, an new helper function called serve_as_process() is added to do this automatically and correctly. Thanks: Scott Murray for helping debug (Bitbake rev: ef2865efa98ad20823267364f2159d8d8c931400) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Use generic ConnectionError

2021-04-27T14:12:57+00:00

The Python built-in ConnectionError type can be used instead of a custom HashConnectionError type. This will make code refactoring simpler. (Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: server: Support searching upstream for outhash

2021-02-10T23:48:16+00:00

Use the new get-outhash message to perform a read-only query against an upstream server (if present) when a reported taskhash/outhash combination is not found in the current database. If a matching entry is found upstream it is copied into the current database so it can be found by future queries. (Bitbake rev: 2be4f7f0d2ccb09917398289e8140e1467e84bb2) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: Support read-only server

2021-02-10T23:48:16+00:00

The -r/--readonly argument is added to the bitbake-hashserv app. If this argument is given then clients may only perform read operations against the server. The read-only mode is implemented by simply not installing handlers for write operations, this keeps the permission model simple and reduces the risk of accidentally allowing write operations. As a sqlite database can be safely opened by multiple processes in parallel, it's possible to start two hashserv instances against a single database if you wish to export both a read-only port and a read-write port. (Bitbake rev: 492bb02eb0e071c792407ac3113f92492da1a9cc) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: hashserv: Fix broken AF_UNIX path length limit

2020-12-10T13:39:15+00:00

Fixes the bug were long paths would break Unix domain socket clients (for real this time; the previous attempt was missing os.path.basename). Adds some tests to prevent regressions (Bitbake rev: 77790e3656048eff5cb1a086c727d86d32773b68) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: hashserve: Add support for readonly upstream

2020-11-24T15:26:12+00:00

Adds support for an upstream server to be specified. The upstream server will be queried for equivalent hashes whenever a miss is found in the local server. If the server returns a match, it is merged into the local database. In order to keep the get stream queries as fast as possible since they are the critical path when bitbake is preparing the run queue, missing tasks provided by the server are not immediately pulled from the upstream server, but instead are put into a queue to be backfilled by a worker task later. (Bitbake rev: e6d6c0b39393e9bdf378c1eba141f815e26b724b) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: hashserv: Fix localhost sometimes resolved to a wrong IP

2020-09-23T19:55:53+00:00

Using localhost for direct builds on host is fine. A case with a misbehavior has been sighted on a Docker build. Even when the host supports IPv6, but Docker is not configured correspondingly - some versions of the asyncio Python module seem to misbehave and try to use IPv6 where it's not supported in the container. This happens at least on some Ubuntu 18.04 based containers, resolving the IP explicitly appears to be the fix. (Bitbake rev: 0e20f91c11afdc17ea776aa02e0cc8b0d59a23d4) Signed-off-by: Anatol Belski Signed-off-by: Richard Purdie