linux/poky.git/bitbake/lib/hashserv/init.py, branch yocto-5.0.3

bitbake: asyncrpc: include parse_address from hashserv

2024-04-14T05:31:45+00:00

Moving the code and related definitions from hashserv/__init__.py to asyncrpc/client.py, allowing this function to be used in other asyncrpc clients. (Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3) Signed-off-by: Michael Opdenacker Suggested-by: Joshua Watt Cc: Tim Orling Cc: Thomas Petazzoni Signed-off-by: Richard Purdie

bitbake: hashserv: Add user permissions

2023-11-09T17:33:03+00:00

Adds support for the hashserver to have per-user permissions. User management is done via a new "auth" RPC API where a client can authenticate itself with the server using a randomly generated token. The user can then be given permissions to read, report, manage the database, or manage other users. In addition to explicit user logins, the server supports anonymous users which is what all users start as before they make the "auth" RPC call. Anonymous users can be assigned a set of permissions by the server, making it unnecessary for users to authenticate to use the server. The set of Anonymous permissions defines the default behavior of the server, for example if set to "@read", Anonymous users are unable to report equivalent hashes with authenticating. Similarly, setting the Anonymous permissions to "@none" would require authentication for users to perform any action. User creation and management is entirely manual (although bitbake-hashclient is very useful as a front end). There are many different mechanisms that could be implemented to allow user self-registration (e.g. OAuth, LDAP, etc.), and implementing these is outside the scope of the server. Instead, it is recommended to implement a registration service that validates users against the necessary service, then adds them as a user in the hash equivalence server. (Bitbake rev: 69e5417413ee2414fffaa7dd38057573bac56e35) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add SQLalchemy backend

2023-11-09T17:33:02+00:00

Adds an SQLAlchemy backend to the server. While this database backend is slower than the more direct sqlite backend, it easily supports just about any SQL server, which is useful for large scale deployments. (Bitbake rev: e0b73466dd7478c77c82f46879246c1b68b228c0) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Abstract database

2023-11-09T17:33:02+00:00

Abstracts the way the database backend is accessed by the hash equivalence server to make it possible to use other backends (Bitbake rev: 04b53deacf857488408bc82b9890b1e19874b5f1) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Add websocket connection implementation

2023-11-09T17:33:02+00:00

Adds support to the hash equivalence client and server to communicate over websockets. Since websockets are message orientated instead of stream orientated, and new connection class is needed to handle them. Note that websocket support does require the 3rd party websockets python module be installed on the host, but it should not be required unless websockets are actually being used. (Bitbake rev: 56dd2fdbfb6350a9eef43a12aa529c8637887a7e) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: asyncrpc: Abstract sockets

2023-11-09T17:33:02+00:00

Rewrites the asyncrpc client and server code to make it possible to have other transport backends that are not stream based (e.g. websockets which are message based). The connection handling classes are now shared between both the client and server to make it easier to implement new transport mechanisms (Bitbake rev: 2aaeae53696e4c2f13a169830c3b7089cbad6eca) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Fix diverging report race condition

2021-10-11T10:00:06+00:00

Fixes the hashequivalence server to resolve the diverging report race error. This error occurs when the same task(hash) is run simultaneous on two different builders, and then the results are reported back but the hashes diverge (e.g. have different outhashes), and one outhash is equivalent to a hash and another is not. If taskhash was not originally in the database, the client will fallback to using the taskhash as the suggested unihash and the server will see reports come in like: taskhash: A unihash: A outhash: B taskhash: C unihash: C outhash: B taskhash: C unihash: C outhash: D Note that the second and third reports are the same taskhash, with diverging outhashes. Taskhash C should be equivalent to taskhash (and unihash) A because they share an outhash B, but the server would not do this when tasks were reported in the order shown. It became clear while trying to fix this that single large table to store all reported hashes was going to make these updates difficult since updating the unihash of all entries would be complex and time consuming. Instead, it makes more sense to split apart the database into two tables: One that maps taskhashes to unihashes and one that maps outhashes to taskhashes. This should hopefully improve the parsing query times as well since they only care about the taskhashes to unihashes table, at the cost of more complex INNER JOIN queries on the lesser used API. Note this change does delete existing hash equivlance data and starts a new database table rather than converting existing data. (Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: hashserv: Support read-only server

2021-02-10T23:48:16+00:00

The -r/--readonly argument is added to the bitbake-hashserv app. If this argument is given then clients may only perform read operations against the server. The read-only mode is implemented by simply not installing handlers for write operations, this keeps the permission model simple and reduces the risk of accidentally allowing write operations. As a sqlite database can be safely opened by multiple processes in parallel, it's possible to start two hashserv instances against a single database if you wish to export both a read-only port and a read-write port. (Bitbake rev: 492bb02eb0e071c792407ac3113f92492da1a9cc) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: bitbake: hashserve: Add support for readonly upstream

2020-11-24T15:26:12+00:00

Adds support for an upstream server to be specified. The upstream server will be queried for equivalent hashes whenever a miss is found in the local server. If the server returns a match, it is merged into the local database. In order to keep the get stream queries as fast as possible since they are the critical path when bitbake is preparing the run queue, missing tasks provided by the server are not immediately pulled from the upstream server, but instead are put into a queue to be backfilled by a worker task later. (Bitbake rev: e6d6c0b39393e9bdf378c1eba141f815e26b724b) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: hashserve: Add async client

2020-11-24T15:26:12+00:00

Adds support for create a client that operates using Python asynchronous I/O. (Bitbake rev: cf9bc0310b0092bf52b61057405aeb51c86ba137) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

linux/poky.git/bitbake/lib/hashserv/__init__.py, branch yocto-5.0.3

bitbake: asyncrpc: include parse_address from hashserv

bitbake: hashserv: Add user permissions

bitbake: hashserv: Add SQLalchemy backend

bitbake: hashserv: Abstract database

bitbake: hashserv: Add websocket connection implementation

bitbake: asyncrpc: Abstract sockets

bitbake: hashserv: Fix diverging report race condition

bitbake: hashserv: Support read-only server

bitbake: bitbake: hashserve: Add support for readonly upstream

bitbake: bitbake: hashserve: Add async client

linux/poky.git/bitbake/lib/hashserv/init.py, branch yocto-5.0.3