linux/poky.git/bitbake/lib/bb/tests, branch 2.8_M3

bitbake: bitbake: Rework hash equivalence

2019-09-18T16:52:01+00:00

Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: tests/fetch: Resolve fetch error in bitbake-selftest

2019-09-01T21:30:42+00:00

FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) AssertionError: '' is not true : Could not find upstream version for db [YOCTO #13496] The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget. Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the recipe. Also change the version being checked. (Bitbake rev: 4cf5bb761c561ddea86f2875be35d05abc8486e1) Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: tests/runqueue: Fix tests

2019-08-15T07:43:22+00:00

There were paths being accidentally included in some of the hashserv tests. Remove that and update the hashes so the tests work independently of paths. (Bitbake rev: 6ddb9f09cb60c2354fa6a67cce412c4dc1e7dc2d) Signed-off-by: Richard Purdie

bitbake: runqueue: Fix event timing race

2019-08-14T16:28:23+00:00

The event from the task notifiing of hash equivalency should only be processed when the task completes. This can otherwise result in a race where a dependent task may run before the original task completes causing various failures. To make this work reliably, the code had to be restructured quite a bit. (Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259) Signed-off-by: Richard Purdie

bitbake: tests/runqueue: Add further hash equivalence tests

2019-08-14T16:28:23+00:00

Add some extra hash equivalence runqueue tests based on recent scenarios that caused problems during testing. (Bitbake rev: 373b085ead992a725b2230ededd992b4c61a1a05) Signed-off-by: Richard Purdie

bitbake: tests/runqueue: Add hashserv+runqueue test

2019-08-06T10:21:32+00:00

Add a test which tests the runqueue adaptations for hash equivalency. (Bitbake rev: 477321d0780df177c1582db119c2bb6795912fc6) Signed-off-by: Richard Purdie

bitbake: siggen: Clean up task reference formats

2019-08-06T10:21:32+00:00

Currently siggen uses the format "." for referencing tasks whilst runqueue uses ":". This converts to use ":" as the separator everywhere. This is an API breaking change since the cache is affected, as are siginfo files and any custom signature handlers such as those in OE-Core. Ultimately this will let us clean up and the accessor functions from runqueue, removing all the ".rsplit(".", 1)[0]" type code currently all over the place. Once a standard is used everwhere we can update the code over time to be more optimal. (Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496) Signed-off-by: Richard Purdie

bitbake: runqueue: Clean up BB_HASHCHECK_FUNCTION API

2019-08-06T10:21:32+00:00

This function uses an old API which uses offsets into lists as a communication mechanism. Update the API to use "tid" which is used universally in runqueue now. We can also add kwargs support to the funciton definition to drop some of the backwards compaiblility hoops we had to jump though with different function argument combinations. (Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6) Signed-off-by: Richard Purdie

bitbake: runqueue: Fix non setscene tasks targets being lost

2019-07-18T11:16:19+00:00

If you specify both setscene and non-setscene tasks on the commandline, the non-setscene tasks could be missed, e.g. "bitbake X:do_patch X:do_populate_sysroot" and do_patch would fail to run. Fix the problem in runqueue and add a testcase. (Bitbake rev: 75292fdec5d9c0b5b3c554c4b7474a63656f7e12) Signed-off-by: Richard Purdie

bitbake: tests/runqueue: Allow common sstate tasks to become valid

2019-07-16T12:53:17+00:00

As the logic in bitbake improves, the logic in the tests needs to as well. Afer we built a task for the first time, allow its setscene hash verification status to change, mirroring what would happen in a multiconfig build. (Bitbake rev: 27ec2e69ab3e32972caf8b072b2945736696d83d) Signed-off-by: Richard Purdie