linux/poky.git/bitbake/lib/bb/runqueue.py, branch zeus-22.0.0

bitbake: runqueue: Small performance optimisation

2019-09-30T16:23:35+00:00

A minor performance optmisation to keep lists smaller when running large builds. We can do this since once a task has been built, we don't need to worry about it. This improves a major bottleneck that shows up on performance profile charts in dryruns. (Bitbake rev: cd6b89230823707c3c9bb9e6883bf5a971916581) Signed-off-by: Richard Purdie

bitbake: runqueue: Save unihashes more frequently

2019-09-30T16:23:35+00:00

There are some runqueue code paths where the unihash cache would not be saved where for example only parsing or an occurred. Save the cache at the end of runqueue generation to ensure entries are cached. (Bitbake rev: 9eee0d36870c11dd303894a6151c33a83bd3a1bc) Signed-off-by: Richard Purdie

bitbake: siggen/runqueue: Fix signature mismatch issues

2019-09-30T16:23:35+00:00

We need to set the setscene tasklist before we call into the taskhash/unihash code else the behaviour is inconsistent. Avoid reporting hashes for non setscene tasks since we'd never query that. (Bitbake rev: 419a7840b8627278db694029c25df00214d01d96) Signed-off-by: Richard Purdie

bitbake: runqueue: Change task migration behaviour for rerunning setscene tasks

2019-09-30T16:23:35+00:00

Currently runqueue will rerun setscene tasks multiple times as hashes change. This has caused numerous problems since a setscene task may become "unavailable" for some future signature combination and the code then can't easily "unskip" tasks its already passed into the execution queue. At least for now, only run setscene once and assume they're equivalent at that point. In practise that has been much more stable in testing. Tweak the test to match the change in behaviour. (Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6) Signed-off-by: Richard Purdie

bitbake: runqueue: Fix task migration problems

2019-09-30T16:23:35+00:00

Tasks were not migrating consistently, particularly: * if a task was rehashed which had already run * if a task which was valid became invalid due to a rehash We need to always run the migration code for rehashed tasks and then reprocess them for hash validity. This means rearranging the code. It also means several tests are no longer correct and can't be written correctly to work on all possible workflows so those are removed. (Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d) Signed-off-by: Richard Purdie

bitbake: runqueue: validate_hashes(): currentcount should be a number

2019-09-19T19:30:35+00:00

According to sstate_checkhashes which is defined in sstate.bbclass, the currentcoun should be a number (0, not None). Fixed: $ bitbake base-files -Sprintdiff > bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete)) TypeError: %d format: a number is required, not NoneType (Bitbake rev: 45cb73e2846eaffe8964a573875f54808e8f3633) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

bitbake: bitbake: Rework hash equivalence

2019-09-18T16:52:01+00:00

Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue/siggen: Optimise hash equiv queries

2019-09-16T22:02:45+00:00

We only have hash equivalence for setscene tasks so only query the server for those, reducing the number of connections needed. (Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db) Signed-off-by: Richard Purdie

bitbake: bitbake: respect force flag in runall and runonly

2019-08-28T15:44:09+00:00

Specifying the force flag will now cause runall and runonly to invalidate the tasks before running them. This allows a --runall or --runonly to force the tasks to run, even if they would have otherwise been skipped, e.g.: bitbake -f --runall fetch Will run all do_fetch tasks even if they wouldn't be necessary (for example, skipped by setscene) (Bitbake rev: 71e52d3822016027106f2a2e74b8dfdf20f5dc1e) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue: Optimise build_taskdepdata slightly

2019-08-21T14:30:13+00:00

Rather than repeatedly calling mc_from_tid() do this in the parent, removing around a million function calls. Takes time spent in this function from 40s to 36s. (Bitbake rev: 28b3f0d8867804799420689c314ac4a8f01efb8c) Signed-off-by: Richard Purdie