summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/runqueue.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Use a set for the setscene tasks listRichard Purdie2020-01-281-2/+2
| | | | | | | | | | | | | This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). (Bitbake rev: 1a026d4aca47ed1b0b74a8a818635e7520e8f9c8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f5daef68703481a3c243dfecc7de404e6ebfdbb6) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task dependency corner case in sanity testRichard Purdie2020-01-111-1/+4
| | | | | | | | | | | | | | A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly. (Bitbake rev: 39cc4930676970b2e51b99a778f5d5d9b5259991) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure task dependencies are run correctlyRichard Purdie2020-01-111-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run. This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption. Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures. There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified. (Bitbake rev: 182b2ae7bab4a943978154be3a65c018aaf21fdc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7a92b7f58ab187eddfe550bd6fb687240c7b11bb) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix equiv hash handling build failuresRichard Purdie2020-01-111-1/+2
| | | | | | | | | | | | | | | | Regardless of whether we remapped the hash on the server or not, we need to have bitbake work as if we did as we need to match how the stamp files look. This change resolves build failures where tasks were rerunning when they shouldn't. (Bitbake rev: 10058fe590c56ee3b4d2136b6e247c2d29ae47e6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 40928f6991436cf687821015324483b205abfcb1) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Only call into the migrations function if migrations activeRichard Purdie2020-01-111-1/+2
| | | | | | | | | | | This doesn't save much time but does make the profile counts for the function more accurate which is in itself useful. (Bitbake rev: 53a3cba93401c902d1d214cafe0bc036e1b101e5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d446fa89d206fbc6d098215163c968ea5a8cf4a9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task filteringRichard Purdie2020-01-111-3/+2
| | | | | | | | | | | We were seeing this running thousands of times with hashequiv, do the filtering where it makes more sense and make it persist. (Bitbake rev: 7fa3ede953971f761d8cafafa37d1bcb65ad6a77) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise out pointless loop iterationRichard Purdie2020-01-111-9/+3
| | | | | | | | (Bitbake rev: 14babbf0a32637db640e0494b0331397dcb0f715) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 105d1f0748edde7753a4063e6fdc758ffc8a8a9e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task migration code slightlyRichard Purdie2020-01-111-8/+9
| | | | | | | | | | | | | | | Move the calls to difference_update out a code level which improves efficiency significantly. Also further combine the outer loop for efficiency too. These two changes remove a bottleneck from the performance charts. (Bitbake rev: 57df7f191755dd887827d51b125d246c1af3e1b6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e28ec69356f1797de3e4e3fca0fef710bc4564de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sstate task iteration performanceRichard Purdie2020-01-111-1/+5
| | | | | | | | | | | | | Creating a new sorted list of sstate tasks each iteration through runqueue is extremely ineffecient and was compounded by the recent change from a list to set. Create one sorted list instead of recreating it each time. (Bitbake rev: a80024a4397485f96693c4df108b40f6b1bdf32b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit de18824996841c3f35f54ff5ad12f94f6dc20d88) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Split get_tashhash for performanceRichard Purdie2020-01-111-0/+1
| | | | | | | | | | | | | | | | | There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash. Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does. This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing. (Bitbake rev: c4ce216b4c5a1626764752edd21005750d05a0c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6a32af2808d748819f4af55c443578c8a63062b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task mismatch failures from incorrect logicRichard Purdie2020-01-111-1/+1
| | | | | | | | | | | | The "no dependencies" task case was not being correctly considered in this code and seemed to be the cause of occasionaly task hash mismatch errors that were being seen as the dependencies were never accounted for properly. (Bitbake rev: 480939538f5887a70d11c2f7cf74446100e25fab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 608b9f821539de813bfbd9e65950dbc56a274bc2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Rework process_possible_migrations() to improve performanceRichard Purdie2020-01-111-47/+56
| | | | | | | | | | | | | | | | | The looping over multiple changed hashes causes many calls to get_taskhash and get_unihash which are potentially slow and then overwritten. Instead, batch up all the tasks which have changed unihashes and then do one big loop over the changed tasks rather than each in turn. This makes worlds of difference to the performance graphs and should speed up build where many tasks are being rehashed. (Bitbake rev: c9ab598f6f1ea3ae3a0713dc6692b4c4bafbfb50) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c9c68d898985cf0bec6fc95f54c151cc50255cac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Batch scenequeue updatesJoshua Watt2020-01-111-4/+7
| | | | | | | | | | | | | | Batch all updates to scenequeue data together in a single invocation instead of checking each task serially. This allows the checks for sstate object to happen in parallel, and also makes sure the log statement only happens once (per set of rehashes). (Bitbake rev: a7426c73a8e9fae468414a2c32a533d9c3729405) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit db033a8f8a276d864bdb2e1eef159ab5794a0658) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add extra debugging when locked sigs mismatches occurRichard Purdie2020-01-111-0/+2
| | | | | | | | (Bitbake rev: 6f0b82627edc82601f80f0f096bf96db43afefa8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3aad9978be2a40d4c535a5ae092f374ba2a5f627) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Allow handling of equivalent hashesRichard Purdie2020-01-111-12/+19
| | | | | | | | | | | | | | | Based on the hashserv's new ability to accept hash mappings, update runqueue to use this through a helper function in siggen. This addresses problems with meta-extsdk-toolchain and its dependency on gdb-cross which caused errors when building eSDK. See the previous commit for more details. (Bitbake rev: 222df6d6b832868c6e87334f8acdd74b730a91d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: not show warning for deferred multiconfig taskKai Kang2019-11-291-1/+1
| | | | | | | | | | | | | | | | | | | When follow the instructions of multiconfig from Yocto dev manual that set in core-image-sato recipe: do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs" it show too many annoying warnings look like: | WARNING: Deferring mc:x86:virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot | after mc:arm: virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot Treat them as infomations rather than warnings. (Bitbake rev: cfa307aabf710d79c404a8571b4158b864a94727) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure failed harddependencies in scenequeue are ↵Richard Purdie2019-11-251-1/+8
| | | | | | | | | | | | | | | | | | accounted for in migrations Setscene hard dependencies were not being correctly handled during task migration. For example, do_package of recipe X might become valid due to hashequiv yet we're still rebuilding pseudo-native, a harddep of do_package. This would mean it would try to execute that setscene task despite pseudo not being present. Fix this by ignoring tasks with failed harddependencies. This does mean stlightly more rebuilds than is optimal but it avoids build crashes. Ultimately the new runqueue model can likely better handle these cases than the older codebase could but that is for another more invasive patch. (Bitbake rev: 9a1072060350dc2e0eee14a5cc5af20c900f8a6d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve sstate rehashing outputRichard Purdie2019-11-251-12/+16
| | | | | | | | | | | | | | Bibake is currently too 'chatty' when hash equivalence is enabled. Fix this by only printing the log output if a rehash happens and it matches an sstate object. Also, pass a summary option to the hash checking function. This was already changed to a mechanism which allows addition of new parameters so this should be backwards and forwards compatible. (Bitbake rev: c5c5d786ca968d0e48002fe8acbcc8a63a954b67) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix hash equivalence duplicate tasks runningRichard Purdie2019-11-251-4/+10
| | | | | | | | | | | | | | | | | The key problem is that currently running setscene tasks are not accounted for when processing task migrations. This means can allow two of the same task to execute at the same time with unpredictable effects. This change allows us to stop doing that and refactor the code slightly to make it clearer that these conditions don't arrive even with deferred tasks. (Bitbake rev: 33ffc2128b1a74fa7179a8341db68cddf402536f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Small performance optimisationRichard Purdie2019-09-301-0/+2
| | | | | | | | | | | A minor performance optmisation to keep lists smaller when running large builds. We can do this since once a task has been built, we don't need to worry about it. This improves a major bottleneck that shows up on performance profile charts in dryruns. (Bitbake rev: cd6b89230823707c3c9bb9e6883bf5a971916581) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Save unihashes more frequentlyRichard Purdie2019-09-301-0/+1
| | | | | | | | | | There are some runqueue code paths where the unihash cache would not be saved where for example only parsing or an occurred. Save the cache at the end of runqueue generation to ensure entries are cached. (Bitbake rev: 9eee0d36870c11dd303894a6151c33a83bd3a1bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue: Fix signature mismatch issuesRichard Purdie2019-09-301-1/+2
| | | | | | | | | | | | We need to set the setscene tasklist before we call into the taskhash/unihash code else the behaviour is inconsistent. Avoid reporting hashes for non setscene tasks since we'd never query that. (Bitbake rev: 419a7840b8627278db694029c25df00214d01d96) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change task migration behaviour for rerunning setscene tasksRichard Purdie2019-09-301-0/+4
| | | | | | | | | | | | | | | | | Currently runqueue will rerun setscene tasks multiple times as hashes change. This has caused numerous problems since a setscene task may become "unavailable" for some future signature combination and the code then can't easily "unskip" tasks its already passed into the execution queue. At least for now, only run setscene once and assume they're equivalent at that point. In practise that has been much more stable in testing. Tweak the test to match the change in behaviour. (Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task migration problemsRichard Purdie2019-09-301-44/+55
| | | | | | | | | | | | | | | | | Tasks were not migrating consistently, particularly: * if a task was rehashed which had already run * if a task which was valid became invalid due to a rehash We need to always run the migration code for rehashed tasks and then reprocess them for hash validity. This means rearranging the code. It also means several tests are no longer correct and can't be written correctly to work on all possible workflows so those are removed. (Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: validate_hashes(): currentcount should be a numberRobert Yang2019-09-191-3/+3
| | | | | | | | | | | | | | | | According to sstate_checkhashes which is defined in sstate.bbclass, the currentcoun should be a number (0, not None). Fixed: $ bitbake base-files -Sprintdiff > bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete)) TypeError: %d format: a number is required, not NoneType (Bitbake rev: 45cb73e2846eaffe8964a573875f54808e8f3633) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rework hash equivalenceJoshua Watt2019-09-181-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Optimise hash equiv queriesRichard Purdie2019-09-161-0/+1
| | | | | | | | | We only have hash equivalence for setscene tasks so only query the server for those, reducing the number of connections needed. (Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: respect force flag in runall and runonlyJoshua Watt2019-08-281-14/+18
| | | | | | | | | | | | | | | | | Specifying the force flag will now cause runall and runonly to invalidate the tasks before running them. This allows a --runall or --runonly to force the tasks to run, even if they would have otherwise been skipped, e.g.: bitbake -f --runall fetch Will run all do_fetch tasks even if they wouldn't be necessary (for example, skipped by setscene) (Bitbake rev: 71e52d3822016027106f2a2e74b8dfdf20f5dc1e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise build_taskdepdata slightlyRichard Purdie2019-08-211-6/+6
| | | | | | | | | | Rather than repeatedly calling mc_from_tid() do this in the parent, removing around a million function calls. Takes time spent in this function from 40s to 36s. (Bitbake rev: 28b3f0d8867804799420689c314ac4a8f01efb8c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Further optimise holdoff tasksRichard Purdie2019-08-211-24/+25
| | | | | | | | | | There are other data structures which can be reprocessed at the same time as holdoff_tasks, further improving build efficiency in various places. (Bitbake rev: 02090b3456b7a2de12e72dfeaabfd3b631609924) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise holdoff task handlingRichard Purdie2019-08-211-2/+10
| | | | | | | | | | | We don't need to process the holdoff task list until we're executing tasks which saves some data manipulation, at the cost of some data structures not being correct at all times. This saves significant amounts of time in various profile charts of larger builds. (Bitbake rev: 270f076111b12eab358417b0c4cf9c70d7cc787a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Small but critical fixRichard Purdie2019-08-211-0/+1
| | | | | | | | | | | | | We've observed do_package and do_package_setscene running in parallel. The reason is that holdoff_tasks wasn't getting updated. Looking at the code, it would seem the reason is that the task was in pending_migrations and hence changed wasn't set and holdoff_tasks wasn't updated. Fix this. It only affects builds with rehashing enabled. (Bitbake rev: e26e61e84575669bd223f6ab316798097ed95ec8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure target_tids is filteredRichard Purdie2019-08-161-0/+1
| | | | | | | | | | | bitbake <target> --runonly=fetch failed as the target_tids list included entries which were no longer targeted task ids. Fix this. (Bitbake rev: 94e848ae6544e628a19cb97115279b0b1678967c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure data is handled correctlyRichard Purdie2019-08-151-0/+2
| | | | | | | | | This doesn't appear to have ill effects right now but there is a correctness issue which this so fix it. (Bitbake rev: a5e084a266f63c2fd370122327615e49beaeb94e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix data corruption problemRichard Purdie2019-08-151-0/+3
| | | | | | | | | This was overwriting data in the parent which caused all kinds of odd/weird failures. (Bitbake rev: 4c5aeb424247a9d0c907524ffacd9c61fcdc0852) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure we clear the stamp cacheRichard Purdie2019-08-141-0/+3
| | | | | | | | | | When the task hashes change we need to ensure the stampcache is cleared out else tasks don't rerun when they should as we're basing decisions on stale cache data. (Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add missing setscene task corner caseRichard Purdie2019-08-141-0/+2
| | | | | | | | | We weren't marking this special case of setscene task as buildable leading to runqueue task failures. (Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add further debug informationRichard Purdie2019-08-141-0/+11
| | | | | | | | | Further testing shows we should test some extra datastructures to help pinpoint logic errors more precisely. This adds some further data structure sanity checks. (Bitbake rev: 83c4370b25c3a14cc946965c5c5f83ea28f488a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop debug statement causing performance issuesRichard Purdie2019-08-141-2/+0
| | | | | | | | | | | | This debug statement could result in a long list of tasks which when repeatedly sent over our IPC, slowed down the builds immensely. Remove it in favour of other more targeted debugging added recently, bringing back some lost performance, particularly on builds with large numbers of tasks. (Bitbake rev: 85fe627fdb6510f0942917964386fad9d8c479c8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix event timing raceRichard Purdie2019-08-141-69/+72
| | | | | | | | | | | | The event from the task notifiing of hash equivalency should only be processed when the task completes. This can otherwise result in a race where a dependent task may run before the original task completes causing various failures. To make this work reliably, the code had to be restructured quite a bit. (Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Recompute holdoff tasks from scratchRichard Purdie2019-08-141-1/+1
| | | | | | | | | | | | The changed_setscene variable here is just odd and not needed. Worse, it could prevent some tasks from being removed from the holdoff tasks list. The list is being rebuilt and should work as intended just from the other data, this is a leftover from previous versions of the code as far as I can tell. (Bitbake rev: 030b9f2b3ce6ed40e79304eb0ffee6c6613f43be) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve scenequeue debuggingRichard Purdie2019-08-141-19/+42
| | | | | | | | | | | | Whilst we had good runqueue failure mode debug, it hadn't adapted to the scenequeue changes. Run the scenequeue sanity tests at the end of a build and output the results regardless of whether all setscene tasks completed or not. This *massively* improves the ability to debug runqueue problems. (Bitbake rev: b9b2177473c0b95a23bd519a201e1d2ba101c6c1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix next_buildable_task performance problemRichard Purdie2019-08-141-5/+7
| | | | | | | | | | | | | | Looking at the profile information, a lot of time is being spent in next_buildable_task. This is probably due to the generator expressions not working well with the empty test. The easiest way to improve things is to switch to using set manipulations. We also don't need to update self.buildable the way the original code did as we don't rely on that anywhere. (Bitbake rev: 3bcf9ad4964b7e42d1a02ce231e9db42a81ead2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Wait for covered tasks to complete before trying setsceneRichard Purdie2019-08-141-0/+4
| | | | | | | | | If tasks are in the covered list of tasks for a given setscene task, it needs to wait for those to complete before we can start. (Bitbake rev: fdee640c26750b852eb68f5c80437377aa300ed8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve setscene task handling logicRichard Purdie2019-08-141-75/+34
| | | | | | | | | | | | | | The previous tasks_covered and tasks_notcovered were basically unstable data structures. We couldn't always tell whether tasks should be covered or not when trying to repair the sturcture if sstate tasks reran. In the end its simpler to throw the lists away and rebuild them based upon current data rather than trying to patch it adhoc. This turns out to be simpler and much more reliable and I've much more confidence in this code. (Bitbake rev: 52ee2ba2c617d928569f5afa404925c8b6f317bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix corruption issueRichard Purdie2019-08-141-1/+1
| | | | | | | | | | | We need to copy this set, not modify the original else all kinds of weird and bad things break, mostly from circular references. We'll not go into how much sleep I lost tracking down the fallout from this. (Bitbake rev: 49927546d2b306830c98f6f9da4a6ad828f6a3a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Clean up task reference formatsRichard Purdie2019-08-061-11/+8
| | | | | | | | | | | | | | | | | | Currently siggen uses the format "<filename>.<taskname>" for referencing tasks whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the separator everywhere. This is an API breaking change since the cache is affected, as are siginfo files and any custom signature handlers such as those in OE-Core. Ultimately this will let us clean up and the accessor functions from runqueue, removing all the ".rsplit(".", 1)[0]" type code currently all over the place. Once a standard is used everwhere we can update the code over time to be more optimal. (Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Clean up BB_HASHCHECK_FUNCTION APIRichard Purdie2019-08-061-41/+13
| | | | | | | | | | | | | This function uses an old API which uses offsets into lists as a communication mechanism. Update the API to use "tid" which is used universally in runqueue now. We can also add kwargs support to the funciton definition to drop some of the backwards compaiblility hoops we had to jump though with different function argument combinations. (Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/hashserv: Allow autostarting of a local hash server using ↵Richard Purdie2019-08-061-0/+1
| | | | | | | | | | | | | | | | | BB_HASHSERVE Its useful, particularly in the local developer model of usage, for bitbake to start and stop a hash equivalence server on local port, rather than relying on one being started by the user before the build. The new BB_HASHSERVE variable supports this. The database handling is moved internally into the hashserv code so that different threads/processes can be used for the server without errors. (Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve determinismRichard Purdie2019-08-061-13/+13
| | | | | | | | | | | Whilst this isn't strictly necessary, its helpful if the log output is consistent and its also helpful if bugs either appear or don't appear for a specific configuration. Ensuring the various iterations we make are deterministic (sorted) helps with this. (Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-20 21:51:10 +0000