linux/poky.git/bitbake/lib/bb/runqueue.py, branch yocto-3.3.6

bitbake: runqueue: Fix runall option handling

2021-11-21T11:35:37+00:00

The previous fix for runall option handling had a small bug in it, it didn't clear the originally processed task list which meant it was running too many tasks. Fix this so the list is reset and rebuild correctly. (Bitbake rev: 15dadb69b4c5d29b80770e55d1e9afbe47084aa4) Signed-off-by: Richard Purdie (cherry picked from commit 87c9e120897ed04dfc64d4752fc602f9bfcb8645) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Fix runall option task deletion ordering issue

2021-11-21T11:35:37+00:00

The runbuild option handling in runqueue was flawed as items deleted from the main task list may be dependencies and hence cause index errors. Rather than modify runtaskentries straight away, compute a new shorted list and use that as an input to the second phase. This avoids the need to add tasks back to the list meaning delcount can be simplifed to a simple counter. The second use case in runonly doen't re-add items so doesn't have this issue. (Bitbake rev: 44574295d54ec9e48c9ecdd4eb869c4494e77679) Signed-off-by: Richard Purdie (cherry picked from commit 3428e3c54eb5cc03ff96f9cee6dc839afee7a419) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue/knotty: Improve UI handling of setscene task counting

2021-09-24T09:13:32+00:00

The recent fixes to merge setscene and normal task accounting in runqueue fixed some display issues but broke the task numbering of setscene tasks. Add new accounting methods to the stats structure specifically designed for setscene. This accounts for the fact that setscene tasks can rerun multiple times in the build. Then use the new data in the UI to correctly display the numbers the user wants to see to understand progress. (Bitbake rev: 7d938703d9321cde5a32e4dff005f07e8821b704) Signed-off-by: Richard Purdie (cherry picked from commit ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Clean up task stats handling

2021-09-24T09:13:32+00:00

When we parallelised normal and setscene tasks, the task stats handling was left separate pending further thought. We had to remove handling of the setscene tasks from the UI in order to maintain consistent task numbering. Currently, "0 of 0" tasks can be shown as setscene tasks execute until the first normal task runs. The only use left for sq_stats is in the active task numbers which we can use the length of sq_ive for instead. We can therefore drop it and return stats in all cases. This removes the "0 of 0" task problem since the stats in all normal and setscene tasks matches. [YOCTO #14479] (Bitbake rev: fac319e2d5463be2a82335b9cb348c7893a1e65f) Signed-off-by: Richard Purdie (cherry picked from commit eae6e947e37e18cded053814bd2a268b44fb25cd) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Improve multiconfig deferred task issues

2021-09-16T21:17:48+00:00

The previous patches have exposed new issues with this code path, the issues being around what should happen when the hash of a task changes and the task is or is not on the deferred task list. Rather than rebuilding the deferred task list during each rehash event, build it once at the start of a build. This avoids the problem of tasks being added back after they have run and also avoids problems of always ensuring the same task is deferred. It also allows the 'outrightfail' codepath to be handled separately as the conditions are subtly differnt. One significant win for the new approch is the build is not continually printing out lists of deferred tasks, that list remains fairly static from the start of the build. Logic is added in to ensure a rehashed task with a hash matching other deferred tasks is deferred along with them as a small optimization. An interesting test case for this code was reported by Mark Hatle with four multiconfigs, each the same apart from TMPDIR and running a build of: bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal which is interesting in that the build of buildtools partially overlaps core-image-minimal and the build has a rehash event for qemuwrapper-cross even without any external hash equivalence server or preexisting data. (Bitbake rev: 1775bcb95f61af21712bc1ba5098a49df8f21128) Signed-off-by: Richard Purdie (cherry picked from commit bb424e0a6d274d398f434f7df63951da9ce305b3) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Fix issues with multiconfig deferred task deadlock messages

2021-09-16T21:17:48+00:00

In multiconfig builds with large numbers of identical tasks, builds were deadlocking after recent runqueue changes upon rebuilds where there was heavy sstate usage (i.e. on second builds after a first completed). The issue was that deferred tasks were being left indefinitely on the deferred list. The deadlock handler was then "breaking" things by failing tasks that had already succeeded, leading to the task being on both covered and not covered lists, giving a further error. The fix is to clean up the deferred task list when each setscene task completes. I'd previously been hoping to avoid iterating that list but it appears unavoidable. [YOCTO #14342] (Bitbake rev: 27d5b55bb33c319ca595c192c910eac91a5d9428) Signed-off-by: Richard Purdie (cherry picked from commit ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Avoid deadlock avoidance task graph corruption

2021-09-16T21:17:48+00:00

If the deferred task deadlock avoidance code triggers, it could mark an executed task as failed which leads to "covered and not covered" error messages. Improve the logic so if the deadlock code is triggered, it doesn't cause the errors. (Bitbake rev: 72f51767da47af12d2c56bd3ee87cb0018b16705) Signed-off-by: Richard Purdie (cherry picked from commit 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Handle deferred task rehashing in multiconfig builds

2021-05-06T10:22:44+00:00

If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. (Bitbake rev: b67476d4758915db7a5d9f58bc903ae7501a1774) Signed-off-by: Richard Purdie (cherry picked from commit 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue

2021-05-06T10:22:44+00:00

We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. (Bitbake rev: e70cba8d5861d79ed0da9e760e618af8b759c8a9) Signed-off-by: Richard Purdie (cherry picked from commit edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Fix deferred task issues

2021-04-27T14:22:58+00:00

In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: 1ec855731800cf8e2bae2b1e7241640e0bad8aae) Signed-off-by: Richard Purdie (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie