linux/poky.git/bitbake/lib/bb/runqueue.py, branch yocto-3.1.11

bitbake: runqueue: Fix deferred task issues

2021-05-13T12:12:00+00:00

In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: fa068b5a3430b1b580cacfaf9011cdc3324d5844) Signed-off-by: Richard Purdie (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Add setscene task overlap sanity check

2021-03-24T14:36:21+00:00

We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. (Bitbake rev: da306cd4430a6fcf6b62ed0e6e5310432aa4c4c4) Signed-off-by: Richard Purdie (cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Fix task execution corruption issue

2021-03-24T14:36:21+00:00

We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. (Bitbake rev: 8a13955d8da86b976bb136ba1e724a02f7953bca) Signed-off-by: Richard Purdie (cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Avoid unpickle errors in rare cases

2020-07-02T15:11:40+00:00

In rare cases the pickled data from a task contains "" which causes backtrace. This can be reproduced with something like: do_unpack_prepend () { bb.warn("") } There are several solutions but the easiest is to catch this exception and look for the next marker instead as this should be the only way such an unpickle error could occur. This fixes rare exceptions seen on the autobuilder. Also add in other potential exceptions listed in the pickle manual page so that better debug is obtained should there be an error in this code path in future. exitcode doesn't need the same handling since we control what is in that data field and it could never contain (Bitbake rev: 6d780fe3a111adbf3f3d2dda22d5a0787b195b62) Signed-off-by: Richard Purdie (cherry picked from commit 5ada512d6f9cbbdf1172ff7818117c38b12225ca) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Lower setscene complete logging level

2020-03-13T11:29:23+00:00

Lowers the level of the log message when setscene tasks have completed. This message can occur multiple times when hash equivalence is enabled, since the runqueue switches between executing setscene tasks and normal tasks. Since this is primarily of use when debugging hash equivalence, use the hash equivalence logger at VERBOSE level. [YOCTO #13813] (Bitbake rev: 7dd5b3900622008ff34ec70d71c6e994f460a46f) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue/siggen: Lower hash equivalence logging

2020-03-13T11:29:23+00:00

Lowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json) [YOCTO #13813] (Bitbake rev: 2ddb649ea31afe052f08e3969e36abf6fb515bc2) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue/siggen: Log hash equivalence with a different logger

2020-03-13T11:29:23+00:00

Switches the hash equivalence logging to use a different logger so that it can be easily filtered out with python's structured logging. (Bitbake rev: 20bb29ef973e9c5483eb50a74550ea207637367b) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: lib/bb/msg: Use log level instead of debug count

2020-03-13T11:29:23+00:00

Passes around the actual logging level as the default log level variable instead of the debug count. This makes it easier to deal with logging levels since the conversion from debug count and verbose flag only has to occur once when logging is initialized and after that actual log levels can be used (Bitbake rev: 41bd155faf7f65cb0727fcce972715769b26ca89) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: runqueue: teach runonly/runall to accept "do_task" as well as "task"

2020-03-07T16:07:00+00:00

Previously --runonly=do_task would give a misleading error like: ERROR: Could not find any tasks with the tasknames ['do_task'] to run within the recipes of the taskgraphs of the targets... The problem is that BitBake tried to find "do_do_task". So teach it to only add the do_ prefix if it's not already there. (Bitbake rev: 694904bde980606dc67c201da61f4fb685679b17) Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie

bitbake: lib: remove unused imports

2020-01-19T13:31:05+00:00

removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews Signed-off-by: Richard Purdie