linux/poky.git/bitbake/lib/bb/runqueue.py, branch hardknott-3.3.3

bitbake: runqueue: Handle deferred task rehashing in multiconfig builds

2021-05-06T10:22:44+00:00

If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. (Bitbake rev: b67476d4758915db7a5d9f58bc903ae7501a1774) Signed-off-by: Richard Purdie (cherry picked from commit 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue

2021-05-06T10:22:44+00:00

We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. (Bitbake rev: e70cba8d5861d79ed0da9e760e618af8b759c8a9) Signed-off-by: Richard Purdie (cherry picked from commit edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Fix deferred task issues

2021-04-27T14:22:58+00:00

In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: 1ec855731800cf8e2bae2b1e7241640e0bad8aae) Signed-off-by: Richard Purdie (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Further fixes for confused setscene tasks

2021-04-06T08:31:24+00:00

There is further evidence of tasks ending up being "covered" and "notcovered" which shouldn't happen and is bad. The code that caused this problem last time appears to have issues where stamps for tasks already exist. Split out the setscene stamp checking code to a separate function and use this when checking "hard dependencies" (like pseudo-native) so that if the stamps exist and it will be "covered", it is not put on the notcovered list. (Bitbake rev: a1848a481e36b729c8e4130c394b1d462d4b488a) Signed-off-by: Richard Purdie

bitbake: runqueue/event: Add an event for notifying of stale setscene tasks

2021-03-23T22:51:25+00:00

Use the new functionality in build.py to identify stale setscene tasks and send an event to the metadata listing them. The metadata then has the option of performing cleanup operations if it thinks that appropriate. (Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761) Signed-off-by: Richard Purdie

bitbake: runqueue: Print pseudo.log if fakeroot task failed

2021-03-11T14:04:45+00:00

Currently if pseudo fails we can only see the path to pseudo.log. If we have no access to server and can only rely on bitbake log then debugging becomes impossible. This printing needs to be added in runqueue level, not inside task execution, because in some cases task fails with pseudo abort really early and we don't even see any log. In this change I'm adding pseudo log printing in every fakeroot task failure that logged `mismatch`, `error` or `fatal` to logfile, because we have no other way to communicate with pseudo if it failed or not. Only lines from last pseudo server execution will be printed. (Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6) Signed-off-by: Tomasz Dziendzielski Signed-off-by: Jan Brzezanski Signed-off-by: Adrian Walag Signed-off-by: Paulo Neves Signed-off-by: Mikolaj Lasota Signed-off-by: Wiktor Baura Signed-off-by: Kamil Kwiek Signed-off-by: Richard Purdie

bitbake: runqueue: Add setscene task overlap sanity check

2021-03-09T00:02:05+00:00

We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. (Bitbake rev: 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Richard Purdie

bitbake: runqueue: Fix task execution corruption issue

2021-03-09T00:02:05+00:00

We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. (Bitbake rev: f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Richard Purdie

bitbake: bitbake-worker/runqueue: Add support for BB_DEFAULT_UMASK

2021-02-16T11:26:11+00:00

Currently each task has to have a umask specified individually. This is leading to determinism issues since it is easy to miss specifying this for an extra task. Add support for specifing the default task umask globally which simplifies the problem. (Bitbake rev: 3e664599fd54a8a37ce587022fcbce5ca26f2ed3) Signed-off-by: Richard Purdie

bitbake: event: Prevent bitbake from executing event handler for wrong multiconfig target

2021-02-16T11:26:11+00:00

When multiconfig is used bitbake might try to run events that don't exist for specific mc target. In cooker.py we pass `self.databuilder.mcdata[mc]` data that contains names of events' handlers per mc target, but fire_class_handlers uses global _handlers variable that is created during parsing of all the targets. This leads to a problem where bitbake runs event handler that don't exist for a target or even overrides them - if multiple targets use event handler with the same name but different code then only one version will be executed for all targets. See [YOCTO #13071] for detailed bug information. Add mc target name as a prefix to event handler name so there won't be two different handlers with the same name. Add internal __BBHANDLERS_MC variable to have the handlers lists per machine. (Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b) Signed-off-by: Tomasz Dziendzielski Signed-off-by: Richard Purdie