linux/poky.git/bitbake, branch yocto-3.1.28

bitbake: runqueue: Fix deferred task/multiconfig race issue

2023-07-12T15:13:58+00:00

If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: b60c7085ec370473bea9b3b4b65826a17638837f) Signed-off-by: Richard Purdie (cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Steve Sakoman

bitbake: tests/fetch.py: fix link to project documentation

2023-05-02T10:18:39+00:00

(Bitbake rev: e16a9ca7e9286790ac37a067fdc8fde3a35a1c44) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: cooker: Drop sre_constants usage

2023-02-22T10:12:33+00:00

As reported by Martin Jansa : bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated import sre_constants it's deprecated since 3.11 with: https://github.com/python/cpython/issues/91308 The correct replacement for our usage is re.error so use that instead. (Bitbake rev: a4cd5b0b4b355b7b75fb48c61289700e3e908b2a) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Ensure deferred tasks are sorted by multiconfig

2023-02-22T10:12:33+00:00

We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. (Bitbake rev: 657940c7c2a9dea4963a5063e4bf900d6b454903) Signed-off-by: Richard Purdie (cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Fix issues with multiconfig deferred task deadlock messages

2023-02-22T10:12:33+00:00

In multiconfig builds with large numbers of identical tasks, builds were deadlocking after recent runqueue changes upon rebuilds where there was heavy sstate usage (i.e. on second builds after a first completed). The issue was that deferred tasks were being left indefinitely on the deferred list. The deadlock handler was then "breaking" things by failing tasks that had already succeeded, leading to the task being on both covered and not covered lists, giving a further error. The fix is to clean up the deferred task list when each setscene task completes. I'd previously been hoping to avoid iterating that list but it appears unavoidable. [YOCTO #14342] (Bitbake rev: 03cf0d9fedfef1ae43b3c3cac07710487857af36) Signed-off-by: Richard Purdie (cherry picked from commit ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Avoid deadlock avoidance task graph corruption

2023-02-22T10:12:33+00:00

If the deferred task deadlock avoidance code triggers, it could mark an executed task as failed which leads to "covered and not covered" error messages. Improve the logic so if the deadlock code is triggered, it doesn't cause the errors. (Bitbake rev: 2e1354525217505ce34fe775ee6ec8af46ff5324) Signed-off-by: Richard Purdie (cherry picked from commit 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Improve multiconfig deferred task issues

2023-02-22T10:12:33+00:00

The previous patches have exposed new issues with this code path, the issues being around what should happen when the hash of a task changes and the task is or is not on the deferred task list. Rather than rebuilding the deferred task list during each rehash event, build it once at the start of a build. This avoids the problem of tasks being added back after they have run and also avoids problems of always ensuring the same task is deferred. It also allows the 'outrightfail' codepath to be handled separately as the conditions are subtly differnt. One significant win for the new approch is the build is not continually printing out lists of deferred tasks, that list remains fairly static from the start of the build. Logic is added in to ensure a rehashed task with a hash matching other deferred tasks is deferred along with them as a small optimization. An interesting test case for this code was reported by Mark Hatle with four multiconfigs, each the same apart from TMPDIR and running a build of: bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal which is interesting in that the build of buildtools partially overlaps core-image-minimal and the build has a rehash event for qemuwrapper-cross even without any external hash equivalence server or preexisting data. (Bitbake rev: 4316e9f60ce5fd250a16586a1772dcc0adfeb932) Signed-off-by: Richard Purdie (cherry picked from commit bb424e0a6d274d398f434f7df63951da9ce305b3) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Handle deferred task rehashing in multiconfig builds

2023-02-22T10:12:32+00:00

If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. (Bitbake rev: 01936b4dd8e680f1f8035ff2d6231673f61efeab) Signed-off-by: Richard Purdie (cherry picked from commit 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue

2023-02-22T10:12:32+00:00

We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. (Bitbake rev: f95ed69e6c91ca6aa18d6fa9a6ac6319035c4661) Signed-off-by: Richard Purdie (cherry picked from commit edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Fabio Berton Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch2/git: Clarify the meaning of namespace

2023-02-13T07:48:16+00:00

Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: c527976e28165de9606497c48d834872fe68372e) Signed-off-by: Marek Vasut Signed-off-by: Richard Purdie (cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie