linux/poky.git/bitbake/lib, branch yocto-3.1.10

bitbake: cooker: Avoid parser deadlocks

2021-06-19T16:06:18+00:00

If you make parsing fail (e.g. add something like: X := "${@d.getVar('MCMACHINES').split()[1]}" to meson.bbclass, then run "while true; do bitbake -g bash; done" it will eventually hang. It appears the cancel_join_thread() call the parsing failure triggers, breaks the results_queue badly enough that it sits in read() indefintely (called from self.result_queue.get(timeout=0.25)). The timeout only applies to lock aquisition, not the read call. I've tried various other approaches such as using cancel_join_thread() in other places but the only way things don't lock up is to avoid cancel_join_thread() entirely for results_queue. I do have a concern that this may adversely affect Ctrl+C handling but equally, its broken now already and this appears to improve things. [YOCTO #14034] (Bitbake rev: 0e0af15b84e07e6763300dcd092b980086b9b9c4) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: cooker: Ensure parser is cleaned up

2021-06-19T16:06:18+00:00

During cooker shutdown, its possible the parser isn't cleaned up. Fix this (which may partially explain why threads were left hanging around at exit). (Bitbake rev: fe1b57f4d39c287526353387e164c863eb9df421) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: cooker: Explictly shut down the sync thread

2021-06-19T16:06:18+00:00

Hongxu Jia reported a problem where the bb_cache files were not always being written out correctly. This was due to the sync thread being terminated prematurely. Whilst the preceeding changes mean the exit handler for this thread is now correctly called since we switch to using sys.exit() instead of os._exit(), this write can happen after we drop the bitbake lock, leading to potential races. Avoid that headache by adding in explicit thread join() calls before we drop the lock (which atexit or Finalize can't do). (Bitbake rev: 6d2dd16b87ce62086f956912e9a7335b2adfcc94) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: cooker: Ensure parse_quit thread is closed down

2021-06-19T16:06:18+00:00

Each run through the parser would leak a thread from the queue created to shut the parser down. Close this down correctly and clean up the code flow slightly whilst in the area, making sure this thread does shut down correctly (we don't care if it loses data). (Bitbake rev: 1dcc4e29dee3fd0dd4d6dc2fc9ccdc5888fcfcf3) Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: providers: selected version not available should be a warning

2021-05-22T11:11:06+00:00

If the selected version if not available, bitbake will happily attempt to build something else. This should be a loud warning not a small note. (Bitbake rev: 078f3164dcb1de7a141bec3a8fd52631d0362631) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie (cherry picked from commit 78cd63285713fde59506eb2e71a7b7ee59a594ff) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: fetch/gitsm: Fix crash when using git LFS and submodules

2021-05-13T12:12:00+00:00

Gitsm fetcher crashes when cloning a repository that contains LFS files. This happens because the unpack method is called during download, but the submodules have not been downloaded yet at this point. This issue was introduced in this commit: 977b7268bf4fd425cb86d4a57500350c9b829162 [YOCTO #14283] (Bitbake rev: b4acaa314cd8bc86c50e14464ff01c2d110122c4) Signed-off-by: Niels Avonds Signed-off-by: Richard Purdie (cherry picked from commit 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie (cherry picked from commit e05d79a6ed92c9ce17b90fd5fb6186898a7b3bf8) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: runqueue: Fix deferred task issues

2021-05-13T12:12:00+00:00

In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: fa068b5a3430b1b580cacfaf9011cdc3324d5844) Signed-off-by: Richard Purdie (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bitbake: tests/fetch: remove write protected files too

2021-05-13T12:12:00+00:00

For some reason several git-annex files in Debian 10 buster are read-only and removing them with "rm -rf" fails. Fixes test failures like: $ bitbake-selftest ... rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied EE..................................ssss.sssssssssssssss.sssss....................................................................................................... ====================================================================== ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex fetcher, ud = self.fetch_shallow(uri) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow bb.utils.remove(ud.clonedir, recurse=True) File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path)) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1. Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod the whole temporary directory which should cover any directory name differences between different git-annex versions. Fixes tests failing after chmod call: Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/ (Bitbake rev: 1fae1c812138f35c35ea4c0586e21f022524c5f1) Signed-off-by: Mikko Rapeli Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bitbake: tests/fetch: fix test execution without .gitconfig

2021-05-13T12:12:00+00:00

A CI user validating changes does not have any git push rights or even a .gitconfig file so fix tests so that they run by setting the user.name and user.email for the repo before committing changes. Fixes errors like: ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_no r_shallow_tarball_exist self.add_empty_file('a') File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file self.git(['commit', '-m', msg, path], cwd) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git return bb.process.run(cmd, cwd=cwd)[0] File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run raise ExecutionError(cmd, pipe.returncode, stdout, stderr) bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128: *** Please tell me who you are. Run git config --global user.email "you@example.com" git config --global user.name "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. (Bitbake rev: e341afcce40edf8078661630af1e0a780e8c1910) Signed-off-by: Mikko Rapeli Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: Force parser shutdown after catching an exception

2021-04-06T21:45:36+00:00

Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. (Bitbake rev: 017a39ed05d065bf28fd38f91bcde8a098300551) Signed-off-by: Jan Brzezanski Signed-off-by: Richard Purdie (cherry picked from commit 915330e1dbae1ee8fd9a0358decf2c294f771961) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie