summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: providers: selected version not available should be a warningRoss Burton2021-05-221-2/+2
| | | | | | | | | | | | | If the selected version if not available, bitbake will happily attempt to build something else. This should be a loud warning not a small note. (Bitbake rev: 078f3164dcb1de7a141bec3a8fd52631d0362631) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 78cd63285713fde59506eb2e71a7b7ee59a594ff) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/gitsm: Fix crash when using git LFS and submodulesNiels Avonds2021-05-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | Gitsm fetcher crashes when cloning a repository that contains LFS files. This happens because the unpack method is called during download, but the submodules have not been downloaded yet at this point. This issue was introduced in this commit: 977b7268bf4fd425cb86d4a57500350c9b829162 [YOCTO #14283] (Bitbake rev: b4acaa314cd8bc86c50e14464ff01c2d110122c4) Signed-off-by: Niels Avonds <niels@codebits.be> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e05d79a6ed92c9ce17b90fd5fb6186898a7b3bf8) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix deferred task issuesRichard Purdie2021-05-131-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: fa068b5a3430b1b580cacfaf9011cdc3324d5844) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: remove write protected files tooMikko Rapeli2021-05-131-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some reason several git-annex files in Debian 10 buster are read-only and removing them with "rm -rf" fails. Fixes test failures like: $ bitbake-selftest ... rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied EE..................................ssss.sssssssssssssss.sssss....................................................................................................... ====================================================================== ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex fetcher, ud = self.fetch_shallow(uri) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow bb.utils.remove(ud.clonedir, recurse=True) File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path)) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1. Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod the whole temporary directory which should cover any directory name differences between different git-annex versions. Fixes tests failing after chmod call: Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/ (Bitbake rev: 1fae1c812138f35c35ea4c0586e21f022524c5f1) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: fix test execution without .gitconfigMikko Rapeli2021-05-131-1/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A CI user validating changes does not have any git push rights or even a .gitconfig file so fix tests so that they run by setting the user.name and user.email for the repo before committing changes. Fixes errors like: ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_no r_shallow_tarball_exist self.add_empty_file('a') File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file self.git(['commit', '-m', msg, path], cwd) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git return bb.process.run(cmd, cwd=cwd)[0] File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run raise ExecutionError(cmd, pipe.returncode, stdout, stderr) bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128: *** Please tell me who you are. Run git config --global user.email "you@example.com" git config --global user.name "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. (Bitbake rev: e341afcce40edf8078661630af1e0a780e8c1910) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Force parser shutdown after catching an exceptionJan Brzezanski2021-04-061-5/+5
| | | | | | | | | | | | | | | | | | | Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. (Bitbake rev: 017a39ed05d065bf28fd38f91bcde8a098300551) Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 915330e1dbae1ee8fd9a0358decf2c294f771961) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add setscene task overlap sanity checkRichard Purdie2021-03-241-0/+4
| | | | | | | | | | | | | We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. (Bitbake rev: da306cd4430a6fcf6b62ed0e6e5310432aa4c4c4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task execution corruption issueRichard Purdie2021-03-241-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. (Bitbake rev: 8a13955d8da86b976bb136ba1e724a02f7953bca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: __init__.py: Fix bitbake debug log handlingRichard Purdie2021-03-091-1/+1
| | | | | | | | | | | | For a while I've been puzzled as to why debug logging from runqueue wouldn't appear on the console with -DD. The logic in the bbdebug handling is inverted so fix it and now we see the expected messages from runqueue with -D and -DD. This should then let us debug other issues using those log messages. (Bitbake rev: e0dd20a006ce6fdc656e6be6ac581e93e2ed4a95) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/fetch2/__init__.py: drop _PYTHON_SYSCONFIGDATA_NAME unsettingAlexander Kanavin2021-02-181-5/+0
| | | | | | | | | | | | | With introduction of python3targetconfig class in core this is no longer needed. (Bitbake rev: fa94374baea75a94e3a488126ca7d8e241a77acd) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 47b64cfacd7c498ef9ed5486d117f2d69a39f225) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: download LFS content too during do_fetchMatt Hoosier2021-02-052-11/+61
| | | | | | | | | | | | | | | | | Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] (Bitbake rev: 2351b496bb63b96920d4ae67bec816f00d510df2) Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git.py: Use the correct branch to check if the repository has LFS ↵Mauro Queirós2021-02-051-2/+9
| | | | | | | | | | | | | | | | objects. Function "contains_lfs" was only looking at the master branch when searching for LFS content. LFS may be configured in specific branches only, so we need to use the correct branch. (Bitbake rev: 274f790ac9161187810df3242d3570261e407b11) Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4fa67c2af830035a1ddedc14592ee25a15ebff22) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git.py: LFS bitbake note should not be printed if need_lfs is not set.Mauro Queirós2021-02-051-1/+1
| | | | | | | | | | | | | | The message "Repository %s has LFS content but it is not being fetched" was being printed, even when Git-LFS was available and "lfs=1" was set. In those situations, we want to fetch LFS content, so that message would not make sense. (Bitbake rev: 2795c6150c0b3218716195c0d10ff2843b1ef695) Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 45028dfda5a29a34ab408cb3f11d72ae17963340) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git.py: skip smudging if lfs=0 is setMauro Queirós2021-02-051-0/+3
| | | | | | | | | | | | | | Git-LFS objects were being fetched even when lfs=0 was not set. This patch disables LFS smudging when lfs=0. That way, only the LFS pointers are downloaded during checkout. (Bitbake rev: 3d09608cabd404733b95ea92d5b00830ded77834) Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 646d86df7de774255246a3d7051c308e43eb257d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Ensure hash reflects vardepvalue flags correctlyRichard Purdie2021-01-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | The get_hash() function is used to decide if the base configuration has changed and hence whether a reparse is required. The vardepvalue flag's value was not expanded but it is often used in contexts like: METADATA_REVISION = "${@base_detect_revision(d)}" METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" which in it's unexpanded form means reparsing doesn't happen when it should as the data appears unchanged. Update get_hash to expand the values of vardepvalue so reparsing works as expected. This avoids basehash mismatch errors such as the one recently caused by using METADATA_REVISION in poky.conf's DISTRO_VERSION variable. The issue there could be exposed by a recipe using DISTRO_VERSION with the sequence: bitbake os-release <change the revision of the metadata with a dummy commit> bitbake os-release -C install which was caused because METADATA_REVISION changed but the metadata didn't reparse. (Bitbake rev: 418c00c570a60845556204b4f52de047b284dd8e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/conf.py: add missing import sysMert Kirpici2021-01-041-0/+1
| | | | | | | | | | | | | | Due to the calls to sys.stderr.write() and sys.exit() in exception handling in case of sphinx_rtd_theme not being installed, the following exception is raised by Python due to the fact that sys module not being imported. (Bitbake rev: a394eeec6aeb98c6395cb5da2036ce936bdaa0a0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5ddf0e5bee0de59d07295fc5693e20b1a0380fde) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: partial undo (bitbake-user-manual: update perforce fetcher ↵Nicolas Dechesne2021-01-041-31/+0
| | | | | | | | | | | | | | docs) This change is not applicable to 1.46/dunfell branch, and was brought by the automatic backport of Sphinx docs, so let's undo it. Note that this patch also included a general statement which applies to 1.46, so this snippet was not removed. (Bitbake rev: b164cf6790e47271a0848104a6c9507dd35c7f8f) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: undo (bitbake-user-manual: Remove TERM from ↵Nicolas Dechesne2021-01-041-1/+1
| | | | | | | | | | | | BB_HASHBASE_WHITELIST example) This change is not applicable to 1.46/dunfell branch, and was brought by the automatic backport of Sphinx docs, so let's undo it. (Bitbake rev: 07bc934fab62c0a238ec62a8516396191095a231) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: import sphinx docsNicolas Dechesne2021-01-0441-11846/+6539
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Bitbake manual was migrated to Sphinx in Yocto Project 3.2. Since the docs between 3.2 and 3.1 are "similar", and since 3.1 is an LTS release, we agreed to backport the documentation onto 3.1. If we look at all docs changes in 3.1 and 3.2, we have the following: === Changes in 3.1 git log --oneline b94dec477a8d48ebceec91952ba290798c56c1f5..origin/1.46 -- doc/ ==== 324aaa7f bitbake-user-manual-metadata.xml: fix a minor error f92e19a3 doc: More explanation to tasks that recursively depend on themselves e4695176 doc: Clarify how task dependencies relate to RDEPENDS 25c5c79b user manual: properly tag content as <replaceable> be367887 docs: delete reference to obsolete recipe-depends.dot === Changes in 3.2/master git log --oneline b94dec477a8d48ebceec91952ba290798c56c1f5..origin/master -- doc/ ==== a7c47f1e sphinx: rename Makefile.sphinx 427721d8 sphinx: remove DocBook files d52190ea docs: static: theme_overrides.css: fix responsive design on <640px screens 9ae5cce7 docs: sphinx: report errors when dependencies are not met ec4c481a docs: update README file after migrationg to Sphinx c87cc35a docs: sphinx: replace special quotes with double quotes ebdeef2c docs: ref-variables: add links to terms in glossary 29081375 bitbake-user-manual: fix bad links a0f37789 sphinx: theme_override: Use bold for emphasis text cbc5ca48 sphinx: theme_override: properly set font for verbatim text 08b1ae23 sphinx: remove leading '/' 99ba6fe9 sphinx: update style for important, caution and warnings d99760cc sphinx: last manual round of fixes/improvements 4f94633a sphinx: bitbake-user-manual: insert additional blank line after title 63adcaa5 sphinx: add releases page 3e940d93 sphinx: conf: enable extlinks extension 9921c652 sphinx: index: move the boilerplate at the end of the page 4e461224 sphinx: add SPDX headers cb19159c sphinx: Enhance the sphinx experience/nagivation with: 10a54678 sphinx: tweak html output a bit 219b2348 sphinx: Makefile.sphinx: add clean and publish targets 35fdc185 sphinx: fixes all remaining warnings e11d2dd1 sphinx: fix links inside notes 57300955 sphinx: fixup for links fa304c01 sphinx: override theme CSS 29af1cd2 sphinx: switch to readthedocs theme e8359fd8 sphinx: bitbake-user-manual: use builtin sphinx glossary 6bf6c8d6 sphinx: initial sphinx support 84ccba0f sphinx: add initial build infrastructure 44b57216 bitbake-user-manual: update perforce fetcher docs 9186ca47 bitbake-user-manual: Add BBFILES_DYNAMIC 7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example 06b5cf0a bitbake-user-manual-metadata.xml: fix a minor error c92a266c doc: More explanation to tasks that recursively depend on themselves caf42243 doc: Clarify how task dependencies relate to RDEPENDS 647c13d4 user manual: properly tag content as <replaceable> 2effbb6e docs: delete reference to obsolete recipe-depends.dot We can conclude the following commits exist in 3.2 and not in 3.1 (if we filter out sphinx changes) 44b57216 bitbake-user-manual: update perforce fetcher docs 9186ca47 bitbake-user-manual: Add BBFILES_DYNAMIC 7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example Out of these 3 changes, the following patches are for 3.2 only: 44b57216 bitbake-user-manual: update perforce fetcher docs 7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example To backport the Sphinx docs, we then need to cherry-pick all docs patches from 3.2/1.48 and 'undo' the two patches above. This first patch is the first step that imports all Sphinx files, and remove Docbook files. It was done with the following command: git cherry-pick -n \ $(git log --reverse --oneline \ b94dec477a8d48ebceec91952ba290798c56c1f5..origin/master -- doc/ \ | cut -f1 -d' ') (Bitbake rev: cd68f14031eb45006b44d10b348e35c69ac21ad0) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskexp: update for GTK API changesRoss Burton2020-12-011-1/+6
| | | | | | | | | | | | | | | | | | | | | | The introspected API for GtkTreeModel and friends had some unexpected quirks which have now been fixed, see[1] for details. However, for example Ubuntu 20.04 has the fixed GTK but not an updated pygobject which means taskexp raises an exception on startup. Solve by manually looking at what functions are present and calling the right one. [ YOCTO #14055 ] [1] https://gitlab.gnome.org/GNOME/pygobject/-/commit/9cdbc56fbac4db2de78dc080934b8f0a7efc892a (Bitbake rev: d171188cf966852326916f726cbd3ca9627b831c) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ac7d1114a7e99e6efd6a37b03d170faf678513fb) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update upstream master->main branchname transitionRichard Purdie2020-10-261-1/+1
| | | | | | (Bitbake rev: 89fc9450abefd682266efcbfa031d1ef115ff1a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: add unit tests for SRC_URI with spaces in urlCharlie Davies2020-10-131-0/+47
| | | | | | | | | | (Bitbake rev: d8daad57bdbceec041c4c2d288ed5d487ad8e6dc) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e7dab75c8d1923abcbbc7c9ac7de215d720ccf26) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch/git: add support for SRC_URI containing spaces in urlCharlie Davies2020-10-131-6/+6
| | | | | | | | | | | | | | | | | | | | | | Microsoft's TFS VCS system allows for spaces in a git repository url. An example of a valid url is: ssh://tfs-my-company.org:22/tfs/My Projects/FooBar This commit adds support for such urls by implementing two changes. Firstly, when bitbake makes a git command line call the url is surrounded by quotes so that the url, regardless of spaces, is treated as one argument. Secondly, additional parsing of various filepath variables, which are based off of the url, are now completed with any spaces in the url replaced with underscores. (Bitbake rev: c4a51b7f4fd8a3c7d63c184675c014ec955d2606) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit eb38b6f0935763f7ba19e5618f376fcae1dac41a) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Add BBFILES_DYNAMICKonrad Weihmann2020-10-131-0/+61
| | | | | | | | | | | | | - add missing entry for BBFILES_DYNAMIC, ported from yocto-docs - add description for the new inverse mode (Bitbake rev: c27a5cc80852595549ad4156e4bb7f5a05e4cd15) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9186ca47ce73b4d1c87eb69163698a04679fb55c) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Move away from problematic freedesktop.org urlsRichard Purdie2020-09-181-2/+4
| | | | | | | | | | | | | We're either hitting rate limiting with freedesktop.org or the servers have intermittent network connections. Use our own mirror of these repositories instead. (Bitbake rev: 18e1957337fd9f06bc673d28dd4f8277321d07bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a1b7ab5c9d5e64969f5ca0e41c0ac13c723e3761) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Drop cups.org from wget status checksRichard Purdie2020-09-171-3/+1
| | | | | | | | | | | | Its becomming clear the upstream server doesn't like this, drop these two urls from the test, not sure we need them here anyway. (Bitbake rev: 16774d4d41f2011e8db7d2986a1a701df123dfc1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ab2ef942dc21f9639793c972f2e546edf9444783) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: clean_basepath: remove recipe full path when virtual:xyz ↵Jean-Francois Dagenais2020-09-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | present Before this fix, this example basepath (a): virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile would get incorrectly "cleaned" into: helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb When searching backwards in `a` trying to isolate the 'virtual:xyz' to add it to the end of the string, we need to consider `a` still has the recipe path and taskname. So stoping the rsplit after only 1 split is not enough. We want to reach the second ':' from the end. This way, we obtain: helloworld/helloworld_1.2.3.bb:do_compile:virtual:native reviewed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> (Bitbake rev: befc11acd25a9d9a2d44c20a0e33ada740407af7) Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d193d93422a0ad62aa35b5d4ca5da8d422f72180) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Account for xmlrpc connectionsRichard Purdie2020-07-291-1/+1
| | | | | | | | | | | | | UI control can happen via the xmlrpc connection. Account for this when timing out UI connections. This was causing issues for toaster on systems where it couldn't parse the metadata within the timeout. (Bitbake rev: 838a89141fd6a75da9f982ad2b035e823b692d2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit fa85a8263971c25e67fa3b421c686a90e46acd87) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix UI first connection trackingRichard Purdie2020-07-291-1/+3
| | | | | | | | | | | | | We're only meant to be doing UI connection timeouts on the first connection but haveui changes for each connection. We need to add a specific variable to track this correctly and get the intended behaviour. (Bitbake rev: fda107a4a1033ac7111a3289dbe6e48cea986927) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e7c387c2e2fb2cc3ca1dc9d2029362909c326d72) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle multiconfig name mappings correctlyRichard Purdie2020-07-231-1/+1
| | | | | | | | | | | | | | "bitbake mc:arm:bash mc:arm:busybox" works but "bitbake multiconfig:arm:bash multiconfig:arm:busybox" does not. The reason is the list is modified whilst iterating. Don't do that. [YOCTO #13607] (Bitbake rev: cc11dfa4eb3616547a8a3909f89da0cc4f35dc57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix note reference -> infoRichard Purdie2020-07-211-1/+1
| | | | | | | | Its bb.note or logger.info, this avoids a backtrace. (Bitbake rev: fb80389d2ce62e15e354c6ea288c9a4bbe0040f6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Ensure UI-less servers don't sit in infinite loopsRichard Purdie2020-07-191-4/+15
| | | | | | | | | | | | | | | | | | | | | | | If server startup is broken for some reason (e.g. lockfile issues) and no UI connection is made, the server will just sit inifinitely waiting. Add a timeout upon startup in the non-memory resident case so that such infinite waits are avoided. In the memory resident case, the server wouldn't have shut down in the first place or will timeout according to configuration. Since any race may mean the socket file is no longer present, ensure the unlink doesn't fault upon exit, thus ensuring any hashequiv or PRServ is removed from memory, allowing all processes to exit cleanly in such scenarios. (Bitbake rev: 81cad9b8c4df15218d1a419c6b8e3ac73b54600c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 39888b750df12478e8bdea6727cca112dce1df85) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix a rare lockfile raceRichard Purdie2020-07-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We're seeing rare occasional races on the autobuilder as if two server processes have the lockfile at the same time. We need to be extremely careful this does not happen. I think there is a potential race in this shutdown code since we delete the lockfile, then call unlockfile() which also tries to delete it. This means we may remove a lock file now held by another process if we're unlucky. Since unlockfile removes the lockfile when it can, just rely on that and remove any possible race window. An example cooker-deamonlog: --- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 --- Started bitbake server pid 2266 Entering server connection loop Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([]) Processing Client Connecting Client Running command ['setFeatures', [2]] Running command ['updateConfig', XXX] Running command ['getVariable', 'BBINCLUDELOGS'] Running command ['getVariable', 'BBINCLUDELOGS_LINES'] Running command ['getSetVariable', 'BB_CONSOLELOG'] Running command ['getSetVariable', 'BB_LOGCONFIG'] Running command ['getUIHandlerNum'] Running command ['setEventMask', XXXX] Running command ['getVariable', 'BB_DEFAULT_TASK'] Running command ['setConfig', 'cmd', 'build'] Running command ['getVariable', 'BBTARGETS'] Running command ['parseFiles'] --- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 --- Started bitbake server pid 8252 --- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 --- Started bitbake server pid 13278 Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}] Running command ['getRecipes', ''] Running command ['clientComplete'] Processing Client Disconnecting Client No timeout, exiting. Exiting where it looks like there are two server processes running which should not be. In that build there was a process left sitting in memory with its bitbake.sock file missing but holding the lock (not sure why it wouldn't timeout/exit). (Bitbake rev: de919782f488a83b80d7c40896bf5b2596f1f65f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e1a7c1821483031b224a1570bfe834da755219cc) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: pyshyacc: allow double COMMA statementsKonrad Weihmann2020-07-191-0/+1
| | | | | | | | | | | | | this allows shell statements like '; ;' to pass the parser. As it may be bad code but still valid enough to execute (Bitbake rev: 85a499f8915dd6596d56808e6e1c536015e4eb34) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b7732b1b5085bea73e17d112e1bd9ac3d4dc34fb) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Change git fetcher not to destroy old referencesRichard Purdie2020-07-191-1/+1
| | | | | | | | | | | | | | | | | | | It looks like we're about to see a lot of changes in branch names in repos. If we have the prune option here, those old names are lost, the changes propagate to our source mirrors and our old releases break. We have the force option so any replaced references should be replaced, its only orphaned branches which will now be preserved. I believe this behaviour will cause us fewer problems given the changes that look likely to happen. (Bitbake rev: 6e527ea72b0816bf5f454473faf6523c365daa18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 820ab886e79eea516560c0c008e4cf059c6e11a3) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Increase timeout for commandsRichard Purdie2020-07-191-1/+3
| | | | | | | | | | | | | We're running into this timeout on loaded autobuilders in situations where things should otherwise succeed. Log a note in these cases and continue to try for longer. (Bitbake rev: 344d3006a243f5f7d8f551556bab721b2ac0a2aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e567743e70f426786ae54dcb5ab550748d9266e4) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/teamcity: don't use removed logging classesChris Laplante2020-07-191-2/+0
| | | | | | | | | | | | Allows the TeamCity frontend to be used again. (Bitbake rev: cea130b238e874bae30e13b3242a00eaff7f853b) Signed-off-by: Chris Laplante <mostthingsweb@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c5477ba79fcad4a887808dd0df9cfe3554e2c17a) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/ui/taskexp: Fix missing Gtk importDavid Khouya2020-07-191-0/+1
| | | | | | | | | | | | | | Adding back gtk objects import. Fix bug introduce when adding validation on gtk import. (Bitbake rev: 44e24fffc568dfee054b5feff2d9b382b9ce4bb5) Signed-off-by: David Khouya <dakhouya@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 765be5ef60668f8a1cfbcba248f4995725807196) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/ui/taskexp: Validate gi importDavid Khouya2020-07-191-3/+9
| | | | | | | | | | | | | | | | When running bitbake -g -u taskexp without having gi python module or and invalid gtk version, bitbake fails with a stack trace. In case of import or version error, bitbake should exit with an error message instead of a stack trace. (Bitbake rev: 45e32f2651d35bf7a315af95cfe0b1070e038c9e) Signed-off-by: David Khouya <dakhouya@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2a2c507f239b047f34765312df4168030e38b90d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: msg: Avoid issues where paths have relative componentsRichard Purdie2020-07-021-1/+1
| | | | | | | | | | | | | The autobuilder can end up using build/../ syntax which is an issue if the build directory is cleaned. Avoid this by using normpath() on the file path passed in. (Bitbake rev: 1d3892d93ee9688d69f5bb5ea6aef8b1152bd1e3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 41988fec47eb196ab7195a75330a6d98de19101b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid unpickle errors in rare casesRichard Purdie2020-07-021-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In rare cases the pickled data from a task contains "</event>" which causes backtrace. This can be reproduced with something like: do_unpack_prepend () { bb.warn("</event>") } There are several solutions but the easiest is to catch this exception and look for the next marker instead as this should be the only way such an unpickle error could occur. This fixes rare exceptions seen on the autobuilder. Also add in other potential exceptions listed in the pickle manual page so that better debug is obtained should there be an error in this code path in future. exitcode doesn't need the same handling since we control what is in that data field and it could never contain </exitcode> (Bitbake rev: 6d780fe3a111adbf3f3d2dda22d5a0787b195b62) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5ada512d6f9cbbdf1172ff7818117c38b12225ca) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix error when hash equivalence has an exceptionJoshua Watt2020-07-021-0/+1
| | | | | | | | | | | | | The code that handled exceptions from the hash equivalence client was raising an exception itself because hashserv.client wasn't imported (Bitbake rev: a7fc2ae3fd40b7de36f7f9b3da64a610a698b4ed) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a76290dfc6f34ff9f6efdb13a6db74b6b4759daf) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Chunkify large messagesJoshua Watt2020-07-024-41/+152
| | | | | | | | | | | | | | | | | | | The hash equivalence client and server can occasionally send messages that are too large for the server to fit in the receive buffer (64 KB). To prevent this, support is added to the protocol to "chunkify" the stream and break it up into manageable pieces that the server can each side can back together. Ideally, this would be negotiated by the client and server, but it's currently hard coded to 32 KB to prevent the round-trip delay. (Bitbake rev: 1a7bddb5471a02a744e7a441a3b4a6da693348b0) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e27a28c1e40e886ee68ba4b99b537ffc9c3577d4) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: test/fetch: change to better svn sourceakuster2020-07-021-1/+1
| | | | | | | | | | | | | | | | | | fixes: svn: warning: W175002: Unexpected HTTP status 504 'Gateway Timeout' on '/openembedded/bitbake/!svn/vcc/default' svn: E205011: Failure occurred processing one or more externals definitions picked pcre2 [Yocto #13948] (Bitbake rev: 767aa9316603a1c92c9c433a0c11ae98089bbc3a) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1483d17108da02f5d615e83403d5fd6288ca957c) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Switch from git.infradead.org to a YP mirrorRichard Purdie2020-06-221-1/+2
| | | | | | | | | | Upstream is unavailable, breaking tests. Switch to a YP mirror since if we can't reach that there are bigger problems. This should remove a source of intermittent failures on the autobuilder. (Bitbake rev: 83296870bede70e31bdf6e73683bcc30681023fc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster-requirements.txt: require Django 2.2Tim Orling2020-06-051-1/+1
| | | | | | | | | | | | | | | In commit 9730f95686b2ac72cf1fa513c555f7c7787e2667 Django 2.2 was enabled. Django 1.11 was EOL on April 1, 2020 (Bitbake rev: 6cc09fa33131f71a3fd0e336ff07a4186b41bf8f) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ee15e78c6f9b59c221b1e43973ee4db20c5b443b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual-metadata.xml: fix a minor errorKai Kang2020-06-051-1/+1
| | | | | | | | | | | | | In the '_remove' example in bitbake-user-manual-metadata.xml, there is no 'jkl' in the original value of FOO2. So remove it from result. (Bitbake rev: 324aaa7f8d6d83e1e00b8054dac44df561588be8) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 06b5cf0ab6c6e518ac780d081fab5546334c5c7d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: More explanation to tasks that recursively depend on themselvesJacob Kroon2020-06-051-6/+8
| | | | | | | | | | (Bitbake rev: f92e19a3b3d89eb26eeb74b18ca01248767035b5) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c92a266c8e452833f2a590721aa1c2bd6fbeb2e0) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: Clarify how task dependencies relate to RDEPENDSJacob Kroon2020-06-051-0/+3
| | | | | | | | | | | | | | Clarify that BitBake knows how to map entries defined in the runtime dependency namespace back to build-time dependencies (recipes) in which tasks are defined. (Bitbake rev: e4695176ffdc5eb959f71a08f77ff6a8e028ffa9) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit caf422435ad64aacbdab8a94da3115599dd0938b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: user manual: properly tag content as <replaceable>Robert P. J. Day2020-06-051-2/+2
| | | | | | | | | | | | | Tag a couple fields as replaceable to be consistent with rest of manual. (Bitbake rev: 25c5c79bbe814eaff03c72cc2680414a73cff7f4) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 647c13d4ae746a1bb9bd76ff318477dadb4d292f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-08 13:07:20 +0000