linux/poky.git/bitbake/lib, branch thud-next

bitbake: tests/fetch: Allow wget upgrade tests to run against a local server

2020-09-09T08:44:53+00:00

Currently these tests rely upon multiple uptream webservers which may change or be unavailable. Add local copies of the test data, copy the httpserver from OE-Core (used for testing there) and run these tests against a local server instead. (Bitbake rev: a21671e8a483ba8a6986d961987eda2d36ec61ca) Signed-off-by: Richard Purdie

bitbake: fetch2: Change git fetcher not to destroy old references

2020-09-08T13:46:13+00:00

It looks like we're about to see a lot of changes in branch names in repos. If we have the prune option here, those old names are lost, the changes propagate to our source mirrors and our old releases break. We have the force option so any replaced references should be replaced, its only orphaned branches which will now be preserved. I believe this behaviour will cause us fewer problems given the changes that look likely to happen. (Bitbake rev: 12d8cc3fecd550c4aadf0519e80711d755ee75ba) Signed-off-by: Richard Purdie

bitbake: tests/fetch: Switch from git.infradead.org to a YP mirror

2020-07-10T20:10:54+00:00

Upstream is unavailable, breaking tests. Switch to a YP mirror since if we can't reach that there are bigger problems. This should remove a source of intermittent failures on the autobuilder. (Bitbake rev: f4e60b29df88393302957c5bbdbe24ca38c4633c) Signed-off-by: Richard Purdie

bitbake: runqueue: fix multiconfig task dependency filtering

2019-12-12T20:02:51+00:00

multiconfig dependencies should be excluded from BB_TASKDEPDATA. However in thud, multiconfig filtering on task dependencies doesn't happen until after deps has already been added to taskdepdata. One manifestation of this results in multiconfig dependencies leaking into staging processing. File: 'exec_python_func() autogenerated', lineno: 2, function: 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/home/user/thud/meta/classes/staging.bbclass', lineno: 344, function: extend_recipe_sysroot 0340: #bb.note(" start is %s" % str(start)) 0341: 0342: # Direct dependencies should be present and can be depended upon 0343: for dep in set(start): *** 0344: if setscenedeps[dep][1] == "do_populate_sysroot": 0345: if dep not in configuredeps: 0346: configuredeps.append(dep) 0347: bb.note("Direct dependencies are %s" % str(configuredeps)) 0348: #bb.note(" or %s" % str(start)) Exception: KeyError: 'multiconfig:musl:/home/user/thud/meta/recipes-kernel/linux/linux-yocto_4.18.bb:do_deploy' This can be reproduced on thud by backporting the multiconfig.MultiConfig.test_multiconfig test and mcextend bbclass from warrior. d22b6e03a5504145abed7c2ca44cf12854df85da mcextend: Add helper class useful for multiconfig d9018a3d9c828551c465b68b27920ec4681524ae selftest: Add multiconfig test Flipping the ordering to match warrior's behavior fixes the test case. (Bitbake rev: b690030efc87850951e8e3ecf4ae3c1dd1dc9b63) Signed-off-by: Kyle Russell Signed-off-by: Richard Purdie

bitbake: fetch2: Ensure cached url data is matched to a datastore

2019-11-13T22:00:33+00:00

There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 97067634b1f149b56844b10e3a5e8d0d980b6e34) Signed-off-by: Richard Purdie

bitbake: cookerdata: Add mc conffiles hashes to cache hash

2019-10-10T15:53:26+00:00

The variable values that result from parsing multiconfig should be included in the cooker data hash, otherwise changes to these files won't be detected, which will allow the parsing cache to be loaded with the old values for the multiconfigs. This can either manifest as the variable values simply not updating, or getting basehash changed errors when building. This bug was previously undetected because all of the multiconfig base files were a direct file dependency in all parsed recipes. This was fixed in 34137a00f60 ("bitbake: bitbake: cooker: Rename __depends in all multiconfigs"), exposing this bug. [YOCTO #13541] (Bitbake rev: 6b045e074c6fea97d4e305a5a3c8bf82135d95eb) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: tests/fetch: Resolve fetch error in bitbake-selftest

2019-09-09T08:58:09+00:00

FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) AssertionError: '' is not true : Could not find upstream version for db [YOCTO #13496] The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget. Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the recipe. Also change the version being checked. (Bitbake rev: 8a58c3c64240c6ab14858d18e6b89febdb315311) Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: fetch2: show warning when renaming the archive with bad checksum failed

2019-09-06T10:58:56+00:00

* noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: d36438759344caa447d9a0bf30749a0aa31d1fba) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: utils: Fix movefile() exception handling with python3

2019-09-06T10:58:56+00:00

* with python3 this fails with: File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile 0795: try: 0796: os.rename(src, destpath) 0797: renamefailed = 0 0798: except Exception as e: *** 0799: if e[0] != errno.EXDEV: 0800: # Some random error. 0801: print("movefile: Failed to move", src, "to", dest, e) 0802: return None 0803: # Invalid cross-device-link 'bind' mounted or actually Cross-Device Exception: TypeError: 'OSError' object is not subscriptable (Bitbake rev: 9f92322fa8d6f1a68c0c3f4984afdf65126b51dc) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: gitsm: Add need_update method to determine when we are going to a new SRCREV

2019-06-18T10:31:10+00:00

If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly. The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch. To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present. This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work. This issue was discovered by William A. Kennington III . The included test case was also written by him, and included unmodified. (Bitbake rev: 4ce92f43eeac6a4bfd06e8567fa6891614b5b3b0) Signed-off-by: Mark Hatle Signed-off-by: Richard Purdie