linux/poky.git/bitbake, branch warrior-21.0.3

bitbake: cooker/siggen: Empty siggen cache during parsing

2020-02-18T13:59:11+00:00

When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. (Bitbake rev: d411b097d810c386d35dc561f8812bb3f35c9a36) Signed-off-by: Richard Purdie

bitbake: cooker: Keep track of watched files using a set instead of a list

2020-01-28T11:51:10+00:00

When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. (Bitbake rev: aa770081d7d3ff678b18117a4b4e9b3da09b72be) Signed-off-by: Peter Kjellerstedt Signed-off-by: Richard Purdie (cherry picked from commit 1e96df260e47d160dbd36bfc92c31ef06266f662) Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: fetch2: Ensure cached url data is matched to a datastore

2019-11-13T22:00:27+00:00

There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 4ce50bbd34eefeabfeca89a6a66c71598d3c58f6) Signed-off-by: Richard Purdie

bitbake: bitbake-worker child process create group before registering SIGTERM handler

2019-11-07T19:46:57+00:00

The bitbake-worker child on the SIGTERM signal handling send the SIGTERM to all processes in it's process group. In cases when the bitbake-worker child got SIGTERM after registering own SIGTERM handler and before the os.setsid() call it can send SIGTERM to unwanted processes. In the worst case during SIGTERM processing the bitbake-worker child can be in the group of the process that started BitBake itself. As a result it can kill processes that not related to BitBake at all. (Bitbake rev: b7483a7738daf69902ef590a8144a557576bbce0) Signed-off-by: Ivan Efimov Signed-off-by: Richard Purdie

bitbake: cookerdata: Add mc conffiles hashes to cache hash

2019-10-10T15:53:33+00:00

The variable values that result from parsing multiconfig should be included in the cooker data hash, otherwise changes to these files won't be detected, which will allow the parsing cache to be loaded with the old values for the multiconfigs. This can either manifest as the variable values simply not updating, or getting basehash changed errors when building. This bug was previously undetected because all of the multiconfig base files were a direct file dependency in all parsed recipes. This was fixed in 34137a00f60 ("bitbake: bitbake: cooker: Rename __depends in all multiconfigs"), exposing this bug. [YOCTO #13541] (Bitbake rev: 75d6648f232a06b99c54a1e33324a7fc1cd15b38) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: tests/fetch: Resolve fetch error in bitbake-selftest

2019-09-09T08:58:02+00:00

FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) AssertionError: '' is not true : Could not find upstream version for db [YOCTO #13496] The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget. Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the recipe. Also change the version being checked. (Bitbake rev: 164c0df429fc9be771c54c241b17d7afb8849afc) Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: fetch2: show warning when renaming the archive with bad checksum failed

2019-09-06T10:58:50+00:00

* noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: 24e44c412ac36987c1f65ced50030064ec58100d) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: utils: Fix movefile() exception handling with python3

2019-09-06T10:58:50+00:00

* with python3 this fails with: File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile 0795: try: 0796: os.rename(src, destpath) 0797: renamefailed = 0 0798: except Exception as e: *** 0799: if e[0] != errno.EXDEV: 0800: # Some random error. 0801: print("movefile: Failed to move", src, "to", dest, e) 0802: return None 0803: # Invalid cross-device-link 'bind' mounted or actually Cross-Device Exception: TypeError: 'OSError' object is not subscriptable (Bitbake rev: 16415c6e0f9e3d5c9fd81c9aabaea11d61b14187) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: bitbake: Drop duplicate license boilerplace text

2019-06-19T17:04:01+00:00

With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 34ed28a412af642a993642c14bd8b95d5ef22cd8) Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie

bitbake: bitbake: Add initial pass of SPDX license headers to source code

2019-06-19T17:04:01+00:00

This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ac556588fac55e91b7ce4839a975eb9ebb5aa192) Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie