linux/poky.git/meta/classes-global, branch styhead-5.1.4

do_package/sstate/sstatesig: Change timestamp clamping to hash output only

2024-12-09T14:25:53+00:00

The code was changing the timestamps of the files in the do_package output, particularly the files added for debug sources. This was to do two things: a) make do_package sstate more reproducible b) ensure better hash equivalence matching Unfortuately the debug source files are hardlinks into the source tree for efficiency so touching these, touches a lot of files in ${B} and ${S}. This causes unpredictable effects if compile is run again for example, or could cause compiling in the install task. The hash equivalence matching is of key importance but we can mimic that using clamping of the file timestamps in the depsig output used to generate the hashes. This patch drops the global timestamp clamping, instead allowing the files to retain their creation timestamps into sstate. This makes do_package sstate slightly less reproducibile. We could clamp the sstate timestamps but that would lead to two different sets of timestamps depending on whether the data came from sstate or not. I'd prefer to have consistent code behaviour, rather than differing behavhour depending on whether data came from sstate or not. If we wanted to have reproducibiliy and fix the "corruption" of S/B and have consistent codepaths, the only other option would be two copies of the sources, which could end up huge and seems the least desireable option. This patch therefore drops the timestamp clamping in the sstate files and tweaks the depsig data generation to clamp the timestamps for do_package instead since this seems the best compromise. I validated that rpm/deb/ipk files still generate correctly as before. (From OE-Core rev: 0e6b2c761f6d727fe21a0ce2803a0f0aef236f59) Signed-off-by: Richard Purdie (cherry picked from commit 475759fdab7200488b2a568b2ba1aa31a456d113) Signed-off-by: Steve Sakoman

package_rpm: restrict rpm to 4 threads

2024-11-30T13:41:59+00:00

TL;DR version: with this, and the previous compression level changes I am seeing drastic speedups in package_write_rpm completion times: webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!) The long version: rpm uses multithreading for two purposes: - spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was) - parallel file classification While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently: https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4 On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes 5m20s with 256 threads (default on my machine!) 1m49s with 64 threads 59s with 16 threads 48s with 8 threads Even a more typical recipe like webkitgtk is affected: 47s with 256 threads 32s with 64 threads 27s with 16 or 8 threads I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time. (From OE-Core rev: ac480775440fba812fd5aa9da73e0e5bc60d46d6) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package_rpm: use zstd's default compression level

2024-11-30T13:41:59+00:00

zstd uses 3 by default, while 19 is the highest and slowest. It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking: https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66 This brings significant speedups in rpm's compression step: for example compressing webkitgtk takes 11s instead of 36s. The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff. (From OE-Core rev: 9ec8b99470ed744c8eca0b23f99ba54210984080) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package_rpm: Check if file exists before open()

2024-11-26T13:37:09+00:00

Exception handler tries to read() /etc/passwd file in sysroot and if file doesn't exist for any reason then it raises FileNotFoundError exception which mask the original source of the problem and makes debugging of the issue more difficult. Fixes: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/codebuild/output/src1899304708/src/build/tmp-container/work/core2-64-oe-linux/emqx-bin/4.3.12/recipe-sysroot/etc/passwd' (From OE-Core rev: 17c8560f323d619b4a98d2276b4c0bf04c8196fd) Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie (cherry picked from commit 4ad9a0e0b11eb7bc5a3dd45fc8945e094ea949e9) Signed-off-by: Steve Sakoman

sanity: check for working user namespaces

2024-09-13T09:28:10+00:00

If user namespaces are not available (typically because AppArmor is blocking them), alert the user. We consider network isolation sufficiently important that this is a fatal error, and the user will need to configure AppArmor to allow bitbake to create a user namespace. [ YOCTO #15592 ] (From OE-Core rev: b6af956fe6e876957a49d4abf425e8c789bf0459) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

insane: Remove dependency on TCLIBC from QA test

2024-09-05T20:50:10+00:00

The TCLIBC value is already encoded into build paths through the triplet so no need to encode it here where it can cause problems for allarch output that span multiple libcs. (From OE-Core rev: ea8c7a457a79589c35ca80b2f265799164855674) Signed-off-by: Richard Purdie

sstate: Make do_recipe_qa and do_populate_lic non-arch specific

2024-09-05T20:48:47+00:00

The sstate functions currently pull in STAMP and SSTATE_PKG which end up pulling in DEFAULTTUNE and other variables. The location on disk encodes all the "architecture" information we need so clean up the dependencies of these tasks and make them non-architecture specific. (From OE-Core rev: 65df61ccff6781906449bfea386a8dd13112a51c) Signed-off-by: Richard Purdie

license: Fix directory layout issues

2024-09-05T20:48:47+00:00

There are several issues: a) pointless empty directories were being created as a path wasn't fixed in a previous commit b) SSTATE_PKGARCH wasn't being captured into the task signature since it is in the ignore list by default. We want to capture the absolute value, not the dependencies c) with those issues fixed, cross/native issues became apparent so those need to be fixed too. (From OE-Core rev: f68aed302a0e4b86fb8c16a6ef4e7295bed48b86) Signed-off-by: Richard Purdie

sstate: Drop SSTATEPOSTINSTFUNC support

2024-09-04T11:38:44+00:00

This was deprecated with the introduction of postfunc support for tasks in general and only used by buildhistory. Now that usage has been removed, drop the code from sstate.bbclass. Any other users should be able to use postfuncs too. (From OE-Core rev: 74e08170a5584d83f5f03cd8a71978b5e0895c1d) Signed-off-by: Richard Purdie

buildhistory: Simplify intercept call sites and drop SSTATEPOSTINSTFUNC usage

2024-09-04T11:38:44+00:00

We planned to drop SSTATEPOSTINSTFUNC some time ago with the introduction of postfuncs. Finally get around to doing that which should make the buildhistory code a little more readable. Unfortunately ordering the buildhistory function calls after the sstate ones is difficult without coding that into the sstate class. This patch does that to ensure everything functions as expected until we can find a better way. This is still likely preferable than the generic sstate postfuncs support since the function flow is much more readable. (From OE-Core rev: c9e2a8fa2f0305ef1247ec405555612326f798f8) Signed-off-by: Richard Purdie