linux/poky.git/meta/classes, branch yocto-4.0.24

package.bbclass: Use shlex instead of deprecated pipes

2024-12-20T14:01:45+00:00

The pipes library is deprecated in Python 3.11 and will be removed in Python 3.13. pipes.quote is just an import of shlex.quote anyway. (From OE-Core rev: d167661bceebebafb04dca3bf6a888003f46e6c9) Signed-off-by: Ernst Persson Signed-off-by: Steve Sakoman

sanity: check for working user namespaces

2024-12-16T13:58:03+00:00

If user namespaces are not available (typically because AppArmor is blocking them), alert the user. We consider network isolation sufficiently important that this is a fatal error, and the user will need to configure AppArmor to allow bitbake to create a user namespace. [ YOCTO #15592 ] (From OE-Core rev: a069b9f9ee6708022e12970d53262d966ee806ba) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie (cherry picked from commit b6af956fe6e876957a49d4abf425e8c789bf0459) Signed-off-by: Steve Sakoman

rootfs-postcommands.bbclass: make opkg status reproducible

2024-12-16T13:58:03+00:00

opkg stores the current time as Installed-Time in its status file when installing packages to the rootfs. Make this reproducible by replacing Installed-Time with ${REPRODUCIBLE_TIMESTAMP_ROOTFS}, which then also matches the files' datestamps. Based on OpenWrt's approach for the issue [1]. [1] https://github.com/openwrt/openwrt/blob/main/include/rootfs.mk#L103 (From OE-Core rev: 61a9b1b1cb618ce90ba7886036f41263075c07df) (From OE-Core rev: bfa9c2f15ac275fceccf22084bed9a064304eb6e) Signed-off-by: Jonas Gorski Signed-off-by: Richard Purdie Signed-off-by: Peter Marko Signed-off-by: Steve Sakoman

do_package/sstate/sstatesig: Change timestamp clamping to hash output only

2024-12-09T15:54:03+00:00

The code was changing the timestamps of the files in the do_package output, particularly the files added for debug sources. This was to do two things: a) make do_package sstate more reproducible b) ensure better hash equivalence matching Unfortuately the debug source files are hardlinks into the source tree for efficiency so touching these, touches a lot of files in ${B} and ${S}. This causes unpredictable effects if compile is run again for example, or could cause compiling in the install task. The hash equivalence matching is of key importance but we can mimic that using clamping of the file timestamps in the depsig output used to generate the hashes. This patch drops the global timestamp clamping, instead allowing the files to retain their creation timestamps into sstate. This makes do_package sstate slightly less reproducibile. We could clamp the sstate timestamps but that would lead to two different sets of timestamps depending on whether the data came from sstate or not. I'd prefer to have consistent code behaviour, rather than differing behavhour depending on whether data came from sstate or not. If we wanted to have reproducibiliy and fix the "corruption" of S/B and have consistent codepaths, the only other option would be two copies of the sources, which could end up huge and seems the least desireable option. This patch therefore drops the timestamp clamping in the sstate files and tweaks the depsig data generation to clamp the timestamps for do_package instead since this seems the best compromise. I validated that rpm/deb/ipk files still generate correctly as before. (From OE-Core rev: 0c93bb692b39af51f0ca109dfd1f949abe7eea9c) Signed-off-by: Richard Purdie (cherry picked from commit 475759fdab7200488b2a568b2ba1aa31a456d113) Signed-off-by: Steve Sakoman

package_rpm: restrict rpm to 4 threads

2024-12-02T14:23:20+00:00

TL;DR version: with this, and the previous compression level changes I am seeing drastic speedups in package_write_rpm completion times: webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!) The long version: rpm uses multithreading for two purposes: - spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was) - parallel file classification While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently: https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4 On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes 5m20s with 256 threads (default on my machine!) 1m49s with 64 threads 59s with 16 threads 48s with 8 threads Even a more typical recipe like webkitgtk is affected: 47s with 256 threads 32s with 64 threads 27s with 16 or 8 threads I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time. (From OE-Core rev: 896192604d84a6f77095f23cd13232e249b7aac5) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package_rpm: use zstd's default compression level

2024-12-02T14:23:20+00:00

zstd uses 3 by default, while 19 is the highest and slowest. It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking: https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66 This brings significant speedups in rpm's compression step: for example compressing webkitgtk takes 11s instead of 36s. The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff. (From OE-Core rev: c377ced95ef7fd060316db1325529826d0985790) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package: Switch debug source handling to use prefix map

2024-11-11T14:19:18+00:00

Reproducible builds are no longer a configuration option but are required. We also rely on the prefix mapping capability of the compilers now. As such, rewrite the source locating code to use the prefix maps instead of taking a guess about WORKDIR which isn't correct for kernels, gcc, externalsrc and probably more. Instead, iterate the maps to locate any matching source code, keeping in mind that multiple maps may map to one target location. (From OE-Core rev: 80289f49d0c5ca98da1d1558728b8a468aab4326) Signed-off-by: Richard Purdie (cherry picked from commit cbd6144a9769d21371ae0fe04db2adc05f6eed02) Signed-off-by: Steve Sakoman

cve_check: Use a local copy of the database during builds

2024-11-11T14:19:18+00:00

Rtaher than trying to use a sqlite database over NFS from DL_DIR, work from a local copy in STAGING DIR after fetching. (From OE-Core rev: 9b6363994e5715f1d08b98956befd8915c128e85) Signed-off-by: Richard Purdie (cherry picked from commit 03596904392d257572a905a182b92c780d636744) Signed-off-by: Steve Sakoman

overlayfs-etc: add option to skip creation of mount dirs

2024-11-02T13:32:36+00:00

The 'preinit' script can't create mount directories when rootfs is read-only. Add an option to skip this step. The user must make sure that all required directories are already in the rootfs directory layout. Cc: Vyacheslav Yurkov (From OE-Core rev: 302dd4a63f97e23631a62a0b902cc253f6843ab0) Signed-off-by: Baruch Siach Signed-off-by: Alexandre Belloni (cherry picked from commit 3d433d8559467d255bd19af2d0999c65ea24a48d) Signed-off-by: Steve Sakoman

cve-check: add support for cvss v4.0

2024-11-02T13:32:36+00:00

https://nvd.nist.gov/general/news/cvss-v4-0-official-support CVSS v4.0 was released in November 2023 NVD announced support for it in June 2024 Current stats are: * cvss v4 provided, but also v3, so cve-check showed a value sqlite> select count(*) from nvd where scorev4 != 0.0 and scorev3 != 0.0; 2069 * only cvss v4 provided, so cve-check did not show any sqlite> select count(*) from nvd where scorev4 != 0.0 and scorev3 = 0.0; 260 (From OE-Core rev: 358dbfcd80ae1fa414d294c865dd293670c287f0) (From OE-Core rev: 8c20a7badb6e5d6c6c90176e45e90f776df25298) Signed-off-by: Peter Marko Signed-off-by: Mathieu Dubois-Briand Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman