linux/poky.git/meta/classes/package_rpm.bbclass, branch yocto-4.0.24

package_rpm: restrict rpm to 4 threads

2024-12-02T14:23:20+00:00

TL;DR version: with this, and the previous compression level changes I am seeing drastic speedups in package_write_rpm completion times: webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!) The long version: rpm uses multithreading for two purposes: - spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was) - parallel file classification While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently: https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4 On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes 5m20s with 256 threads (default on my machine!) 1m49s with 64 threads 59s with 16 threads 48s with 8 threads Even a more typical recipe like webkitgtk is affected: 47s with 256 threads 32s with 64 threads 27s with 16 or 8 threads I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time. (From OE-Core rev: 896192604d84a6f77095f23cd13232e249b7aac5) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package_rpm: use zstd's default compression level

2024-12-02T14:23:20+00:00

zstd uses 3 by default, while 19 is the highest and slowest. It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking: https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66 This brings significant speedups in rpm's compression step: for example compressing webkitgtk takes 11s instead of 36s. The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff. (From OE-Core rev: c377ced95ef7fd060316db1325529826d0985790) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package_rpm: Allow compression mode override

2023-11-01T15:01:26+00:00

Commit 4a4d5f78a6962dda5f63e9891825c80a8a87bf66 ("package_rpm: use zstd instead of xz") changed the rpm package compressor from 'xz' to 'zstd' which results in decompression failure with BusyBox-provided 'rpm2cpio' applet and 'rpm' applet when given the '-i' (Install package) option: rpm2cpio: no gzip/bzip2/xz magic Introduce a variable which makes it possible to use a different compression mode, making it possible to override the default value for example like RPMBUILD_COMPMODE = "${@'w6T%d.xzdio' % int(d.getVar('XZ_THREADS'))}" to enable rpm decompression without including the full rpm package in the resulting root filesystem. (From OE-Core rev: a40d9258148e28cbee2168c93179cd4c1232fb62) (From OE-Core rev: ad4ea9f225b0dd6396088cc70b34f886c5fa62b4) Signed-off-by: Niko Mauno Signed-off-by: Richard Purdie Signed-off-by: Steve Sakoman

package_rpm: Do not replace square brackets in %files

2022-08-31T15:54:17+00:00

This reverts commit f95adb749619e70920c6cc6cd01c6d02cd348fd8. At some point RPM stopped treating "[]?" characters in quoted text as a glob This causes rpmbuild error [1] in oe-core if package contains filenames with such characters [Yocto #13746]. Reverting commit which replaces "[]" with "?" fixes the issue. It should be safe now becuase rpm tries to use filename directly if not found by glob [2]. [1] | error: File not found: /mnt/builds/yocto/build/build/tmp/work/core2-64-poky-linux/testrpm/0.0.1-r0/package/42 All-Time Classics (E) ?v1.1?.cht | File not found: /mnt/builds/yocto/build/build/tmp/work/core2-64-poky-linux/testrpm/0.0.1-r0/package/42 All-Time Classics (E) ?v1.1?.cht | [2] https://github.com/rpm-software-management/rpm/commit/c16c70cbd6b31cd93541d5c22d23ba98d212ad3d (From OE-Core rev: c17806bcd3b16b7c13606325195270d7c799a5c9) Signed-off-by: Pavel Zhukov Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie (cherry picked from commit 9916212b4810280108107d94cd2c0cd3e07ae6ed) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

package_deb/ipk/rpm: Add more minimal do_build dependencies back

2022-01-04T23:14:05+00:00

The dependencies for do_build became a little too minimal after the removal of recrdeptask since "bitbake go" would not package go-runtime despite it being in DEPENDS and the resulting package having a dependency on it. A reasonable compromise is probably rdeptask instead of recrdeptask which is a lot lower overhead but makes the build target more useful and importantly, lets world builds do what you'd expect them to. (From OE-Core rev: f3d02c328f3f182340528d11c7b10454e3f6a54b) Signed-off-by: Richard Purdie

reproducible: Merge code into base.bbclass

2021-10-16T16:41:59+00:00

Reproducibility is here to stay and needs to be part of our default workflow. Move the remaining code to base.bbclass so it is always a first class citizen and it is clear people need to be mindful of it. (From OE-Core rev: abb0671d2cebfd7e8df94796404bbe9c7f961058) Signed-off-by: Richard Purdie

package_rpm: use zstd instead of xz

2021-10-11T17:41:37+00:00

zstd has similar time and space performance in compression but is vastly faster in decompression, which benefits rootfs creation (especially when installing very large packages) and on-target package installation. Also, ensure ZSTD_THREADS doesn't change sstate checksums. The detailed explanation is in the commit making similar change for XZ_THREADS. (From OE-Core rev: e72c0b94554a9bc293844ec2bddb0c04ea19791d) Signed-off-by: Alexander Kanavin Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

reproducible_build/package_XXX: Ensure SDE task is in dependency chain

2021-09-23T12:49:11+00:00

(From OE-Core rev: 938e4dfb68a465f61cb754b2cd28d0728616b5a9) Signed-off-by: Richard Purdie

package_ipk/deb/rpm: Drop recursive do_build task dependencies

2021-09-23T12:49:11+00:00

This is a controversial change which removes the recursive dependencies from the do_build target of packaging tasks of recipes. Currently this means when you "bitbake " or "bitbake ", the packaging tasks run for all packaging backends enabled for all recipes in the dependency chain. The same therefore then applies to images. We don't actually need that, it is a convinience thing. Removing it massively simplifies the task graph and causes much fewer tasks to execute in many common scenarios. It also means less sstate is fetched for example when building an image. This means when building a recipe all package formats would be built but when building an image, only the format used by the image would be generated. This should be an improvement in most cases but some CI systems may need to be explict about what they're building. 'bitbake XXX --runall build' would give the previous behaviour. (From OE-Core rev: 220d29a5abb5be8b80c64d8f2f48949bc8c534a3) Signed-off-by: Richard Purdie

package_rpm/update-alternatives: fix package's provides

2021-08-22T21:21:46+00:00

Currently we have: $ rpm -qp --provides tmp/deploy/rpm/core2_64/bash-5.1.8-r0.core2_64.rpm bash = 5.1.8-r0 This is incorrect as bash provides /bin/bash and /bin/sh. This is caused by incomplete conversion of new override syntax, which breaks the per-file parsing of package_rpm. With this patch, we have: $ rpm -qp --provides tmp/deploy/rpm/core2_64/bash-5.1.8-r0.core2_64.rpm /bin/bash /bin/sh bash = 5.1.8-r0 (From OE-Core rev: f186d32ff8bf176ad0bd8d49e47cdb017c701a46) Signed-off-by: Chen Qi Signed-off-by: Richard Purdie