linux/poky.git/meta/classes-global/package_rpm.bbclass, branch styhead

package_rpm: restrict rpm to 4 threads

2024-11-30T13:41:59+00:00

TL;DR version: with this, and the previous compression level changes I am seeing drastic speedups in package_write_rpm completion times: webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!) The long version: rpm uses multithreading for two purposes: - spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was) - parallel file classification While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently: https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4 On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes 5m20s with 256 threads (default on my machine!) 1m49s with 64 threads 59s with 16 threads 48s with 8 threads Even a more typical recipe like webkitgtk is affected: 47s with 256 threads 32s with 64 threads 27s with 16 or 8 threads I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time. (From OE-Core rev: ac480775440fba812fd5aa9da73e0e5bc60d46d6) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package_rpm: use zstd's default compression level

2024-11-30T13:41:59+00:00

zstd uses 3 by default, while 19 is the highest and slowest. It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking: https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66 This brings significant speedups in rpm's compression step: for example compressing webkitgtk takes 11s instead of 36s. The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff. (From OE-Core rev: 9ec8b99470ed744c8eca0b23f99ba54210984080) Signed-off-by: Alexander Kanavin Signed-off-by: Steve Sakoman

package_rpm: Check if file exists before open()

2024-11-26T13:37:09+00:00

Exception handler tries to read() /etc/passwd file in sysroot and if file doesn't exist for any reason then it raises FileNotFoundError exception which mask the original source of the problem and makes debugging of the issue more difficult. Fixes: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/codebuild/output/src1899304708/src/build/tmp-container/work/core2-64-oe-linux/emqx-bin/4.3.12/recipe-sysroot/etc/passwd' (From OE-Core rev: 17c8560f323d619b4a98d2276b4c0bf04c8196fd) Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie (cherry picked from commit 4ad9a0e0b11eb7bc5a3dd45fc8945e094ea949e9) Signed-off-by: Steve Sakoman

lib: package: Add file_reverse_translate

2024-06-03T06:47:04+00:00

Adds API to reverse the removal of special characters from file names so it can be correctly done in multiple places without open-coding it. Replace the translation done in the package_rpm.bbclass with the new API (From OE-Core rev: 4cb7e93c624987d146aaf626ce8e99568e938a70) Signed-off-by: Joshua Watt Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

package_rpm: remove support for DIRFILES

2024-04-23T12:40:24+00:00

This was added here: https://git.yoctoproject.org/poky/commit/?id=2f42ef8d8fb1febf28252b98884cebabc931f720 It's undocumented, untested, rpm-specific, and currently broken, and to begin with looks like a workaround for Tizen/smack security label conflicts elsewhere. (From OE-Core rev: ea8af19bb909c7e3633d82bec9d925c8f42ec860) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

package_rpm: add RPMBUILD_EXTRA_PARAMS variable

2024-02-08T10:53:13+00:00

* e.g. for DISTROs which define extra user-defined fields with PACKAGE_ADD_METADATA/PACKAGE_ADD_METADATA_RPM as undefined fields in packagedata are fatal error for rpmbuild: "error: line 9: Unknown tag: Author: Unspecified" as shown in: http://errors.yoctoproject.org/Errors/Details/751706/ with "Author" field added with: PACKAGE_CLASSES = "package_rpm" PACKAGE_ADD_AUTHOR_METADATA = "test-author" PACKAGE_ADD_METADATA = "Author: ${PACKAGE_ADD_AUTHOR_METADATA}" to fix rpm build you can use: RPMBUILD_EXTRA_PARAMS = " --define '_Author Author'" keep in mind that this doesn't cause this Author field to be added in .rpm, it just avoids the BUILDSPEC failure. and for ipk build: OPKG_MAKE_INDEX_EXTRA_PARAMS = "-f" alternatively you can avoid additional packagedata fields ending in the package manager (if you use them only with buildhistory or packagedata) with: PACKAGE_ADD_METADATA_RPM = "" PACKAGE_ADD_METADATA_IPK = "" PACKAGE_ADD_METADATA_DEP = "" (From OE-Core rev: 31030e7a19a27ad424b997fad36ee4f633ce0d63) Signed-off-by: Martin Jansa Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

classes/package_rpm: streamline the logic in one of the condition blocks

2024-02-08T10:53:13+00:00

(From OE-Core rev: e6a8ca554509c0edf9fd36ced88165dc3caf0e87) Signed-off-by: Alexander Kanavin Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

classes/package_rpm: additionally escape \ and " in filenames

2024-02-08T10:53:13+00:00

As specified in: https://github.com/rpm-software-management/rpm/blob/rpm-4.19.x/docs/manual/spec.md#shell-globbing (From OE-Core rev: 78d22c80c3f3a9f82c9f0d3dd1c591d395e02918) Signed-off-by: Alexander Kanavin Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

classes/package_rpm: correctly escape percent characters

2024-02-02T11:06:17+00:00

This many characters doesn't work with rpm 4.19 packaging (as shown by nodejs recipes), and per documentation a single escape is enough: https://github.com/rpm-software-management/rpm/blob/rpm-4.19.x/docs/manual/spec.md#shell-globbing It also should be done in a function, and just before writing out the corrected filename to .spec, not earlier where the path may still be needed for file operations (such as gettings file attributes). (From OE-Core rev: 6d9fe2623c37e405a80acf71633f7291ecdde533) Signed-off-by: Alexander Kanavin Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

classes/package_rpm: set bogus locations for passwd/group files

2024-01-27T08:20:02+00:00

Since https://github.com/rpm-software-management/rpm/commit/f3eaeeb7341085e1850e914350cf1f33d538320d rpm does its own parsing of /etc/passwd and /etc/group instead of relying on getpwnam() and friends. This has an unfortunate effect of leaking build host uid/gid values for users and groups into the cpio header inside rpm file (where previously those were always zero). Installation of rpm packages relies on rpm header to set files ownership, and that is a different structure that is build from .spec information, so we can avoid host contamination by setting the paths to something bogus. (From OE-Core rev: ac8ea64bb39a5e56599e078c7e6cd056a2aa4144) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie