diff options
author | Alexander Kanavin <alex@linutronix.de> | 2024-11-21 14:42:42 +0100 |
---|---|---|
committer | Steve Sakoman <steve@sakoman.com> | 2024-11-26 06:11:30 -0800 |
commit | 7bbee0e49e4de46875a557383bc38c017ac536b0 (patch) | |
tree | c96027f9441e93c612c2aa60fa8eab095d63ab97 /bitbake/lib/bb/exceptions.py | |
parent | 71da054b97d002d38aa0cf65d3756a4b147936f7 (diff) | |
download | poky-7bbee0e49e4de46875a557383bc38c017ac536b0.tar.gz |
package_rpm: restrict rpm to 4 threads
TL;DR version:
with this, and the previous compression level changes
I am seeing drastic speedups in package_write_rpm completion times:
webkitgtk goes from 78 seconds to 37 seconds
glibc-locale goes from 399 seconds to 58 seconds (!)
The long version:
rpm uses multithreading for two purposes:
- spawning compressors (which are nowadays themselves
multi-threaded, so the feature is not as useful as it once
was)
- parallel file classification
While the former behaves well on massively parallel CPUs
(it was written and verified here :), the latter was then added
by upstream and only benchmarked on their very old, slow laptop,
apparently:
https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4
On anything more capable it starts showing pathologic behavior,
presumably from spawning massive amount of very short-lived threads,
and then having to synchronize them. For example classifying glibc-locale
takes
5m20s with 256 threads (default on my machine!)
1m49s with 64 threads
59s with 16 threads
48s with 8 threads
Even a more typical recipe like webkitgtk is affected:
47s with 256 threads
32s with 64 threads
27s with 16 or 8 threads
I have found that the optimal amount is actually four: this also
means that only four compressors are running at a time, but
as they're themselves using threads, and typical recipes are dominated
by just two or three large packages, this does not affect overall
completion time.
(From OE-Core rev: 8cbf3a2cddbff1728bd3d1382c3fec00b604501e)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Diffstat (limited to 'bitbake/lib/bb/exceptions.py')
0 files changed, 0 insertions, 0 deletions