summaryrefslogtreecommitdiffstats
path: root/meta/lib
diff options
context:
space:
mode:
authorRoss Burton <ross@burtonini.com>2021-11-24 17:15:29 +0000
committerRichard Purdie <richard.purdie@linuxfoundation.org>2021-11-25 21:55:10 +0000
commitc6f23f1f0fad29da4dee27a9cb8219ae05a8bfd5 (patch)
tree287aa74517c481c4b2f0b257a4a3a916a7758450 /meta/lib
parent317511ce1fa8bf2270fa86798dc53353e6335087 (diff)
downloadpoky-c6f23f1f0fad29da4dee27a9cb8219ae05a8bfd5.tar.gz
oe/utils: by default cap cpu_count() to 64 cores
Larger systems may have large numbers of cores, but beyond a certain point they can't all be used for compiling: whilst purely compute-intensive jobs can be parallelised to hundreds of cores, operations such as compressing (needs lots of RAM) or compiling (lots of I/O) don't scale linearly. For example, the Marvel ThunderX2 has 32 cores, each capable of executing four threads, and can be configured with two sockets, making 256 CPUs according to Linux. Zstd using 256 threads has been seen to fail to allocate memory during even small recipes such as iso-codes. Add a default cap of 64 CPUs to the cpu_count() method so that extreme parallisation is limited. 64 is high enough that meaningful gains beyond it are unlikely, but high enough that most systems won't be effected. (From OE-Core rev: 765d0f25ce48636b1838a5968e2dc15de2127428) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'meta/lib')
-rw-r--r--meta/lib/oe/utils.py4
1 files changed, 2 insertions, 2 deletions
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 7982b2b511..136650e6f7 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -248,9 +248,9 @@ def trim_version(version, num_parts=2):
248 trimmed = ".".join(parts[:num_parts]) 248 trimmed = ".".join(parts[:num_parts])
249 return trimmed 249 return trimmed
250 250
251def cpu_count(at_least=1): 251def cpu_count(at_least=1, at_most=64):
252 cpus = len(os.sched_getaffinity(0)) 252 cpus = len(os.sched_getaffinity(0))
253 return max(cpus, at_least) 253 return max(min(cpus, at_most), at_least)
254 254
255def execute_pre_post_process(d, cmds): 255def execute_pre_post_process(d, cmds):
256 if cmds is None: 256 if cmds is None: