linux/poky.git/meta/lib/oe/utils.py, branch 3.5_M2

oe/utils: by default cap cpu_count() to 64 cores

2021-11-25T21:55:10+00:00

Larger systems may have large numbers of cores, but beyond a certain point they can't all be used for compiling: whilst purely compute-intensive jobs can be parallelised to hundreds of cores, operations such as compressing (needs lots of RAM) or compiling (lots of I/O) don't scale linearly. For example, the Marvel ThunderX2 has 32 cores, each capable of executing four threads, and can be configured with two sockets, making 256 CPUs according to Linux. Zstd using 256 threads has been seen to fail to allocate memory during even small recipes such as iso-codes. Add a default cap of 64 CPUs to the cpu_count() method so that extreme parallisation is limited. 64 is high enough that meaningful gains beyond it are unlikely, but high enough that most systems won't be effected. (From OE-Core rev: 765d0f25ce48636b1838a5968e2dc15de2127428) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

oe/utils: allow naming threads in ThreadedPool

2021-11-25T21:55:10+00:00

When looking at logs involving thread pools it is useful if the threads can be named. (From OE-Core rev: 18342945b021608794d83ecf567afd43f4379b24) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

oe/utils: log exceptions in ThreadedWorker functions

2021-10-18T12:48:16+00:00

If the function a ThreadedWorker is executing raises an exception, don't use print() as that mostly disappears. Instead, output it to the logger. This is done using bb.mainlogger.debug directly instead of bb.debug() as this allows us to pass the exception instance directly, which is then incorporated into the log stream. (From OE-Core rev: 2f1ea25c222b344dd8b784b2bc73a6540ab30274) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

buildhistory: Label packages providing per-file dependencies in depends.dot

2021-09-01T13:06:18+00:00

Currently, depends.dot includes per-file dependencies but not the packages providing those files. This makes it hard to obtain all package dependencies by just looking at depends.dot. Parse the RPROVIDES and FILERPROVIDES fields from pkgdata to map each of their values to the package providing the component. Include runtime packages as dependencies in depends.dot, together with the component provided by the package as a label. (From OE-Core rev: 2ba33093017574bbe29eeba699eb90628614d03a) Signed-off-by: Andres Beltran Signed-off-by: Richard Purdie

lib/oe/utils: add directory size function

2021-03-28T21:28:26+00:00

For the purpose of image construction using du on a rootfs directory isn't entirely satisfactory. Bare "du" will report the actual disk usage so file systems which can compress the data will report less than the actual space required. Using "du --apparent-size" will report the actual space used, but as this simply sums the bytes used for content across an entire file system can result in significant under-reporting due to block size overhead. Attempt to solve these problems by implementing our own function to calculate how large a rootfs will be. This function handles hardlinks correctly but rounds up all sizes to multiples of the block size (currently, 4KB is the hard-coded block size). (From OE-Core rev: 6ca53ad7b26ee2b4e6d2c121c6f6d6eed7f6b56f) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

lib/oe/utils: Return empty string in parallel_make

2021-01-16T22:39:35+00:00

In cmake.bbclass we set CMAKE_BUILD_PARALLEL_LEVEL using parallel_make function and if PARALLEL_MAKE is set to empty string then this variable is exported as "None" causing cmake to fail with: "'CMAKE_BUILD_PARALLEL_LEVEL' environment variable invalid number 'None' given." (From OE-Core rev: 2f790ded554a52ac18d1c28002142f9c62abec8b) Signed-off-by: Tomasz Dziendzielski Signed-off-by: Richard Purdie

utils: respect scheduler affinity in cpu_count()

2020-09-30T14:01:51+00:00

cpu_count() returns multiprocessing.cpu_count() but that is simply returns os.cpu_count() so we could use that directly. However this returns the number of CPUs on the host, not the number of usable CPUs on the host. If the user is using scheduler affinity then the number of usable CPUs may be less, so when determining how many cores we can use check the affinity instead. (From OE-Core rev: e2e8ccbe410b2f38bcd9525982b2261cf71aab60) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

lib/oe/utils.py: add get_host_compiler_version()

2020-04-01T10:44:24+00:00

Add helper function to get the host compiler and version. Do not assume compiler is gcc. NOTE: cannot set env to d.getVar("PATH") as that does not contain the session PATH which was set by environment-setup-... which breaks the install-buildtools use-case (From OE-Core rev: 88712929354ff9c876bb1e48b6f15c33af5f2bbc) Signed-off-by: Tim Orling Signed-off-by: Richard Purdie

utils: fix gcc 10 version detection

2020-03-29T11:05:31+00:00

Utils can not detect GCC 10 correctly due to wrong regex. It generates this error "ERROR: Can't get compiler version from gcc --version output" Sub-version numbers should be 1 or more digits instead of 1 only. (From OE-Core rev: 186fe4a3d390a52b87282c3e694ce3251e45ee78) Signed-off-by: Charles-Antoine Couret Signed-off-by: Richard Purdie

lib/oe/utils: allow to set a lower bound on returned cpu_count()

2020-03-06T08:19:17+00:00

This will be needed for making xz compression more deterministic, as xz archives are created differently in single- vs multi-threaded modes. This means that due to bitbake's default of using as many threads as there are cores in the system, files compressed with xz will be different if built on a multi-core system compared to single-core systems. Allowing cpu_count() here to return a lower bound, will allow forcing xz to always use multi-threaded operation. (From OE-Core rev: c6dda204e15c5b27559f7617b18a7b3ce604e2f8) Signed-off-by: André Draszik Signed-off-by: Richard Purdie