linux/poky.git/meta/classes/sstate.bbclass, branch 2.7_M1

sstate.bbclass: Only remove sstate file when task is existed

2018-11-23T23:35:19+00:00

This can improve the performance a lot for "bitbake -ccleansstate" when there are a lot of sstate files. For example: * Before $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files: - Without disk caches # echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches (e.g., run it in the second time) $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308s So the 4 removing *package* commands cost more than 20s or 272s in theory. * After $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* We can see that it saved 20s or 272s in theory. (From OE-Core rev: bb2d6349ea87f090c58001f0d4348b24c2982cde) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly

2018-11-23T23:35:18+00:00

The glob.glob("/sstate/*/*/") is very time consuming, set SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time when there are many sstate files. For example, I have more than 600,000 sstate files: * Before - Without disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 0m4.111s user 0m2.348s sys 0m1.756s * After $ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' - Without disk caches: real 0m7.928s user 0m0.172s sys 0m0.124s - With disk caches: real 0m0.131s user 0m0.088s sys 0m0.044s We can see that it saves about 3.8s with disk caches, and saves about 264s without disk caches. (From OE-Core rev: 8b31c919814b8bdf25b3381053656523c001ae0d) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

sstate.bbclass: update SSTATE_DUPWHITELIST

2018-09-13T06:42:28+00:00

Update SSTATE_DUPWHITELIST in sstate.bbclass. * remove ${DEPLOY_DIR_RPM}/noarch/ which is not overwritten any more * add directories for package target-sdk-provides-dummy (From OE-Core rev: 6d3ca476dbc2059f4b7fa3dfd73de6bbfed49198) Signed-off-by: Kai Kang Signed-off-by: Richard Purdie

sstate.bbclass: make SSTATE_PRUNE_OBSOLETEWORKDIR could be overwritten

2018-08-28T09:30:28+00:00

Define variable SSTATE_PRUNE_OBSOLETEWORKDIR with '?=' in sstate.bbclass, then it could be overwritten by user configuration. (From OE-Core rev: 22af59c9bfec31b31027ebd2a4da162f481aa6b5) Signed-off-by: Kai Kang Signed-off-by: Richard Purdie

sstate: add intel-microcode to SSTATE_DUPWHITELIST

2018-08-23T06:50:00+00:00

intel-microcode multilib recipes can generate identical overlapping files: microcode.cpio. (From OE-Core rev: 4b27da3334aff7f9f03ae934bbab7e7af07df3f6) Signed-off-by: Yongxin Liu Signed-off-by: Richard Purdie

sstate: Optimise SSTATE_EXCLUDEDEPS_SYSROOT handling

2018-08-16T08:49:39+00:00

Using re.compile() is around six times faster than recompiling the regexp each time so maintain a cache. (From OE-Core rev: 41eb382737706e245f2b7104e313c8dfaa370945) Signed-off-by: Richard Purdie

sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping base-passwd|shadow-sysroot recipes

2018-08-16T08:49:39+00:00

Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the base-passwd|shadow-sysroot recipes to be excluded from a recipe sysroot. (From OE-Core rev: 68e502e9063a88532fe0154f152ba408f0091900) Signed-off-by: André Draszik Signed-off-by: Richard Purdie

sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping *-initial recipes

2018-08-16T08:49:39+00:00

Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the *-initial recipes to be excluded from a recipe sysroot. (From OE-Core rev: 6706bad52f9311ea79c534ee90014c3216992999) Signed-off-by: André Draszik Signed-off-by: Richard Purdie

sstate: allow specifying indirect dependencies to exclude from sysroot

2018-08-16T08:49:39+00:00

Currently, a dependency on any -native recipe will pull in all dependencies of that -native recipe in the recipe sysroot. This behaviour might not always be wanted, e.g. when that -native recipe depends on build-tools that are not relevant for the current recipe. This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable, which will be evaluated for such recursive dependencies to be excluded. The idea is similar to http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html except that the list is not hard-coded anymore. SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular expressions of recipe and dependency to ignore, e.g. in the above flex-native / bison-native use-case, one would specify SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native" in layer.conf. The existing special handling of "-initial" as well as "base-passwd" and "shadow-sysroot" could also be streamlined: SSTATE_EXCLUDEDEPS_SYSROOT += "\ .*->.*-initial.* \ .*(base-passwd|shadow-sysroot)->.* \ " Another anticipated user is meta-java, where certain newer JDKs can only be bootstrapped (built) using older JDKs, but it doesn't make much sense to copy all those older JDKs and their own build tools (ant, etc.) into the sysroot of recipes wanting to be built using the newer JDK (only), e.g.: SSTATE_EXCLUDEDEPS_SYSROOT += "\ openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \ " (From OE-Core rev: 92c5131a2feae2036c71a36c18bb9175bb2856dc) Signed-off-by: André Draszik Signed-off-by: Richard Purdie

sstate/lib.oe.path: Ensure file sparseness is preserved

2018-08-15T08:44:33+00:00

Files when restored from sstate were missing their sparseness. Fix up various functions to preserve this and make things more deterministic. (From OE-Core rev: 055402e5504f041c346571e243c7cf0894955cad) Signed-off-by: Richard Purdie