summaryrefslogtreecommitdiffstats
path: root/meta/classes/sstate.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* sstate.bbclass: make sure changes to SSTATE_SCAN_FILES are not ignoredAlexander Kanavin2019-01-241-0/+1
| | | | | | | | | | | | | When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild, so if there's a sstate-cache available with "bad" sstate data in it that will still be used even though the recipe is updated to address this. [YOCTO #13144] (From OE-Core rev: ea3526961920a229e0bb5fb459952be89fce2255) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Update output hashJoshua Watt2019-01-221-62/+3
| | | | | | | | | | | | | | | | | Updates the output hash calculation for determining if tasks are equivalent. The new algorithm does the following based on feedback: 1) The output hash function was moved to the OE library. 2) All files are printed in a single line tabular format 3) Prints the file type and mode in a user-friendly ls-like format 4) Includes the file owner and group (by name, not ID). These are only included if the task is run under pseudo since that is the only time they can be consistently determined. 5) File size is included for regular files (From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: remove dependencies of SSTATE_EXTRAPATHWILDCARDRobert Yang2019-01-161-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | There might be do_cleansstate errors sometimes: ERROR: When reparsing /path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate, the basehash value changed from b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata is not deterministic and this needs to be fixed. The stable reproducer is: - Initial a fresh build, this is a must, otherwise we may can't reproduce it $ . oe-init-build-env build $ bitbake quilt-native -ccleansstate This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g., ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as SSTATE_EXTRAPATH did can fix the problem. (From OE-Core rev: 781117f9f02c0080dadc8797a8f8f9377a99b164) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar calls (again)André Draszik2019-01-141-2/+2
| | | | | | | | | | | | | | | | A couple have still been missed in the past despite multiple attempts at doing so (or simply have re-appeared?). Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Implement hash equivalence sstateJoshua Watt2019-01-081-8/+97
| | | | | | | | | | | | | | | | | | | | | Converts sstate so that it can use a hash equivalence server to determine if a task really needs to be rebuilt, or if it can be restored from a different (equivalent) sstate object. The unique hashes are cached persistently using persist_data. This has a number of advantages: 1) Unique hashes can be cached between invocations of bitbake to prevent needing to contact the server every time (which is slow) 2) The value of each tasks unique hash can easily be synchronized between different threads, which will be useful if bitbake is updated to do on the fly task re-hashing. [YOCTO #13030] (From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Handle unihash in hash checkJoshua Watt2019-01-081-1/+1
| | | | | | | | | | | | Handles the argument that passes task unique hash in the hash check function, as it is now required by bitbake [YOCTO #13030] (From OE-Core rev: 1c14b6969e58b51a325c1c1acf9c96e55675035c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add support for caching shared workdir tasksMichael Ho2019-01-031-0/+6
| | | | | | | | | | | | | | | | | The sstate bbclass uses workdir as a hardcoded string in path manipulations. This means that the sstate caching mechanism does not work for the work-shared directory which the kernel uses to share its build configuration and source files for out of tree kernel modules. This commit modifies the path manipulation mechanism to use the work-shared directory if detected in the paths when handling the sstate cache packages. (From OE-Core rev: 27642449f95e38598f9c83948ce109c5891e5877) Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: catch ValueError from incorrectly written sstate manifestsMartin Jansa2019-01-031-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * in some rare cases the file format might be broken, e.g. when you run out of disk space while writing to it (hint: make sure to use BB_DISKMON_DIRS on _all_ the builds, to make sure that they can stop gracefully before this happens). * will show error like this: ERROR: Invalid line '/OE/build/luneos-thud/webos-ports' in sstate manifest '/OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch' when the file isn't complete, like in my case: $ tail -n 2 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch /OE/build/luneos-thud/webos-ports/tmp-glibc/stamps/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/manifest-allarch-org.webosports.app.calculator /OE/build/luneos-thud/webos-ports/tmp-glibc/work/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports * instead of much longer exception which doesn't really show what's wrong to selectively fix that (other than removing while TMPDIR): ERROR: Command execution failed: Traceback (most recent call last): File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/command.py", line 113, in runAsyncCommand self.cooker.updateCache() File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/cooker.py", line 1562, in updateCache bb.event.fire(event, self.databuilder.mcdata[mc]) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 225, in fire fire_class_handlers(event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 134, in fire_class_handlers execute_handler(name, handler, event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 106, in execute_handler ret = handler(event) File "/OE/build/luneos-thud/webos-ports/openembedded-core/meta/classes/sstate.bbclass", line 1083, in sstate_eventhandler2 (stamp, manifest, workdir) = l.split() ValueError: not enough values to unpack (expected 3, got 1) (From OE-Core rev: 870ba5d80e76e0f989971532fc2adc0ebe811ab6) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Only remove sstate file when task is existedRobert Yang2018-11-231-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk> -ccleansstate" when there are a lot of sstate files. For example: * Before $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files: - Without disk caches # echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches (e.g., run it in the second time) $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308s So the 4 removing *package* commands cost more than 20s or 272s in theory. * After $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* We can see that it saved 20s or 272s in theory. (From OE-Core rev: bb2d6349ea87f090c58001f0d4348b24c2982cde) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitlyRobert Yang2018-11-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The glob.glob("/sstate/*/*/") is very time consuming, set SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time when there are many sstate files. For example, I have more than 600,000 sstate files: * Before - Without disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 0m4.111s user 0m2.348s sys 0m1.756s * After $ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' - Without disk caches: real 0m7.928s user 0m0.172s sys 0m0.124s - With disk caches: real 0m0.131s user 0m0.088s sys 0m0.044s We can see that it saves about 3.8s with disk caches, and saves about 264s without disk caches. (From OE-Core rev: 8b31c919814b8bdf25b3381053656523c001ae0d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: update SSTATE_DUPWHITELISTKai Kang2018-09-131-1/+3
| | | | | | | | | | | | Update SSTATE_DUPWHITELIST in sstate.bbclass. * remove ${DEPLOY_DIR_RPM}/noarch/ which is not overwritten any more * add directories for package target-sdk-provides-dummy (From OE-Core rev: 6d3ca476dbc2059f4b7fa3dfd73de6bbfed49198) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make SSTATE_PRUNE_OBSOLETEWORKDIR could be overwrittenKai Kang2018-08-281-1/+1
| | | | | | | | | | Define variable SSTATE_PRUNE_OBSOLETEWORKDIR with '?=' in sstate.bbclass, then it could be overwritten by user configuration. (From OE-Core rev: 22af59c9bfec31b31027ebd2a4da162f481aa6b5) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add intel-microcode to SSTATE_DUPWHITELISTYongxin Liu2018-08-231-1/+2
| | | | | | | | | | intel-microcode multilib recipes can generate identical overlapping files: microcode.cpio. (From OE-Core rev: 4b27da3334aff7f9f03ae934bbab7e7af07df3f6) Signed-off-by: Yongxin Liu <yongxin.liu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Optimise SSTATE_EXCLUDEDEPS_SYSROOT handlingRichard Purdie2018-08-161-3/+10
| | | | | | | | | Using re.compile() is around six times faster than recompiling the regexp each time so maintain a cache. (From OE-Core rev: 41eb382737706e245f2b7104e313c8dfaa370945) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping ↵André Draszik2018-08-161-3/+0
| | | | | | | | | | | | base-passwd|shadow-sysroot recipes Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the base-passwd|shadow-sysroot recipes to be excluded from a recipe sysroot. (From OE-Core rev: 68e502e9063a88532fe0154f152ba408f0091900) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping *-initial recipesAndré Draszik2018-08-161-3/+0
| | | | | | | | | | Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the *-initial recipes to be excluded from a recipe sysroot. (From OE-Core rev: 6706bad52f9311ea79c534ee90014c3216992999) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: allow specifying indirect dependencies to exclude from sysrootAndré Draszik2018-08-161-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, a dependency on any -native recipe will pull in all dependencies of that -native recipe in the recipe sysroot. This behaviour might not always be wanted, e.g. when that -native recipe depends on build-tools that are not relevant for the current recipe. This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable, which will be evaluated for such recursive dependencies to be excluded. The idea is similar to http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html except that the list is not hard-coded anymore. SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular expressions of recipe and dependency to ignore, e.g. in the above flex-native / bison-native use-case, one would specify SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native" in layer.conf. The existing special handling of "-initial" as well as "base-passwd" and "shadow-sysroot" could also be streamlined: SSTATE_EXCLUDEDEPS_SYSROOT += "\ .*->.*-initial.* \ .*(base-passwd|shadow-sysroot)->.* \ " Another anticipated user is meta-java, where certain newer JDKs can only be bootstrapped (built) using older JDKs, but it doesn't make much sense to copy all those older JDKs and their own build tools (ant, etc.) into the sysroot of recipes wanting to be built using the newer JDK (only), e.g.: SSTATE_EXCLUDEDEPS_SYSROOT += "\ openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \ " (From OE-Core rev: 92c5131a2feae2036c71a36c18bb9175bb2856dc) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/lib.oe.path: Ensure file sparseness is preservedRichard Purdie2018-08-151-2/+2
| | | | | | | | | Files when restored from sstate were missing their sparseness. Fix up various functions to preserve this and make things more deterministic. (From OE-Core rev: 055402e5504f041c346571e243c7cf0894955cad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure a given machine only removes things which it createdRichard Purdie2018-08-151-1/+16
| | | | | | | | | | | | | | | | | | | | | | Currently if you build qemux86 and then generic86, the latter will remove all of the former from deploy and workdir. This is because qemux86 is i586, genericx86 is i686 and the architctures are compatible therefore the sstate 'cleaup' code kicks in. There was a valid reason for this to ensure i586 packages didn't get into an i686 rootfs for example. With the rootfs creation being filtered now, this is no longer necessary. Instead, save out a list of stamps which a give machine has ever seen in a given build and only clean up these things if they're no longer "reachable". In particular this means the autobuilder should no longer spend a load of time deleting files when switching MACHINE, improving build times. (From OE-Core rev: 5634f2fb1740732056d2c1a22717184ef94405bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Remove DEPLOY_DIR_IMAGE from SSTATE_DUPWHITELISTRichard Purdie2018-08-141-1/+5
| | | | | | | | | | | | Replace the generic whitelist entry with entries for the three specific 'problem' cases in OE-Core. This means the general DEPLOY_DIR_IMAGE entry doesn't mask problems for others as was recently encoutered by users reported on irc. In the whitelisted cases they occur only in multilib builds and the files are identical. (From OE-Core rev: 05f6042a40bb772f7ce8d6819c5b2937d8c9808d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add sstate usage summary to the buildRichard Purdie2018-08-081-0/+12
| | | | | | | | | | | | | | | | | Currently the user has no indication of how much sstate was already present or that would be used by the build. This change adds some summary information so that the user can see how much reuse is occurring. To fully work it needs some extra information from a recent bitbake commit but this is optional. When combined with bitbake --dry-run this feature can be used to check if sstate would be reused in a build. [YOCTO #12749] (From OE-Core rev: 596f76029ccb6f87c3b049552bd08f5034c41d9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/bitbake.conf: Use pigz if availableRichard Purdie2018-07-241-2/+9
| | | | | | | | | | | | | | Currently the compression of sstate objects is single threaded. In the case of ltp, this takes around 33s. If we add pigz into the list of non-fatal HOSTTOOLS and then use if it available when building the sstate object, this time drops to around 6s. Since pigz is now widely available this is an optimisation we should utilise. (From OE-Core rev: 2de56aa0792ec93445130d801936a8ea643fad27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Remove unused argumentJoshua Watt2018-07-061-2/+2
| | | | | | | | | Removes an unused argument to pstaging_fetch() (From OE-Core rev: ab0eb2bf0db6e7da7c9b9bb4d46621dbf76dcc2a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Remove trailing whitespaceJoshua Watt2018-07-061-1/+1
| | | | | | | (From OE-Core rev: dc7c181f231db1290ea7d7d5e42476e70a488d73) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: drop obsolete codesChen Qi2018-05-041-5/+0
| | | | | | | | | | | | | | | The SSTATECLEANFUNCS mechanism was introduced to solve user/group deletion problem. After RSS mechanism was introduced, there's no need to do so. There was a patch to remove these obsolete codes for useradd.bbclass, but the codes in sstate.bbclass were not removed. So clean it up. (From OE-Core rev: 215b83ce892a7002ed0b1bd7b82a08e67ae15121) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Add ${PACKAGE_EXTRA_ARCHS} to SSTATE_ARCHSPeter Kjellerstedt2018-04-101-2/+3
| | | | | | | | | | | | | | | | | This makes sure files provided by packages that use any of the extra architectures defined using ${PACKAGE_EXTRA_ARCHS} are cleaned from tmp/sysroot-components when sstate_eventhandler2() executes. Without this, changing a package from using one of the extra architectures to another architecture would lead to files being leftover in tmp/sysroot-components, which could later be picked up when creating the RSS for another package rather than the files that belonged to the updated package. (From OE-Core rev: b959b3bd92aa2939d407800fc4167b0535fd4674) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: sstate_hardcode_path(): fix for multilibRobert Yang2018-01-111-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It only substituted staging_target for target recipe which didn't work for multilib, for example, postinst-useradd-lib32-polkit: * No multilib: PATH=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot-native/bin staging_target=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot The PATH would be substituted to: FIXMESTAGINGDIRTARGET-native/bin Not the funny "-native/bin", this works well. * When multilib: PATH=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/recipe-sysroot-native/bin staging_target=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/lib32-recipe-sysroot Now staging_target endswith "/lib32-recipe-sysroot", so it can't replace '/recipe-sysroot-native' in PATH , so PATH can't be fixed, and there would be build errors when building multilib + rm_work, for example: chown: invalid user: ‘polkitd:root’ Substitute staging_host for target recipe can fix the problem, now all of native, cross and target need substitute staging_host, so we can simply the code a little. (From OE-Core rev: 087510795331fa21ff52f103269087c06b1660fa) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* nativesdk/sdk: Update sdk dummy providersRichard Purdie2018-01-111-0/+3
| | | | | | | | | | | | | | | | | | When we migrated rpm v5 -> v4, we lost the ability to drop "per file" dependencies from the rpm backend for things like "/bin/bash" and "/usr/bin/env" which meant the sdks were becomming 'bloated'. This restores the functionality using a dummy package, similarly to the way the buildtools perl issue was addressed. It also removes the non-functional old code so as not to confuse people in future. I ran into this problem trying to filter dependencies to only rpms a build directly depends upon and it turns out we have some determinism issues in this area so this is something key to fix. (From OE-Core rev: 9d490dc01dcedb216129b22cbe17a6c99efc4f5c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Add progress bars when removing from the sysrootsPeter Kjellerstedt2017-12-101-14/+22
| | | | | | | | | | | If there are many recipes that should be cleaned out from the sysroots, it can actually take some time. This adds a progress bar to give a visual clue that the clean up is actually progressing. (From OE-Core rev: d9b59df1230a20c7a5c9f4fb0325bb9216025a16) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: provide an exception for useradd scenarioMaxin B. John2017-10-161-1/+2
| | | | | | | | | | | | | | | Packages, which depend on users/groups created from other packages, needs "shadow-native" as a build time dependency. So, add an exception to the "shadow-native" from otherwise discarded native/cross tools dependency. Fixes [YOCTO #11960] (From OE-Core rev: 979699b55214933e0f11727a2fb9bfda8a3a3870) Signed-off-by: Maxin B. John <maxin.john@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: rework source extraction so that dependencies are handledPaul Eggleton2017-09-181-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since it was first implemented, devtool's source extraction (as used by the devtool modify, extract and upgrade subcommands) ignored other recipe dependencies - so for example if you ran devtool modify on a recipe that fetches from svn or is compressed using xz then it would fail if those dependencies hadn't been built first. Now that we can execute tasks in the normal way (i.e. tinfoil.build_targets()) then we can rework it to use that. This is slightly tricky in that the source extraction needs to insert some logic in between tasks; luckily we can use a helper class that conditionally adds prefuncs to make that possible. Some side-effects / aspects of this change worth noting: * Operations are a little slower because we have to go through the task dependency graph generation and other startup processing. There's not really any way to avoid this though. * devtool extract didn't used to require a workspace, now it does because it needs to create a temporary bbappend for the recipe. (As with other commands the workspace be created on the fly if it doesn't already exist.) * I want any existing sysroot files and stamps to be left alone during extraction since we are running the tasks off to the side, and especially devtool extract should be able to be used without touching these. However, this was hampered by the automatic removal process in sstate.bbclass triggered by bb.event.ReachableStamps when the task signatures change, thus I had to introduce a way to disable this removal on a per-recipe basis (we still want it to function for any dependencies that we aren't working on). To implement this I elected to use a file written to tmp/sstate-control which gets deleted automatically after reading so that there's less chance of stale files affecting future sessions. I could have used a variable but this would have needed to be whitelisted and I'd have to have poked its value in using the setVariable command. Fixes [YOCTO #11198]. (From OE-Core rev: 830dbd66992cbb9e731b48d56fddf8f220349666) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve SSTATE_SKIP_CREATIONRichard Purdie2017-07-311-6/+4
| | | | | | | | | | | | | | | | Currently, dummy archives are created for skipped sstate tasks. We relied on these never being installed (the setscene task is missing) however this leads to odd behaviour as for example the setscene stamps are never looked at. Rather than trying to continue with the two separate behaviours, really skip package creation. We do need the file manipulation steps to install files under sstate control from a manifest perspective so we only skip at the final creation step. (From OE-Core rev: 1e95fc5419a131a5e4091b8ff01639de99284399) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: Fix return value checks from subprocess.call()'sMikko Rapeli2017-06-281-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Python function subprocess.call() returns the return value of the executed process. If return values are not checked, errors may go unnoticed and bad things can happen. Change all callers of subprocess.call() which do not check for the return value to use subprocess.check_call() which raises CalledProcessError if the subprocess returns with non-zero value. https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module All users of the function were found with: $ git grep "subprocess\.call" | \ egrep -v 'if.*subprocess\.call|=\ +subprocess\.call|return.*subprocess\.call' Tested similar patch on top of yocto jethro. Only compile tested core-image-minimal on poky master branch. (From OE-Core rev: 578c8205fd14c48c6d30ef2889d86f1b4aee060a) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure native/cross recipes have relocation of HOSTTOOLS_DIRRichard Purdie2017-05-121-1/+1
| | | | | | | | | | The previous change to relocate HOSTTOOLS wasn't complete as some files, particularly in gcc stashed build directories were not being correctly relocated. This patch addresses the issue. (From OE-Core rev: 21dd36cc12a033b012544c5d15a6f8afd84dabc9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass, staging.bbclass: Handle HOSTTOOLS_DIR when restoring statePeter Kjellerstedt2017-05-011-1/+1
| | | | | | | | | | | Paths to host tools that have been copied to ${HOSTTOOLS_DIR} may end up in the sstate cache. They thus need to be corrected when restoring from the sstate cache. (From OE-Core rev: f8671aecf05a286dd2b34b07bb5fbbe0c31e26d0) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base-passwd/useradd: Various improvements to useradd with RSSRichard Purdie2017-04-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently there are multiple issues with useradd: * If base-passwd rebuilds, it wipes out recipe specific user/group additions to sysroots and causes errors * If recipe A adds a user and recipe B depends on A, it can't see any of the users/groups A adds. This patch changes base-passwd so it always works as a postinst script within the sysroot and copies in the master files, then runs any postinst-useradd-* scripts afterwards to add additional user/groups. The postinst-useradd-* scripts are tweaked so that if /etc/passwd doesn't exist they just exit, knowning they'll be executed later. We also add a dummy entry to the dummy passwd file from pseudo so we can avoid this too. There is a problem where if recipe A adds a user and recipe B depends on A but doesn't care about users, it may not have a dependency on the useradd/groupadd tools which would therefore not be available in B's sysroot. We therefore also tweak postinst-useradd-* scripts so that if the tools aren't present we simply don't add users. If you need the users, you add a dependency on the tools in the recipe and they'll be added. We add postinst-* to SSTATE_SCAN_FILES since almost any postinst script of this kind is going to need relocation help. We also ensure that the postinst-useradd script is written into the sstate object as the current script was only being added in a recipe local way. Thanks to Peter Kjellerstedt <pkj@axis.com> and Patrick Ohly for some pieces of this patch. [Yocto #11124] (From OE-Core rev: 1b5afaf437f7a1107d4edca8eeb668b9618a5488) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: check if mirror directory is writableEd Bartosh2017-04-131-1/+1
| | | | | | | | | | | | | | | | | | | | Commit 51edde653707e7a3cd2186082458f01f32cd1996 makes a wrong assumption that SSTATE_MIRRORS have write permissions. A mirror is by definition outside of it's user control. In my use case it happens I does not have permissions to update the access time of the dereferenced symbolic-link file. Checked if file is writable before changing its atime. Thanks to Paulo Neves for the patch. [YOCTO #11307] (From OE-Core rev: b8f26c011d5ed888d85fef040bd821400d54c8fe) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Skip glibc do_stash_locale and gcc do_gcc_stash_builddir tasksRichard Purdie2017-04-101-0/+4
| | | | | | | | | | We never need these tasks as dependencies of other sstate tasks since they're only ever needed to build artefacts so we can always skip them and save some time/space. (From OE-Core rev: 246df3df4b7da4b75de0745938438124c2b1d4a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure installation directory is empty before executionRichard Purdie2017-03-221-0/+5
| | | | | | | | | | | | | | | | | | When you enable the systemd DISTRO_FEATURE, opkg-native contains systemd units which have a relocation fixme list. When systemd isn't in DISTRO_FEATURES, there are no fixmes required. Unfortunately as sstate isn't cleaning up its installation directory before use, if you install the systemd version, then install the non-systemd version from sstate, it would leave behind the fixme file from the systemd version and breakage results as it would try and fixup files which don't exist. The solution is to ensure the unpack/install directory is clean before use. It does raise other questions about opkg-native, systemd and DISTRO_FEATURES but there is an underlying sstate issue here too which would cause missing file failures. (From OE-Core rev: d1d55041e38b12d40f896834b56475ea19a6047f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_rpm.bbclass: make architecture-independent .rpm packages "noarch" ↵Alexander Kanavin2017-03-141-1/+1
| | | | | | | | | | | instead of "all" Too many places in dnf/rpm4 stack make that assumption; let's not fight against it. (From OE-Core rev: 341810aff923ace6b1cc1e15e19383c4f8773b51) Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: update .siginfo atimeEd Bartosh2017-03-101-0/+2
| | | | | | | | | | | | | | | | | | | .siginfo files are not being accessed from local or NFS-mounted sstate mirrors when sstate package is installed, so their atime is not updated. If sstate mirror is cleaned based on access time, they get deleted, even though they are still being used. Updated atime of .siginfo symlinks with 'touch -a'. This command dereferences symlinks pointing to the local mirror and updates atime of the .siginfo file on the mirror. [YOCTO #10857] (From OE-Core rev: fb1499a42756faeef025122bbde98bc14f4ae61e) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* kernel, license, sstate, rootfs.py: Remove deploy directory READMEMike Crowe2017-03-011-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | It isn't clear that the README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt file in the deploy directory warrants the complexity it brings elsewhere. Let's just remove it entirely. In particular, if two do_image_complete tasks run in parallel they risk both trying to put their image into ${DEPLOY_DIR_IMAGE} at the same time. Both will contain a README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt file. In theory this should be safe because "cp -alf" will just cause one to overwrite the other. Unfortunately, coreutils cp also has a race[1] which means that if one copy creates the file at just the wrong point the other will fail with: cp: cannot create hard link ‘..../tmp-glibc/deploy/images/pantera/README_-_DO_NOT_DELETE_FILES_IN_THIS_D.txt’ to +‘..../tmp-glibc/work/rage_against-oe-linux-gnueabi/my-own-image/1.0-r0/deploy-my-own-image-complete/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt’: File exists [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25680 (From OE-Core rev: 71e9e88847d7000781642ea6187ebd8f40dfdcfe) Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: Drop now unneeded update_data callsRichard Purdie2017-02-151-2/+0
| | | | | | | | | | Now that the datastore works dynamically we don't need the update_data calls so we can just remove them. They're not actually done anything at all for a while. (From OE-Core rev: 8de0c5d3bd01919e2bf0394f9c485936d6098cec) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Make absolute symlinks an errorRichard Purdie2017-02-091-30/+12
| | | | | | | | | | | | The current relocation code is broken, at least in the native case. Fixing it would mean trying pass in new data on sstate tasks about the relative positioning of symlinks compared to the sstate relocation paths. Whilst we could do this, right now I'm favouring making this an error and fixing the small number of problematic recipes we have in OE-Core (3). (From OE-Core rev: cf94de4ddee3e5072da8608c9151301fcec02cd0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/staging: Batch log messages for performanceRichard Purdie2017-02-011-4/+10
| | | | | | | | | | | | | | | | | | | According to profile data, repeated calls to bb.debug and bb.note in the extend_recipe_sysroot() codepath were accounting for 75% of the time (1.5s) in calls from tasks like do_image_complete. This batches up the log messages into one call into the logging system which gives similar behaviour to disabling the logging but retains the debug information. Since setscene_depvalid is also called from bitbake's setscene code, we have to be a little creative with the function parameters and leave the other debug output mechanism in place. This should hopefully speed up recipe specific sysroots. (From OE-Core rev: 3b0af8dc0f796345d1f1ba77ea35bbd090a5feb3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Make SSTATE_SCAN_CMD vars configurable using weak defaults.Kristian Amlie2017-01-311-7/+2
| | | | | | | | | | | | | | | For this we move them out of the python section so they can be overridden on a per-recipe basis. The motivation for this change is that not all tool chains need the path modifications provided by the command, and these will provide alternative or empty commands. The Go compiler is such an example. (From OE-Core rev: 7d2a2160336413736dd4640f5b84ba4d74bb00f5) Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Allow sstate_clean_manifest to take a prefixRichard Purdie2017-01-281-1/+3
| | | | | | | | | | Manifest files containing the same duplicated prefix are wasteful on space and ultimately this costs build time. Add support for manifest files with common prefixes removed and use the prefix if the path isn't absolute. (From OE-Core rev: 71e8c98238d627f9665faee9d7c226b9c07bd256) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/oeqa: Replace subprocess.check_call() with check_output()Richard Purdie2017-01-231-2/+2
| | | | | | | | | | If you use subprocess.check_output() the traceback will contain the output when the command fails which is very useful for debugging. There is no good reason not to use this everywhere. (From OE-Core rev: ad750dd1cc9d789abe723daddd098ce41d8547f5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Switch to Recipe Specific SysrootsRichard Purdie2017-01-231-41/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch is comparatively large and invasive. It does only do one thing, switching the system to build using recipe specific sysroots and where changes could be isolated from it, that has been done. With the current single sysroot approach, its possible for software to find things which aren't in their dependencies. This leads to a determinism problem and is a growing issue in several of the market segments where OE makes sense. The way to solve this problem for OE is to have seperate sysroots for each recipe and these will only contain the dependencies for that recipe. Its worth noting that this is not task specific sysroots and that OE's dependencies do vary enormously by task. This did result in some implementation challenges. There is nothing stopping the implementation of task specific sysroots at some later point based on this work but that as deemed a bridge too far right now. Implementation details: * Rather than installing the sysroot artefacts into a combined sysroots, they are now placed in TMPDIR/sysroot-components/PACKAGE_ARCH/PN. * WORKDIR/recipe-sysroot and WORKDIR/recipe-sysroot-native are built by hardlinking in files from the sysroot-component trees. These new directories are known as RECIPE_SYSROOT and RECIPE_SYSROOT_NATIVE. * This construction is primarily done by a new do_prepare_recipe_sysroot task which runs before do_configure and consists of a call to the extend_recipe_sysroot function. * Other tasks need things in the sysroot before/after this, e.g. do_patch needs quilt-native and do_package_write_deb needs dpkg-native. The code therefore inspects the dependencies for each task and adds extend_recipe_sysroot as a prefunc if it has populate_sysroot dependencies. * We have to do a search/replace 'fixme' operation on the files installed into the sysroot to change hardcoded paths into the correct ones. We create a fixmepath file in the component directory which lists the files which need this operation. * Some files have "postinstall" commands which need to run against them, e.g. gdk-pixbuf each time a new loader is added. These are handled by adding files in bindir with the name prefixed by "postinst-" and are run in each sysroot as its created if they're present. This did mean most sstate postinstalls have to be rewritten but there shouldn't be many of them. * Since a recipe can have multiple tasks and these tasks can run against each other at the same time we have to have a lock when we perform write operations against the sysroot. We also have to maintain manifests of what we install against a task checksum of the dependency. If the checksum changes, we remove its files and then add the new ones. * The autotools logic for filtering the view of m4 files is no longer needed (and was the model for the way extend_recipe_sysroot works). * For autotools, we used to build a combined m4 macros directory which had both the native and target m4 files. We can no longer do this so we use the target sysroot as the default and add the native sysroot as an extra backup include path. If we don't do this, we'd have to build target pkg-config before we could built anything using pkg-config for example (ditto gettext). Such dependencies would be painful so we haven't required that. * PKDDATA_DIR was moved out the sysroot and works as before using sstate to build a hybrid copy for each machine. The paths therefore changed, the behaviour did not. * The ccache class had to be reworked to function with rss. * The TCBOOTSTRAP sysroot for compiler bootstrap is no longer needed but the -initial data does have to be filtered out from the main recipe sysroots. Putting "-initial" in a normal recipe name therefore remains a bad idea. * The logic in insane needed tweaks to deal with the new path layout, as did the debug source file extraction code in package.bbclass. * The logic in sstate.bbclass had to be rewritten since it previously only performed search and replace on extracted sstate and we now need this to happen even if the compiled path was "correct". This in theory could cause a mild performance issue but since the sysroot data was the main data that needed this and we'd have to do it there regardless with rss, I've opted just to change the way the class for everything. The built output used to build the sstate output is now retained and installed rather than deleted. * The search and replace logic used in sstate objects also seemed weak/incorrect and didn't hold up against testing. This has been rewritten too. There are some assumptions made about paths, we save the 'proper' search and replace operations to fixmepath.cmd but then ignore this. What is here works but is a little hardcoded and an area for future improvement. * In order to work with eSDK we need a way to build something that looks like the old style sysroot. "bitbake build-sysroots" will construct such a sysroot based on everything in the components directory that matches the current MACHINE. It will allow transition of external tools and can built target or native variants or both. It also supports a clean task. I'd suggest not relying on this for anything other than transitional purposes though. To see XXX in that sysroot, you'd have to have built that in a previous bitbake invocation. * pseudo is run out of its components directory. This is fine as its statically linked. * The hacks for wayland to see allarch dependencies in the multilib case are no longer needed and can be dropped. * wic needed more extensive changes to work with rss and the fixes are in a separate commit series * Various oe-selftest tweaks were needed since tests did assume the location to binaries and the combined sysroot in several cases. * Most missing dependencies this work found have been sent out as separate patches as they were found but a few tweaks are still included here. * A late addition is that extend_recipe_sysroot became multilib aware and able to populate multilib sysroots. I had hoped not to have to add that complexity but the meta-environment recipe forced my hand. That implementation can probably be neater but this is on the list of things to cleanup later at this point. In summary, the impact people will likely see after this change: * Recipes may fail with missing dependencies, particularly native tools like gettext-native, glib-2.0-native and libxml2.0-native. Some hosts have these installed and will mask these errors * Any recipe/class using SSTATEPOSTINSTFUNCS will need that code rewriting into a postinst * There was a separate patch series dealing with roots postinst native dependency issues. Any postinst which expects native tools at rootfs time will need to mark that dependency with PACKAGE_WRITE_DEPS. There could well be other issues. This has been tested repeatedly against our autobuilders and oe-selftest and issues found have been fixed. We believe at least OE-Core is in good shape but that doesn't mean we've found all the issues. Also, the logging is a bit chatty at the moment. It does help if something goes wrong and goes to the task logfiles, not the console so I've intentionally left this like that for now. We can turn it down easily enough in due course. (From OE-Core rev: 809746f56df4b91af014bf6a3f28997d6698ac78) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Drop the depchain isPostDep() checksRichard Purdie2017-01-201-9/+7
| | | | | | | | | | | | | The dependencies of do_package_write_* tasks are either going to be packaging tools needed to build the packages, or, native tools needed at postinst time. Now we've formalised this dependency pattern, drop the hardcoded list and work based on the rule. The package creation tools are usually the same tools needed at rootfs/postinst time anyway so the difference is moot. (From OE-Core rev: 8082c6aabf838a2cc5253d2bb1bd8867f2e1ba6a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-16 12:28:59 +0000