summaryrefslogtreecommitdiffstats
path: root/meta/classes/sstate.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* sstate.bbclass: Fix for consistent whitespaceMartin Jansa2020-01-111-6/+6
| | | | | | | | | | (From OE-Core rev: a15c05fa17bcce345ce254f02c0d6dcd97f60cb8) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e874d067ace03774132ba3030a898678057c1304) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure mkdir happens before mktempRichard Purdie2020-01-111-3/+2
| | | | | | | | | | | | | This avoids a directory not present error. Fix a comment typo whilst here. (From OE-Core rev: c5f285f83ea1e322d5124847904adf488d3e790a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1360d8d7d99b70a80c8cdbc1fc6d9e6752483139) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure target sstate directory exists if unihash changesRichard Purdie2020-01-111-2/+2
| | | | | | | | | | | | The previous patches meant the mkdir might no longer match the final target directory. Fix this. (From OE-Core rev: 7ce42a30a7508e8fcb496ba05cf6967dc04a988e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0af4dae84099e8632a9ea6a4afdbea2f232bb170) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure SSTATE_PKG is reloaded when handling siginfoRichard Purdie2020-01-111-1/+2
| | | | | | | | | | | | STATE_PKG may have been changed by sstate_report_unihash so don't cache the variable's value. (From OE-Core rev: bfed8be91525478ecdf6b64e6308958b3271c8cc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit be29a25400c4ea285ab3f588c5831f00ba5d4f63) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Stop overwriting SSTATE_PKGRichard Purdie2020-01-111-4/+4
| | | | | | | | | | | | | | | Its rather antisocial to overwrite SSTATE_PKG with an expanded form for the variable and it stops the value of BB_UNIHASH being changed when the package is written out. Instead of expanding the variable, append to it instead to avoid this rather hard to figure out behaviour and allow the siggen code to behave as expected. (From OE-Core rev: fe454e9c7837a903ffcff4c6f8a25070a0f0af82) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 62eca02024b4c44d618ab9bcf87a3166c886dadb) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add ability to hide summary output for sstateRichard Purdie2020-01-111-11/+12
| | | | | | | | | | | | | Its confusing to keep seeing sstate summary messages when hash equivalency is active. This adds an option to control it. A default value is given which maintains compatibility with different bitbake versions. (From OE-Core rev: a67bd96de2b253c1d4c3bd82120f28dee06a4bf6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 038004866ff6650bcff7bb1bde36de6c0f451d29) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Remove SSTATE_HASHEQUIV_SERVERJoshua Watt2019-09-181-5/+0
| | | | | | | | | | Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This variable is redundant now that BB_HASHSERVE is present. (From OE-Core rev: 54b3adb6bc90a8e4b9e92952688772ee074d36e5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/sstatesig: Update to new form of BB_HASHCHECK_FUNCTIONRichard Purdie2019-08-061-42/+38
| | | | | | | | | Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION, adapt to this cleanup. This is an API breaking change. (From OE-Core rev: a4d413d8d809132b0e0a5dd673a36e2bd0e0be4c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Reduce race windowsRichard Purdie2019-08-031-1/+11
| | | | | | | | | | | | | | When we write to the sstate directory we try to do so atomically so consumers either see one version or another but never an imcomplete file. Unfortunately this is reliant on filesystem support and with some NFS configurations a replaced file would be lost from memory even if users held open descriptors. It makes sense to try and avoid replacing existing files where we can. (From OE-Core rev: 18cdc087fd5da30e2b31f3d4e81b153cd36ca844) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: regenerate sstate when signing enabledJoshua Lock via Openembedded-core2019-07-311-8/+12
| | | | | | | | | | | | | | | | This change ensures that the task signatures changes, and therefore sstate tasks are rerun, when signing is enabled. This has the positive outcome that if signing is enabled new signed shared state objects will be produced, rather than just signing shared state objects for tasks where no work has been performed yet. The downside of this change is that enabling/disabling sstate object signing alters the taskhash and results in rebuilding the world. (From OE-Core rev: ef60c65e3856682b90b17ae06a1b675756fd4496) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: don't use unsigned sstate when verification enabledJoshua Lock via Openembedded-core2019-07-271-1/+2
| | | | | | | | | | | | When signature verification of shared state objects is enabled (SSTATE_VERIFY_SIG) use of an unsigned object, even though it produces a warning, seems unexpected. Instead skip unsigned objects and force the non-accelerated task to be run. (From OE-Core rev: 571235978d98552e3734bf382454dd51272db782) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix log messageJoshua Lock via Openembedded-core2019-07-271-1/+1
| | | | | | | | | | | Referring to the sstate object as a staging package is an artefact of the code's origins. Switch to referring to an "Sstate package" in order to be more accurate and consistent with the rest of the file. (From OE-Core rev: 06559c1ed86dbce53505f9ed98111fe9d0b97ed7) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add tweak to avoid multiple sstate stats messagesRichard Purdie2019-07-151-0/+4
| | | | | | | | | | | | After the recent changes in bitbake to runqueue, we need to recheck sstate validity, particularly in multiconfig builds where tasks have the same checksum. Avoid printing summary messages in this case. Also avoid multiple events to toaster which may not be expecting that at later points in the code. (From OE-Core rev: 227125b96ad6fb0cf6e259e787d83415993db847) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Use bb.utils.to_boolean() for BB_NO_NETWORKRobert Yang2019-04-101-2/+4
| | | | | | | | | Make it consistent with bitbake (From OE-Core rev: 7fb540c3199bc2b82d60fff678b5e588ab4d1ad6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make sure changes to SSTATE_SCAN_FILES are not ignoredAlexander Kanavin2019-01-241-0/+1
| | | | | | | | | | | | | When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild, so if there's a sstate-cache available with "bad" sstate data in it that will still be used even though the recipe is updated to address this. [YOCTO #13144] (From OE-Core rev: ea3526961920a229e0bb5fb459952be89fce2255) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Update output hashJoshua Watt2019-01-221-62/+3
| | | | | | | | | | | | | | | | | Updates the output hash calculation for determining if tasks are equivalent. The new algorithm does the following based on feedback: 1) The output hash function was moved to the OE library. 2) All files are printed in a single line tabular format 3) Prints the file type and mode in a user-friendly ls-like format 4) Includes the file owner and group (by name, not ID). These are only included if the task is run under pseudo since that is the only time they can be consistently determined. 5) File size is included for regular files (From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: remove dependencies of SSTATE_EXTRAPATHWILDCARDRobert Yang2019-01-161-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | There might be do_cleansstate errors sometimes: ERROR: When reparsing /path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate, the basehash value changed from b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata is not deterministic and this needs to be fixed. The stable reproducer is: - Initial a fresh build, this is a must, otherwise we may can't reproduce it $ . oe-init-build-env build $ bitbake quilt-native -ccleansstate This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g., ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as SSTATE_EXTRAPATH did can fix the problem. (From OE-Core rev: 781117f9f02c0080dadc8797a8f8f9377a99b164) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar calls (again)André Draszik2019-01-141-2/+2
| | | | | | | | | | | | | | | | A couple have still been missed in the past despite multiple attempts at doing so (or simply have re-appeared?). Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Implement hash equivalence sstateJoshua Watt2019-01-081-8/+97
| | | | | | | | | | | | | | | | | | | | | Converts sstate so that it can use a hash equivalence server to determine if a task really needs to be rebuilt, or if it can be restored from a different (equivalent) sstate object. The unique hashes are cached persistently using persist_data. This has a number of advantages: 1) Unique hashes can be cached between invocations of bitbake to prevent needing to contact the server every time (which is slow) 2) The value of each tasks unique hash can easily be synchronized between different threads, which will be useful if bitbake is updated to do on the fly task re-hashing. [YOCTO #13030] (From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Handle unihash in hash checkJoshua Watt2019-01-081-1/+1
| | | | | | | | | | | | Handles the argument that passes task unique hash in the hash check function, as it is now required by bitbake [YOCTO #13030] (From OE-Core rev: 1c14b6969e58b51a325c1c1acf9c96e55675035c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add support for caching shared workdir tasksMichael Ho2019-01-031-0/+6
| | | | | | | | | | | | | | | | | The sstate bbclass uses workdir as a hardcoded string in path manipulations. This means that the sstate caching mechanism does not work for the work-shared directory which the kernel uses to share its build configuration and source files for out of tree kernel modules. This commit modifies the path manipulation mechanism to use the work-shared directory if detected in the paths when handling the sstate cache packages. (From OE-Core rev: 27642449f95e38598f9c83948ce109c5891e5877) Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: catch ValueError from incorrectly written sstate manifestsMartin Jansa2019-01-031-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * in some rare cases the file format might be broken, e.g. when you run out of disk space while writing to it (hint: make sure to use BB_DISKMON_DIRS on _all_ the builds, to make sure that they can stop gracefully before this happens). * will show error like this: ERROR: Invalid line '/OE/build/luneos-thud/webos-ports' in sstate manifest '/OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch' when the file isn't complete, like in my case: $ tail -n 2 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch /OE/build/luneos-thud/webos-ports/tmp-glibc/stamps/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/manifest-allarch-org.webosports.app.calculator /OE/build/luneos-thud/webos-ports/tmp-glibc/work/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports * instead of much longer exception which doesn't really show what's wrong to selectively fix that (other than removing while TMPDIR): ERROR: Command execution failed: Traceback (most recent call last): File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/command.py", line 113, in runAsyncCommand self.cooker.updateCache() File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/cooker.py", line 1562, in updateCache bb.event.fire(event, self.databuilder.mcdata[mc]) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 225, in fire fire_class_handlers(event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 134, in fire_class_handlers execute_handler(name, handler, event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 106, in execute_handler ret = handler(event) File "/OE/build/luneos-thud/webos-ports/openembedded-core/meta/classes/sstate.bbclass", line 1083, in sstate_eventhandler2 (stamp, manifest, workdir) = l.split() ValueError: not enough values to unpack (expected 3, got 1) (From OE-Core rev: 870ba5d80e76e0f989971532fc2adc0ebe811ab6) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Only remove sstate file when task is existedRobert Yang2018-11-231-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk> -ccleansstate" when there are a lot of sstate files. For example: * Before $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files: - Without disk caches # echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches (e.g., run it in the second time) $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308s So the 4 removing *package* commands cost more than 20s or 272s in theory. * After $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* We can see that it saved 20s or 272s in theory. (From OE-Core rev: bb2d6349ea87f090c58001f0d4348b24c2982cde) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitlyRobert Yang2018-11-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The glob.glob("/sstate/*/*/") is very time consuming, set SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time when there are many sstate files. For example, I have more than 600,000 sstate files: * Before - Without disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 0m4.111s user 0m2.348s sys 0m1.756s * After $ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' - Without disk caches: real 0m7.928s user 0m0.172s sys 0m0.124s - With disk caches: real 0m0.131s user 0m0.088s sys 0m0.044s We can see that it saves about 3.8s with disk caches, and saves about 264s without disk caches. (From OE-Core rev: 8b31c919814b8bdf25b3381053656523c001ae0d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: update SSTATE_DUPWHITELISTKai Kang2018-09-131-1/+3
| | | | | | | | | | | | Update SSTATE_DUPWHITELIST in sstate.bbclass. * remove ${DEPLOY_DIR_RPM}/noarch/ which is not overwritten any more * add directories for package target-sdk-provides-dummy (From OE-Core rev: 6d3ca476dbc2059f4b7fa3dfd73de6bbfed49198) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make SSTATE_PRUNE_OBSOLETEWORKDIR could be overwrittenKai Kang2018-08-281-1/+1
| | | | | | | | | | Define variable SSTATE_PRUNE_OBSOLETEWORKDIR with '?=' in sstate.bbclass, then it could be overwritten by user configuration. (From OE-Core rev: 22af59c9bfec31b31027ebd2a4da162f481aa6b5) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add intel-microcode to SSTATE_DUPWHITELISTYongxin Liu2018-08-231-1/+2
| | | | | | | | | | intel-microcode multilib recipes can generate identical overlapping files: microcode.cpio. (From OE-Core rev: 4b27da3334aff7f9f03ae934bbab7e7af07df3f6) Signed-off-by: Yongxin Liu <yongxin.liu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Optimise SSTATE_EXCLUDEDEPS_SYSROOT handlingRichard Purdie2018-08-161-3/+10
| | | | | | | | | Using re.compile() is around six times faster than recompiling the regexp each time so maintain a cache. (From OE-Core rev: 41eb382737706e245f2b7104e313c8dfaa370945) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping ↵André Draszik2018-08-161-3/+0
| | | | | | | | | | | | base-passwd|shadow-sysroot recipes Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the base-passwd|shadow-sysroot recipes to be excluded from a recipe sysroot. (From OE-Core rev: 68e502e9063a88532fe0154f152ba408f0091900) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping *-initial recipesAndré Draszik2018-08-161-3/+0
| | | | | | | | | | Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the *-initial recipes to be excluded from a recipe sysroot. (From OE-Core rev: 6706bad52f9311ea79c534ee90014c3216992999) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: allow specifying indirect dependencies to exclude from sysrootAndré Draszik2018-08-161-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, a dependency on any -native recipe will pull in all dependencies of that -native recipe in the recipe sysroot. This behaviour might not always be wanted, e.g. when that -native recipe depends on build-tools that are not relevant for the current recipe. This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable, which will be evaluated for such recursive dependencies to be excluded. The idea is similar to http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html except that the list is not hard-coded anymore. SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular expressions of recipe and dependency to ignore, e.g. in the above flex-native / bison-native use-case, one would specify SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native" in layer.conf. The existing special handling of "-initial" as well as "base-passwd" and "shadow-sysroot" could also be streamlined: SSTATE_EXCLUDEDEPS_SYSROOT += "\ .*->.*-initial.* \ .*(base-passwd|shadow-sysroot)->.* \ " Another anticipated user is meta-java, where certain newer JDKs can only be bootstrapped (built) using older JDKs, but it doesn't make much sense to copy all those older JDKs and their own build tools (ant, etc.) into the sysroot of recipes wanting to be built using the newer JDK (only), e.g.: SSTATE_EXCLUDEDEPS_SYSROOT += "\ openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \ " (From OE-Core rev: 92c5131a2feae2036c71a36c18bb9175bb2856dc) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/lib.oe.path: Ensure file sparseness is preservedRichard Purdie2018-08-151-2/+2
| | | | | | | | | Files when restored from sstate were missing their sparseness. Fix up various functions to preserve this and make things more deterministic. (From OE-Core rev: 055402e5504f041c346571e243c7cf0894955cad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure a given machine only removes things which it createdRichard Purdie2018-08-151-1/+16
| | | | | | | | | | | | | | | | | | | | | | Currently if you build qemux86 and then generic86, the latter will remove all of the former from deploy and workdir. This is because qemux86 is i586, genericx86 is i686 and the architctures are compatible therefore the sstate 'cleaup' code kicks in. There was a valid reason for this to ensure i586 packages didn't get into an i686 rootfs for example. With the rootfs creation being filtered now, this is no longer necessary. Instead, save out a list of stamps which a give machine has ever seen in a given build and only clean up these things if they're no longer "reachable". In particular this means the autobuilder should no longer spend a load of time deleting files when switching MACHINE, improving build times. (From OE-Core rev: 5634f2fb1740732056d2c1a22717184ef94405bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Remove DEPLOY_DIR_IMAGE from SSTATE_DUPWHITELISTRichard Purdie2018-08-141-1/+5
| | | | | | | | | | | | Replace the generic whitelist entry with entries for the three specific 'problem' cases in OE-Core. This means the general DEPLOY_DIR_IMAGE entry doesn't mask problems for others as was recently encoutered by users reported on irc. In the whitelisted cases they occur only in multilib builds and the files are identical. (From OE-Core rev: 05f6042a40bb772f7ce8d6819c5b2937d8c9808d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add sstate usage summary to the buildRichard Purdie2018-08-081-0/+12
| | | | | | | | | | | | | | | | | Currently the user has no indication of how much sstate was already present or that would be used by the build. This change adds some summary information so that the user can see how much reuse is occurring. To fully work it needs some extra information from a recent bitbake commit but this is optional. When combined with bitbake --dry-run this feature can be used to check if sstate would be reused in a build. [YOCTO #12749] (From OE-Core rev: 596f76029ccb6f87c3b049552bd08f5034c41d9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/bitbake.conf: Use pigz if availableRichard Purdie2018-07-241-2/+9
| | | | | | | | | | | | | | Currently the compression of sstate objects is single threaded. In the case of ltp, this takes around 33s. If we add pigz into the list of non-fatal HOSTTOOLS and then use if it available when building the sstate object, this time drops to around 6s. Since pigz is now widely available this is an optimisation we should utilise. (From OE-Core rev: 2de56aa0792ec93445130d801936a8ea643fad27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Remove unused argumentJoshua Watt2018-07-061-2/+2
| | | | | | | | | Removes an unused argument to pstaging_fetch() (From OE-Core rev: ab0eb2bf0db6e7da7c9b9bb4d46621dbf76dcc2a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Remove trailing whitespaceJoshua Watt2018-07-061-1/+1
| | | | | | | (From OE-Core rev: dc7c181f231db1290ea7d7d5e42476e70a488d73) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: drop obsolete codesChen Qi2018-05-041-5/+0
| | | | | | | | | | | | | | | The SSTATECLEANFUNCS mechanism was introduced to solve user/group deletion problem. After RSS mechanism was introduced, there's no need to do so. There was a patch to remove these obsolete codes for useradd.bbclass, but the codes in sstate.bbclass were not removed. So clean it up. (From OE-Core rev: 215b83ce892a7002ed0b1bd7b82a08e67ae15121) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Add ${PACKAGE_EXTRA_ARCHS} to SSTATE_ARCHSPeter Kjellerstedt2018-04-101-2/+3
| | | | | | | | | | | | | | | | | This makes sure files provided by packages that use any of the extra architectures defined using ${PACKAGE_EXTRA_ARCHS} are cleaned from tmp/sysroot-components when sstate_eventhandler2() executes. Without this, changing a package from using one of the extra architectures to another architecture would lead to files being leftover in tmp/sysroot-components, which could later be picked up when creating the RSS for another package rather than the files that belonged to the updated package. (From OE-Core rev: b959b3bd92aa2939d407800fc4167b0535fd4674) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: sstate_hardcode_path(): fix for multilibRobert Yang2018-01-111-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It only substituted staging_target for target recipe which didn't work for multilib, for example, postinst-useradd-lib32-polkit: * No multilib: PATH=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot-native/bin staging_target=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot The PATH would be substituted to: FIXMESTAGINGDIRTARGET-native/bin Not the funny "-native/bin", this works well. * When multilib: PATH=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/recipe-sysroot-native/bin staging_target=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/lib32-recipe-sysroot Now staging_target endswith "/lib32-recipe-sysroot", so it can't replace '/recipe-sysroot-native' in PATH , so PATH can't be fixed, and there would be build errors when building multilib + rm_work, for example: chown: invalid user: ‘polkitd:root’ Substitute staging_host for target recipe can fix the problem, now all of native, cross and target need substitute staging_host, so we can simply the code a little. (From OE-Core rev: 087510795331fa21ff52f103269087c06b1660fa) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* nativesdk/sdk: Update sdk dummy providersRichard Purdie2018-01-111-0/+3
| | | | | | | | | | | | | | | | | | When we migrated rpm v5 -> v4, we lost the ability to drop "per file" dependencies from the rpm backend for things like "/bin/bash" and "/usr/bin/env" which meant the sdks were becomming 'bloated'. This restores the functionality using a dummy package, similarly to the way the buildtools perl issue was addressed. It also removes the non-functional old code so as not to confuse people in future. I ran into this problem trying to filter dependencies to only rpms a build directly depends upon and it turns out we have some determinism issues in this area so this is something key to fix. (From OE-Core rev: 9d490dc01dcedb216129b22cbe17a6c99efc4f5c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Add progress bars when removing from the sysrootsPeter Kjellerstedt2017-12-101-14/+22
| | | | | | | | | | | If there are many recipes that should be cleaned out from the sysroots, it can actually take some time. This adds a progress bar to give a visual clue that the clean up is actually progressing. (From OE-Core rev: d9b59df1230a20c7a5c9f4fb0325bb9216025a16) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: provide an exception for useradd scenarioMaxin B. John2017-10-161-1/+2
| | | | | | | | | | | | | | | Packages, which depend on users/groups created from other packages, needs "shadow-native" as a build time dependency. So, add an exception to the "shadow-native" from otherwise discarded native/cross tools dependency. Fixes [YOCTO #11960] (From OE-Core rev: 979699b55214933e0f11727a2fb9bfda8a3a3870) Signed-off-by: Maxin B. John <maxin.john@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: rework source extraction so that dependencies are handledPaul Eggleton2017-09-181-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since it was first implemented, devtool's source extraction (as used by the devtool modify, extract and upgrade subcommands) ignored other recipe dependencies - so for example if you ran devtool modify on a recipe that fetches from svn or is compressed using xz then it would fail if those dependencies hadn't been built first. Now that we can execute tasks in the normal way (i.e. tinfoil.build_targets()) then we can rework it to use that. This is slightly tricky in that the source extraction needs to insert some logic in between tasks; luckily we can use a helper class that conditionally adds prefuncs to make that possible. Some side-effects / aspects of this change worth noting: * Operations are a little slower because we have to go through the task dependency graph generation and other startup processing. There's not really any way to avoid this though. * devtool extract didn't used to require a workspace, now it does because it needs to create a temporary bbappend for the recipe. (As with other commands the workspace be created on the fly if it doesn't already exist.) * I want any existing sysroot files and stamps to be left alone during extraction since we are running the tasks off to the side, and especially devtool extract should be able to be used without touching these. However, this was hampered by the automatic removal process in sstate.bbclass triggered by bb.event.ReachableStamps when the task signatures change, thus I had to introduce a way to disable this removal on a per-recipe basis (we still want it to function for any dependencies that we aren't working on). To implement this I elected to use a file written to tmp/sstate-control which gets deleted automatically after reading so that there's less chance of stale files affecting future sessions. I could have used a variable but this would have needed to be whitelisted and I'd have to have poked its value in using the setVariable command. Fixes [YOCTO #11198]. (From OE-Core rev: 830dbd66992cbb9e731b48d56fddf8f220349666) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve SSTATE_SKIP_CREATIONRichard Purdie2017-07-311-6/+4
| | | | | | | | | | | | | | | | Currently, dummy archives are created for skipped sstate tasks. We relied on these never being installed (the setscene task is missing) however this leads to odd behaviour as for example the setscene stamps are never looked at. Rather than trying to continue with the two separate behaviours, really skip package creation. We do need the file manipulation steps to install files under sstate control from a manifest perspective so we only skip at the final creation step. (From OE-Core rev: 1e95fc5419a131a5e4091b8ff01639de99284399) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: Fix return value checks from subprocess.call()'sMikko Rapeli2017-06-281-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Python function subprocess.call() returns the return value of the executed process. If return values are not checked, errors may go unnoticed and bad things can happen. Change all callers of subprocess.call() which do not check for the return value to use subprocess.check_call() which raises CalledProcessError if the subprocess returns with non-zero value. https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module All users of the function were found with: $ git grep "subprocess\.call" | \ egrep -v 'if.*subprocess\.call|=\ +subprocess\.call|return.*subprocess\.call' Tested similar patch on top of yocto jethro. Only compile tested core-image-minimal on poky master branch. (From OE-Core rev: 578c8205fd14c48c6d30ef2889d86f1b4aee060a) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure native/cross recipes have relocation of HOSTTOOLS_DIRRichard Purdie2017-05-121-1/+1
| | | | | | | | | | The previous change to relocate HOSTTOOLS wasn't complete as some files, particularly in gcc stashed build directories were not being correctly relocated. This patch addresses the issue. (From OE-Core rev: 21dd36cc12a033b012544c5d15a6f8afd84dabc9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass, staging.bbclass: Handle HOSTTOOLS_DIR when restoring statePeter Kjellerstedt2017-05-011-1/+1
| | | | | | | | | | | Paths to host tools that have been copied to ${HOSTTOOLS_DIR} may end up in the sstate cache. They thus need to be corrected when restoring from the sstate cache. (From OE-Core rev: f8671aecf05a286dd2b34b07bb5fbbe0c31e26d0) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base-passwd/useradd: Various improvements to useradd with RSSRichard Purdie2017-04-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently there are multiple issues with useradd: * If base-passwd rebuilds, it wipes out recipe specific user/group additions to sysroots and causes errors * If recipe A adds a user and recipe B depends on A, it can't see any of the users/groups A adds. This patch changes base-passwd so it always works as a postinst script within the sysroot and copies in the master files, then runs any postinst-useradd-* scripts afterwards to add additional user/groups. The postinst-useradd-* scripts are tweaked so that if /etc/passwd doesn't exist they just exit, knowning they'll be executed later. We also add a dummy entry to the dummy passwd file from pseudo so we can avoid this too. There is a problem where if recipe A adds a user and recipe B depends on A but doesn't care about users, it may not have a dependency on the useradd/groupadd tools which would therefore not be available in B's sysroot. We therefore also tweak postinst-useradd-* scripts so that if the tools aren't present we simply don't add users. If you need the users, you add a dependency on the tools in the recipe and they'll be added. We add postinst-* to SSTATE_SCAN_FILES since almost any postinst script of this kind is going to need relocation help. We also ensure that the postinst-useradd script is written into the sstate object as the current script was only being added in a recipe local way. Thanks to Peter Kjellerstedt <pkj@axis.com> and Patrick Ohly for some pieces of this patch. [Yocto #11124] (From OE-Core rev: 1b5afaf437f7a1107d4edca8eeb668b9618a5488) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-24 19:56:10 +0000