summaryrefslogtreecommitdiffstats
path: root/meta/classes/sstate.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* sstate.bbclass: When siginfo or sig files are missing, stop fetcher errorsMark Hatle2020-06-051-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prior to fetching, the system checks if the sstate file is present either locally or on the mirror. If it is, then it goes to the fetch stage. Up to three files can be fetched, sstate, sstate.siginfo and sstate.sig (if signature validation is enabled). The previous pstaging_fetch function would iterate over these, and if a download error occurred would spew forth a great amount of fetcher failure messages as well as stop fetching the next item in the set. This was resolved by adding a fetcher.checkstatus() call prior to the download. If the file isn't present, then the exception will be triggered, and no fetcher failure messages will reach the user. The exception handler is then modified to be a pass so that it will loop and pull the rest of the files that that are requested. Additionally, a check for the existance of the .sig file was added to the sstate_installpkg to avoid an error trying to load the .sig if it wasn't downloaded. (From OE-Core rev: ec58532ab6fc6343144da67789c928c751d36c06) Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a9085140434e2d26c0bb75bb53fcb7f7c19ef86d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Do not fail if files cannot be touchedPeter Kjellerstedt2020-05-221-4/+11
| | | | | | | | | | | | It may be that a file is not allowed to be touched, e.g., if it is a symbolic link into a global sstate cache served over NFS. (From OE-Core rev: f528d6ffc9649536d21d625f65b415bfec7db258) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Drop obsolete check in hash validationRichard Purdie2020-03-061-4/+0
| | | | | | | | | | | | | | Now this function has a summary parameter we can drop this check. It could well be why the mysterious "locked sigs" selftest fails intermittently if this function were called with a single hash to check. [YOCTO #13605] (with luck) (From OE-Core rev: 02aabe0e59f73bf206d9bada1e7089832ceed254) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix issue while handling long sstate filenamesJaewon Lee2020-02-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | When moving to python3, divison using '/' now returns float instead of an integer. In upstream commit b8025e972081b70960ffcbcbe43a7118041556a1 sstate filenames longer than the limit are changed to just include necessary info + 3 fields just for information. The space left over after the necessary info is divided into 3 for each of the fields. Using '//' instead to do the division to solve the following error message: avail = (254 - len(hash + "_" + taskname + extension) - len(components[0]) - len(components[1]) - len(components[5]) - len(components[6]) - 7) / 3 > components[2] = components[2][:avail] components[3] = components[3][:avail] TypeError: slice indices must be integers or None or have an __index__ method (From OE-Core rev: 2acfee61a062c6520a413b2a797544d968bb0c76) Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Only show sstate mirror progress bar for >= 100 objectsPeter Kjellerstedt2020-01-271-4/+8
| | | | | | | | | | | | | | | With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly. Most of the times the number of objects scanned is very low and the progress bar completes almost immediately. To avoid all these unnecessary progress bars, set the minimum number of objects to 100 before the progress bar is shown. (From OE-Core rev: 114340e8d393cf60d0a596cb0800cf1b7fd198df) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve move into place op and touch fileRichard Purdie2020-01-131-1/+5
| | | | | | | | | | | | | | | | | Depending on the kernel and coreutils version, mv operations test for existence of files and can potentially race. It also leads to the file always changing which leads to potential problems if using and NFS share and there are other readers. Using ln instead means we don't overwrite the file if it already exists meaning other readers aren't disrupted and should work more reliably on NFS which is used for sstate on the autobuilder. Since we're not overwriting files, touch the file to show activity as would have been done it it were reused from sstate. (From OE-Core rev: d8e9a22a4e23616ad01627c1e472296b1e26f13c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Avoid races over rewritten siginfo filesRichard Purdie2020-01-061-3/+10
| | | | | | | | | | | | | | Try and avoid errors like: Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/x86_64-linux/libxslt-native/1.1.34-r0/temp/run.sstate_unpack_package.12343' failed with exit code 1: touch: setting times of 'build/sstate_devtool/80/4d/sstate:libxslt-native::1.1.34:r0::3:804d33f3f8d032b01ae92207669dd0a8f95a84917d563d122a77df19e786d73c_populate_lic.tgz.siginfo': Stale file handle WARNING: exit code 1 from a shell command. by not overwriting existing files. (From OE-Core rev: b2a389ad5111d587db3f95c6ce13fc2eafc22f27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Handle sstate filenames longer than 255 charactersRichard Purdie2020-01-061-1/+18
| | | | | | | | | | | | Many filesystems can't cope with filenames longer that 255 characters. Add code to detect this and truncate non-essential elements of the filename to stay within the limit. [YOCTO #13268] (From OE-Core rev: 90cc3d1ed1a12294a2d3ac97c1ba528ab315605d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Merge file name generation into single functionRichard Purdie2020-01-061-10/+12
| | | | | | | | | Move the task handling and extension handling into one common filename construction function. (From OE-Core rev: c0c158d38583648a801e959d91371f7b43a98da5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve SSTATE_PKG handlingRichard Purdie2020-01-061-10/+11
| | | | | | | | | | | Move the task handling code into the SSTATE_PKGNAME variable using a temporary variable. This makes the code more understandable as as well as allowing the length of the final sstate filename to be more easily accesses for following patches. (From OE-Core rev: be603dad6817948cfa09621d2311046631ec4ab2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add extra directory levelRichard Purdie2020-01-061-2/+2
| | | | | | | | | | | | | | | | | | We're having speed issues on the autobuilder due to the numbers of files in sstate directories. We previously split these by the first two characters of the hash. This change extends this to split by the next two characters as well, creating more layers of directories. This should signifiantly speed up eSDK builds on the autobuilder as the current sstate layout simply isn't scaling there but addresses a general complaint. gen-lockedsig-cache needed to be updated for the new split level sstate. Also update tests for new layout. (From OE-Core rev: d05bde16bdad761ed8f4c0a48de60c649aa33e85) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Fix for consistent whitespaceMartin Jansa2019-12-301-6/+6
| | | | | | | (From OE-Core rev: e874d067ace03774132ba3030a898678057c1304) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure mkdir happens before mktempRichard Purdie2019-12-111-3/+2
| | | | | | | | | | This avoids a directory not present error. Fix a comment typo whilst here. (From OE-Core rev: 1360d8d7d99b70a80c8cdbc1fc6d9e6752483139) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure target sstate directory exists if unihash changesRichard Purdie2019-12-111-2/+2
| | | | | | | | | The previous patches meant the mkdir might no longer match the final target directory. Fix this. (From OE-Core rev: 0af4dae84099e8632a9ea6a4afdbea2f232bb170) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure SSTATE_PKG is reloaded when handling siginfoRichard Purdie2019-12-101-1/+2
| | | | | | | | | STATE_PKG may have been changed by sstate_report_unihash so don't cache the variable's value. (From OE-Core rev: be29a25400c4ea285ab3f588c5831f00ba5d4f63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Stop overwriting SSTATE_PKGRichard Purdie2019-12-101-4/+4
| | | | | | | | | | | | Its rather antisocial to overwrite SSTATE_PKG with an expanded form for the variable and it stops the value of BB_UNIHASH being changed when the package is written out. Instead of expanding the variable, append to it instead to avoid this rather hard to figure out behaviour and allow the siggen code to behave as expected. (From OE-Core rev: 62eca02024b4c44d618ab9bcf87a3166c886dadb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add ability to hide summary output for sstateRichard Purdie2019-11-071-11/+12
| | | | | | | | | | Its confusing to keep seeing sstate summary messages when hash equivalency is active. This adds an option to control it. A default value is given which maintains compatibility with different bitbake versions. (From OE-Core rev: 038004866ff6650bcff7bb1bde36de6c0f451d29) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Remove SSTATE_HASHEQUIV_SERVERJoshua Watt2019-09-181-5/+0
| | | | | | | | | | Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This variable is redundant now that BB_HASHSERVE is present. (From OE-Core rev: 54b3adb6bc90a8e4b9e92952688772ee074d36e5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/sstatesig: Update to new form of BB_HASHCHECK_FUNCTIONRichard Purdie2019-08-061-42/+38
| | | | | | | | | Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION, adapt to this cleanup. This is an API breaking change. (From OE-Core rev: a4d413d8d809132b0e0a5dd673a36e2bd0e0be4c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Reduce race windowsRichard Purdie2019-08-031-1/+11
| | | | | | | | | | | | | | When we write to the sstate directory we try to do so atomically so consumers either see one version or another but never an imcomplete file. Unfortunately this is reliant on filesystem support and with some NFS configurations a replaced file would be lost from memory even if users held open descriptors. It makes sense to try and avoid replacing existing files where we can. (From OE-Core rev: 18cdc087fd5da30e2b31f3d4e81b153cd36ca844) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: regenerate sstate when signing enabledJoshua Lock via Openembedded-core2019-07-311-8/+12
| | | | | | | | | | | | | | | | This change ensures that the task signatures changes, and therefore sstate tasks are rerun, when signing is enabled. This has the positive outcome that if signing is enabled new signed shared state objects will be produced, rather than just signing shared state objects for tasks where no work has been performed yet. The downside of this change is that enabling/disabling sstate object signing alters the taskhash and results in rebuilding the world. (From OE-Core rev: ef60c65e3856682b90b17ae06a1b675756fd4496) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: don't use unsigned sstate when verification enabledJoshua Lock via Openembedded-core2019-07-271-1/+2
| | | | | | | | | | | | When signature verification of shared state objects is enabled (SSTATE_VERIFY_SIG) use of an unsigned object, even though it produces a warning, seems unexpected. Instead skip unsigned objects and force the non-accelerated task to be run. (From OE-Core rev: 571235978d98552e3734bf382454dd51272db782) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix log messageJoshua Lock via Openembedded-core2019-07-271-1/+1
| | | | | | | | | | | Referring to the sstate object as a staging package is an artefact of the code's origins. Switch to referring to an "Sstate package" in order to be more accurate and consistent with the rest of the file. (From OE-Core rev: 06559c1ed86dbce53505f9ed98111fe9d0b97ed7) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add tweak to avoid multiple sstate stats messagesRichard Purdie2019-07-151-0/+4
| | | | | | | | | | | | After the recent changes in bitbake to runqueue, we need to recheck sstate validity, particularly in multiconfig builds where tasks have the same checksum. Avoid printing summary messages in this case. Also avoid multiple events to toaster which may not be expecting that at later points in the code. (From OE-Core rev: 227125b96ad6fb0cf6e259e787d83415993db847) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Use bb.utils.to_boolean() for BB_NO_NETWORKRobert Yang2019-04-101-2/+4
| | | | | | | | | Make it consistent with bitbake (From OE-Core rev: 7fb540c3199bc2b82d60fff678b5e588ab4d1ad6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make sure changes to SSTATE_SCAN_FILES are not ignoredAlexander Kanavin2019-01-241-0/+1
| | | | | | | | | | | | | When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild, so if there's a sstate-cache available with "bad" sstate data in it that will still be used even though the recipe is updated to address this. [YOCTO #13144] (From OE-Core rev: ea3526961920a229e0bb5fb459952be89fce2255) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Update output hashJoshua Watt2019-01-221-62/+3
| | | | | | | | | | | | | | | | | Updates the output hash calculation for determining if tasks are equivalent. The new algorithm does the following based on feedback: 1) The output hash function was moved to the OE library. 2) All files are printed in a single line tabular format 3) Prints the file type and mode in a user-friendly ls-like format 4) Includes the file owner and group (by name, not ID). These are only included if the task is run under pseudo since that is the only time they can be consistently determined. 5) File size is included for regular files (From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: remove dependencies of SSTATE_EXTRAPATHWILDCARDRobert Yang2019-01-161-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | There might be do_cleansstate errors sometimes: ERROR: When reparsing /path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate, the basehash value changed from b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata is not deterministic and this needs to be fixed. The stable reproducer is: - Initial a fresh build, this is a must, otherwise we may can't reproduce it $ . oe-init-build-env build $ bitbake quilt-native -ccleansstate This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g., ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as SSTATE_EXTRAPATH did can fix the problem. (From OE-Core rev: 781117f9f02c0080dadc8797a8f8f9377a99b164) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar calls (again)André Draszik2019-01-141-2/+2
| | | | | | | | | | | | | | | | A couple have still been missed in the past despite multiple attempts at doing so (or simply have re-appeared?). Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Implement hash equivalence sstateJoshua Watt2019-01-081-8/+97
| | | | | | | | | | | | | | | | | | | | | Converts sstate so that it can use a hash equivalence server to determine if a task really needs to be rebuilt, or if it can be restored from a different (equivalent) sstate object. The unique hashes are cached persistently using persist_data. This has a number of advantages: 1) Unique hashes can be cached between invocations of bitbake to prevent needing to contact the server every time (which is slow) 2) The value of each tasks unique hash can easily be synchronized between different threads, which will be useful if bitbake is updated to do on the fly task re-hashing. [YOCTO #13030] (From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Handle unihash in hash checkJoshua Watt2019-01-081-1/+1
| | | | | | | | | | | | Handles the argument that passes task unique hash in the hash check function, as it is now required by bitbake [YOCTO #13030] (From OE-Core rev: 1c14b6969e58b51a325c1c1acf9c96e55675035c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add support for caching shared workdir tasksMichael Ho2019-01-031-0/+6
| | | | | | | | | | | | | | | | | The sstate bbclass uses workdir as a hardcoded string in path manipulations. This means that the sstate caching mechanism does not work for the work-shared directory which the kernel uses to share its build configuration and source files for out of tree kernel modules. This commit modifies the path manipulation mechanism to use the work-shared directory if detected in the paths when handling the sstate cache packages. (From OE-Core rev: 27642449f95e38598f9c83948ce109c5891e5877) Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: catch ValueError from incorrectly written sstate manifestsMartin Jansa2019-01-031-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * in some rare cases the file format might be broken, e.g. when you run out of disk space while writing to it (hint: make sure to use BB_DISKMON_DIRS on _all_ the builds, to make sure that they can stop gracefully before this happens). * will show error like this: ERROR: Invalid line '/OE/build/luneos-thud/webos-ports' in sstate manifest '/OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch' when the file isn't complete, like in my case: $ tail -n 2 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch /OE/build/luneos-thud/webos-ports/tmp-glibc/stamps/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/manifest-allarch-org.webosports.app.calculator /OE/build/luneos-thud/webos-ports/tmp-glibc/work/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports * instead of much longer exception which doesn't really show what's wrong to selectively fix that (other than removing while TMPDIR): ERROR: Command execution failed: Traceback (most recent call last): File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/command.py", line 113, in runAsyncCommand self.cooker.updateCache() File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/cooker.py", line 1562, in updateCache bb.event.fire(event, self.databuilder.mcdata[mc]) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 225, in fire fire_class_handlers(event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 134, in fire_class_handlers execute_handler(name, handler, event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 106, in execute_handler ret = handler(event) File "/OE/build/luneos-thud/webos-ports/openembedded-core/meta/classes/sstate.bbclass", line 1083, in sstate_eventhandler2 (stamp, manifest, workdir) = l.split() ValueError: not enough values to unpack (expected 3, got 1) (From OE-Core rev: 870ba5d80e76e0f989971532fc2adc0ebe811ab6) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Only remove sstate file when task is existedRobert Yang2018-11-231-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk> -ccleansstate" when there are a lot of sstate files. For example: * Before $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files: - Without disk caches # echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches (e.g., run it in the second time) $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308s So the 4 removing *package* commands cost more than 20s or 272s in theory. * After $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* We can see that it saved 20s or 272s in theory. (From OE-Core rev: bb2d6349ea87f090c58001f0d4348b24c2982cde) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitlyRobert Yang2018-11-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The glob.glob("/sstate/*/*/") is very time consuming, set SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time when there are many sstate files. For example, I have more than 600,000 sstate files: * Before - Without disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 0m4.111s user 0m2.348s sys 0m1.756s * After $ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' - Without disk caches: real 0m7.928s user 0m0.172s sys 0m0.124s - With disk caches: real 0m0.131s user 0m0.088s sys 0m0.044s We can see that it saves about 3.8s with disk caches, and saves about 264s without disk caches. (From OE-Core rev: 8b31c919814b8bdf25b3381053656523c001ae0d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: update SSTATE_DUPWHITELISTKai Kang2018-09-131-1/+3
| | | | | | | | | | | | Update SSTATE_DUPWHITELIST in sstate.bbclass. * remove ${DEPLOY_DIR_RPM}/noarch/ which is not overwritten any more * add directories for package target-sdk-provides-dummy (From OE-Core rev: 6d3ca476dbc2059f4b7fa3dfd73de6bbfed49198) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make SSTATE_PRUNE_OBSOLETEWORKDIR could be overwrittenKai Kang2018-08-281-1/+1
| | | | | | | | | | Define variable SSTATE_PRUNE_OBSOLETEWORKDIR with '?=' in sstate.bbclass, then it could be overwritten by user configuration. (From OE-Core rev: 22af59c9bfec31b31027ebd2a4da162f481aa6b5) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add intel-microcode to SSTATE_DUPWHITELISTYongxin Liu2018-08-231-1/+2
| | | | | | | | | | intel-microcode multilib recipes can generate identical overlapping files: microcode.cpio. (From OE-Core rev: 4b27da3334aff7f9f03ae934bbab7e7af07df3f6) Signed-off-by: Yongxin Liu <yongxin.liu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Optimise SSTATE_EXCLUDEDEPS_SYSROOT handlingRichard Purdie2018-08-161-3/+10
| | | | | | | | | Using re.compile() is around six times faster than recompiling the regexp each time so maintain a cache. (From OE-Core rev: 41eb382737706e245f2b7104e313c8dfaa370945) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping ↵André Draszik2018-08-161-3/+0
| | | | | | | | | | | | base-passwd|shadow-sysroot recipes Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the base-passwd|shadow-sysroot recipes to be excluded from a recipe sysroot. (From OE-Core rev: 68e502e9063a88532fe0154f152ba408f0091900) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping *-initial recipesAndré Draszik2018-08-161-3/+0
| | | | | | | | | | Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying the *-initial recipes to be excluded from a recipe sysroot. (From OE-Core rev: 6706bad52f9311ea79c534ee90014c3216992999) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: allow specifying indirect dependencies to exclude from sysrootAndré Draszik2018-08-161-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, a dependency on any -native recipe will pull in all dependencies of that -native recipe in the recipe sysroot. This behaviour might not always be wanted, e.g. when that -native recipe depends on build-tools that are not relevant for the current recipe. This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable, which will be evaluated for such recursive dependencies to be excluded. The idea is similar to http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html except that the list is not hard-coded anymore. SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular expressions of recipe and dependency to ignore, e.g. in the above flex-native / bison-native use-case, one would specify SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native" in layer.conf. The existing special handling of "-initial" as well as "base-passwd" and "shadow-sysroot" could also be streamlined: SSTATE_EXCLUDEDEPS_SYSROOT += "\ .*->.*-initial.* \ .*(base-passwd|shadow-sysroot)->.* \ " Another anticipated user is meta-java, where certain newer JDKs can only be bootstrapped (built) using older JDKs, but it doesn't make much sense to copy all those older JDKs and their own build tools (ant, etc.) into the sysroot of recipes wanting to be built using the newer JDK (only), e.g.: SSTATE_EXCLUDEDEPS_SYSROOT += "\ openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \ " (From OE-Core rev: 92c5131a2feae2036c71a36c18bb9175bb2856dc) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/lib.oe.path: Ensure file sparseness is preservedRichard Purdie2018-08-151-2/+2
| | | | | | | | | Files when restored from sstate were missing their sparseness. Fix up various functions to preserve this and make things more deterministic. (From OE-Core rev: 055402e5504f041c346571e243c7cf0894955cad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure a given machine only removes things which it createdRichard Purdie2018-08-151-1/+16
| | | | | | | | | | | | | | | | | | | | | | Currently if you build qemux86 and then generic86, the latter will remove all of the former from deploy and workdir. This is because qemux86 is i586, genericx86 is i686 and the architctures are compatible therefore the sstate 'cleaup' code kicks in. There was a valid reason for this to ensure i586 packages didn't get into an i686 rootfs for example. With the rootfs creation being filtered now, this is no longer necessary. Instead, save out a list of stamps which a give machine has ever seen in a given build and only clean up these things if they're no longer "reachable". In particular this means the autobuilder should no longer spend a load of time deleting files when switching MACHINE, improving build times. (From OE-Core rev: 5634f2fb1740732056d2c1a22717184ef94405bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Remove DEPLOY_DIR_IMAGE from SSTATE_DUPWHITELISTRichard Purdie2018-08-141-1/+5
| | | | | | | | | | | | Replace the generic whitelist entry with entries for the three specific 'problem' cases in OE-Core. This means the general DEPLOY_DIR_IMAGE entry doesn't mask problems for others as was recently encoutered by users reported on irc. In the whitelisted cases they occur only in multilib builds and the files are identical. (From OE-Core rev: 05f6042a40bb772f7ce8d6819c5b2937d8c9808d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add sstate usage summary to the buildRichard Purdie2018-08-081-0/+12
| | | | | | | | | | | | | | | | | Currently the user has no indication of how much sstate was already present or that would be used by the build. This change adds some summary information so that the user can see how much reuse is occurring. To fully work it needs some extra information from a recent bitbake commit but this is optional. When combined with bitbake --dry-run this feature can be used to check if sstate would be reused in a build. [YOCTO #12749] (From OE-Core rev: 596f76029ccb6f87c3b049552bd08f5034c41d9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/bitbake.conf: Use pigz if availableRichard Purdie2018-07-241-2/+9
| | | | | | | | | | | | | | Currently the compression of sstate objects is single threaded. In the case of ltp, this takes around 33s. If we add pigz into the list of non-fatal HOSTTOOLS and then use if it available when building the sstate object, this time drops to around 6s. Since pigz is now widely available this is an optimisation we should utilise. (From OE-Core rev: 2de56aa0792ec93445130d801936a8ea643fad27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Remove unused argumentJoshua Watt2018-07-061-2/+2
| | | | | | | | | Removes an unused argument to pstaging_fetch() (From OE-Core rev: ab0eb2bf0db6e7da7c9b9bb4d46621dbf76dcc2a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Remove trailing whitespaceJoshua Watt2018-07-061-1/+1
| | | | | | | (From OE-Core rev: dc7c181f231db1290ea7d7d5e42476e70a488d73) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: drop obsolete codesChen Qi2018-05-041-5/+0
| | | | | | | | | | | | | | | The SSTATECLEANFUNCS mechanism was introduced to solve user/group deletion problem. After RSS mechanism was introduced, there's no need to do so. There was a patch to remove these obsolete codes for useradd.bbclass, but the codes in sstate.bbclass were not removed. So clean it up. (From OE-Core rev: 215b83ce892a7002ed0b1bd7b82a08e67ae15121) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-24 18:00:21 +0000