linux/poky.git/meta/classes/sstate.bbclass, branch yocto-2.7

sstate.bbclass: Use bb.utils.to_boolean() for BB_NO_NETWORK

2019-04-10T12:46:17+00:00

Make it consistent with bitbake (From OE-Core rev: 7fb540c3199bc2b82d60fff678b5e588ab4d1ad6) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

sstate.bbclass: make sure changes to SSTATE_SCAN_FILES are not ignored

2019-01-24T17:45:25+00:00

When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild, so if there's a sstate-cache available with "bad" sstate data in it that will still be used even though the recipe is updated to address this. [YOCTO #13144] (From OE-Core rev: ea3526961920a229e0bb5fb459952be89fce2255) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

classes/sstate: Update output hash

2019-01-22T14:35:58+00:00

Updates the output hash calculation for determining if tasks are equivalent. The new algorithm does the following based on feedback: 1) The output hash function was moved to the OE library. 2) All files are printed in a single line tabular format 3) Prints the file type and mode in a user-friendly ls-like format 4) Includes the file owner and group (by name, not ID). These are only included if the task is run under pseudo since that is the only time they can be consistently determined. 5) File size is included for regular files (From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

sstate.bbclass: remove dependencies of SSTATE_EXTRAPATHWILDCARD

2019-01-16T15:35:07+00:00

There might be do_cleansstate errors sometimes: ERROR: When reparsing /path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate, the basehash value changed from b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata is not deterministic and this needs to be fixed. The stable reproducer is: - Initial a fresh build, this is a must, otherwise we may can't reproduce it $ . oe-init-build-env build $ bitbake quilt-native -ccleansstate This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g., ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as SSTATE_EXTRAPATH did can fix the problem. (From OE-Core rev: 781117f9f02c0080dadc8797a8f8f9377a99b164) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

meta: remove True option to getVar calls (again)

2019-01-14T11:35:56+00:00

A couple have still been missed in the past despite multiple attempts at doing so (or simply have re-appeared?). Search & replace made using the following command: sed -e 's|$d\.getVar \?$( \?$[^,()]*$, \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?$ ?([^,()]*), ?True$' \ | cut -d':' -f1 \ | sort -u) (From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba) Signed-off-by: André Draszik Signed-off-by: Richard Purdie

sstate: Implement hash equivalence sstate

2019-01-08T11:16:44+00:00

Converts sstate so that it can use a hash equivalence server to determine if a task really needs to be rebuilt, or if it can be restored from a different (equivalent) sstate object. The unique hashes are cached persistently using persist_data. This has a number of advantages: 1) Unique hashes can be cached between invocations of bitbake to prevent needing to contact the server every time (which is slow) 2) The value of each tasks unique hash can easily be synchronized between different threads, which will be useful if bitbake is updated to do on the fly task re-hashing. [YOCTO #13030] (From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

classes/sstate: Handle unihash in hash check

2019-01-08T11:16:44+00:00

Handles the argument that passes task unique hash in the hash check function, as it is now required by bitbake [YOCTO #13030] (From OE-Core rev: 1c14b6969e58b51a325c1c1acf9c96e55675035c) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

sstate: add support for caching shared workdir tasks

2019-01-03T21:16:00+00:00

The sstate bbclass uses workdir as a hardcoded string in path manipulations. This means that the sstate caching mechanism does not work for the work-shared directory which the kernel uses to share its build configuration and source files for out of tree kernel modules. This commit modifies the path manipulation mechanism to use the work-shared directory if detected in the paths when handling the sstate cache packages. (From OE-Core rev: 27642449f95e38598f9c83948ce109c5891e5877) Signed-off-by: Michael Ho Signed-off-by: Richard Purdie

sstate.bbclass: catch ValueError from incorrectly written sstate manifests

2019-01-03T12:35:53+00:00

* in some rare cases the file format might be broken, e.g. when you run out of disk space while writing to it (hint: make sure to use BB_DISKMON_DIRS on _all_ the builds, to make sure that they can stop gracefully before this happens). * will show error like this: ERROR: Invalid line '/OE/build/luneos-thud/webos-ports' in sstate manifest '/OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch' when the file isn't complete, like in my case: $ tail -n 2 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch /OE/build/luneos-thud/webos-ports/tmp-glibc/stamps/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/manifest-allarch-org.webosports.app.calculator /OE/build/luneos-thud/webos-ports/tmp-glibc/work/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports * instead of much longer exception which doesn't really show what's wrong to selectively fix that (other than removing while TMPDIR): ERROR: Command execution failed: Traceback (most recent call last): File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/command.py", line 113, in runAsyncCommand self.cooker.updateCache() File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/cooker.py", line 1562, in updateCache bb.event.fire(event, self.databuilder.mcdata[mc]) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 225, in fire fire_class_handlers(event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 134, in fire_class_handlers execute_handler(name, handler, event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 106, in execute_handler ret = handler(event) File "/OE/build/luneos-thud/webos-ports/openembedded-core/meta/classes/sstate.bbclass", line 1083, in sstate_eventhandler2 (stamp, manifest, workdir) = l.split() ValueError: not enough values to unpack (expected 3, got 1) (From OE-Core rev: 870ba5d80e76e0f989971532fc2adc0ebe811ab6) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

sstate.bbclass: Only remove sstate file when task is existed

2018-11-23T23:35:19+00:00

This can improve the performance a lot for "bitbake -ccleansstate" when there are a lot of sstate files. For example: * Before $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files: - Without disk caches # echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches (e.g., run it in the second time) $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308s So the 4 removing *package* commands cost more than 20s or 272s in theory. * After $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* We can see that it saved 20s or 272s in theory. (From OE-Core rev: bb2d6349ea87f090c58001f0d4348b24c2982cde) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie