linux/poky.git/meta/classes, branch uninative-2.2

license_image: Fix race

2018-07-25T15:48:27+00:00

The current code pokes into do_deploy manifests from do_image_complete when the do_image_complete task may or may not depend upon the do_deploy tasks in question. Often it gets lucky, sometimes it results in build failures. To fix this, split the functionality to its own task which can have the correct task dependencies. This means the data in BB_TASKDEPDATA is definitive, the other code can be dropped, as can the IMAGE_EXTRATYPES do_populate_lic dependencies from image.bbclass. This fixes bugs which show up as: NOTE: recipe linux-yocto-4.14.48+gitAUTOINC+d64aec9793_97c8063d2d-r0: task do_deploy: Started ERROR: core-image-minimal-1.0-r0 do_image_complete: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: 0001: *** 0002:write_deploy_manifest(d) 0003: File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest 0029: 'w+').write(output) 0030:} 0031: 0032:python write_deploy_manifest() { *** 0033: license_deployed_manifest(d) 0034:} 0035: 0036:python license_create_manifest() { 0037: import oe.packagedata File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 191, function: license_deployed_manifest 0187: # It is necessary to mark this will be used for image manifest 0188: man_dic[dep]["IMAGE_MANIFEST"] = True 0189: man_dic[dep]["PN"] = dep 0190: man_dic[dep]["FILES"] = \ *** 0191: " ".join(get_deployed_files(dep_dic[dep])) 0192: with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: 0193: for line in f.readlines(): 0194: key,val = line.split(": ", 1) 0195: man_dic[dep][key] = val[:-1] File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 289, function: get_deployed_files 0285: """ 0286: 0287: dep_files = [] 0288: excluded_files = [] *** 0289: with open(man_file, "r") as manifest: 0290: all_files = manifest.read() 0291: for f in all_files.splitlines(): 0292: if ((not (os.path.islink(f) or os.path.isdir(f))) and 0293: not os.path.basename(f) in excluded_files): Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build-st-730/tmp/sstate-control/manifest-qemux86_64-linux-yocto.deploy' ERROR: core-image-minimal-1.0-r0 do_image_complete: Function failed: write_deploy_manifest ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build-st-730/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/log.do_image_complete.50537 NOTE: recipe core-image-minimal-1.0-r0: task do_image_complete: Failed ERROR: Task (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/images/core-image-minimal.bb:do_image_complete) failed with exit code '1' NOTE: recipe linux-yocto-4.14.48+gitAUTOINC+d64aec9793_97c8063d2d-r0: task do_deploy: Succeeded (From OE-Core rev: b54cdaea7844ee3bf0c39eb97cc7c4c17ed5818c) Signed-off-by: Richard Purdie

license: Split image license functions to a separate class

2018-07-25T15:48:27+00:00

This means the image code is only included in image recipes through the IMAGE_CLASSES variable. This sets things up to allow us to fix image deploy dependency problems. (From OE-Core rev: fd44b8b4b2484f2d35c7a0e749e7dc316d601989) Signed-off-by: Richard Purdie

package: Use subprocess to be consistent with the rest of the class

2018-07-25T15:48:27+00:00

Using 'sub' in one function just confuses things, standardise and fix formatting of the parameters too. (From OE-Core rev: a740c638148ed7236c49eed55ae9a15b94e55b9f) Signed-off-by: Richard Purdie

Revert "relocate_sdk.py: remove hardcoded SDK path"

2018-07-25T15:46:56+00:00

This reverts commit 6671a4d980c8bef8f402780a308f6c43a25044aa. This breaks uninative tarball since the call of relocate_sdk.py from uninative.bbclass wasn't updated to account for this change. It isn't clear what value that code could pass in and this isn't simple to fix so revert until a better fix can be found that doesn't break uninative. (From OE-Core rev: b247392b4ced57cfe694656032f6a6723740a9e8) Signed-off-by: Richard Purdie

package/package_manager: multiprocess_exec -> multiprocess_launch

2018-07-24T10:52:27+00:00

After this replacement, the parent exception handling works so we don't need subprocess wrapping with bb.error in the underlying functions. The underlying contexts also have better module handling so the imports can be cleaned up. (From OE-Core rev: aa8260adf53139d776a2affe6118d28b295c1fab) Signed-off-by: Richard Purdie

package: Clean up getstatusoutput usage

2018-07-24T10:52:27+00:00

Replace usage of oe.utils.getstatusoutput() with direct subprocess calls. The call is just a wraper to the subprocess call of the same name and requires the caller to handle exceptions themselves. We usually do this badly, failing to show the output or the command or the return code. Its much safer to rely on a call like subprocess.check_output() instead. This also makes it easier to spot and remove cases where shell=True isn't needed in a later cleanup. (From OE-Core rev: 9f058857fb692f1251deb43bcaa7ed0120140093) Signed-off-by: Richard Purdie

sstate/bitbake.conf: Use pigz if available

2018-07-24T10:52:27+00:00

Currently the compression of sstate objects is single threaded. In the case of ltp, this takes around 33s. If we add pigz into the list of non-fatal HOSTTOOLS and then use if it available when building the sstate object, this time drops to around 6s. Since pigz is now widely available this is an optimisation we should utilise. (From OE-Core rev: 2de56aa0792ec93445130d801936a8ea643fad27) Signed-off-by: Richard Purdie

package: Call file to determine elf status in parallel

2018-07-24T10:52:27+00:00

This allows the calls to is_elf (which calls file) to happen in parallel allowing a speedup of do_package and do_populate_sysroot for native recipes. (From OE-Core rev: bbe0d3e26484f3f347262d40a8a9d415ce21fb43) Signed-off-by: Richard Purdie

package: Refactor to remove isElf/is_elf function duplication

2018-07-24T10:52:27+00:00

There are probably further cleanups needed here but this at least removes the major code duplication between these two similar funcitons, keeping the kernel module ".ko" extension check for efficiency to avoid opening and reading file contents in the general case. (From OE-Core rev: 7ad0c0d6ab12bebeac097fc0f5210c876dcfe9be) Signed-off-by: Richard Purdie

package: Allow parallel processing of shlib analysis

2018-07-24T10:52:27+00:00

This function is a bit more invasive to add parallelism to but allows the shlibs analysis to happen in multiple threads. In order to return values correctly/safely the data types needed tweaking to avoid lists and use immutable objects. (From OE-Core rev: b5788fb1f795f2f35d1788d8311e12984ffb2122) Signed-off-by: Richard Purdie