linux/poky.git/bitbake/lib/bb/build.py, branch krogoth-15.0.1

bitbake: build/utils: Allow python functions to execute with real exception handling

2016-03-31T08:13:04+00:00

With the code as it stands today it not possible to execute a python function and get "normal" python exception handling behaviour. If a python function raises an exception, it forces a traceback to be printed and the exception becomes a FuncFailed exception. This adds in a parameter 'pythonexception' which allows standard python exceptions to be passed unchanged with no traceback. Ultimately we may want to change to this convention in various places but at least it means we can start to add sane functions now. (Bitbake rev: 85cf22fd0ed26bb7dc7738ef2a10441891f38ae2) Signed-off-by: Richard Purdie

bitbake: build: Improve python execution tracebacks

2016-02-10T15:47:11+00:00

If something fails in a exec_func_python() the current stack trace shows incorrect filenames and linenumbers. For example: The stack trace of python calls that resulted in this exception/failure was: File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: 0196: chksum = bb.utils.sha256_file(fn) 0197: f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath))) 0198: 0199: *** 0200:copy_buildsystem(d) 0201: File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem 0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs" 0006: 0007:LICENSE = "MIT" 0008: *** 0009:inherit core-image 0010: 0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games" File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call 0531: The arguments are the same as for the Popen constructor. Example: 0532: 0533: check_call(["ls", "-l"]) 0534: """ *** 0535: retcode = call(*popenargs, **kwargs) 0536: if retcode: 0537: cmd = kwargs.get("args") 0538: if cmd is None: 0539: cmd = popenargs[0] The problem is the use of "FILE" to obtain the current filename. Instead, we therefore inject the function being executed into the methodpool which allows us to correct its linenumber and filename information. We can then clearly mark the initial piece as autogenerated and the rest of the linenumber and filename information should be correct. Afterwards the trace starts: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: 0001: *** 0002:copy_buildsystem(d) 0003: File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem 0062: import glob 0063: import oe.copy_buildsystem 0064: import subprocess 0065: *** 0066: subprocess.check_call("foo") 0067: 0068: oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True) 0069: 0070: conf_bbpath = '' File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call 0531: The arguments are the same as for the Popen constructor. Example: 0532: 0533: check_call(["ls", "-l"]) 0534: """ *** 0535: retcode = call(*popenargs, **kwargs) 0536: if retcode: 0537: cmd = kwargs.get("args") 0538: if cmd is None: 0539: cmd = popenargs[0] We can't inject into methodpool at parsing time, since there may be _append or other override operations against the function before its execution. (Bitbake rev: fae153095d23157dd7e72c29f683f86149ee33a8) Signed-off-by: Richard Purdie

bitbake: build/data: Don't expand python functions before execution [API change]

2016-02-10T15:47:11+00:00

Right now, if you have some python code like: X = "a" def somefunction(d): d.setVar("X", "b") d.setVar("Y", "${X}") then any sane person would expect that Y = "b" at the end of the function. This is not the case, Y = "a". This is due to the python function being expanded before execution, the executed code would read d.setVar("Y", "a"). This understandably confuses people, it also makes it near impossible to write ${} in a python function without unintended things happening. I think there is general agreement we should fix this and standardise on non-expansion of python functions. We already don't expand anonymous python (mostly). I've checked OE-Core with buildhistory before and after this change and there were a small number of issues this exposed which I've sent patches for. I propose we default to not expanding python code and then deal with any consequences from that if/as/where identified. This will improve new user understanding and usability of the system, it also allows several long standing weird expansion issues to be fixed. (Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb) Signed-off-by: Richard Purdie

bitbake: lib/bb: Add expansion parameter to getVarFlag

2016-02-04T23:47:49+00:00

This sets the scene for removing the default False for expansion from getVarFlag. This would later allow True to become the expand default. On the most part this is an automatic translation with: sed -e 's:$\.getVarFlag([^,()]*, [^,()]*$):\1, False):g' -i `grep -ril getVar *` There should be no functional change from this patch. (Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913) Signed-off-by: Richard Purdie

bitbake: build.py: minor shell_trap_code() formatting tweaks

2016-01-30T11:43:55+00:00

Fix quoting of $BASH_COMMAND and avoid wrapping at 80 columns (the script which follows is likely to contain some very long lines, so line wrapping in bb_exit_handler() looks somewhat out of place). (Bitbake rev: 8e12c8f8441a7c6a03e603c5789d6037945704c1) Signed-off-by: Andre McCurdy Signed-off-by: Richard Purdie

bitbake: bitbake: clean up stamp-base related codes

2016-01-11T23:28:55+00:00

The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful, clean them up. [YOCTO #8468] (Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e) Signed-off-by: Chen Qi Signed-off-by: Richard Purdie

bitbake: build/utils: Add BB_TASK_IONICE_LEVEL support

2015-10-29T07:35:52+00:00

Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio of tasks to be adjusted. This is in response to various qemu runtime timeouts which have been witnessed on the autobuilder, seemingly due to IO starvation (we already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal with certain 'random' failures we're seeing on the autobuilders in testing. The format of the data in the variable is BB_TASK_IONICE_LEVEL = ".". For , 2 is best effort (the default), 1 is real time and 3 is idle. You'd need superuser privileges to use realtime. The value is a default of 4, and can be set between 0 and 7 with 7 being lowest priority and 0 the highest. The user can set this freely with normal privileges Note that in order for this to take effect, you need the cfq scheduler selected for the backing block device. We could use nice wrapper functions for ioprio from modules like psutil however that would complicate bitbake dependencies. This version has some magic numbers but works on the main 32 and 64 bit x86 build architectures and can easily be extended if ever needed. When we move to python 3.x, we can likely replace this with standard calls. (Bitbake rev: b9471ad147b102c45d65f5ffd9521864df7ff9c1) Signed-off-by: Richard Purdie

bitbake: build: delete tasks thoroughly

2015-09-03T11:33:08+00:00

We want addtask to be able to bring back a deleted task, but we don't want its previous dependencies to come back with it, so rather than marking a task as deleted and then skipping tasks marked as such, actually delete the task and its dependency information in deltask. While we're in that part of the code, also fix a couple 'not foo in bar' instances to 'foo not in bar', which is preferred in python. (Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1) Signed-off-by: Christopher Larson Signed-off-by: Richard Purdie

bitbake: build: Exit scripts with result of last shell function

2015-07-27T22:29:16+00:00

Since shell scripts run with 'set -e' the final exit at the end of the script can only be returning 0. However, for correctness and to follow the original intention of the 'cleanup' commands, let's fix the typo and return the success of the last shell function rather than the success of unhooking the exit trap handler. (Bitbake rev: bef724057f1ea81571dd3ac5a9cf862f818434b5) Signed-off-by: Andre McCurdy Signed-off-by: Richard Purdie

bitbake: lib/bb: provide mechanism to bypass UI log suppression

2015-07-16T14:09:26+00:00

The recent change to connect through the shell logging functions had an unexpected side-effect - bb.error() and bb.fatal() cause a flag to be set internally such that BitBake's UI will not print the full task log on failure; unfortunately we have in places within the OpenEmbedded metadata called these shell logging functions under error situations where we still want to see the full log (i.e., the message we're sending doesn't include the full error). Thus, provide a mechanism to fatally exit with an error but unset the flag, utilising the built-in python logging functionality that allows extra values to be passed in the log record. (Bitbake rev: e561b997c55e8537d82aa1339adfff4505cc38b7) Signed-off-by: Paul Eggleton Signed-off-by: Richard Purdie