summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/build.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: build: Improve python execution tracebacksRichard Purdie2016-02-101-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If something fails in a exec_func_python() the current stack trace shows incorrect filenames and linenumbers. For example: The stack trace of python calls that resulted in this exception/failure was: File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: <module> 0196: chksum = bb.utils.sha256_file(fn) 0197: f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath))) 0198: 0199: *** 0200:copy_buildsystem(d) 0201: File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem 0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs" 0006: 0007:LICENSE = "MIT" 0008: *** 0009:inherit core-image 0010: 0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games" File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call 0531: The arguments are the same as for the Popen constructor. Example: 0532: 0533: check_call(["ls", "-l"]) 0534: """ *** 0535: retcode = call(*popenargs, **kwargs) 0536: if retcode: 0537: cmd = kwargs.get("args") 0538: if cmd is None: 0539: cmd = popenargs[0] The problem is the use of "FILE" to obtain the current filename. Instead, we therefore inject the function being executed into the methodpool which allows us to correct its linenumber and filename information. We can then clearly mark the initial piece as autogenerated and the rest of the linenumber and filename information should be correct. Afterwards the trace starts: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:copy_buildsystem(d) 0003: File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem 0062: import glob 0063: import oe.copy_buildsystem 0064: import subprocess 0065: *** 0066: subprocess.check_call("foo") 0067: 0068: oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True) 0069: 0070: conf_bbpath = '' File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call 0531: The arguments are the same as for the Popen constructor. Example: 0532: 0533: check_call(["ls", "-l"]) 0534: """ *** 0535: retcode = call(*popenargs, **kwargs) 0536: if retcode: 0537: cmd = kwargs.get("args") 0538: if cmd is None: 0539: cmd = popenargs[0] We can't inject into methodpool at parsing time, since there may be _append or other override operations against the function before its execution. (Bitbake rev: fae153095d23157dd7e72c29f683f86149ee33a8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/data: Don't expand python functions before execution [API change]Richard Purdie2016-02-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Right now, if you have some python code like: X = "a" def somefunction(d): d.setVar("X", "b") d.setVar("Y", "${X}") then any sane person would expect that Y = "b" at the end of the function. This is not the case, Y = "a". This is due to the python function being expanded before execution, the executed code would read d.setVar("Y", "a"). This understandably confuses people, it also makes it near impossible to write ${} in a python function without unintended things happening. I think there is general agreement we should fix this and standardise on non-expansion of python functions. We already don't expand anonymous python (mostly). I've checked OE-Core with buildhistory before and after this change and there were a small number of issues this exposed which I've sent patches for. I propose we default to not expanding python code and then deal with any consequences from that if/as/where identified. This will improve new user understanding and usability of the system, it also allows several long standing weird expansion issues to be fixed. (Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add expansion parameter to getVarFlagRichard Purdie2016-02-041-7/+7
| | | | | | | | | | | | | | | This sets the scene for removing the default False for expansion from getVarFlag. This would later allow True to become the expand default. On the most part this is an automatic translation with: sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *` There should be no functional change from this patch. (Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: minor shell_trap_code() formatting tweaksAndre McCurdy2016-01-301-3/+2
| | | | | | | | | | | Fix quoting of $BASH_COMMAND and avoid wrapping at 80 columns (the script which follows is likely to contain some very long lines, so line wrapping in bb_exit_handler() looks somewhat out of place). (Bitbake rev: 8e12c8f8441a7c6a03e603c5789d6037945704c1) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: clean up stamp-base related codesChen Qi2016-01-111-4/+4
| | | | | | | | | | | | The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful, clean them up. [YOCTO #8468] (Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Add BB_TASK_IONICE_LEVEL supportRichard Purdie2015-10-291-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio of tasks to be adjusted. This is in response to various qemu runtime timeouts which have been witnessed on the autobuilder, seemingly due to IO starvation (we already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal with certain 'random' failures we're seeing on the autobuilders in testing. The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>". For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd need superuser privileges to use realtime. The <prio> value is a default of 4, and can be set between 0 and 7 with 7 being lowest priority and 0 the highest. The user can set this freely with normal privileges Note that in order for this to take effect, you need the cfq scheduler selected for the backing block device. We could use nice wrapper functions for ioprio from modules like psutil however that would complicate bitbake dependencies. This version has some magic numbers but works on the main 32 and 64 bit x86 build architectures and can easily be extended if ever needed. When we move to python 3.x, we can likely replace this with standard calls. (Bitbake rev: b9471ad147b102c45d65f5ffd9521864df7ff9c1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: delete tasks thoroughlyChristopher Larson2015-09-031-10/+13
| | | | | | | | | | | | | | | We want addtask to be able to bring back a deleted task, but we don't want its previous dependencies to come back with it, so rather than marking a task as deleted and then skipping tasks marked as such, actually delete the task and its dependency information in deltask. While we're in that part of the code, also fix a couple 'not foo in bar' instances to 'foo not in bar', which is preferred in python. (Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Exit scripts with result of last shell functionAndre McCurdy2015-07-271-1/+1
| | | | | | | | | | | | | Since shell scripts run with 'set -e' the final exit at the end of the script can only be returning 0. However, for correctness and to follow the original intention of the 'cleanup' commands, let's fix the typo and return the success of the last shell function rather than the success of unhooking the exit trap handler. (Bitbake rev: bef724057f1ea81571dd3ac5a9cf862f818434b5) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: provide mechanism to bypass UI log suppressionPaul Eggleton2015-07-161-1/+6
| | | | | | | | | | | | | | | | | | The recent change to connect through the shell logging functions had an unexpected side-effect - bb.error() and bb.fatal() cause a flag to be set internally such that BitBake's UI will not print the full task log on failure; unfortunately we have in places within the OpenEmbedded metadata called these shell logging functions under error situations where we still want to see the full log (i.e., the message we're sending doesn't include the full error). Thus, provide a mechanism to fatally exit with an error but unset the flag, utilising the built-in python logging functionality that allows extra values to be passed in the log record. (Bitbake rev: e561b997c55e8537d82aa1339adfff4505cc38b7) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: set up infrastructure for shell message functionsPaul Eggleton2015-07-121-7/+43
| | | | | | | | | | | | | | | | | Create a fifo under ${T} for each running task that can be used by the task to send events back to BitBake. The purpose of this is to allow OpenEmbedded's logging.bbclass (which provides shell function equivalents for bb.warn(), bb.note() etc.) to have those functions trigger the appropriate events within BitBake so that messages are shown through the BitBake UI rather than just in the task log; to do that we just call the python functions. Part of the fix for [YOCTO #5275]. (Bitbake rev: bae330a1f8a59a912eca71177b3c6ba7c92a2f38) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add explict getVar param for (non) expansionRichard Purdie2015-06-231-4/+4
| | | | | | | | | | | | | | Rather than just use d.getVar(X), use the more explict d.getVar(X, False) since at some point in the future, having the default of expansion would be nice. This is the first step towards that. This patch was mostly made using the command: sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *` (Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: reset build mtime cache before the buildEd Bartosh2015-04-291-1/+16
| | | | | | | | | | | | | Introduced build mtime cache structure. Reset it before the build to prevent bitbake from crashing when build/tmp/stamps hierarchy is removed. [YOCTO: #7562] (Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/data: Write out more complete python run filesRichard Purdie2014-08-281-1/+1
| | | | | | | | | | | | | Currently the output in the python task/function run files is rather incomplete and effectively useless. This enhances the code to take advantage of the bitbake's dependency tracking and extend the output to include dependencies. This makes the files more usable for debugging purposes. Since this only happens at python function execution time, the overhead is minimal in the grand scheme of things. (Bitbake rev: 02667e048c3e632f857c87177c0022eaf5481802) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/*.py: Typo fixes/grammar/comment fixes, nothing functional.Robert P. J. Day2014-08-251-6/+5
| | | | | | | (Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Ensure shared work siginfo files are written to the ↵Richard Purdie2014-06-191-2/+6
| | | | | | | | | | | | | correct location Right now shared work signature data is saved to the non-shared directory which is confusing to everyone including bitbake. Whilst its messy, extra the stampbase data instead, which ensures the sig data is written to the correct location alongside its corresponding stamp file. (Bitbake rev: 7ae1d4844d9d3a76f86ef32c5a794e51e334e588) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Fix broken exception handlingRichard Purdie2014-06-011-3/+2
| | | | | | | | | | Checking for explicit exception names is bad, we also want to be able top rely on inheritance. Fix these checks to be part of the real except clauses so SkipPackage is recognised as being inherited from SkipRecipe. (Bitbake rev: b131229145e1f2c372d6230a7b554e436c13c3f9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Add SkipRecipe event to replace SkipPackageRichard Purdie2014-06-011-1/+1
| | | | | | | | | | | | In the depths of time we were rather confused about naming. bb files are recipes, the event to skip parsing them should be SkipRecipe, not SkipPackage. This changes bitbake to use the better name but leaves the other around for now. We can therefore start removing references to it from the metadata. (Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: measure task duration with server-side timestampsAlexandru DAMIAN2014-03-211-0/+2
| | | | | | | | | | | | | | | | | | | | | The buildstats and toaster use separate time markers to measure the duration of task execution. This causes a mismatch in the time measured by buildstats class and the time measured in toaster. The solution implemented here is to timestamp the creation of every TaskBase event on the bitbake server side and calculate the execution duration as the difference between creation time of TaskSucceeded and TaskStarted events. Based on an original patch by Marius Avram. [YOCTO #5485] (Bitbake rev: 7a08282c074c264f414cf7665dc873f51245072c) Signed-off-by: Marius Avram <marius.avram@intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: filter out expanded empty strings for lockfiles flagStefan Stanacar2014-02-281-1/+1
| | | | | | | | | | | | | | | | | | | | | | If we have something similar to: LOCKFILES = "${TMPDIR}/my.lock" LOCKFILES_qemuall = "" do_task[lockfiles] += "${LOCKFILES}" when expanded, lockfiles will be empty for qemu, resulting in File "/home/stefans/yocto/poky/bitbake/lib/bb/utils.py", line 630, in mkdirhier raise e OSError: [Errno 2] No such file or directory: '' This should filter out the empty expansions. (Bitbake rev: 7813e1bfd08cd48871f8c03cae2810265590105d) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: fix handling of task override for tasks with underscores in ↵Paul Eggleton2014-02-171-1/+1
| | | | | | | | | | | | | | their names Tasks whose names contain underscores (such as do_populate_sdk in OE) when converted to a task override do not function properly. If we replace underscores with hyphens we can still have a working override for these tasks. (Bitbake rev: cf90bd6b2a0ab7dce922bffb500d6a2ff2ff10e2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: build.py: create separate function for shell trap creation ↵Laurentiu Palcu2014-02-091-17/+20
| | | | | | | | | | | | | | | code Currently, the shell trap code was created in exec_func_shell(). Split the function so that we can create the code separately. Also, some whitespaces were automatically deleted by my editor. Since this is not necessarily a bad thing, leave these changes too. (Bitbake rev: c712e622d20c61a07c9c172b60e9dc6beae14197) Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/lib/bb/build.py: fix the task flags cleandirsRobert Yang2014-01-071-0/+1
| | | | | | | | | | | | | | | The user manual said: 'cleandirs' - directories which should created before the task runs but should be empty But it only removes the dir, doesn't create it [YOCTO #5703] (Bitbake rev: 0636797d75874ce4577f29011d69c56a4c6b9e89) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/ast: Create strong task add/del API in bb.buildRichard Purdie2013-12-181-5/+32
| | | | | | | | | | | | | | | Currently its near impossible to control task addition/deletion from metadata context. This adds stong add/deltask API to bb.build which is traditionally where it resided. The rather broken remove_tasks function was removed, it didn't appear to do anything useful or have any users. This allows us to clean up hacks currently in use in metadata and use standard API for it instead. (Bitbake rev: bf7138dd38fc1f8efca80891198e3422fef64093) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/BBHandler/build: Add support for removing tasks (deltask)Richard Purdie2013-12-181-1/+5
| | | | | | | | | | | | | | | | | Back in the depths of time we did support task removal. In the pre AST days it was nearly impossible to continue supporting it, it wasn't used so it was dropped. With the modern codebase we can easily now support deltask and it would be very useful within the metadata since it can massively simplify dependency trees. As an example, a core-image-sato had 47703 inter task dependencies before this patch and a patch to native.bbclass, afterwards with the noexec tasks deleted, we had 29883. Such a significant simplification is worthwhile and justifies adding a deltask operation to the system. (Bitbake rev: acecbde6fb70ff3c96deab3cdf819d8442e87ed4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: add single-quotes around already-expanded directory namePeter Seebach2013-11-201-1/+1
| | | | | | | | | | | | | | | | | | | If the computed name of a directory contains an undefined variable reference, bitbake dutifully creates a directory with a name that has ${...} in it. However, the actual task script created then tries to cd to that directory, and the cd command fails, because no such directory exists -- because the shell has helpfully removed the ${...} which did not match any actual variables. Since we want the name to be used exactly-as-is, add single quotes around the name so this doesn't cause strange failures running tasks, which allows us to progress past such failures and get to a point where they can be diagnosed. (Bitbake rev: 2809c2e6f2f35f9b08058950be896947ab5a0284) Signed-off-by: Peter Seebach <peter.seebach@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build, toaster: record proper task typeAlexandru DAMIAN2013-11-041-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Bitbake tasks may be of type 'python' or 'shell', or they may not be executed at all, which is record as task type 'noexec'. In order to record proper task type, this patch: * creates no exec task type as the default value in the toaster model definition * adds full task flags to the bb.build.TaskStarted event in build.py * if the task actually starts, the toaster ui will record the type of the task as either 'python' or 'shell' based on the task flags. [YOCTO #5073] [YOCTO #5075] [YOCTO #5327] (Bitbake rev: 6648c57e6d369fc009ea3a9fe939def5d2c67bf5) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Add BB_TASK_NICE_LEVEL to task codeRichard Purdie2013-09-221-0/+8
| | | | | | | | | | | | | | | | | | On Linux its not possible for processes to regain a previous nice level after it has changed. Its therefore not possible to have a core low priority and then raise the priorities of individual tasks. This variable allows us to do something like: BB_TASK_NICE_LEVEL = "5" BB_TASK_NICE_LEVEL_task-testimage = "0" to give priority to specific tasks which the BB_NICE_LEVEL functionality doesn't give us the option of. (Bitbake rev: 94d82997220c6cfc7028f76719df028ba8254a5c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: build, runqueue: adds info to the *runQueue* eventsAlexandru DAMIAN2013-09-221-0/+2
| | | | | | | | | | | | | | | | | | | This patch adds task identifying information for all runQueue and sceneQueue events, and for bb.build.Task* events. This will allow matching event to specific tasks in the UI handlers processing these events. Adds RunQueueData functions to get the task name and task file for usage with the runQueue* events. Adds taskfile and taskname properties to bb.build.TaskBase. Adds taskfile and taskname properties to the *runQueue* events (Bitbake rev: b4a5e4be50d871a80dbe0993117d73f5ad82e38f) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Add logfile to add TaskBase eventsRichard Purdie2013-09-161-10/+6
| | | | | | | | | | | We add the path to the logfile for all Task events except TaskInvalid so that we can trace back the logfile locations at some future point. TaskInvalid doesn't ever have a logfile. (Bitbake rev: 8344d84c609446f59f9619cc7ca0d693b7e2bbd6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Fix profile file namesRichard Purdie2013-09-011-1/+1
| | | | | | | | | | | | | Using the basename of the .bb file is not unique, for example xxx-native and xxx can overwrite each other. If this happens whilst running, you can get odd backtraces as one file is parsed as another tries to write out new data. Avoid issues by using PN for the output filename instead. (Bitbake rev: c9534f8e59d44b885334607ed90a3be2e492ec69) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: create symlink for run.do_xxx scriptsNicolas Dechesne2013-08-231-0/+14
| | | | | | | | | | | | | | | | | | | | | | The 'courtesy' symlink for log.do_xxx are quite useful when debugging, so with this commit, we now get similar 'courtesy' symlink for run.do_xxx scripts. We only create symlink for tasks, not individual functions. The symlink is create right before the actual runfile is created, indeed we cannot create the symlink right after running the task since a failure or execption can happen, in which case the symlink wouldn't be created, and symlink are particularely useful when the task failed! Another option would be create the symlink after the runfile is created, and before the script is executed, but that means we need to duplicate the code in case of Shell vs Python task. (Bitbake rev: a672b39c5d529ba85d72eee8fef4c4273eaa5397) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Provide useful diagnostics when exiting.Peter Seebach2013-08-201-1/+24
| | | | | | | | | | | Running scripts with 'set -e' produces silent failures with no diagnostic. Add an exit handler which produces diagnostics, including details of what was running if the shell seems to be bash. (Bitbake rev: e213e6a4c297a4f1c22eed15bd7b4cbc0e9eab4f) Signed-off-by: Peter Seebach <peter.seebach@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Cleanup data store referencesRichard Purdie2013-06-201-22/+22
| | | | | | | | Clean up a number of old style accesses to the datastore. (Bitbake rev: d872fef2c38749c3c6f5d84344db3ec2f9f134ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/build: Add recideptask flagRichard Purdie2013-06-201-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | Currently, tasks like fetchall are slightly broken since if a recipe has specific [depends] which occur after do_fetch and add items not listed in DEPENDS and RDEPENDS, they are not caught by recrdeptask. We've gone around in circles on this issue (e.g http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/runqueue.py?id=5fa6036d49ed7befe6ad50ec95c61a50aec48195 ) and in many cases the behaviour of recrdepends is correct but tasks like fetchall need the other behaviour. To address this we add a recideptask flag which can be used in conjuction with the recrdeptask flag to specify which task to to the inspection upon. This means entries like do_rootfs[depends] which have do_fetch tasks are caught and run. I'm not 100% happy with needing another flag but I don't see any rational way to get the correct behaviour in all cases without it. [YOCTO #4597] (Bitbake rev: f8c9b292b02ce2c28741b74901205f5e5807ca87) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Avoid empty promisesRichard Purdie2013-05-241-1/+1
| | | | | | | | | | | | The "see xxx for further information" is misleading since it is just the same information. Clarify just to mention the that this is the location of the logfile without any empty promise. [YOCTO #4343] (Bitbake rev: 7088c0e8553dd3c408b5bc06f8c34d5b72e9ea9a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: Clean up various file access syntaxRichard Purdie2013-05-091-8/+7
| | | | | | | | | | | | | Python 3 is stricter about how files are accessed. Specficially: * Use open(), not file() * Use binary mode for binary files (when checksumming) * Use with statements to ensure files get closed * Add missing file close statements (Bitbake rev: 9f08b901375ba640f47596f1bcf43f98a931550f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: avoid deleting taint files when writing stampsPaul Eggleton2013-02-221-1/+4
| | | | | | | | | | | | | | | | | | | | The stamp cleaning process that occurs before writing out new stamps for a task was deleting taint files as well. This resulted in tasks that were forcibly re-executed using the -f or -C command line options to have their previous output restored from shared state when called upon a second time, because the taint value was no longer incorporated into the task signature and thus it was reverting to its previous value. This also affected the kernel menuconfig command in OE-Core. Note that the taint file *is* still deleted when doing -c clean, which is the desired behaviour. Fixes [YOCTO #3919]. (Bitbake rev: e6db0ee31178d4386802e720d75303ec7dc21519) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Dump out performance data of individual tasksRichard Purdie2013-01-281-2/+16
| | | | | | | (Bitbake rev: 32aa49519e4f015e3c21466a7e5dc939f6369851) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Fix traceback when there are no dependenciesRichard Purdie2013-01-071-3/+4
| | | | | | | | | A recipe with no dependencies results in a traceback (e.g. all in ASSUME_PROVIDED). This shouldn't happen and this patch fixes it. (Bitbake rev: dee7decf87dfa8cb966fe40846d27f3e6ab1846b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Preserve sigdata files in the stamps directoryRichard Purdie2012-12-031-2/+7
| | | | | | | | | Leaving the sigdata files around can aid debugging and doesn't harm anything. This is the easiest way to allow this to happen. (Bitbake rev: 1f500149ecd533a6edbeea902c3f1e009c755154) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen.py: Avoid removing too many stamps when cleaningRichard Purdie2012-11-281-4/+6
| | | | | | | | | | | | | | | | The "*" part of the mask is to ensure we clean both any stamp, and any setscene varient. It turns out we would also trample other tasks, e.g. do_package_write could trample do_package_write_rpm. do_package also tramples do_package_write_* but this is less of an issue since the other tasks depend on it. Rather than use the wildcard, we can just use a list instead. [YOCTO #3484] (Bitbake rev: c14d831ea3f625e9a47266a0c4e6deefc924ca5a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen: Add support for stamp 'clean' masksRichard Purdie2012-09-201-0/+29
| | | | | | | | | | | | | | | | | | | | | Currently when we execute a task, we don't remove other potentially stale stamps. This can mean if you switch between two different versions of a recipe without a clean, the build can get very confused. This patch adds in functionality to allow a wildcard expression of stamp files to be removed when creating a new stamp file. This patch adds in the core of the code to enable this but it also requires metadata support to enable it. When writing this improvement I went through several different options but this was the only way I could find to allow things like noexec tasks to function correctly (where stamps need to be created without the data store). [YOCTO #2961] (Bitbake rev: e026469b307522e5b6a680e0ae5587749d33dcae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Correct package to recipe in TaskBase events to use ↵Richard Purdie2012-08-181-1/+1
| | | | | | | | consistent terminology (Bitbake rev: e5045429bce15b66c4355be214db3982ac7761f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: build.py: Add stampdir argument to cached_mtime_noerrorAndrei Gherzan2012-08-161-1/+1
| | | | | | | | | | | | After commit 2718537b4b04eb3d80ab4d74171b58e7b8dd68b8 (bitbake: build.py: Only execute mkdirhier if stampdir doesn't exist) build failes as cached_mtime_noerror needs an argument - stamp dir. This argument was forgotten. (Bitbake rev: e69103c4f7e45c24f1fbe9df0383f39e877abcb4) Signed-off-by: Andrei Gherzan <andrei.gherzan@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Only execute mkdirhier if stampdir doesn't existRichard Purdie2012-08-161-1/+3
| | | | | | | | | | | | | I noticed this was showing up on profile logs as a sigificant time user in "bitbake bash" when bash was already built. It reduces the time from 5.2 to 4.5 seconds in my test environment. We make use of the parser's mtime cache as once a directory exists, we can assume it continues to exist and this avoids syscalls. (Bitbake rev: 769b694eeb617bb793bd79d0d7b29c43d2646ece) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Improve _logged_communicate bufferingRichard Purdie2012-06-251-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | There are two problems with the _logged_communicate that are both caused as a result of buffering I/O issues: 1) log truncation when python fails 2) While a bitbake task is running it is impossible to see what is going on if it is only writing a small incremental log that is smaller than the buffer, or you get only a partial log, up until the task exists. It is worse in the case that stderr and stdout are separate file handles, because previous code blocks on the read of stdout and then stderr, serially. The right approach is simply to use select() to determine if there is data available and also flush the log before exiting. This is based on a patch from Jason Wessel <jason.wessel@windriver.com> with some changes to flush upon exit, abstract the non blocking file descriptor setup and drop the buffer size parameter. (Bitbake rev: 361fb71e907aa84c28ecec79fefc6ca39c39172f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskdata.py: Add support for rdepends task flagRichard Purdie2012-06-251-0/+1
| | | | | | | | | | Currently its not possible to add arbitrary RDEPENDS to a specific task. This can be useful and this patch adds functionality equivalent to the 'depends' task flag. (Bitbake rev: db65080a6199baecc5c422294a4c4a9fd12dc29e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: ensure -f causes dependent tasks to be re-runPaul Eggleton2012-06-211-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If -f is specified, force dependent tasks to be re-run next time. This works by changing the force behaviour so that instead of deleting the task's stamp, we write a "taint" file into the stamps directory, which will alter the taskhash randomly and thus trigger the task to re-run next time we evaluate whether or not that should be done as well as influencing the taskhashes of any dependent tasks so that they are similarly re-triggered. As a bonus because we write this file as <stamp file name>.taskname.taint, the existing code which deletes the stamp files in OE's do_clean will already handle removing it. This means you can now do the following: bitbake somepackage [ change the source code in the package's WORKDIR ] bitbake -c compile -f somepackage bitbake somepackage and the result will be that all of the tasks that depend on do_compile (do_install, do_package, etc.) will be re-run in the last step. Note that to operate in the manner described above you need full hashing enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler that inherits from BasicHash). If this is not the case, -f will just delete the stamp for the specified task as it did before. This fix is required for [YOCTO #2615] and [YOCTO #2256]. (Bitbake rev: f7b55a94226f9acd985f87946e26d01bd86a35bb) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Add support for log and run filename changesMark Hatle2012-05-231-3/+17
| | | | | | | | | | | | | | | | | | | | | | | The format of the log file and run file are now selectable using BB_LOGFMT and BB_RUNFMT, respectively. The following values may be used: {task} - task name {taskfunc} - task.func or func, if task==func {func} - function name, only available in BB_RUNFMT {pid} - pid The log/run files may be placed into a subdirectory that is relative to T. Default BB_LOGFMT is: log.{task}.{pid} Default BB_RUNFMT is: run.{func}.{pid} (Bitbake rev: 588da606eb81c52cb92d29041e1c67897427bfdf) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Add additional debug messagesMark Hatle2012-05-231-0/+8
| | | | | | | | | | | We now add a debug message when entering and exiting a python or shell function. This makes it easier to inspect a log and figure out the run order from the logs. (Bitbake rev: a0f554d0e722b6705844c6031fdcafa5d1a1c8a7) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-18 16:51:45 +0000