summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/build.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: process: Improve _logged_communicate bufferingRichard Purdie2012-06-251-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | There are two problems with the _logged_communicate that are both caused as a result of buffering I/O issues: 1) log truncation when python fails 2) While a bitbake task is running it is impossible to see what is going on if it is only writing a small incremental log that is smaller than the buffer, or you get only a partial log, up until the task exists. It is worse in the case that stderr and stdout are separate file handles, because previous code blocks on the read of stdout and then stderr, serially. The right approach is simply to use select() to determine if there is data available and also flush the log before exiting. This is based on a patch from Jason Wessel <jason.wessel@windriver.com> with some changes to flush upon exit, abstract the non blocking file descriptor setup and drop the buffer size parameter. (Bitbake rev: 361fb71e907aa84c28ecec79fefc6ca39c39172f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskdata.py: Add support for rdepends task flagRichard Purdie2012-06-251-0/+1
| | | | | | | | | | Currently its not possible to add arbitrary RDEPENDS to a specific task. This can be useful and this patch adds functionality equivalent to the 'depends' task flag. (Bitbake rev: db65080a6199baecc5c422294a4c4a9fd12dc29e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: ensure -f causes dependent tasks to be re-runPaul Eggleton2012-06-211-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If -f is specified, force dependent tasks to be re-run next time. This works by changing the force behaviour so that instead of deleting the task's stamp, we write a "taint" file into the stamps directory, which will alter the taskhash randomly and thus trigger the task to re-run next time we evaluate whether or not that should be done as well as influencing the taskhashes of any dependent tasks so that they are similarly re-triggered. As a bonus because we write this file as <stamp file name>.taskname.taint, the existing code which deletes the stamp files in OE's do_clean will already handle removing it. This means you can now do the following: bitbake somepackage [ change the source code in the package's WORKDIR ] bitbake -c compile -f somepackage bitbake somepackage and the result will be that all of the tasks that depend on do_compile (do_install, do_package, etc.) will be re-run in the last step. Note that to operate in the manner described above you need full hashing enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler that inherits from BasicHash). If this is not the case, -f will just delete the stamp for the specified task as it did before. This fix is required for [YOCTO #2615] and [YOCTO #2256]. (Bitbake rev: f7b55a94226f9acd985f87946e26d01bd86a35bb) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Add support for log and run filename changesMark Hatle2012-05-231-3/+17
| | | | | | | | | | | | | | | | | | | | | | | The format of the log file and run file are now selectable using BB_LOGFMT and BB_RUNFMT, respectively. The following values may be used: {task} - task name {taskfunc} - task.func or func, if task==func {func} - function name, only available in BB_RUNFMT {pid} - pid The log/run files may be placed into a subdirectory that is relative to T. Default BB_LOGFMT is: log.{task}.{pid} Default BB_RUNFMT is: run.{func}.{pid} (Bitbake rev: 588da606eb81c52cb92d29041e1c67897427bfdf) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Add additional debug messagesMark Hatle2012-05-231-0/+8
| | | | | | | | | | | We now add a debug message when entering and exiting a python or shell function. This makes it easier to inspect a log and figure out the run order from the logs. (Bitbake rev: a0f554d0e722b6705844c6031fdcafa5d1a1c8a7) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Cleanup exec_func_shellMark Hatle2012-05-231-5/+5
| | | | | | | | | | | exec_func_python and exec_func_shell are similar, but variable usage has diverged sync the two up. Since exec_func_python is first use that as the guide for the later exec_func_shell variable naming. (Bitbake rev: ccfe1a3a2419172799957676107f240badf3f062) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Add a log to capture task execution orderMark Hatle2012-05-231-1/+15
| | | | | | | | | | | The new log.task_order contains an ordered list of the tasks as they were executed in any given recipe. The format of the lines is <task> <pid>: <log file> (Bitbake rev: 8662b43dcbd6e38a5b2ab9d6529af1fb08c146bf) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bb.build: use relative log links, not absoluteChristopher Larson2012-03-291-2/+3
| | | | | | | | | | | | Using symlinks to absolute paths isn't gaining us anything, and can cause problems in situations such as automated build systems when they try to resolve the link, depending on the mechanism used to pull the build artifacts (e.g. nfs). (Bitbake rev: f9cf2671c0ecad153db11bca1aebe151213bfb42) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/classes: Convert to use appendVar and appendVarFlagsRichard Purdie2012-03-051-2/+2
| | | | | | (From OE-Core rev: 3b57de68e70e77dbc03c0616a83a29a2e99e40b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build: report TaskFailedSilent events as "Failed"Paul Eggleton2012-03-011-1/+8
| | | | | | | | | | | | The change for setscene events to fire a TaskFailedSilent event instead of TaskFailed resulted in "FailedSilent" being reported in the task finish note log entry, which is not really desirable, so change it back to reporting "Failed" again. (Bitbake rev: 224bc74d4e901b7886b845fbb3b5fe7564a2f6cb) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: implement BB_VERBOSE_LOGSPaul Eggleton2012-02-281-1/+2
| | | | | | | | | | Enable configuring whether "set +x" is added to all shell tasks rather than forcing it; this is enabled by setting BB_VERBOSE_LOGS to 1. (Bitbake rev: 659411b6bb30e1a8355afc1c29b8170a8f2b55ac) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fix setscene task start/failure handlingPaul Eggleton2012-02-281-1/+9
| | | | | | | | | | | | | * When a setscene task starts, print out that it's starting in the UI (ensuring we get the correct task name) * When a setscene task fails, ensure we remove it from the list of running tasks so that if you break out any time afterwards it is not still listed. (Bitbake rev: e8a3499c95a6d4f2b8fed002fb9504733c5be3c6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Hob: A new implemetation (v2)Dongxiao Xu2012-02-241-4/+6
| | | | | | | | | | | | | | | | | | | | | This commit implements a new design for hob Some of the new features: - Friendly new designed GUI. Quick response to user actions. - Two step builds support package generation and image generation. - Support running GUI seprarately from bitbake server. - Recipe/package selection and deselection. - Accurate customization for image contents and size. - Progress bars showing the parsing and build status. - Load/save user configurations from/into templates. (Bitbake rev: 4dacd29f9c957d20f4583330b51e5420f9c3338d) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Shane Wang <shane.wang@intel.com> Signed-off-by: Liming An <limingx.l.an@intel.com> Signed-off-by: Fengxia Hua <fengxia.hua@intel.com> Designed-by: Belen Barros Pena <belen.barros.pena@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/knotty: avoid printing full task log when error already printedPaul Eggleton2012-01-151-3/+15
| | | | | | | | | | | | | | If a task has logged an ERROR then don't print the contents of the task's log file in knotty (the default terminal UI). As a side-effect we now also respect BBINCLUDELOGS in knotty; if it is false we never print the log (but the pointer to the log file is always printed). (Bitbake rev: b9746b7e4d7aa5c34eba15a61427bfc6949af123) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/fetch2: improve error formatting for fetcher errorsPaul Eggleton2012-01-151-1/+1
| | | | | | | | | | | | | | | * The "name" argument to FuncFailed is rarely used as a name in actual usage within bitbake, so don't treat it as one in the output. * Don't print URL for FetchError if it was not specified (i.e. don't output "Fetcher failure for URL 'None'") * Don't include URL in "unable to fetch from any source" message since we supply it to FetchError and it will be printed anyway. * Don't include URL in "checksum failed" message for the same reason (Bitbake rev: 86811bd85e2e453ee92a05fe60160d9b49ac69e8) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Be determistic about a function's cwdRichard Purdie2011-12-011-2/+1
| | | | | | | | | | | | | | | | | There is a subtle but nasty problem that a function's cwd can vary depending on whether ${B} (often ${S}) exists before the funciton is called or not. Most functions in the system can cope with this but its bad practise and I've just witnessed build failures resulting from this during image generation from bootimg.bbclass. I also suspect this could explain some odd fetcher behaviour witnessed in the past. This change ensures we always call funcitons with a specific build directory making things deterministic. (Bitbake rev: ef0888f83fa4408eb768257d7e03700202faad18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update users of getVar/setVar to use the data store functions directlyRichard Purdie2011-11-271-2/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/lib/bb/msg.py: fix setting debug and verbosity levelsPaul Eggleton2011-09-251-2/+2
| | | | | | | | | | | | | | The debug and verbosity levels (as set by the -D and -v command line options respectively) were not being passed through within msg.py since bitbake revision 45aad2f9647df14bcfa5e755b57e1ddab377939a due to incorrect variable names. Fixes [YOCTO #1513]. (Bitbake rev: c6e88b7c0e61f9586a275df53f48b90687c5f92f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Fix cut and paste errorRichard Purdie2011-08-151-2/+2
| | | | | | (Bitbake rev: a1980fc89e45be6fbd4a193aaa8142e8bebcf3f4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/logging: Overhaul internal logging processRichard Purdie2011-08-151-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At the moment it bugs me a lot that we only have one effective logging level for bitbake, despite the logging module having provision to do more advanced things. This patch: * Changes the core log level to the lowest level we have messages of (DEBUG-2) so messages always flow through the core logger * Allows build.py's task logging code to log all the output regardless of what output is on the console and sets this so log files now always contain debug level messages even if these don't appear on the console * Moves the verbose/debug/debug-domains code to be a UI side setting * Adds a filter to the UI to only print the user requested output. The result is more complete logfiles on disk but the usual output to the console. There are some behaviour changes intentionally made by this patch: a) the -v option now controls whether output is tee'd to the console. Ultimately, we likely want to output a message to the user about where the log file is and avoid placing output directly onto the console for every executing task. b) The functions get_debug_levels, the debug_levels variable, the set_debug_levels, the set_verbosity and set_debug_domains functions are removed from bb.msg. c) The "logging" init function changes format. d) All messages get fired to all handlers all the time leading to an increase in inter-process traffic. This could likely be hacked around short term with a function for a UI to only request events greater than level X. Longer term, having masks for event handlers would be better. e) logger.getEffectiveLevel() is no longer a reliable guide to what will/won't get logged so for now we look at the default log levels instead. [YOCTO #304] (Bitbake rev: 45aad2f9647df14bcfa5e755b57e1ddab377939a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add task specific stamp file supportRobert Yang2011-06-281-2/+2
| | | | | | | | | | | | | | | | | | This patch, based on proof of concept code from Richard adds code to bitbake to allow individual tasks to optionally specify their stamp file using the stamp-base flag. This takes the same form as the STAMP variable but can be specified on a per task basis. Code is also added to runqueue to ensure that if two tasks share the same stamp file, only one will be executed at once. A significant usecase for this code is to share source code (${S}) between recipes where separate build directories (${B}) are used. (Bitbake rev: 41bef02bef8379590ba012319aebe05068a8081e) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue.py: Add umask task controlMark Hatle2011-06-241-0/+1
| | | | | | | | | | | | | | | | The umask for a task can now be set as: task[umask] = 022 task[umask] = '022' If specified as a text string, it must be octal. (This is due to recipe parsing where it's always set to a string.) [RP tweaked to use None instead of -1] (Bitbake rev: a5caaaaba8f0db1af5d8f2e610021d6d4b56894e) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build.py: Drop exec_shell environment as its now unneededRichard Purdie2011-06-081-6/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cooker.py: Misc sync with upstreamRichard Purdie2011-06-081-0/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build/siggen: Ensure a task signature file is generated for each task that ↵Richard Purdie2011-05-251-0/+6
| | | | | | | | | | | | | | | | | | is executed Tracing down signature changes and dependencies is hard unless the complete task chain can be evaultated. Since we have the data available at task exeuction time, writing it to disk makes sense. This change is suboptimal internal knowledge stamp_internal() has leaks into make_stamp() and the concepts used by this code need revisiting but the change brings enough usability benefit to warrant its inclusion despite that. Addresses issue [YOCTO #1074] (Bitbake rev: 139b8a625818225c358a1b8363518d7ed6913188) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build: fix dir removal tracebackChris Larson2011-05-061-6/+9
| | | | | | | | | | This one is to cover the case where the current directory vanishes out from under us, so os.getcwd() raises an OSError. (Bitbake rev: 7a29ab534388c0095f7f826b16c5cff343927d10) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build: in exec_func, mkdirhier ${T}Chris Larson2011-05-061-1/+2
| | | | | | | | | This should fix the -c clean traceback people are seeing. (Bitbake rev: 416d24912fcef1d82ce2c02855accd86a29e76b2) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Initial work on getting bitbake working under pypyChris Larson2011-05-061-1/+2
| | | | | | | | | | | - use os.chmod, not os.fchmod, as the latter is missing under pypy - rearrange our imports a bit - don't die if sqlite3 is missing shared cache support (Bitbake rev: f229824dc9c453adf6067500e2bf6761536e4f2f) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "bitbake/gcc: Enable a shared common source tree"Richard Purdie2011-03-291-2/+2
| | | | | This reverts commit 12b163dbd81cafafec1ebe3c4039c65af60ee261 as it was an unintended change for master
* bitbake/gcc: Enable a shared common source treeRichard Purdie2011-03-281-2/+2
| | | | | | | | | | | | | | | | | | | | | This patch is a quick proof of concept to show how source code could be shared between recipes which use ${B} to have a separate build directory compared to source directory ${S}. Issues: a) gcc uses sed and creates config files against ${S} which means the directory should not be shared. Need to change the way that works. b) Could be extended to cover eglibc except there is a patch applied against nativesdk versions which again makes the source incompatible. c) Need to clean up the layout in work-shared and make a directory level deeper to ensure patch separation. d) clean task does not remove stamps Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build: add missing newlineChris Larson2011-03-031-1/+1
| | | | | | | (Bitbake rev: a7aa0415bdaa458a941004bf8dd8bbfeddd6ef5a) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build: switch to old cwd handlingChris Larson2011-03-031-3/+3
| | | | | | | | | We want this to ensure the user can run the run. script from anywhere. (Bitbake rev: a600b79ecefc95eeb266c3f362c7160fa8c948c1) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build.py: Fix TaskFailed parametersRichard Purdie2011-02-231-2/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/build: Drop stdout specific handling for python messages as ↵Richard Purdie2011-02-071-7/+0
| | | | | | this is no longer needed with newer log handling Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build.py: When looking up extra stamp info for setscene tasks, use ↵Richard Purdie2011-01-271-2/+6
| | | | | | | | the real taskname (Fixed based on a bug reported by Dongxiao Xu <dongxiao.xu@intel.com>) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Introduce stamp-extra-info task flag into stamp filenamesDongxiao Xu2011-01-181-3/+9
| | | | | | | | | | | For certain tasks, we need additional information in build stamp file other than the task name and file name. stamp-extra-info is introduced as a task flag which is appended to the stamp file name. [Code simplifcations/tweaks from Richard] Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build.py: Allow FuncFailed error to have an optional messageRichard Purdie2011-01-171-2/+5
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build.py: Rename message field to avoid DeprecationWarning: ↵Richard Purdie2011-01-171-3/+3
| | | | | | BaseException.message warning Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build: remove duplicate import of utilsBernhard Reutner-Fischer2011-01-121-1/+0
| | | | | | | (Bitbake rev: 86c6434f093ce5e55d1febfb424a8e595ffc2d17) Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* *: use utils.remove() some moreBernhard Reutner-Fischer2011-01-121-4/+2
| | | | | | | (Bitbake rev: d3489b141cac1197324661680fe38b8a88bc49b4) Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build.py: Fix del_stamp work correctly after recent stamp function ↵Richard Purdie2011-01-071-1/+1
| | | | | | changes Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake build/siggen/runqueue: Fix stampfile parametersRichard Purdie2011-01-061-2/+3
| | | | | | | | The current parameters are not useful to the stampfile generator function as they can't uniquely define a task. This updated things so the parameters can identify unique tasks. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake build.py: Stamp handling improvementsRichard Purdie2011-01-061-5/+11
| | | | | | | * Move stamp file deletion out of the internal stamp helper function * Add a new function to return the path to a stamp for a given task Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake build.py: Use localdata for stamp handling, not dRichard Purdie2011-01-061-2/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/build.py: Ditch the extract_stamp function as its just going to ↵Richard Purdie2011-01-061-10/+9
| | | | | | complicate improving the stamp handling functions Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build: use logger.isEnabledFor, not getEffectiveLevelChris Larson2011-01-061-2/+2
| | | | | | | | | | I think this is less confusing, and avoids needing to know about the *range* of logging levels, instead simply asking what we really want to know. (Bitbake rev: dc2264387617586b5c0a61e126c75edde5e7abcd) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: Logging of None was losing the logs so force this to ↵Richard Purdie2011-01-051-1/+1
| | | | | | stdout so we can see function execution failures Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Misc syncups with bitbake upstream, mostly whitespace and a missed ↵Richard Purdie2011-01-051-0/+1
| | | | | | line from the last UI commit Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: build.py logging updatesRichard Purdie2011-01-051-16/+23
| | | | | | | | | | | | python tasks calling shell functions using exec_func() would show the log file as /dev/null. It makes most sense for all the task logging to be setup centrally by exec_task(), at least with the current code base in Poky. This commit will need discussion in relation to upstream bitbake and the IO redirection could be better handled using a context manager (although task contexts shouldn't ever nest). Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Use os.devnull, not /dev/nullChris Larson2011-01-041-2/+1
| | | | | | | (Bitbake rev: ce5ab0fc524a1c2c48c4c39d6fc8aae23019207b) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-11 19:58:50 +0000