summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: bb.cookerdata: include useful traceback for ExpansionError/ParseErrorChristopher Larson2015-09-041-1/+10
| | | | | | | | | | Show the user only the portion of the traceback which was from the metadata, nothing from bitbake's internal calls. (Bitbake rev: c45054aef03393fa0bf70e853ddcfc55988493cf) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.data_smart: retain traceback from ExpansionErrorChristopher Larson2015-09-041-1/+2
| | | | | | | | | | This gives us the needed context of the original ExpansionError, which is invaluable when we have a chain of function calls in the expansion. (Bitbake rev: c514b6fbea77ede1b7871b89592a33ed39b1d71c) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/runqueue: Allow bitbake commands starting with do_Alex Franco2015-09-042-4/+12
| | | | | | | | | | | | | | | The output of "bitbake, -c listtasks pkg" lists tasks with their real names (starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always gets unconditionally prepended to task names. This patch handles this error by checking whether a task starts with "do_" prior to prepending it with it when the task runlist is being constructed (and a few other corner cases). [YOCTO #7818] (Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df) Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/server: Add _uiready flag to handle missing error messagesRichard Purdie2015-09-043-5/+9
| | | | | | | | | | | | | | | | | | | | | If you start and suspend a bitbake execution so the bitbake lock is held, then try and run "bitbake -w '' X", you will see bitbake return an error exit code but print no message about what happened at all. The reason is that the -w option creates a "UI" which swallows the messages. The code which handles this exit failure mode thinks a UI has printed the messages and therefore doesn't do so. This adds in an extra parameter to the UI registration code so that we can figure out whether its a primary UI or not and base decisions on whether to display information on that instead. This fixes the error shown above and some bizarre failures on the Yocto Project Autobuilder. [YOCTO #8239] (Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix a potential error in movefileBenjamin Esquivel2015-09-041-3/+6
| | | | | | | | | | | bitbake utils' movefile is now prone to malform the destination file with duplicated file name strings. Fixing it to force a file name append iff the dest argument is a dir not a file name (Bitbake rev: 38dd27f7191da002a16c561be3790ce487045b01) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Improve how default project is identified and fetchedElliot Smith2015-09-031-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Command line builds are associated with a "default project" (as we currently require a build to have a project). This acts as a container for builds initiated outside Toaster. Currently, this project is marked as the default by its ID being 0. However, this doesn't work with MySQL, as MySQL won't allow 0 in a foreign key which references an autoincrement field. Instead, use an is_default field to track the default Project for builds initiated outside Toaster. Add a method to fetch this default project, rather than fetching a project with a magic ID. Add this default project in a migration, rather than as a side effect of a get_or_create() style method. Also ensure that builds always have a project explicitly assigned to avoid any magic with a build's project foreign key defaulting to 0 (as it no longer does). [YOCTO #7932] (Bitbake rev: 71b709a1bbc26d89d61873763b467d21e625b274) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Specify dest file name in movefile()Benjamin Esquivel2015-09-031-1/+3
| | | | | | | | | | | | When moving a file via the os.rename function, it was missing the destination file name which caused an OSError [YOCTO#8180] (Bitbake rev: b147ba0341d87e077bd2b09ef4355976ecd2d26b) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: delete tasks thoroughlyChristopher Larson2015-09-032-12/+14
| | | | | | | | | | | | | | | We want addtask to be able to bring back a deleted task, but we don't want its previous dependencies to come back with it, so rather than marking a task as deleted and then skipping tasks marked as such, actually delete the task and its dependency information in deltask. While we're in that part of the code, also fix a couple 'not foo in bar' instances to 'foo not in bar', which is preferred in python. (Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/cooker: add interrupted flag to BuildCompleted eventPaul Eggleton2015-08-292-3/+10
| | | | | | | | | | | | | Allow any listeners for this event (such as buildhistory.bbclass in OpenEmbedded) to find out if the build was interrupted rather than completing normally. The value will be 0 if not interrupted, 1 if interrupted waiting for remaining tasks to complete, or 2 if force interrupted (stopping any running tasks immediately). (Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/main: avoid importing all server/UI modules on every executionPaul Eggleton2015-08-241-22/+38
| | | | | | | | | | | | | | | | We're importing the server and UI modules in order to check they are valid, but it turns out that that has some nasty side-effects. We don't actually need to do this except when --help is passed or the module doesn't exist, so rearrange the code so that we only do the module listing in those two cases. Additionally, let's just go ahead and catch all errors on import; we really don't care to have them cause a failure now. (Bitbake rev: c9dc6d9c86e8b887821a6d00346bd0b09e1da97c) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Handle RuntimeError exception in list_extension_modulesRandy Witt2015-08-201-1/+1
| | | | | | | | | | | This exception was triggered in toaster from recent changes and is completely breaking the whole of bitbake. Add the exception to the list so at least only toaster is affected. (Bitbake rev: f64def7cb6069dc1134fcd546bb59e4030c7376f) Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster logger: fix pylint issuesAlexandru DAMIAN2015-08-192-89/+98
| | | | | | | | | | | | | | | | | | | | This patch fixes pylint issues in the toaster build data logger, toasterui. The following types of warnings are touched here: * fixing imports * unused variables are set to _ * logger calls now use lazy evaluation instead of formatting the string * correct whitespace identation * removes unneeded "pass" statements, and extra parantheses * disable specific pylint warnings when decideing to override them (Bitbake rev: 947d47f15048baa967f88e03d80014e88ce152aa) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fix pylint errorsAlexandru Damian2015-08-191-2/+3
| | | | | | | | | | | | | | | | | | | | Prompted by issues discovered during running pylint on the toaster sources, this patch fixes: * missing import in toaster ui * improper call of function in toaster ui (logger.debug) * improper function definitions in bbcontroller * invalid references to objects in bldcontrol.models * proper initialization of object fields in ToasterTables Also inhibiting specific pylint errors in files where the problems are mis-identified. (Bitbake rev: 1c71955c416fb68455f7f70669aba4202c411807) Signed-off-by: Alexandru Damian <alexandru.damian@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-layers: fix mapping files to layersPaul Eggleton2015-08-192-8/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | bitbake-layers needs to map recipe and class files to the layer they came from within the show-recipes and show-overlayed commands. However, it turns out that mapping a file to the layer it came from is not as trivial as it might seem. To do it properly we need to match the path to an entry in BBFILES then map that to the collection name using BBFILE_PATTERN, then map that to the actual layer using variable history. If it doesn't match any entry in BBFILES, then we can fall back to BBFILE_PATTERN (to handle classes and conf files). This fixes the layer name not showing up properly in the output of the show-recipes and show-overlayed commands for recipes in layers such as meta-intel that have subdirectories in BBFILE_PATTERN. It also fixes the priority not showing up in show-layers for such layers. As part of this I've added a function to VariableHistory which for a space-separated list variable gives you a dict mapping the items added to the files in which they were added. I've also fixed bb.utils.get_file_layer() and reduced some of the duplication by using this function in bitbake-layers. Also fixes the priority not showing up for layers such as meta-intel Fixes [YOCTO #8160]. (Bitbake rev: e852f6cabd7489585477ab567a1afeb2252377ac) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/main: consolidate UI/server extension listing and loadingPaul Eggleton2015-08-191-31/+54
| | | | | | | | | | Provide us with a means of showing the list of UIs / server choices for the command line help, and do the processing in one place for both. (Bitbake rev: 24035c1daad5a904c3372d21d44191ee8182338f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/parse: properly handle OSError when updating mtime cachePaul Eggleton2015-08-191-1/+6
| | | | | | | | | | | If a file no longer exists, drop it from the cache silently instead of generating a traceback. This was visible in some cases when a recipe was deleted when bitbake was resident in memory. (Bitbake rev: fe105b9042bdac4afd9f38fcf92bfdc2c04ec23f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: drop appendlistPaul Eggleton2015-08-191-5/+0
| | | | | | | | | | Now we have the bbappends list and all users have been converted over to use it, we don't need this anymore. (Bitbake rev: 279770c42d4c63aa2cebce331b55a92a564b50ac) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: drop appliedappendlistPaul Eggleton2015-08-191-10/+18
| | | | | | | | | | Whilst collecting this list on the fly may be quicker, doing so within a function that's meant to *query* the list of bbappends is poor practice. (Bitbake rev: 5c12aa3b0010d7d1733e54a0ca7d0af465454210) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Fix -m handling if cannot connect to serverPaul Eggleton2015-08-191-2/+0
| | | | | | | | | | | If we can't connect to the server we should error out, because it might not be that the server is actually dead - it might just be unable to execute commands. (Bitbake rev: d4b921676859d6ba4e1922fa4682ee941652f483) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: ensure prefile/postfile can work in memory resident modePaul Eggleton2015-08-192-2/+5
| | | | | | | | | | | | The prefile/postfile options weren't working in memory resident mode because they weren't being passed through to the server, so ensure that they do get passed through and that the server is reset when the values come through. (Bitbake rev: a3f7dc042fc7b1c308bfd248431930eb8ba50326) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: ensure sync commands that read configuration see updatesRichard Purdie2015-08-192-12/+28
| | | | | | | | | | | | Add a means of ensuring that synchronous commands that read the results of the configuration trigger a reparse of the configuration if any underlying files have changed. (Bitbake rev: aaf3cc024315450c1674819edf2a4e5cdf293f35) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: further limit inotify watchesPaul Eggleton2015-08-191-0/+11
| | | | | | | | | | | | | | | | | | | | | | Unfortunately we were acting on inotify notifications about any files changing within the watched directories, not just the ones we actually care about. In OE the build directory is in BBPATH and hence it gets watched, and we write things like bitbake.lock and bitbake-cookerdaemon.log to that directory, hence effectively notifications were being tripped on every bitbake invocation. To avoid this, record which file/subdirectory we're interested in against each watched directory so we can ignore any events for files/subdirectories we don't care about. Additionally, if we move up to the parent dir, ensure we haven't already seen it before adding a watch on it (we were previously calling watcher.add_watch() on the same directory multiple times before in a typical OE configuration). (Bitbake rev: bc39b8da34c046b629c43fd0a8cac2efbf1c060f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: intercept SystemExit to avoid trashing the serverPaul Eggleton2015-08-191-1/+1
| | | | | | | | | | | | If sys.exit() is called within a command run over XMLRPC, the XMLRPC server is effectively trashed (apparently listening but no longer able to respond to commands). We need to intercept the SystemExit exception and deal with it as we would any other exception. (Bitbake rev: 95e391acbc3b4efd6c77637a1ce815012ae0f09b) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Fix default function parameter assignment to a listPaul Eggleton2015-08-199-30/+55
| | | | | | | | | | | | | | With python you should not assign a list as the default value of a function parameter - because a list is mutable, the result will be that the first time a value is passed it will actually modify the default. Reference: http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments (Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: Add provides to taskdepdataMariano Lopez2015-08-101-1/+2
| | | | | | | | | | | | | | | Currently bitbake is the only one that knows the relation between PN and PROVIDES. In some cases it is needed to have this relation in the data store (the bootloader it's a good case). This adds the PROVIDES to the taskdata, so it would be in the data store as one field of BB_TASKDEPDATA. (Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Resolve file monitoring race issues when using memres bitbakeRichard Purdie2015-08-031-3/+4
| | | | | | | | | | | | | | | If you load memory resident bitbake, then change bblayers.conf or other configuration files, then execute your first command, bitbake will not notice the changes. This is because it adds the file watches during inital parsing, which happens at the time the first command is run. To fix this, we move the addition of the file watches to earlier in the process, so bitbake then does track them correctly. This also avoids some issues Paul was seeing with tinfoil2. (Bitbake rev: b9375c73e736003e66575969c9ea244403e47aeb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.parse: properly error out on filesystem errorsChristopher Larson2015-08-012-10/+18
| | | | | | | | | | | | | | | We've had a long-standing bug where a legitimate error reading a file (IOError or OSError) is always suppressed as though it was a 'file not found' case. As a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently not parse local.conf, rather than erroring to let the user know about the problem. Fix this by handling the ENOENT case specifically. (Bitbake rev: e691312a3add222b04e7b2f52f8df6abcb9068bf) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.cookerdata: don't show traceback for ParseError/ExpansionErrorChristopher Larson2015-08-011-2/+5
| | | | | | | | | | | Tracebacks are of extremely limited usefulness in this context. The exceptions carry the necessary context already, and the user doesn't care about the calls in bitbake internals that led to an expansion or parse failure. (Bitbake rev: 9b95fa94eaae452ac7814f1e67c2f7a6314c52f1) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: toaster: Wait for toaster gui to comeEd Bartosh2015-08-011-0/+1
| | | | | | | | | | | | | | | Set logging level for the Toaster logger to get messages to toaster_ui.log Reverted previous workaround. [YOCTO: #7965] (Bitbake rev: f1d8e0ff7233f17cff3a56db10d2443b392b407d) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: fix incorrect regexesAlexander Kanavin2015-08-011-2/+2
| | | | | | | | | | | [\.-_] means "any character between . and _" What was meant here is certainly "any character from the three characters .-_" (Bitbake rev: af13eaba627f199f91c048c435b9dbe19c79527f) Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: properly fix bitbake.lock handlingRichard Purdie2015-08-016-30/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If the PR server or indeed any other child process takes some time to exit (which it sometimes does when saving its database), it can end up holding bitbake.lock after the UI exits, which led to errors if you ran bitbake commands successively - we saw this when running the PR server oe-selftest tests in OE-Core. The recent attempt to fix this wasn't quite right and ended up breaking memory resident bitbake. This time we close the lock file when cooker shuts down (inside the UI process) instead of unlocking it, and this is done in the cooker code rather than the actual UI code so it doesn't matter which UI is in use. Additionally we report that we're waiting for the lock to be released, using lsof or fuser if available to list the processes with the lock open. The 'magic' in the locking is due to all spawned subprocesses of bitbake holding an open file descriptor to the bitbake.lock. It is automatically unlocked when all those fds close the file (as all the processes terminate). We close the UI copy of the lock explicitly, then close the server process copy, any remaining open copy is therefore some proess exiting. (The reproducer for the problem is to set PRSERV_HOST = "localhost:0" and add a call to time.sleep(20) after self.server_close() in lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ). Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>. This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and e97a9f1528d77503b5c93e48e3de9933fbb9f3cd. (Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Exit scripts with result of last shell functionAndre McCurdy2015-07-271-1/+1
| | | | | | | | | | | | | Since shell scripts run with 'set -e' the final exit at the end of the script can only be returning 0. However, for correctness and to follow the original intention of the 'cleanup' commands, let's fix the typo and return the success of the last shell function rather than the success of unhooking the exit trap handler. (Bitbake rev: bef724057f1ea81571dd3ac5a9cf862f818434b5) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Improve override history loggingRichard Purdie2015-07-242-11/+14
| | | | | | | | | | | | | | | | | | Calling record() for each override alteration is slow. Since we now expand overrides dynamically we don't have to record the log data at each alteration, we can instead print it directly from the existing data stores at variable history print time using the exact same data stores. This massively improves performance of the data store when parsing with bitbake -e for example, it will improve memory overhead as well. The only downside is that VariableHistory has to poke into the datastore for some of its data but that seems an acceptable tradeoff rather than double caching. (Bitbake rev: 100b447a161ef20fa559e39516cd32fa78e38262) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Improve performance of infer_caller_details()Richard Purdie2015-07-241-8/+17
| | | | | | | | | | | | | | | | | | | | | | As things stand now, bitbake -e (which turns on all the caller tracking) of OE-Core generates around 9.5 million stat calls which is slow and the largest single thing on the profile data. This is because infer_caller_details() calls traceback.extract_stack() which adds line contents to the traceback. This in turn calls python's internal linecache code which calls stat on every file for every callback. We don't even use that info. We only even want a single frame of the stack. Instead, open code for the pieces of information we need. Also, only obtain the stack once for both halves of the infer_caller_details() code. This reduces the number of stat calls to around 0.5 million and significantly improves parsing with bitbake -e. (Bitbake rev: 7be76d8f79ea92fd4bd36146eb9a4b86551b526d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Clean up datastore accesses and True/False valuesRichard Purdie2015-07-241-20/+20
| | | | | | | | | | | | We should use the d.xxxVar syntax rather than the older function style. Also replace 0/1 with the more pythonic True/False. No functionality changes. (Bitbake rev: 90fdd69cca951f8bd2ff634f3b42fccd4fc03095) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Add newline after the last export in emit_funcAndre McCurdy2015-07-241-1/+2
| | | | | | | | | | | | | Minor formatting improvement in generated shell scripts. Add a newline after the last export emitted by emit_func. Also, remove chance of putting an extra newline in the middle of the exports, since we never want to do that. (Bitbake rev: 5117d9b5d32d5d81adf70fa3e3feac9cef654240) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: Add shutdown methodRichard Purdie2015-07-241-0/+4
| | | | | | | | | | | | One drawback to tinfoil is that once a cooker is created, it will hold the cooker lock and stop any other bitbake execution against a directory. Add a shutdown method to tinfoil, allowing other users to use the build directory after the tinfoil usage is no longer needed. (Bitbake rev: e44ce85fe551677fc0dcc1da4f789a0c13093ff1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Add BBCLASSEXTEND multiple data store corruption ↵Richard Purdie2015-07-231-2/+34
| | | | | | | | | | | | | | | reproducer One data store changing a variable poked through into a different data store. This test case replicates that issue where the value 'B' would become unset/disappear. We also enhance parsehelper to generate files with an optional suffix such as bbclass. (Bitbake rev: 5c4179f58a4e04f1c354df5f17d1860eb403f0ac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add CoW approach for overridedataRichard Purdie2015-07-231-1/+10
| | | | | | | | | | | | | Using deepcopy() caused a major performance regression. Turns out we can work with a shallow copy as long as we force the recreation of any list that we change, effectively a poor mans CoW. This isn't too invasive and avoids the huge overhead of deepcopy so this seems like the best way to have performance and a working data store. (Bitbake rev: 32dff4749c32f32e947c42c86f8357b8b607354b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: Updated test name FetchMethodTest -> ↵Aníbal Limón2015-07-221-1/+1
| | | | | | | | | | | FetchLatestVersionTest Change the test name to be more specific on what is tested. (Bitbake rev: 3e39156bc330c4c726058a5b9c13d33e2daad89f) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: latest_versionstring now returns (version, revision)Aníbal Limón2015-07-222-7/+8
| | | | | | | | | | | | | | | | Now latest_versionstring method returns (version, revision) for comply the new return convention needed by SCM's like git get the current revision. bb/tests/fetch.py: Updated wget latest_versionstring test for comply new convention. [YOCTO #7605] (Bitbake rev: 8d454646cbe1b04758ca178d8c6fcfd02b818b7b) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: latest_versionstring now returns (version, revision)Aníbal Limón2015-07-222-7/+14
| | | | | | | | | | | | | | | | We need to know the revision associated with the upstream version in SCM like git, this change broke return convention, oe-core recipeutils get_recipe_upstream_version need to be updated. tests/fetch.py: Updated git latest_versionstring test for comply new convention. [YOCTO #7605] (Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: latest_versionstring search in all tagsAníbal Limón2015-07-221-1/+1
| | | | | | | | | Don't limit the tag search for only tags end with ^{}. (Bitbake rev: 7006ab313766344cf33481228465082ed5977d28) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Use deepcopy for overridedataRichard Purdie2015-07-211-1/+1
| | | | | | | | | | | | | | | dbus sets RDEPENDS_dbus-ptest_class-target = "X". nativesdk as BBCLASSEXTEND was expanding and setting various RDEPENDS variables which were clearing overrides queued against RDEPENDS. The problem was that this was leaking into other data contexts such as the target case. This was because overridedata was a shallow copy. Replacing the shallow copy with a deep copy avoids this problem of leakage between the data stores at a small performance penalty. (Bitbake rev: 83e2923cd3c8f2e25987132f85b06ce72bc941d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: provide mechanism to bypass UI log suppressionPaul Eggleton2015-07-162-5/+10
| | | | | | | | | | | | | | | | | | The recent change to connect through the shell logging functions had an unexpected side-effect - bb.error() and bb.fatal() cause a flag to be set internally such that BitBake's UI will not print the full task log on failure; unfortunately we have in places within the OpenEmbedded metadata called these shell logging functions under error situations where we still want to see the full log (i.e., the message we're sending doesn't include the full error). Thus, provide a mechanism to fatally exit with an error but unset the flag, utilising the built-in python logging functionality that allows extra values to be passed in the log record. (Bitbake rev: e561b997c55e8537d82aa1339adfff4505cc38b7) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/process: check output of select() before reading extrafilesPaul Eggleton2015-07-161-10/+11
| | | | | | | | | | | | | | | | | We're calling select() to find the fds that have data available, so we really ought to check the return to see if the extra file is in there before trying to read from it. This is part of the fix for the performance regression that this code introduced (5-10 minutes extra in a reasonable size OE build); the rest is down to an issue in the way that pseudo currently handles FIFOs and will need to be addressed there, see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7993 Solution suggested by Richard Purdie <richard.purdie@linuxfoundation.org> (Bitbake rev: 3e2db8d7eaa0f14813213d1c95d3ee9efd97dc9d) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/data: Add new data testsRichard Purdie2015-07-122-0/+93
| | | | | | | | | Add a variety of tests which were found to be useful when working on the data store recently. (Bitbake rev: 5c5f8da509f6bbc1fad263462142519ef3d54a35) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix appendVar/prependVarRichard Purdie2015-07-121-4/+2
| | | | | | | | | | | | | | Now that overrides get expanded 'on the fly', change appendVar and prependVar to work using _append and _prepend, else we'd have to re-implement pieces of getVar and the timing of expansions becomes problematic. Using _append/_prepend equivalence gives the behaviour users likley expect from these functions. (Bitbake rev: 40d661aaf7a563c6447b073310c5f2fdae6ca3d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix data expansion cache issuesRichard Purdie2015-07-121-1/+1
| | | | | | | | | | | The expand cache can be cleared in the middle of getVar now due to the use of operations like delVar. We therefore need to check if variables are in the cache before accessing in case it was cleared. (Bitbake rev: a97ce216cfe44136f742383542954bfce027831e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix keys() missing overridesRichard Purdie2015-07-121-0/+15
| | | | | | | | | | d.keys() was not returning items which had no base content but the variable came into existance through overrides. We have to process self.overridedata to find these other variables. (Bitbake rev: 77fc57c610d7f65f7e328ad555f525b2f8bdf1d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>