summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/codeparser.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: bb/cache: drop some unused argumentsMarkus Lehtonen2016-02-181-4/+4
| | | | | | | | | | Drop unused 'd' argument from the cache save methods, simplifying the API. (Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Improve handling of data.expand() dependenciesRichard Purdie2016-02-041-0/+12
| | | | | | | | | | | | | | | | | Currently bitbake doesn't parse into data.expand() expressions, relying on high level expansion of python code to handle this. One of the tests does however test this works. We don't really want to be doing string expansion on python code, so specifically parse into expand() function calls so that when the high level behaviour is tweaked, the self tests continue to pass and that we do continue to handle expand() function calls as best we can. (Bitbake rev: b12c17be5e4a74c9680876605c87f46501f78d28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Add support for correct linenumbersRichard Purdie2016-01-061-2/+8
| | | | | | | | | | | | | | | Currently, if there is something like a python indentation error in a python function, the linenumbers and even file aren't reported correctly. This allows lineno and filename parameters to be passed in to correct this. The lack of a lineno parameter to python's compile() function is worked around by using empty linefeeds. Ugly, but effective and with minimal performance overhead. (Bitbake rev: 5796ed550d127853808f38257f8dcc8c1cf59342) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Only load the codeparser cache onceRichard Purdie2015-11-161-0/+4
| | | | | | | | | | The server state gets reset multiple times during startup and currently we reload the codeparser cache each time. This is pointless and causes unnecessary interaction time with bitbake. (Bitbake rev: 4f700316933e8e7b2d27366e5ce6176895b913e7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Add repr() methodsRichard Purdie2015-06-271-0/+5
| | | | | | | | | These aid when debugging parts of the codeparser cache since the object contents becomes identifiable. (Bitbake rev: 344b098c7eafc2bcc5c6b44ea47985bc0cb446b5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Allow empty functionsRichard Purdie2015-06-271-0/+3
| | | | | | | | | | | | | | | | | The main parser and other code copes with empty python functions but the python codeparser does not. Fix this to avoid errors with code like: python do_build () { } which currently results in the obtuse: "Failure expanding variable do_build: IndexError: string index out of range" [YOCTO #7829] (Bitbake rev: e4f594c670189e04d58ce7d160fc1d86123620af) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.codeparser.py: Remove reference for oe.utils.containsOtavio Salvador2014-11-201-1/+1
| | | | | | | | | | | | The oe.utils.contains function has been removed from OE-Core metadata as the references for it has been replaced to use the bb.utils.contains. (Bitbake rev: 5cfdebe7a67dccc7552ff80c1ccc970e36d562df) Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser cache improvementsRichard Purdie2014-07-261-34/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out the codeparser cache is the bottleneck I've been observing when running bitbake commands, particularly as it grows. There are some things we can do about this: * We were processing the cache with "intern()" at save time. Its actually much more memory efficient to do this at creation time. * Use hashable objects such as frozenset rather than set so that we can compare objects * De-duplicate the cache objects, link duplicates to the same object saving memory and disk usage and improving speed * Using custom setstate/getstate to avoid the overhead of object attribute names in the cache file To make this work, a global cache was needed for the list of set objects as this was the only way I could find to get the data in at setstate object creation time :(. Parsing shows a modest improvement with these changes, cache load time is significantly better, cache save time is reduced since there is now no need to reprocess the data and cache is much smaller. We can drop the compress_keys() code and internSet code from the shared cache core since its no longer used and replaced by codeparser specific pieces. (Bitbake rev: 4aaf56bfbad4aa626be8a2f7a5f70834c3311dd3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Fix to better catch all getVar referencesRichard Purdie2014-05-031-3/+3
| | | | | | | | | | | | Currently if you do localdata.getVar, the code parser simply ignores the references. Change the code to use endswith() to catch more of the references. These names are probably unique enough to get away with this. Bump the cache version to ensure things get updated. (Bitbake rev: cf763cddc3faa2361b4c4dbd08419e4ebabf208f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils, bb.codeparser: Add bb.utils.contains_anyOtavio Salvador2014-04-291-1/+1
| | | | | | | | | | | | | | | | | This includes contains_any in the special handling code for sstate. It does not take into account the equivalence of the values. In current code, considering 'bb.utils.contains_any("A", "foo bar", ...)': A = "foo" A = "bar" A = "foo bar" All those will get different signatures. (Bitbake rev: d1e3345d715e488ec3f5515fb0e1fb39366346bc) Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: don't interact with the cache for subshellsChristopher Larson2014-04-291-9/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Doing so was causing leakage between the execs of the main value and that of the subshell value, and was causing the cached subshell value to be used for the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of a subshell for that of the main value. Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change. The codeparser cache version is bumped, to ensure we don't use potentially incorrect cached values from previous runs. This should hopefully resolve the difficult-to-reproduce issues we've seen at Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has also been mentioned on the oe devel list, where someone hit a case where oe_runmake was missing from a do_install task (IIRC). Adding debug information showed that bitbake's information about the variable dependencies for this task is inaccurate in the failure cases. (Bitbake rev: 97537e4786a1e3a329249497498b59b8f5174fc3) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Fix var_execs to append to execs, not referencesRichard Purdie2014-03-021-1/+1
| | | | | | | | | | | When using the "execs" information in new code, it became clear that the returned data was incorrect and there were missing exec'd functions. This corrects the error and changes one of the test results to match the correct behaviour. (Bitbake rev: 8a24f2d3b735bbc59ca4a09670cabbadb1868c1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser/data_smart: Optimise parsing speedRichard Purdie2013-12-031-3/+6
| | | | | | | | | | | | | | | | | | | | | The previous "contains" changes caused a ~3% parsing speed impact. Looking at the cause of those changes was interesting: * Use of defaultdict was slower than just checking for missing entries and setting them when needed. * Even the "import collections" adversely affects parsing speed * There was a missing intern function for the contains cache data * Setting up a log object for each variable has noticeable overhead due to the changes in the code paths uses, we can avoid this. * We can call getVarFlag on "_content" directly within VariableParse for a noticeable speed gain since its a seriously hot code path. This patch therefore tweaks the code based on the above observations to get some of the speed back. (Bitbake rev: fca802187a2a30686a8a07d2b6b16a3e5716e293) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/codeparser: Improve handling of contains functionsRichard Purdie2013-11-261-3/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | One of the current frustrations with the sstate checksums is that code like base_contains('X', 'y',...) adds a full dependency on X and varies depend even on whitespace changes in X. This patch adds special handling of the contains functions to expand the first parameter and check for the flag specified by the second parameter (assuming its a string). The result is then appended to the value of the variable with a "Set" or "Unset" status. If the flag is added/removed, the stored variable value changes and hence the checksum changes. No dependency on X is added so it is free to change with regard to other flags or whitespace. This code is far from ideal, ideally we'd have properly typed variables however it fixes a major annoyance of the current checksums and is of enough value its worth adding in a stopgap solution. It shouldn't significantly restrict any propely typed variable implementation in future. (Bitbake rev: ed2d0a22a80299de0cfd377999950cf4b26c512e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Drop unneeded variable separationRichard Purdie2013-11-261-3/+1
| | | | | | | | | There is no good reason to separately track var_references and references so merge them and remove the unneeded variable. (Bitbake rev: 64d4cbd6360c96574cece70205ea3aecc3f8bae6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Track bb.utils.contains usageRichard Purdie2013-02-151-2/+3
| | | | | | | | | | | | | | The bb.utils.contains function usage is getting increasingly used in the metadata but isn't handled automatically by the python dependency tracking code. This patch changes that and also adds the "OE" names for the functions. Whilst there are reasons this is a bad idea, its likely outweighed by the shear number of these references and the current holes in dependency information which we're now relying heavily upon. (Bitbake rev: 0b9d117631ce909312d53b93289e61defc6be01c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Track appendVar and prependVar calls as we do for getVarRichard Purdie2013-02-151-1/+1
| | | | | | | | | We need to track appendVar and prependVar calls just as we do for getVar in order to ensure we're not missing variable dependencies. (Bitbake rev: 767b4751232f4ee3979deb4d3f733fcf9ee2bd44) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: refactor out codeparser cache into a separate classPaul Eggleton2012-05-231-134/+57
| | | | | | | | | | We want to be able to reuse most this functionality for the file checksum cache. (Bitbake rev: 0fe3cb1438d297f90dd0fc6b26362ecbff75c76d) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: Call intern over the set contents for better cache performanceRichard Purdie2012-03-121-0/+21
| | | | | | | | See the comment in the code in the commit for more information. (Bitbake rev: 2d56dc7b1f0d186e14c4c8a949b280b6b3fc31de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: Compute extra cache components on the fly rather than laterRichard Purdie2012-03-121-21/+20
| | | | | | | | | | | | | | | | | | | | In the initial implementation there was a relcutance on my part to generate incremental cache components on the fly since it would lead to some duplicate code. We are now seeing problems where each thread reading in the saved cache file causes significant overhead and can make the process appear to hang on a many core build, particularly when the cache file is large. This patch changes the code to maintain the delta in a separate dict right from the start. The code duplication isn't too bad and could be mitigated in other ways if it becomes an issue. [YOCTO #2039 partial] (Bitbake rev: cdd5d0dee6ab12326b252b6b505a316a52638cac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update users of getVar/setVar to use the data store functions directlyRichard Purdie2011-11-271-2/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: silence non-literal warnings for vardepsChristopher Larson2011-11-101-5/+23
| | | | | | | | | | If the vardeps flag is not None, we now silence the warnings about non-literal usage for that variable. (Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: drop expand trackingChristopher Larson2011-11-101-11/+0
| | | | | | | | | | | | | | | | | | | | | | There are two usual cases involving bb.data.expand: - Calling it with a string literal -- "bb.data.expand('${FOO}/${BAZ}/bleh', d)". - Calling it on getVar results (legacy) -- "bb.data.expand(bb.data.getVar('FOO', d), d)" Nothing in any of the usual layers uses it in any other way, and I'm having trouble coming up with any real use cases beyond this. The first of the above cases is already tracked, via the expandWithRefs called on the python code string. The second didn't emit a warning anyway, since the getVar was already handled. Given this, I see no reason for us to maintain explicit expansion tracking. Further, we weren't using its results anyway (the var_expands member). (Bitbake rev: 405dfe69e6a608826e599ebf2f83ef8cf5083b96) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: accept a name for better messagesChristopher Larson2011-11-101-8/+10
| | | | | | | | | | | | | | | | | | | | - If a name is passed to the parser, prepend the messages with "while parsing <name>:". This gives a bit more context. - Tweak the warning messages slightly (they had to be altered anyway to inject the variable being parsed). Before: DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \ not a literal After: DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \ ''%s' % var' is not a string literal (Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: simplify how we compare the called node namesChristopher Larson2011-11-101-52/+20
| | | | | | | | | | | | | | With the previous method, using the compare_name methods, we split the requested match name by '.', reversed it, then compared them piecemeal during the node traversal. The new method walks the nodes and hands back the name of what's being called, and then we check that. This also consolidates the two different implementations of traversal of the attribute/name nodes (one in compare_name, one for the execs). (Bitbake rev: 84e535b5165c7e936c5b1486bdf4626ed3649f5f) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: merge the nested python parsing classesChristopher Larson2011-11-101-102/+87
| | | | | | | | | | The split is even less necessary now that we use ast.walk rather than an actual NodeVisitor subclass. (Bitbake rev: d6c44fac184abae8395bfa7078f06675218aa534) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: make var_expands actually hold useful informationChristopher Larson2011-11-041-1/+1
| | | | | | | | | | | Previously, it was calling var_expands.update() rather than add(), with a string argument, resulting in adding each character of that string to the var_expands set, rather than the string itself. (Bitbake rev: 8e4e75383e43d6da2c16ec5286186a0d0569b0f8) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: When loading the cache, ignore ValueErrorRichard Purdie2011-06-151-1/+1
| | | | | | (Bitbake rev: 9bff182a4ba9571679985b45b309990a6eddad14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/codeparser: Improve cache handlingRichard Purdie2011-05-271-10/+70
| | | | | | | | | | | | | | | | | | | The current codeparser cache handling hurts performance badly even over a couple of cores and certainly on many core systems, it can spent huge amounts of time in the codeparser cache save functions. This patch reworks the cache handling so that each parsing thread saves out its own "differences" file compared to any existing core cache and then the main bitbake thread picks these up and merges things back together. This was tested on systems with small and large numbers of cores and was found to perform orders of magnitude better in all cases despite the more complex code. (Bitbake rev: 9f27563d66523f5af1028f173d53ee75e0877d46) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser.py: fix syntax error in exception handlingOtavio Salvador2011-05-241-1/+1
| | | | | | | | | | Commit 036cf3cd11b3a6836b77f5ffa760ceee6b71b1ef missed the needed brackets to handle more then a type of exception. (Bitbake rev: abecbb4c0af83c6b4ee248b0f03b779f84b13390) Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser.py: Ignore incomplete cache filesRichard Purdie2011-05-121-1/+1
| | | | | | (Bitbake rev: 036cf3cd11b3a6836b77f5ffa760ceee6b71b1ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/codeparser: Correctly handle a missing/empty cache fileRichard Purdie2011-05-051-2/+5
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cooker/codeparser: Ensure the code parser cache is saved for each ↵Richard Purdie2011-05-051-0/+14
| | | | | | | | | | | | | | parsing process Before this change, the codeparser cache was only being saved for the main server process. This is suboptimal as it leaves code being re-evaluated at task execution time and increases parse time. We use the multiprocess Finalize() functionality to ensure each process saves out its cache. We need to update the cache save function to be multiprocess friendly with locking. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: use ==, not 'is' to compare stringsChris Larson2011-03-311-2/+2
| | | | | | | (Bitbake rev: 8f5cf3a9975d8e6878e403be0e6edc22cc44f396) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/codeparser: fix raising of ShellSyntaxErrorJoshua Lock2011-02-251-2/+2
| | | | | | | To raise the ShellSyntaxError we need to import it's module and reference it by namespace. Signed-off-by: Joshua Lock <josh@linux.intel.com>
* codeparser: fix spacing in diagnostic messagesBernhard Reutner-Fischer2011-01-121-2/+2
| | | | | | | (Bitbake rev: 8c5555f5ed6d61db57de80d2820c8cec64a27239) Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/codeparser: Fix import to match upstream bitbakeRichard Purdie2011-01-101-1/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Sync a load of whitespace and other non-functionality changes with ↵Richard Purdie2011-01-041-17/+23
| | | | | | bitbake uptream Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Move the pysh package into the bb packageChris Larson2011-01-041-1/+1
| | | | | | | | | | The pysh we're using is modified, and we don't want to risk it conflicting with one from elsewhere. (Bitbake rev: 1cbf8a9403b4b60d59bfd90a51c3e4246ab834d6) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* codeparser.py: Fix storing of hash values as object references can be corruptedRichard Purdie2010-09-281-8/+11
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/codeparser: Deal with functions with trailing whitespaceRichard Purdie2010-09-071-1/+8
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/codeparser: Ensure cached sheel entries return the correct dependenciesRichard Purdie2010-09-031-1/+1
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/codeparser: Implement persistent cacheRichard Purdie2010-08-311-10/+56
| | | | | | | For a given input to this code, the output doesn't change to implement a persistent cache of the data to speed up parsing. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Add codeparser for parsing shell and python functionsRichard Purdie2010-08-311-0/+273
This commit is derived from Chris Larson's checksum work, turned into a standalone piece of code for parsing python and shell functions. The deindent code has been replaced with code to work around indentation for speed. The original NodeVisitor in the ast was replaced with a faster class walk call. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-11 23:11:06 +0000