linux/poky.git/bitbake/lib/bb/codeparser.py, branch dizzy-12.0.1

bitbake: codeparser cache improvements

2014-07-26T07:50:14+00:00

It turns out the codeparser cache is the bottleneck I've been observing when running bitbake commands, particularly as it grows. There are some things we can do about this: * We were processing the cache with "intern()" at save time. Its actually much more memory efficient to do this at creation time. * Use hashable objects such as frozenset rather than set so that we can compare objects * De-duplicate the cache objects, link duplicates to the same object saving memory and disk usage and improving speed * Using custom setstate/getstate to avoid the overhead of object attribute names in the cache file To make this work, a global cache was needed for the list of set objects as this was the only way I could find to get the data in at setstate object creation time :(. Parsing shows a modest improvement with these changes, cache load time is significantly better, cache save time is reduced since there is now no need to reprocess the data and cache is much smaller. We can drop the compress_keys() code and internSet code from the shared cache core since its no longer used and replaced by codeparser specific pieces. (Bitbake rev: 4aaf56bfbad4aa626be8a2f7a5f70834c3311dd3) Signed-off-by: Richard Purdie

bitbake: codeparser: Fix to better catch all getVar references

2014-05-03T10:24:48+00:00

Currently if you do localdata.getVar, the code parser simply ignores the references. Change the code to use endswith() to catch more of the references. These names are probably unique enough to get away with this. Bump the cache version to ensure things get updated. (Bitbake rev: cf763cddc3faa2361b4c4dbd08419e4ebabf208f) Signed-off-by: Richard Purdie

bitbake: bb.utils, bb.codeparser: Add bb.utils.contains_any

2014-04-29T22:38:54+00:00

This includes contains_any in the special handling code for sstate. It does not take into account the equivalence of the values. In current code, considering 'bb.utils.contains_any("A", "foo bar", ...)': A = "foo" A = "bar" A = "foo bar" All those will get different signatures. (Bitbake rev: d1e3345d715e488ec3f5515fb0e1fb39366346bc) Signed-off-by: Otavio Salvador Signed-off-by: Richard Purdie

bitbake: codeparser: don't interact with the cache for subshells

2014-04-29T16:34:24+00:00

Doing so was causing leakage between the execs of the main value and that of the subshell value, and was causing the cached subshell value to be used for the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of a subshell for that of the main value. Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change. The codeparser cache version is bumped, to ensure we don't use potentially incorrect cached values from previous runs. This should hopefully resolve the difficult-to-reproduce issues we've seen at Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has also been mentioned on the oe devel list, where someone hit a case where oe_runmake was missing from a do_install task (IIRC). Adding debug information showed that bitbake's information about the variable dependencies for this task is inaccurate in the failure cases. (Bitbake rev: 97537e4786a1e3a329249497498b59b8f5174fc3) Signed-off-by: Christopher Larson Signed-off-by: Richard Purdie

bitbake: codeparser: Fix var_execs to append to execs, not references

2014-03-02T17:25:29+00:00

When using the "execs" information in new code, it became clear that the returned data was incorrect and there were missing exec'd functions. This corrects the error and changes one of the test results to match the correct behaviour. (Bitbake rev: 8a24f2d3b735bbc59ca4a09670cabbadb1868c1a) Signed-off-by: Richard Purdie

bitbake: codeparser/data_smart: Optimise parsing speed

2013-12-03T17:45:53+00:00

The previous "contains" changes caused a ~3% parsing speed impact. Looking at the cause of those changes was interesting: * Use of defaultdict was slower than just checking for missing entries and setting them when needed. * Even the "import collections" adversely affects parsing speed * There was a missing intern function for the contains cache data * Setting up a log object for each variable has noticeable overhead due to the changes in the code paths uses, we can avoid this. * We can call getVarFlag on "_content" directly within VariableParse for a noticeable speed gain since its a seriously hot code path. This patch therefore tweaks the code based on the above observations to get some of the speed back. (Bitbake rev: fca802187a2a30686a8a07d2b6b16a3e5716e293) Signed-off-by: Richard Purdie

bitbake: data/codeparser: Improve handling of contains functions

2013-11-26T23:01:33+00:00

One of the current frustrations with the sstate checksums is that code like base_contains('X', 'y',...) adds a full dependency on X and varies depend even on whitespace changes in X. This patch adds special handling of the contains functions to expand the first parameter and check for the flag specified by the second parameter (assuming its a string). The result is then appended to the value of the variable with a "Set" or "Unset" status. If the flag is added/removed, the stored variable value changes and hence the checksum changes. No dependency on X is added so it is free to change with regard to other flags or whitespace. This code is far from ideal, ideally we'd have properly typed variables however it fixes a major annoyance of the current checksums and is of enough value its worth adding in a stopgap solution. It shouldn't significantly restrict any propely typed variable implementation in future. (Bitbake rev: ed2d0a22a80299de0cfd377999950cf4b26c512e) Signed-off-by: Richard Purdie Signed-off-by: Christopher Larson Signed-off-by: Richard Purdie

bitbake: codeparser: Drop unneeded variable separation

2013-11-26T23:01:32+00:00

There is no good reason to separately track var_references and references so merge them and remove the unneeded variable. (Bitbake rev: 64d4cbd6360c96574cece70205ea3aecc3f8bae6) Signed-off-by: Richard Purdie

bitbake: codeparser: Track bb.utils.contains usage

2013-02-15T12:12:35+00:00

The bb.utils.contains function usage is getting increasingly used in the metadata but isn't handled automatically by the python dependency tracking code. This patch changes that and also adds the "OE" names for the functions. Whilst there are reasons this is a bad idea, its likely outweighed by the shear number of these references and the current holes in dependency information which we're now relying heavily upon. (Bitbake rev: 0b9d117631ce909312d53b93289e61defc6be01c) Signed-off-by: Richard Purdie

bitbake: codeparser: Track appendVar and prependVar calls as we do for getVar

2013-02-15T12:12:35+00:00

We need to track appendVar and prependVar calls just as we do for getVar in order to ensure we're not missing variable dependencies. (Bitbake rev: 767b4751232f4ee3979deb4d3f733fcf9ee2bd44) Signed-off-by: Richard Purdie