summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/data_smart.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: data_smart: skip all interal (underscored) flags in getVarFlagsRoss Burton2012-08-221-1/+1
| | | | | | | (Bitbake rev: 9f631e29a2eebb96a8291839dd8b39aa9126a10e) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data-smart: rename the 'content' internal variable to '_content'Ross Burton2012-08-221-8/+8
| | | | | | | (Bitbake rev: 684cf09aed09ec82c8afb99895f92d73cd0519df) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix unanchored regexp causing strange parsing issueRichard Purdie2012-08-031-1/+1
| | | | | | | | | | | | | | | If this regular expression is unanchored, it would accept strings like: do_install_append1 do_install_appendsomelongstring and treat them like they were do_install_append. Clearly this isn't desirable. Only one instance of this type of issue was found in OE-Core and has been fixed so correcting the regexp should be safe to do. (Bitbake rev: 23bd5300b4a99218a15f4f6b0ab4091d63a602a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix multiple override interaction with append and ↵Richard Purdie2012-07-051-1/+6
| | | | | | | | | | | | | | | | | | | | | prepend operators Variables which used multiple overrides and the append/prepend operators were not functioning correctly. This change fixes that. This fixes the testcase: OVERRIDES = "linux:x86" TESTVAR = "original" TESTVAR_append_x86 = " x86" TESTVAR_append_x86_linux = " x86+linux" TESTVAR_append_linux_x86 = " linux+x86" [YOCTO #2672] (Bitbake rev: dc35a2e506e15fb7ddbf74c3b3280e9e83ab33bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/bb/data_smart.py: don't report variable in ExpansionError if not setPaul Eggleton2012-05-301-1/+4
| | | | | | | | | | If the variable name is not specified then don't confuse the error message by starting off with "Failure expanding variable None...". (Bitbake rev: 9cb16f3c73751e7cf6d495586a6193f06eb97b1f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart: Improve the calculation of config hashDongxiao Xu2012-04-171-3/+4
| | | | | | | | | | | For config hash, we put the keys in structure of "set()", which is not order sensitive. Therefore when calculating the md5 value for config hash, we need to identify the order of the keys. (Bitbake rev: 0f1b142a3f6b8125bf023c2e5ec269618869abf7) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/classes: Convert to use appendVar and appendVarFlagsRichard Purdie2012-03-051-1/+1
| | | | | | (From OE-Core rev: 3b57de68e70e77dbc03c0616a83a29a2e99e40b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart.py: Uses BB_HASHCONFIG_WHITELIST to filter unnecessary variablesDongxiao Xu2012-02-271-12/+4
| | | | | | | | | | | | | Adopt the BB_HASHCONFIG_WHITELIST as a exclusion list for variables that are not needed in cache hash calculation. (Bitbake rev: ae8cf138b5eb8f1f28a7143b8d67ad06cbe43061) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> CC: Christopher Larson <kergoth@gmail.com> CC: Martin Jansa <martin.jansa@gmail.com> CC: Andreas Oberritter <obi@opendreambox.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: Use configuration's hash value to validate cacheDongxiao Xu2012-02-231-0/+21
| | | | | | | | | | | | | | | Previously we use the file time stamp to judge if a cache is valid. Here this commit introduce a new method, which calculates the total hash value for a certain configuration's key/value paris, and tag it into cache filename, for example, bb_cache.dat.xxxyyyzzz. This mechanism also ensures the cache's correctness if user dynamically setting variables from some frontend GUI, like HOB. (Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update users of getVar/setVar to use the data store functions directlyRichard Purdie2011-11-271-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart: Add appendVar/prependVar functionsRichard Purdie2011-11-101-0/+16
| | | | | | | | | | | | | | | | | | | This patch adds appendVar and prependVar functions to the data store meaning python code would no longer have to do the getVar, append and the setVar dance that much of the current python code does. It also adds corresponding variants for flags. Currently there is no spacing added by these functions. That could be added as a parameter if desired. If these functions turn out to be hotspots in the code, there are tricks that could potentially be used to increase the speed of these specific operations within the datastore. (Bitbake rev: 4a4046268f84b85559eea2c4b6a6004ad8cccb77) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: silence non-literal warnings for vardepsChristopher Larson2011-11-101-1/+7
| | | | | | | | | | If the vardeps flag is not None, we now silence the warnings about non-literal usage for that variable. (Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: accept a name for better messagesChristopher Larson2011-11-101-1/+1
| | | | | | | | | | | | | | | | | | | | - If a name is passed to the parser, prepend the messages with "while parsing <name>:". This gives a bit more context. - Tweak the warning messages slightly (they had to be altered anyway to inject the variable being parsed). Before: DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \ not a literal After: DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \ ''%s' % var' is not a string literal (Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart.py: make use of expand cache in getVar()Dongxiao Xu2011-08-291-1/+6
| | | | | | | | | | | | | | | Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache. Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s) (Bitbake rev: 6555a77c199f41bf35460138764e03e30c56d29f) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/ast: Fix ??= vs. ?= handlingRichard Purdie2011-07-051-8/+4
| | | | | | | | | | As the code stands, setting a variable with ??= could result in a ?= variable not overriding it. This patch fixes the issue by allowing the ast to make lookups that ignore any ??= set variables. (Bitbake rev: 32fee2e650dfdd3aa9a7572dad1251e0c24ca34b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/data_smart: Don't track overrides in deleted variable namesRichard Purdie2011-06-281-1/+4
| | | | | | | | | | | | | When we delete a variable we no longer expect it to override other variables. To do this we remove it from the list of active overrides at deletion time. It turns out we already had to do this at override expansion time so this cleans up the code to be more consistent as an added bonus. (Bitbake rev: d924ff9ede57c3dea6e1c738ba3633f18d460b14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/data_smart: Don't export deleted/empty entries in the list of keysRichard Purdie2011-06-281-0/+2
| | | | | | | | | | | | | | | If you d.delVar(), you expect the variable to be gone. Even empty variables continue to exist in the datastore and are still user visible unfortunately. The COW siutation means you can't just remove it since it might unmask a variable from an inner copy. This patch therefore stops empty variables from appearing in key lists exposed to the external world making empty variables an internal implementation detail only. (Bitbake rev: 2b5548c591d4cfde9238d2cc0959c42cfc08f09c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/data_smart: Optimise the data store iteratorRichard Purdie2011-06-281-10/+12
| | | | | | | | | | Since we're going to creat the seen set() anyway, we might as well use it directly. If we don't do this, we see thousands of function calls with associated overhead on profiles. (Bitbake rev: 9d43e3279895639ee4899df635f2546c7ee13737) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/data_smart: Change overrides behaviour to remove expanded variables ↵Richard Purdie2011-06-091-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | from the datastore Currently if you do: OVERRIDES = "z" DEPENDS_prepend = "a " DEPENDS = "b" DEPENDS_z = "c" d.update_data() d.getVar("DEPENDS") gives "a c" d.update_data() d.getVar("DEPENDS") then gives "c" This patch changes the behaviour such that at the time bitbake expands the DEPENDS_z override, it removes "DEPENDS_z" from the data store. In the above example this would mean that it wouldn't matter how often you call d.update_data(), you'd always get "a c" back. See the bitbake-devel mailing list for further discussion and analysis of the potential impact of this change. (Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart: Adding overrides of zero length is pointless so lets notRichard Purdie2011-06-011-3/+4
| | | | | | (Bitbake rev: 67434496108efc3aba9cb1e3640bc712658b1408) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* More 'is' fixupsChris Larson2011-03-311-2/+2
| | | | | | | (Bitbake rev: 967cd1aa2c59f15d805862bd9935f507c635c762) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/data_smart: Improve Variable expansion error handlingRichard Purdie2011-02-231-2/+14
| | | | | | | | | | | | | | | | If expanding a variable triggers an exception the caller currently has no way to supress the error message or otherwise handle the siutation. An example of where this is a problem is "bitbake -e" showing tracebacks and errors for variables like SRCPV in OE/Poky. Secondly in a chained expansion fails, log mesages are recorded for every step of the expansion, not just the innermost error which is where the real failure occured. To fix this we introduce a new exception ExpansionError which callers can handle as appropriate. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/data_smart: Improve the way lazyassignment worksRichard Purdie2011-02-231-0/+2
| | | | | | | | | | | | | | | | | | | | Currently, if a variable has been set with ??= and the code looks it up before the data finalisation phase, no value is found. This is causes serious problems for anonymous python functions which manipulate data, or for the fetcher revision handling code where revisions can be set with ??=. There is also a significant performance implication for processing lazy assignment in finalise. Moving the check for a default value into getVarFlag addresses both the timing issue and the performace. This change gives a 7% real time performance improvement to parsing the Poky metadata. The cost of the check at this point is minimal since we have all the data flags available. This should also fix Yocto bug 752. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Sync a load of whitespace and other non-functionality changes with ↵Richard Purdie2011-01-041-7/+8
| | | | | | bitbake uptream Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Move LAYERDIR expansion hack into DataSmartChris Larson2011-01-041-0/+21
| | | | | | | (Bitbake rev: 40778a6e9e82c7ea4673a74fc19574430fa63e8d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* build: use bb.process instead of os.systemChris Larson2011-01-041-2/+2
| | | | | | | | | | | | (Bitbake rev: 53740977521bc81ffa37adfa7bbeb8f2a80ea165) build: write logfiles per task, not per function Based on d14f9bf6 from poky, reworked for master and other cleanup. (Bitbake rev: beadff2eca1eb95f0411115dd72ddb4c3c44c604) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Access metadata vars as locals in python snippetsChris Larson2011-01-041-1/+14
| | | | | | | | | | | | | Example: FOO = "bar" BAR = "${@FOO + '/baz'}" ${BAR} == "bar/baz" (Bitbake rev: 606fa1fd97cbd47a6a7ebdc7a2e6aa93a8f65cf5) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Fix __getitem__ for DataSmartChris Larson2011-01-041-2/+6
| | | | | | | | | | Ensure it raises KeyError for a missing key, this is required to use this as a mapping in various places, e.g. as locals in an eval. (Bitbake rev: 8d661ce0c303e8d69f17c1d095545d5ed086d1d5) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Switch bitbake internals to use logging directly rather than bb.msgChris Larson2011-01-041-5/+5
| | | | | | | | | | | | | | We use a custom Logger subclass for our loggers This logger provides: - 'debug' method which accepts a debug level - 'plain' method which bypasses log formatting - 'verbose' method which is more detail than info, but less than debug (Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Make DataSmart inherit the MutableMapping ABCChris Larson2011-01-041-14/+20
| | | | | | | | | | | Provide __len__, __iter__, and the getitem/setitem/delitem methods, and its mixed in versions of keys(), values(), items(), etc will automatically behave, making the DataSmart act more like a real mapping. (Bitbake rev: 89b5351c656d263b0ce513cee043bc046d20a01e) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: add optional expansion to getVarFlag()Paul Eggleton2010-12-141-3/+6
| | | | | | | | Add a parameter to getVarFlag() to auto-expand the value of the flag. This makes getVarFlag() more consistent with getVar(), and allows expansion of vardeps and vardepsexclude (which has been done in this commit). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* bitbake/data_smart: Refactor _append/_prepend code to remove duplicationRichard Purdie2010-12-071-36/+23
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/data_smart: Fix append/prepend/override ordering issueRichard Purdie2010-12-071-8/+16
| | | | | | | | | | | | | | | | | | | | | | | | | Where a variable name consisted of an append/prepend combined with an override and there was also an append/prepend to the variable, the override could be lost if the override was not in OVERRIDES. For example: FOO = "A" FOO_append = "B" FOO_append_virtclass-native = "C" could result in "AB" even though virtclass-native was in OVERRIDES. With this patch applied, the result is "ABC" as would be expected. The problem was the deletion of the _append/_prepend flag was happening if *any* append/prepend was procesed, the result should really be that it should contain any unprocessed append/prepend. Kevin Tian deserves credit for looking into this and working out the problem here. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Add codeparser for parsing shell and python functionsRichard Purdie2010-08-311-3/+3
| | | | | | | | | | | This commit is derived from Chris Larson's checksum work, turned into a standalone piece of code for parsing python and shell functions. The deindent code has been replaced with code to work around indentation for speed. The original NodeVisitor in the ast was replaced with a faster class walk call. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/data_smart.py: Allow the data expand function to keep track of ↵Richard Purdie2010-08-311-22/+47
| | | | | | references (including those from python code) Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Revert "In expand, drop the unnecessary second regular expression match"Chris Larson2010-07-021-11/+11
| | | | | | This reverts commit 05462fa7908fc22988b3dc9d376798d0a46ccb5a. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* In expand, drop the unnecessary second regular expression matchChris Larson2010-07-021-11/+11
| | | | | | | | (Bitbake rev: 05462fa7908fc22988b3dc9d376798d0a46ccb5a) Signed-off-by: Imran Mehmood <imran_mehmood@mentor.com> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Simplify _findVar a bitChris Larson2010-07-021-9/+6
| | | | | | | (Bitbake rev: b017acd39b811a00305002a8044e7d02e79f41d7) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Fix major bug that slipped in when moving update_dataChris Larson2010-07-021-9/+10
| | | | | | | | | | | | The root cause is, I was testing the use of renameVar() from finalize, in order to get flags copied over when an override is applied, but renameVar removes the original, whereas the old code did not do so. Going back to the old method, will revisit the override/flags later on. (Bitbake rev: 2f7c498abcf675e5b8de197d8056a0581670c2bd) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* For ${@}, run str() on the resultChris Larson2010-07-021-6/+4
| | | | | | | (Bitbake rev: cf1a51eab11f645c95e5fa278221f78c129b9dd1) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Kill unnecessary usages of the types moduleChris Larson2010-07-021-4/+4
| | | | | | | | | | | | | types.IntType -> int types.StringType -> basestring ... Also moves our ImmutableTypes tuple into our own namespace. (Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Apply some 2to3 transforms that don't cause issues in 2.6Chris Larson2010-07-021-8/+8
| | | | | | | (Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Exception handling fixupsChris Larson2010-07-021-2/+2
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Import fixupsChris Larson2010-07-021-3/+2
| | | | | | | (Bitbake rev: 4fa052f426e3205ebace713eaa22deddc0420e8a) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Formatting cleanupsChris Larson2010-07-021-24/+22
| | | | | | | (Bitbake rev: 2caf134b43a44dad30af4fbe33033b3c58deee57) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Move update_data into the DataSmart class as a finalize() methodChris Larson2010-07-021-0/+65
| | | | | | | (Bitbake rev: ff801397785567cb84b3615de86bff764d65decf) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Consolidate the exec/eval bits, switch anonfunc to better_exec, etcChris Larson2010-07-021-7/+1
| | | | | | | | | | | | The methodpool, ${@} expansions, anonymous python functions, event handlers now all run with the same global context, ensuring a consistent environment for them. Added a bb.utils.better_eval function which does an eval() with the same globals as better_exec. (Bitbake rev: 424d7e267b009cc19b8503eadab782736d9597d0) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Pass an empty locals dict in expansion, since list comprehensions apparently ↵Chris Larson2010-07-021-2/+2
| | | | | | | | | leak their '_' helper variable (Bitbake rev: 97da0cebbaf4dd1b46e58bd2e80cab6c007ae7c9) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Two minor changes to the way python snippet expansion happensChris Larson2010-07-021-8/+8
| | | | | | | | | | | | - Use a single dictionary for the context, both global & local, since for some reason it chokes wanting a global "d" rather than a local in the metadata. - First compile the string into a code object before running eval, so we can include the variable name in an evaluation error. (Bitbake rev: 49534d928a37e0804ca84eed186cd22363023b2e) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* data_smart: be explicit about what we make available to eval'd python codeChris Larson2010-03-251-2/+7
| | | | | | | (Bitbake rev: 6a73dda60f50e9b3e5513795d1ec7207d1446de2) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-02 14:27:43 +0000