linux/poky.git/bitbake/lib/bb/data_smart.py, branch yocto-4.2.4

bitbake: data_smart: Small optimisation to _findVar()

2023-01-24T21:59:44+00:00

Some users of _findVar don't need the override data and even getVarFlag doesn't need it in some common cases (parsing=True). Rearrange the code as the current overridedata approach doesn't need to be in the _findVar code anyway. This removes some search overhead from a critical path. (Bitbake rev: fcb64e1138a20eb19560af3fc5d1fa748cc9cf34) Signed-off-by: Richard Purdie

bitbake: data_smart: Small cache reuse optimization

2022-11-29T10:25:51+00:00

Currently the expand cache doesn't work for "parser" return types, which is the main type used by the build_dependencies() call that we spend most of the time in when parsing. Tweak the code to cache the unexpanded value in the expand cache and hence allow reuse of the parser in other fast path cases for small speed gains. (Bitbake rev: b4a8e5071dbcba2217b79e83e08b275ffcbc0eef) Signed-off-by: Richard Purdie

bitbake: data_smart: Use regex consistently for override matching

2022-11-29T10:25:51+00:00

One section of the code is enforcing lowercase overrides, the other is allowing numeric characters. We should be consistent with one or the other. (Bitbake rev: df5b3b841fd8d6a652d643e9ae2bba09d60043e0) Signed-off-by: 张忠山 Signed-off-by: Richard Purdie

bitbake: data_smart: Add debugging for overrides stability issue

2022-11-27T23:48:15+00:00

If someone is unfortunate enough to run into override recursion issues they're hard to debug with the existing message. We can at least show the values that OVERRIDES takes to show there is some problem and aid debugging. (Bitbake rev: 43035b75201616e7bfd680d3d15c5c0fc7c04eb6) Signed-off-by: Richard Purdie

bitbake: data/data_smart/build: Clean up datastore finalize/update_data references

2022-11-14T16:22:00+00:00

We dropped the update_data calls a while ago. Clean up the code to match the reality and drop the remaining no-op pieces. Update the comments to reflect the slowest operations and let the cookie monster's spirit live on! (Bitbake rev: 584989ed2b5af4e8799571dece0cf94f995ef14e) Signed-off-by: Richard Purdie

bitbake: data_smart: allow python snippets to include a dictionary

2022-11-10T14:43:06+00:00

[YOCTO #14917] Attempting to use a dictionary in a python code snippet for variable assignment results in an error. For example attempting something such as IDX = "green" VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}" produces the error expansion of VAL threw ExpansionError: Failure expanding variable VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]} which triggered exception SyntaxError: '{' was never closed (Var , line 1) The existing __expand_python_regexp__, "\${@.+?}", will match the first close curly bracket encountered, resulting in incomplete and un-parsable code, and thus produce the error. We can correct this by allowing a single depth of nested curly brackets in __expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will match up to and including the matching close curly bracket to the open, '${@', curly bracket, even if there are one or more singly nested curly brackets present. This change allows the usecase described above to function. This change can't be made on its own though. The old regex would, in an obscure way, handle the case where a python snippet contained an unexpandable variable. Since the unexpandable variable is in curly brackets it would cause incomplete/un-parsable python code and thus remain unparsed. So something like VAL = "${@d.getVar('foo') + ${unsetvar}}" would remain unparsed as the close curly bracket in "${unsetvar}" would match and terminate the snippet prematurely. This quirk resulted in the proper handling of python snippets with unexpanded variables. With the change to __expand_python_regexp__ the full snippet will match and be parsed, but to match the old/correct behavior we would not want to parse it until ${unsetvar} can be expanded. To ensure the old/correct behavior for python snippets with unexpanded variables remains in place we add a check for unexpanded variables in the python snippets before running them. This handling of unparsed variables brings two benefits. The first we now have an explicit check visible to all for unexpanded variables instead of a somewhat hidden behavior. The second is that if there are multiple python snippets the old behavior would run the code for each but a single snippet with unexpanded variables would mean all snippets would remain unparsed, meaning more and repeated processing at a later time. For example: "${@2*2},${@d.getVar('foo') ${unsetvar}}" old behavior would give: "${@2*2},${@d.getVar('foo') ${unsetvar}}" new behavior will give: "4,${@d.getVar('foo') ${unsetvar}}" The old behavior would calculate '2*2' but toss the result when the second snippet would fail to parse resulting in future recalculations (or fetching from cache), while the new behavior avoids this. (Bitbake rev: 94e49b9b9e409c29eb04603b1305d96ebe661a4b) Signed-off-by: Mark Asselstine Signed-off-by: Richard Purdie

bitbake: data_smart: Avoid multiple getVarFlag calls

2022-03-18T13:17:10+00:00

We can call getVarFlags() instead of the multiple getVarFlag calls which is a little more efficient. This reduces the number of overall function calls so is good but probably isn't much faster (or slower). (Bitbake rev: 505a4dd34e822bdf902d9b348dbcdf5b2c94e784) Signed-off-by: Richard Purdie

bitbake: data_smart: Skip commonly accessed variables from variable data context lookup

2022-03-18T13:17:10+00:00

The code tries to expand missing entities when they're encountered in python expressions. Looking at traces, these are often things which would not be in the data store such as "bb". Optimise to skip the data store queries for things we know will never be there. The expansion cache usually covers these but skipping entirely cuts a few million function calls parsing OE-Core. (Bitbake rev: 1ae712635a2eef3ecbf541e1f7cc2eeb2f3799c9) Signed-off-by: Richard Purdie

bitbake: data_smart: Avoid exceptions for non string data

2022-02-22T11:38:18+00:00

File "scripts/lib/checklayer/__init__.py", line 49, in _get_layer_collections ldata.expandVarref('LAYERDIR') File "build/bitbake/lib/bb/data_smart.py", line 1007, in expandVarref if referrervalue and ref in referrervalue: TypeError: argument of type 'bool' is not iterable We inject True values as an internal sentinel to the datastore, fix this codepath to handle it. (Bitbake rev: 3b88562d87ac94725c1a683c859c2a6a3287d173) Signed-off-by: Richard Purdie

bitbake: lib/bb: fix exit when found renamed variables

2022-02-21T23:37:26+00:00

Until now, if a renamed variable was found, bitbake exited immediately if it was in a class, but continued after an error message if the variable was in a recipe. This was caused by cookerdata.py CookerDataBuilder::parseBaseConfiguration checking a different DataSmart instance than the variable was set in. To solve the issue, add a special variable and set it when we find a renamed variable. Check for it in ast.py and bail out if needed. (Bitbake rev: d12400600e30549c88dc9e7883dc3d63b1dc1117) Signed-off-by: Marta Rybczynska Signed-off-by: Richard Purdie