linux/poky.git/bitbake/lib/bb/data_smart.py, branch styhead-5.1.4

bitbake: data_smart: Improve performance for VariableHistory

2024-08-03T10:37:47+00:00

Fixed: - BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS $ bitbake -p -P Check profile.log.processed, the record() cost more than 20 seconds, it is less than 1 second when multiconfig is not enabled, and there would be the following error when more muticonfigs are enabled: Timeout while waiting for a reply from the bitbake server Don't change the type of loginfo['detail'] or re-assign it can make record() back to less than 1 second, this won't affect COW since loginfo is a mutable type. The time mainly affected by two factors: 1) The number of enabled layers, nearly 1 second added per layer when the number is larger than 50. 2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer when the layers number is larger than 50. (Bitbake rev: 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

bitbake: data_smart: directly check for methodpool functions in context lookup

2023-08-04T10:37:56+00:00

We previously checked for the existence of the 'func' flag to determine if we should avoid looking up in the metadata. This was done to ensure the user gets the function for 'def' python functions rather than their string contents. We can sidestep the metadata lookup and check our function context directly, instead. (Bitbake rev: 6cac1eac51efa9a54e8457f60ea1ea0e604c50b7) Signed-off-by: Christopher Larson Signed-off-by: Richard Purdie

bitbake: data_smart: check for python builtins directly for context lookup

2023-08-04T10:37:55+00:00

This avoids the need to hardcode a list of python builtins. This also slightly changes behavior, in a case like `${@eval("3")}`, this will ensure we always call the builtin, even if the metadata has an 'eval' variable defined. (Bitbake rev: 9976ae50677b333d646ca0fd395468bd2301d03f) Signed-off-by: Christopher Larson Signed-off-by: Richard Purdie

bitbake: data_smart: Small optimisation to _findVar()

2023-01-24T21:59:44+00:00

Some users of _findVar don't need the override data and even getVarFlag doesn't need it in some common cases (parsing=True). Rearrange the code as the current overridedata approach doesn't need to be in the _findVar code anyway. This removes some search overhead from a critical path. (Bitbake rev: fcb64e1138a20eb19560af3fc5d1fa748cc9cf34) Signed-off-by: Richard Purdie

bitbake: data_smart: Small cache reuse optimization

2022-11-29T10:25:51+00:00

Currently the expand cache doesn't work for "parser" return types, which is the main type used by the build_dependencies() call that we spend most of the time in when parsing. Tweak the code to cache the unexpanded value in the expand cache and hence allow reuse of the parser in other fast path cases for small speed gains. (Bitbake rev: b4a8e5071dbcba2217b79e83e08b275ffcbc0eef) Signed-off-by: Richard Purdie

bitbake: data_smart: Use regex consistently for override matching

2022-11-29T10:25:51+00:00

One section of the code is enforcing lowercase overrides, the other is allowing numeric characters. We should be consistent with one or the other. (Bitbake rev: df5b3b841fd8d6a652d643e9ae2bba09d60043e0) Signed-off-by: 张忠山 Signed-off-by: Richard Purdie

bitbake: data_smart: Add debugging for overrides stability issue

2022-11-27T23:48:15+00:00

If someone is unfortunate enough to run into override recursion issues they're hard to debug with the existing message. We can at least show the values that OVERRIDES takes to show there is some problem and aid debugging. (Bitbake rev: 43035b75201616e7bfd680d3d15c5c0fc7c04eb6) Signed-off-by: Richard Purdie

bitbake: data/data_smart/build: Clean up datastore finalize/update_data references

2022-11-14T16:22:00+00:00

We dropped the update_data calls a while ago. Clean up the code to match the reality and drop the remaining no-op pieces. Update the comments to reflect the slowest operations and let the cookie monster's spirit live on! (Bitbake rev: 584989ed2b5af4e8799571dece0cf94f995ef14e) Signed-off-by: Richard Purdie

bitbake: data_smart: allow python snippets to include a dictionary

2022-11-10T14:43:06+00:00

[YOCTO #14917] Attempting to use a dictionary in a python code snippet for variable assignment results in an error. For example attempting something such as IDX = "green" VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}" produces the error expansion of VAL threw ExpansionError: Failure expanding variable VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]} which triggered exception SyntaxError: '{' was never closed (Var , line 1) The existing __expand_python_regexp__, "\${@.+?}", will match the first close curly bracket encountered, resulting in incomplete and un-parsable code, and thus produce the error. We can correct this by allowing a single depth of nested curly brackets in __expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will match up to and including the matching close curly bracket to the open, '${@', curly bracket, even if there are one or more singly nested curly brackets present. This change allows the usecase described above to function. This change can't be made on its own though. The old regex would, in an obscure way, handle the case where a python snippet contained an unexpandable variable. Since the unexpandable variable is in curly brackets it would cause incomplete/un-parsable python code and thus remain unparsed. So something like VAL = "${@d.getVar('foo') + ${unsetvar}}" would remain unparsed as the close curly bracket in "${unsetvar}" would match and terminate the snippet prematurely. This quirk resulted in the proper handling of python snippets with unexpanded variables. With the change to __expand_python_regexp__ the full snippet will match and be parsed, but to match the old/correct behavior we would not want to parse it until ${unsetvar} can be expanded. To ensure the old/correct behavior for python snippets with unexpanded variables remains in place we add a check for unexpanded variables in the python snippets before running them. This handling of unparsed variables brings two benefits. The first we now have an explicit check visible to all for unexpanded variables instead of a somewhat hidden behavior. The second is that if there are multiple python snippets the old behavior would run the code for each but a single snippet with unexpanded variables would mean all snippets would remain unparsed, meaning more and repeated processing at a later time. For example: "${@2*2},${@d.getVar('foo') ${unsetvar}}" old behavior would give: "${@2*2},${@d.getVar('foo') ${unsetvar}}" new behavior will give: "4,${@d.getVar('foo') ${unsetvar}}" The old behavior would calculate '2*2' but toss the result when the second snippet would fail to parse resulting in future recalculations (or fetching from cache), while the new behavior avoids this. (Bitbake rev: 94e49b9b9e409c29eb04603b1305d96ebe661a4b) Signed-off-by: Mark Asselstine Signed-off-by: Richard Purdie

bitbake: data_smart: Avoid multiple getVarFlag calls

2022-03-18T13:17:10+00:00

We can call getVarFlags() instead of the multiple getVarFlag calls which is a little more efficient. This reduces the number of overall function calls so is good but probably isn't much faster (or slower). (Bitbake rev: 505a4dd34e822bdf902d9b348dbcdf5b2c94e784) Signed-off-by: Richard Purdie