summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/data_smart.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: data_smart: directly check for methodpool functions in context lookupChristopher Larson2023-08-041-3/+4
| | | | | | | | | | | | | We previously checked for the existence of the 'func' flag to determine if we should avoid looking up in the metadata. This was done to ensure the user gets the function for 'def' python functions rather than their string contents. We can sidestep the metadata lookup and check our function context directly, instead. (Bitbake rev: 6cac1eac51efa9a54e8457f60ea1ea0e604c50b7) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: check for python builtins directly for context lookupChristopher Larson2023-08-041-4/+8
| | | | | | | | | This avoids the need to hardcode a list of python builtins. This also slightly changes behavior, in a case like `${@eval("3")}`, this will ensure we always call the builtin, even if the metadata has an 'eval' variable defined. (Bitbake rev: 9976ae50677b333d646ca0fd395468bd2301d03f) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Small optimisation to _findVar()Richard Purdie2023-01-241-7/+9
| | | | | | | | | | | | | Some users of _findVar don't need the override data and even getVarFlag doesn't need it in some common cases (parsing=True). Rearrange the code as the current overridedata approach doesn't need to be in the _findVar code anyway. This removes some search overhead from a critical path. (Bitbake rev: fcb64e1138a20eb19560af3fc5d1fa748cc9cf34) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Small cache reuse optimizationRichard Purdie2022-11-291-3/+7
| | | | | | | | | | | | Currently the expand cache doesn't work for "parser" return types, which is the main type used by the build_dependencies() call that we spend most of the time in when parsing. Tweak the code to cache the unexpanded value in the expand cache and hence allow reuse of the parser in other fast path cases for small speed gains. (Bitbake rev: b4a8e5071dbcba2217b79e83e08b275ffcbc0eef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Use regex consistently for override matching张忠山2022-11-291-1/+1
| | | | | | | | | | | One section of the code is enforcing lowercase overrides, the other is allowing numeric characters. We should be consistent with one or the other. (Bitbake rev: df5b3b841fd8d6a652d643e9ae2bba09d60043e0) Signed-off-by: 张忠山 <zzs213@126.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add debugging for overrides stability issueRichard Purdie2022-11-271-1/+3
| | | | | | | | | | If someone is unfortunate enough to run into override recursion issues they're hard to debug with the existing message. We can at least show the values that OVERRIDES takes to show there is some problem and aid debugging. (Bitbake rev: 43035b75201616e7bfd680d3d15c5c0fc7c04eb6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/data_smart/build: Clean up datastore finalize/update_data ↵Richard Purdie2022-11-141-8/+1
| | | | | | | | | | | | | references We dropped the update_data calls a while ago. Clean up the code to match the reality and drop the remaining no-op pieces. Update the comments to reflect the slowest operations and let the cookie monster's spirit live on! (Bitbake rev: 584989ed2b5af4e8799571dece0cf94f995ef14e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: allow python snippets to include a dictionaryMark Asselstine2022-11-101-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #14917] Attempting to use a dictionary in a python code snippet for variable assignment results in an error. For example attempting something such as IDX = "green" VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}" produces the error expansion of VAL threw ExpansionError: Failure expanding variable VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]} which triggered exception SyntaxError: '{' was never closed (Var <VAL>, line 1) The existing __expand_python_regexp__, "\${@.+?}", will match the first close curly bracket encountered, resulting in incomplete and un-parsable code, and thus produce the error. We can correct this by allowing a single depth of nested curly brackets in __expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will match up to and including the matching close curly bracket to the open, '${@', curly bracket, even if there are one or more singly nested curly brackets present. This change allows the usecase described above to function. This change can't be made on its own though. The old regex would, in an obscure way, handle the case where a python snippet contained an unexpandable variable. Since the unexpandable variable is in curly brackets it would cause incomplete/un-parsable python code and thus remain unparsed. So something like VAL = "${@d.getVar('foo') + ${unsetvar}}" would remain unparsed as the close curly bracket in "${unsetvar}" would match and terminate the snippet prematurely. This quirk resulted in the proper handling of python snippets with unexpanded variables. With the change to __expand_python_regexp__ the full snippet will match and be parsed, but to match the old/correct behavior we would not want to parse it until ${unsetvar} can be expanded. To ensure the old/correct behavior for python snippets with unexpanded variables remains in place we add a check for unexpanded variables in the python snippets before running them. This handling of unparsed variables brings two benefits. The first we now have an explicit check visible to all for unexpanded variables instead of a somewhat hidden behavior. The second is that if there are multiple python snippets the old behavior would run the code for each but a single snippet with unexpanded variables would mean all snippets would remain unparsed, meaning more and repeated processing at a later time. For example: "${@2*2},${@d.getVar('foo') ${unsetvar}}" old behavior would give: "${@2*2},${@d.getVar('foo') ${unsetvar}}" new behavior will give: "4,${@d.getVar('foo') ${unsetvar}}" The old behavior would calculate '2*2' but toss the result when the second snippet would fail to parse resulting in future recalculations (or fetching from cache), while the new behavior avoids this. (Bitbake rev: 94e49b9b9e409c29eb04603b1305d96ebe661a4b) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Avoid multiple getVarFlag callsRichard Purdie2022-03-181-3/+4
| | | | | | | | | | We can call getVarFlags() instead of the multiple getVarFlag calls which is a little more efficient. This reduces the number of overall function calls so is good but probably isn't much faster (or slower). (Bitbake rev: 505a4dd34e822bdf902d9b348dbcdf5b2c94e784) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Skip commonly accessed variables from variable data ↵Richard Purdie2022-03-181-0/+3
| | | | | | | | | | | | | | | | context lookup The code tries to expand missing entities when they're encountered in python expressions. Looking at traces, these are often things which would not be in the data store such as "bb". Optimise to skip the data store queries for things we know will never be there. The expansion cache usually covers these but skipping entirely cuts a few million function calls parsing OE-Core. (Bitbake rev: 1ae712635a2eef3ecbf541e1f7cc2eeb2f3799c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Avoid exceptions for non string dataRichard Purdie2022-02-221-1/+1
| | | | | | | | | | | | | | | File "scripts/lib/checklayer/__init__.py", line 49, in _get_layer_collections ldata.expandVarref('LAYERDIR') File "build/bitbake/lib/bb/data_smart.py", line 1007, in expandVarref if referrervalue and ref in referrervalue: TypeError: argument of type 'bool' is not iterable We inject True values as an internal sentinel to the datastore, fix this codepath to handle it. (Bitbake rev: 3b88562d87ac94725c1a683c859c2a6a3287d173) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: fix exit when found renamed variablesMarta Rybczynska2022-02-211-0/+4
| | | | | | | | | | | | | | | | Until now, if a renamed variable was found, bitbake exited immediately if it was in a class, but continued after an error message if the variable was in a recipe. This was caused by cookerdata.py CookerDataBuilder::parseBaseConfiguration checking a different DataSmart instance than the variable was set in. To solve the issue, add a special variable and set it when we find a renamed variable. Check for it in ast.py and bail out if needed. (Bitbake rev: d12400600e30549c88dc9e7883dc3d63b1dc1117) Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add support to BB_RENAMED_VARIABLES for custom stringsRichard Purdie2022-02-211-14/+21
| | | | | | | | | Add support for custom strings in BB_RENAMED_VARIABLES and use this to show that BB_STAMP_WHITELIST and BB_STAMP_POLICY are no longer supported. (Bitbake rev: 0914011f7647571ab125bbddcd7d68e3da47226a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Allow rename mechanism to show full expressionsRichard Purdie2022-02-211-2/+4
| | | | | | (Bitbake rev: bac6f7acfd2e6b5b4d6d3a8d40beeff76b215751) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename allowed multiple provider variableScott Murray2022-02-211-0/+1
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: MULTI_PROVIDER_WHITELIST -> BB_MULTI_PROVIDER_ALLOWED (Bitbake rev: a09546b725fda13c0279638c7c904110da7bf6cd) (Bitbake rev: d035435c1a4951a45481867cf932faa4a6f8f936) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename setscene enforce filtering variableScott Murray2022-02-211-0/+1
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_SETSCENE_ENFORCE_WHITELIST -> BB_SETSCENE_ENFORCE_IGNORE_TASKS (Bitbake rev: 2e243ac06581c4de8c6e697dfba460ca017d067c) (Bitbake rev: f8f7b80a0df4646247e58238a52a7d85a37116d4) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename configuration hash filtering variableScott Murray2022-02-211-2/+3
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_HASHCONFIG_WHITELIST -> BB_HASHCONFIG_IGNORE_VARS (Bitbake rev: f344246be73d626c215f867718e45fd6cddc2aaf) (Bitbake rev: 371deb3fe8510aadf4455810d7c5243d374e6532) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename environment filtering variablesScott Murray2022-02-211-0/+2
| | | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS (Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a) (Bitbake rev: 87104b6a167188921da157c7dba45938849fb22a) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename basehas and taskhash filtering variablesScott Murray2022-02-211-0/+2
| | | | | | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS the derived code variables basewhitelist and taskwhitelist have been renamed to basehash_ignore_vars and taskhash_ignore_tasks, respectively. [RP: Added compatibility mapping code and tweaked naming] (Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart/cookerdata: Add variable remapping supportRichard Purdie2022-02-211-0/+36
| | | | | | | | | | | | | | | | | | | | This change adds support for improving the user experience when variables are renamed. This isn't as simple as it might first appear since some bitbake variables are used through the environment before the datastore exists, some are bitbake variables which we know about straight away and some are metadata defined which we don't know about until later. This patch adds support for handling these different cases, allowing a list of bitbake renamed variables to be defined in bitbake itself and allows this to be extended through the metadata using BB_RENAMED_VARIABLES. In order to give the best feedback to the user, we default to turning on variable history tracking in the base data store from knotty, which allows filename and line number information to be shown. (Bitbake rev: bd50a5d5e4b4fa90844464396887ebdff0d4e5f7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add hasOverrides method to public datastore APIRichard Purdie2022-02-171-0/+2
| | | | | | | | | | | There are some cases where the metadata needs to check if a variable has any overrides set, even if they are currently inactive. That code currently pokes into datastore internals. Add API instead to replace and avoid that. (Bitbake rev: c885e5542dcf760b8fc5881e385abb4a10020874) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix overrides file/line message additionsRichard Purdie2022-02-171-4/+4
| | | | | | | | | | The overrides warning message is meant to show filename and line numbers but the variable names are incorrect and this wasn't working. Fix it. (Bitbake rev: 551c1cb20fc9b9d0dab5d830182c2bf626e72845) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart.py: Skip old override syntax checking for anonymous ↵Robert Yang2021-11-261-1/+1
| | | | | | | | | | | | | | | | | | | | | functions Fixed when oe-core's path contians append/prepend/remove, e.g.: /path/to/append_test/oe-core/ Initial a build in any build dirs: $ bitbake -p ERROR: Variable __anon_32__buildarea2_xhou_builds_append_test_layers_oe_core_meta_classes_patch_bbclass contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake. The anonymous fuctions has no names, so skip checking for it to fix the issue. (Bitbake rev: ebd00330c41c75797529ff38d6a0955b93f05d1b) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Fix string concatination potential performance issuesRichard Purdie2021-11-031-3/+3
| | | | | | | | | | | | | | | | Python scales badly when concatinating strings in loops. Most of these references aren't problematic but at least one (in data.py) is probably a performance issue as the issue is compounded as strings become large. The way to handle this in python is to create lists which don't reconstruct all the objects when appending to them. We may as well fix all the references since it stops them being copy/pasted into something problematic in the future. This patch was based on issues highligthted by a report from AWS Codeguru. (Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Don't add None to ExpansionError varlistRichard Purdie2021-09-211-2/+3
| | | | | | | | | If a "None" value gets into the varlist, it doesn't display properly. Ensure we don't add one to have the exception display properly. (Bitbake rev: ee26e258888114143e66330c256b5bfe7d071c53) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: adjust parser error check for python 3.10 compatibilityAlexander Kanavin2021-09-171-1/+1
| | | | | | | | | | The change was introduced in https://github.com/python/cpython/commit/a698d52c3975c80b45b139b2f08402ec514dce75 (Bitbake rev: 8d3c6cbbe6ee734495713ae3b99c609527842506) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: correct the collections vs collections.abc deprecationAlexander Kanavin2021-09-171-1/+1
| | | | | | | | | This becomes a hard error in python 3.10. (Bitbake rev: ae219e1f7460077f4492b31ac91cef4cf9b17277) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Improve error display for handled exceptionsRichard Purdie2021-09-111-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We don't need tracebacks for BBHandledException. Reduces confusing output like: ERROR: /meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio ERROR: ExpansionError during parsing /meta/recipes-core/images/core-image-tiny-initramfs.bb Traceback (most recent call last): File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_qa', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_cpio', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1038, in follow_chain(task='do_image_complete', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in chain: > bb.fatal("Circular task dependencies as %s depends itself via the chain %s?!" % (task, " -> ".join(chain))) chain.append(task) File "/bitbake/lib/bb/__init__.py", line 165, in fatal: mainlogger.critical(''.join(args), extra=kwargs) > raise BBHandledException() to the real error: ERROR: /media/build1/poky/meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio (Bitbake rev: 551d4c0576a0a0c3406000029df9238b312f2263) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Make ExpansionErrors more readableRichard Purdie2021-09-111-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds context to ExpansionError messages which show the variable chain for which expansion is being attempted. This should allow users to debug the issues more easily than the current message (the first line alone below). Example output from a SRC_URI which references ${S}: bb.data_smart.ExpansionError: Failure expanding variable PV, expression was 0.1+git${SRCPV} which triggered exception RecursionError: maximum recursion depth exceeded while calling a Python object The variable dependency chain for the failure is: PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> BP -> FILESPATH which is more useful that no output. We could truncate at repetition but I suspect this makes this clearer as it stands so there is little value in complicating the code. (Bitbake rev: 699634bec47964fa7ab18689dc23db6f0bc22fb3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix inactive overide accidental variable value corruptionRichard Purdie2021-08-041-4/+4
| | | | | | | | | | | | | | | | | | | | | Setting something like: BAR:append:unusedoverride should cause BAR to be None, not "" which was what the datastore was returning. This caused problems when mixing variables like: RDEPENDS:${PN}:inactiveoverride RDEPENDS:${BPN} since key expansion would report key overlap when there was none. This is a bug in the datastore. Fix it and add a test too. [YOCTO #14088] (Bitbake rev: 699e36c270d863258502d315ed00a1b940bfbf96) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Switch to using new override syntaxRichard Purdie2021-08-021-58/+55
| | | | | | | | | | | | | | | | | This change updates the datastore to use the new override syntax using colons instead of underscores exclusively. It is expected layers would have to be converted to work with bitbake after this change. Supporting mixed syntax isn't possible, it is only feasible to have one internal representation of overrides. Whilst we can't warn for every possible override that may be set in the old format, show errors for _append/_prepend/_remove since those should never be present. (Bitbake rev: 7dcf317cc141dc980634f8c18bfa84f83e57206a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Allow colon in variable expansion regexRichard Purdie2021-07-291-1/+1
| | | | | | | | | Now that ":" is a valid character in variable key names, it needs to be allowed by the variable expansion code too, to match. (Bitbake rev: 019251649a38754d5877759d13b664e28dea77de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart/parse: Allow ':' characters in variable/function namesRichard Purdie2021-07-201-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It is becomming increasingly clear we need to find a way to show what is/is not an override in our syntax. We need to do this in a way which is clear to users, readable and in a way we can transition to. The most effective way I've found to this is to use the ":" charater to directly replace "_" where an override is being specified. This includes "append", "prepend" and "remove" which are effectively special override directives. This patch simply adds the character to the parser so bitbake accepts the value but maps it back to "_" internally so there is no behaviour change. This change is simple enough it could potentially be backported to older version of bitbake meaning layers using the new syntax/markup could work with older releases. Even if other no other changes are accepted at this time and we don't backport, it does set us on a path where at some point in future we could require a more explict syntax. I've tested this patch by converting oe-core/meta-yocto to the new syntax for overrides (9000+ changes) and then seeing that builds continue to work with this patch. (Bitbake rev: 0dbbb4547cb2570d2ce607e9a53459df3c0ac284) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Ensure hash reflects vardepvalue flags correctlyRichard Purdie2020-12-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | The get_hash() function is used to decide if the base configuration has changed and hence whether a reparse is required. The vardepvalue flag's value was not expanded but it is often used in contexts like: METADATA_REVISION = "${@base_detect_revision(d)}" METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" which in it's unexpanded form means reparsing doesn't happen when it should as the data appears unchanged. Update get_hash to expand the values of vardepvalue so reparsing works as expected. This avoids basehash mismatch errors such as the one recently caused by using METADATA_REVISION in poky.conf's DISTRO_VERSION variable. The issue there could be exposed by a recipe using DISTRO_VERSION with the sequence: bitbake os-release <change the revision of the metadata with a dummy commit> bitbake os-release -C install which was caused because METADATA_REVISION changed but the metadata didn't reparse. (Bitbake rev: 26ccf1575aef2d6e2d7717d3bd10b1ed0d5a777d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: fix most undefined code picked up by pylintFrazer Clews2020-08-251-1/+1
| | | | | | | | | | Correctly import, and inherit functions, and variables. Also fix some typos and remove some Python 2 code that isn't recognised. (Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5) Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Handle hashing of datastores within datastores correctlyRichard Purdie2020-04-241-1/+4
| | | | | | | | | | | | | | | | If there is a datastore within a datastore (e.g. BB_ORIGENV) then get-hash() doesn;t correclty handle the contents using the memory address instead of the contents. This is a patch from dominik.jaeger@nokia.com which addresses this problem. Its been low priority since we don't include BB_ORIGENV anywhere this would cause an issue as standard. [YOCTO #12473] (Bitbake rev: 1a8bcfc1eb89ccff834ba68fb514330b510976a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: Simplify remote datastore connectionsRichard Purdie2020-03-241-50/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current approach to remote datastores used in tinfoil is breaking. For example, adding a devupstream extension to a recipe with a git upstream, making it the preferred version and then running "devtool modify" on it causes get_srcrev() circular dependency issues. The problem is the override handling in the datastore is broken. This gets broken since remotedata:recieve_datastore() sets d.dict but doesn't update d.overridedata (or d.inchistory or d.varhistory). We could play whack-a-mole but the current implementation seems to be flawed to me. It also doesn't cover, or only partially covers some datastore operations and each needs new dedicated command API. Instead, step back and reimplement the way the datastore connector works. With this change, the datastore is either remote or local but the data is not spread on two sides of the connection. All the API is proxied over the connection by a single function for the datastore (and two to support variable history and include history). This code does not support using the datastore as a parameter to any data store functions. We did have one case of that but its just bad code and can be replaced. The result is something which is much simpler and less invasive to the datastore code itself, meaning its behaviour should be much more consistent. The existing tests for the remote data no longer make any sense and are removed. The one bug this code would have is if key/value pairs are returned over the IPC and those values contained a DataSmart object since we don't recurse into return values to find such things. Nothing appears to do that currently so lets worry about it if its ever an issue. This change should simplfy a ton of other issues and avoid a ton of other bugs so is a huge net gain. Tested with bitbake's and OE's selftests. (Bitbake rev: 85e03a64dd0a4ebe71009ec4bdf4192c04a9786e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Don't pass unneeded datastoreRichard Purdie2020-03-241-1/+2
| | | | | | | | | | | | The datastore is already available to this function internally so don't also try and pass the datastore as a parameter. This is clearly broken API when you look at the existing calls to it. This then doesn't break the planned tinfoil data connector changes. (Bitbake rev: af1654498ee5b47368a41dad2d2b7b6abc19ff15) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Don't allow renameVar calls with equivalent keysKyle Russell2020-02-191-0/+4
| | | | | | | | | | | | While usually a programming error, the behavior can cause a Parser instance to eventually gobble up a significant amount of memory, greatly affecting system performance. Try to avoid getting into that situation and alert the user about what they attempted to do. (Bitbake rev: 01bf0912eef5700d61c6e3c9138cb4b6825ee782) Signed-off-by: Kyle Russell <bkylerussell@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-041-2/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-041-12/+0
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+2
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.data_smart: only try to expand refs to valid variable namesChristopher Larson2019-01-211-1/+1
| | | | | | | | | | | | | | | This aligns the behavior of expansion with the recipe parser, only attempting to expand references to valid variable names. This avoids adding references for things like `${foo#${TOPDIR}}` to our vardeps without imposing much additional processing overhead beyond the change to the expansion regexp. YOCTO #12987 (Bitbake rev: df2ac65370aa86cdbc1574fdede25e3519410e45) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/siggen: Switch md5 -> sha256Richard Purdie2019-01-081-1/+1
| | | | | | | | | | Similarly to the codeparser change, change to sha256 hashes due to worries over collisions. The main impact of this change is slightly slower parsing time as well as longer sstate file names. (Bitbake rev: 66f1b766997d53b4375fdd25719b1175f3828903) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Allow numeric characters in overridesRichard Purdie2018-12-141-1/+2
| | | | | | | | | | | We're seeing problems due to the way x86-64 is handled (or not handled) as an override. Relax the containts on overrides from being lowercase to being lowercase or numeric. This fixes problem where MACHINE=qemux86 would work but MACHINE=qemux86-64 would fail the same tests. (Bitbake rev: 3a3be518536acc868c7eeb3c1111ad1b321480b7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add missing regexp markupRichard Purdie2018-12-141-2/+2
| | | | | | | | Fix some further python3 warnings about unescaped regexs. (Bitbake rev: 8667605d016e82add95638fcb15c2bbc1b489ecc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: fix filename for compile()Robert Yang2018-11-191-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: Add the following two lines to conf/local.conf: FOO = "${@foo = 5}" HOSTTOOLS += "${FOO}" * Before the patch $ bitbake -p Check the first lines of bitbake bitbake-cookerdaemon.log [snip] File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 125, in python_sub codeobj = compile(code.strip(), self.varname or "<expansion>", "eval") File "FOO", line 1 [snip] There isn't a file named 'FOO', but a variable name. * After the patch $ bitbake -p [snip] File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 129, in python_sub codeobj = compile(code.strip(), varname, "eval") File "Var <FOO>", line 1 foo = 5 (Bitbake rev: 540b546be55e0f5f5d91695956da3a7732b2f90a) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add original traceback to ExpansionErrorRobert Yang2018-11-191-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This can make it print clearer errors, for exmaple: Add Runtime_error to 'def oe_import(d)" 16 def oe_import(d): 17 import sys 18 Runtime_error [snip] * Before the patch: $ bitbake -p ERROR: Unable to parse /buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py Traceback (most recent call last): File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 430, in DataSmart.expandWithRefs(s='${@oe_import(d)}', varname='OE_IMPORTED[:=]'): except Exception as exc: > raise ExpansionError(varname, s, exc) from exc bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined This error message has two problems: - "Unable to parse data_smart.py": This isn't the real cause. - It pionts to "raise ExpansionError(varname, s, exc) from exc" which isn't clear enough. * After the patch: $ bitbake -p ERROR: Unable to parse OE_IMPORTED[:=] Traceback (most recent call last): File "OE_IMPORTED[:=]", line 1, in <module> File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 18, in oe_import(d=<bb.data_smart.DataSmart object at 0x7f9257e7a0b8>): import sys > Runtime_error bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined This one is more clearer than before. (Bitbake rev: c0fe524c1aeccb24ddd2e1f7bf235c00fdbf79a7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Fix removal handling interaction issue with overridesRichard Purdie2018-10-201-4/+13
| | | | | | | | | | | | | | | | | | | If a variable has a _remove applied to it but that variable is in turn 'renamed' through OVERRIDES, the removal gets lost with the current code. TEST = "foo" TEST_someval = "bar" TEST_someval_remove = "bar" OVERRIDES = "someval" currently gives "bar" for TEST but should give "". This fixes the code to track the removal and adds a test case to ensure this doesn't regress again. (Bitbake rev: 8f55010c18057be040f073d8bcb4c5c2c311d809) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: data: Ensure task checksums account for remove dataRichard Purdie2018-10-181-9/+18
| | | | | | | | | | | | | | | | | | | | | | | | Currently remove operations are not being accounted for in the task checksums. This is a fairly serious oversight and needs to be fixed. To do so, we need internal data from getVarFlag combined with the expanded variable data so that only "active" remove operators are accounted for in the task checksum. We can get this from the new optional removes attribute in the returned parser object. The code can then use the data on active remove operators to account for the removals in task checksum but only when the removal is active. We have to be careful here not to reference any expanded data since this may for example contain build paths. This means we can only map back and reference the unsplit (and hence unexpanded) remove string which may expand to multiple removal values. [YOCTO #12913] (Bitbake rev: 57d2ee17ae83a139a37081eb082e6184fa883581) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-12 02:17:04 +0000