linux/poky.git/bitbake/lib/bb/codeparser.py, branch mickledore

bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache location

2023-01-26T21:50:31+00:00

Currently the codeparser cache is set from CACHE, which is typically in bitbake.conf which means we can't read/write any cache until it is found/read. We may well have python expressions to parse before that happens. The net result is suboptimal functioning of the codeparser cache since it will often be invalidated by data that is never written. This patch changes the codeparser and filechecksum caches to use BB_CACHE as their setting and defaults it to ${TOPDIR}/cache. The patch doesn't change where the "persistent" data such as prserver and hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing cache resists (still currently CACHE). I've left those for a later patch. The patch does ensure data parsed by the core datastore parsing calls is written back since this is now much more useful after this change. (Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628) Signed-off-by: Richard Purdie

bitbake: codeparser/data: Add vardepsexclude support to module dependency code

2022-12-17T08:52:28+00:00

We need to be able to exclude dependencies from the python module dependency code. Add support for the vardepexclude flag for these. It only works from the configuration namespace rather than per recipe for efficiency. (Bitbake rev: 1aa672b01037fda4ca82f2c7e394783287c09ecd) Signed-off-by: Richard Purdie

bitbake: ast/data/codeparser: Add dependencies from python module functions

2022-12-17T08:52:28+00:00

Moving code into python modules is a very effective way to reduce parsing time and overhead in recipes. The downside has always been that any dependency information on which variables those functions access is lost and the hashes can therefore become less reliable. This patch adds parsing of the imported module functions and that dependency information is them injected back into the hash dependency information. Intermodule function references are resolved to the full function call names in our module namespace to ensure interfunction dependencies are correctly handled too. (Bitbake rev: 605c478ce14cdc3c02d6ef6d57146a76d436a83c) (Bitbake rev: 91441e157e495b02db44e19e836afad366ee8924) Signed-off-by: Richard Purdie

bitbake: bitbake: Add copyright headers where missing

2022-08-12T10:49:29+00:00

Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie

bitbake: codeparser: Avoid log bufer overhead in cache case

2022-03-18T13:17:10+00:00

Creating the new log instances triggers a lot of python logging overhead in a commonly called function (about 600,000 for parsing OE-Core). We only need the log functionality if we're parsing, not if we just hit from the cache. Therefore defer the log setup overhead until we know it is a cache miss. Whilst this complicates the code slightly, the performance gain is worth it as for parsing OE-Core we drop 60 million funciton calls (from 225 overall). (Bitbake rev: ac868167ad854f9bb32dcb2e63528870547805a7) Signed-off-by: Richard Purdie

bitbake: codeparser: Fix TypeError in bitbake debug mode

2021-03-15T21:47:42+00:00

Commit 75f87db413 fixed the confusion between bitbake and python logger but in codeparser still old method of setting debug level was used causing TypeError, because debug level value was incorrectly returned and assigned to event.msg. | File "./bitbake/lib/bb/ui/knotty.py", line 660, in main | event.msg = event.fn + ': ' + event.msg | TypeError: can only concatenate str (not "int") to str [YOCTO #14298] (Bitbake rev: bec9ea7ab28a8dfad1a6010ca5c6ec691754748d) Signed-off-by: Tomasz Dziendzielski Signed-off-by: Richard Purdie

bitbake: lib: remove unused imports

2020-01-19T13:31:05+00:00

removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews Signed-off-by: Richard Purdie

bitbake: bitbake: Add initial pass of SPDX license headers to source code

2019-05-04T09:44:04+00:00

This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie

bitbake: codeparser: Switch to sha256 from md5

2019-01-08T11:16:03+00:00

We've reports of hash collision with codeparser. Looking at the way collision problems occur with md5 and the way our function templating works, I can believe we may run into issues. This patch therefore switches to sha256. Performance wise, parse time could appear to rise by 4s in 374s Before: 384329 in 2.966s (md5) After: 349743 in 2.340s (sha256) 34723 in 1.245s (md5) since we still have md5 used elsewhere in the code, something we should look at next (using sha256 everywhere is around 5.3s in total) Unfortunately this does nearly double the size of the codeparser cache file due to the hash size change. (Bitbake rev: 4bed7a97198176503fe8c72d8dd7c36b28fc9dd1) Signed-off-by: Richard Purdie

bitbake: bitbake: pysh: Improve error handling for shell code

2018-12-09T11:07:48+00:00

The p_error() is used for printing errors when parse shell code, but it can't the EOF error correctly - Add the following lines to quilt.inc do_configure_prepend () { find ${s} -name "*.in" -exec sed -i -e "1s,^#\!.*@perl@ -w$,#\! @perl@\nuse warnings;," {} \; if [ hello ]; then } - Before the patch: $ rm -fr cache/ tmp/cache/; bitbake -p [snip] WARNING: /path/to/quilt/quilt-native_0.65.bb: Error during finalise of /path/to/quilt/quilt-native_0.65.bb [snip] bb.pysh.sherrors.ShellSyntaxError: None followed by: We can see that this isn't easy to debug, let p_error() check wheter it is EOF and print appropriate errors can improve the error message. And don't let codeparser.py except pyshlex.NeedMore (in fact, it never worked since p_error() only raise ShellSyntaxError), but make it print the last 5 lines which might be useful for debuging. - After the patch $ rm -fr cache/ tmp/cache/; bitbake -p [snip] ERROR: /path/to/quilt/quilt_0.65.bb: Error during parse shell code, the last 5 lines are: find /path/to/quilt/0.65-r0/quilt-0.65 -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \; if [ hello ]; then autotools_do_configure sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i /path/to/quilt/0.65-r0/quilt-0.65/Makefile [snip] File "/path/to/bb/pysh/pyshyacc.py", line 649, in p_error(p=None): w('Unexpected EOF') > raise sherrors.ShellSyntaxError(''.join(msg)) bb.pysh.sherrors.ShellSyntaxError: Unexpected EOF (Bitbake rev: 44790597951638e32eb1672de2e40bd5a603326b) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie