summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: main/process: Add extra sockname debuggingRichard Purdie2022-12-132-4/+6
| | | | | | | | | | | | | | We're struggling to understand how bitbake.sock can sometimes disappear in live builds when we can't see where it could have been deleted. This causes connection failures to the server and failed builds. Add some extra debugging around the server log and client retry log messages to give more information for the next time this issue occurs. (Bitbake rev: 376a516dc8c96727fd042ada65f803013601ee2d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump minimum python version requirement to 3.8Richard Purdie2022-12-121-2/+2
| | | | | | | | | | | Most of our older distros with python older than 3.8 already need buildtools tarballs apart from Ubuntu 18.04. 3.8 allows us to fix a few things, tidy code in places and is widely available or can be obtained with buildtools. Therefore update our minimum version to 3.8. (Bitbake rev: 744310f360d2288ac2ef07745abc86852126b5b9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen/runqueue: Drop do_setscene referencesRichard Purdie2022-12-113-5/+5
| | | | | | | | | | do_setscene was from a different era before our modern setscene per task code. It hasn't been used for years so remove some old obsolete references to it. (Bitbake rev: ef72282298f7c4db74383c23bb0251dd06d3c6d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Drop non-multiconfig aware siggen supportRichard Purdie2022-12-113-55/+4
| | | | | | | | | All siggens in common use should now support multiconfig, drop the compatibility code. (Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: correctly match versioned directoriesAlexander Kanavin2022-12-115-2/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | | When obtaining latest upstream versions, the code needs to check if the existing tarball is in a versioned directory (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and if it is, it needs to first obtain the list of all such versioned directories and then check all of them by going one step up in the directory hierarchy. Existing code was returning a correct match when the component name did not have numbers, e.g. a check on 'source/epiphany/43/' would return 43, but was stopping too soon when the component name itself had numbers ('source/libxml2/2.10/' would return libxml2). This change ensures the last match is taken instead of the first. Also, adjust the fetcher tests to check that versioned directories are correctly traversed in this case (e.g. the step to go one level up is taken and a new tarball is discovered in a different versioned directory). (Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository ↵Marek Vasut2022-12-091-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | metadata The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> (Bitbake rev: d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: handle username/password in uriKasper Revsbech2022-12-091-1/+2
| | | | | | | | | | | | | | | | | In checkstatus() opener.open() is used to check if an artifact is available. The check fails if the uri contains username password in the format: "username:password@hostname..". Moreover, the checkstatus function already uses the username from the "ud" object to craft a header, is username and password is provided. This fix ensure the uri in the Requests object used does not contain username as password. (Bitbake rev: 88350002d45e0aa85ecd5356da2c8d71e450641e) Signed-off-by: Kasper Revsbech <kasper.revsbech.ext@siemensgamesa.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main/server: Add lockfile debugging upon server retryRichard Purdie2022-12-092-25/+37
| | | | | | | | | | | | We keep seeing server issues where the lockfile is present but we can't connect to it. Reuse the lockfile debugging code from the server to dump better information to the console from the client side when we run into this issue. Whilst not pretty, this might give us a chance of being able to debug the problems further. (Bitbake rev: 22685460b5ecb1aeb4ff3436088ecdacb43044d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Add timestamp to server retry messagesRichard Purdie2022-12-091-3/+7
| | | | | | | | | | | We have timestamps in the server logs but we don't know which UI messages correlate to them. Add some timestamps to the messages which doesn't make them pretty but which might make it possible to debug problems. (Bitbake rev: a1a86f8c311cb1fc4f5562f1c77f82aa95141eee) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Ensure layers use LAYERSERIES_COMPAT fairlyRichard Purdie2022-12-081-2/+27
| | | | | | | | | | | | | | Some layers think they're going to be 'clever' and copy the values from another layer, e.g. using ${LAYERSERIES_COMPAT_core}. The whole point of this mechanism is to make it clear which releases a layer supports and show when a layer master branch is bitrotting and is unmaintained. Therefore add some code to avoid people doing this. I wish we didn't have to but... (Bitbake rev: 6709aedccbb2e7ddbb1b2e7e4893481a7b536436) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: Add support for addpylib conf file directive and ↵Richard Purdie2022-12-085-8/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | BB_GLOBAL_PYMODULES For many years OE-Core has injected it's own python modules into the python namespace using an immediate expansion of a variable in base.bbclass. It also added all entries from BBPATH to sys.path. We really need this to become a first class citizen of the langauge, this new addpylib directive allows that. Usage is of the form: addpylib <directory> <namespace> The namespace is imported and if there is an attribute BBIMPORT, that list of names is iterated and imported too. This mirrors what OE-Core has done for a long time with one difference in implmentation, sys.path is only appended to. This means later imported namespaces can't overwrite an earlier one and can't overwrite the main python module space. In practice we've never done that and it isn't something we should encourage or support. The new directive is only applied for .conf files and not other filetypes as it only makes sense in that context. It is also only allowed in the "base configuration" context of cookerdata since adding it at the recipe level wouldn't work properly due to the way it changes the global namespace. At the same time, move the list of modules to place in the global namespace into a BB_GLOBAL_PYMODULES variable. It is intended that only the core layer should touch this and it is meant to be a very small list, usually os and sys. BB_GLOBAL_PYMODULES is expected to be set before the first addpylib directive. Layers adding a lib directory will now need to use this directive as BBPATH is not going to be added automatically by OE-Core in future. The directives are immediate operations so it does make modules available sooner than the current OE-Core approach. The new code appends to sys.path rather than prepends as core did, as overwriting python standard library modules would be a bad idea and naturally encouraging people to collaborate around our own core modules is desireable. (Bitbake rev: afb8478d3853f6edf3669b93588314627d617d6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the systemFrank de Brabander2022-12-081-0/+16
| | | | | | | | | | | | | | Get rid of the duplicate code and add extra check that the locale en_US.UTF-8 is available on the system. This new helper method is now located right above the method filter_environment() which sets LC_ALL environment variable to 'en_US.UTF-8'. [YOCTO #10165] (Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Start sync thread a little earlierJoshua Watt2022-12-081-6/+7
| | | | | | | | | | | Starts the sync thread slightly earlier to give it some extra time to dump out the caches while the main process tears down the parsing processes (Bitbake rev: 105f2897b0618713b036fc0f7a6e0f3e78d1707a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Use event to terminate parser threadsJoshua Watt2022-12-081-12/+3
| | | | | | | | | | Uses an event to terminate the parser threads instead of a queue. This is not only simpler, but saves about 500ms on shutdown time (Bitbake rev: 2aed34e1d4bf24bba6263f168ff31b55b5fbe982) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/cache: Optionally allow adding siggen hash data to the ↵Richard Purdie2022-12-083-5/+34
| | | | | | | | | | | | | | | bitbake cache Being able to track siggen hash construction data can be useful for cache debugging. For now, add an extra cache class which contains this information. It can be enabled in the same way as the hob data cache through a feature flag to cooker. This allows us to experiment with the data without carrying larger patches around and ultimately may allow use to have a hash mismatch debugging mode that is more easily enabled. (Bitbake rev: 0736a8a03da8b774fafbd28f746bef4705378049) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/siggen: Simplify passing basehash data into the cacheRichard Purdie2022-12-082-3/+6
| | | | | | | | | | | | | The basehash data is really internal bitbake data and passing an object directly is more efficient than creating and then extracting variables. This will match the format of other data we may optionally wish to store in the cache so more to the more efficient method. Nothing I can see is using this data today (and nothing should be). (Bitbake rev: e621093a1bf37cd75ede3fb77ab6845556870fc7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Tweak code layoutRichard Purdie2022-12-081-25/+25
| | | | | | | | | | Make a small change in the layout of the code in build_dependencies which makes subsequent patches easier to read. No functionality change, just moving the function definitions to the start of the function block. (Bitbake rev: fff13b1e5e8397130b4378e0ba2301336ec651a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Small cache reuse optimizationRichard Purdie2022-11-291-3/+7
| | | | | | | | | | | | Currently the expand cache doesn't work for "parser" return types, which is the main type used by the build_dependencies() call that we spend most of the time in when parsing. Tweak the code to cache the unexpanded value in the expand cache and hence allow reuse of the parser in other fast path cases for small speed gains. (Bitbake rev: b4a8e5071dbcba2217b79e83e08b275ffcbc0eef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve error message for missing multiconfigRichard Purdie2022-11-291-0/+2
| | | | | | | | | | | | If you place a multiconfig which isn't enabled into an mcdepends you currently get a traceback from runqueue. We can do better, add some code to tell the user what happened in a more readable way without the traceback. [YOCTO #14970] (Bitbake rev: a4693b70764bb394ee2cf8dd12a5f6fce866008b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Remove pointless global variable declarationsRichard Purdie2022-11-291-2/+2
| | | | | | | | | | These are static regex compilations which don't change or a standard module import (bb). There is noneed to declare them as global so drop the pointless code which doesn't do anything. (Bitbake rev: 09a4c159e3fd184f730821e7bd99916b0d28dc70) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Drop obsolete pydoc/path codeRichard Purdie2022-11-291-5/+0
| | | | | | | | | | This code looks like it is from a time when the data module was executable. Nobody does that now and this usage of pydoc is long since obsolete/broken so clean up the code which doesn't do anything useful. (Bitbake rev: 6f4ef770641a9657edaf8618f4c86dfb1116622c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Use regex consistently for override matching张忠山2022-11-291-1/+1
| | | | | | | | | | | One section of the code is enforcing lowercase overrides, the other is allowing numeric characters. We should be consistent with one or the other. (Bitbake rev: df5b3b841fd8d6a652d643e9ae2bba09d60043e0) Signed-off-by: 张忠山 <zzs213@126.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Allow to_boolean to support int valuesRichard Purdie2022-11-271-0/+3
| | | | | | | | | | | | | | | | Some variables may be set as: X = 1 as well the more usual X = "1" so add support to to_boolean to handle this case. (Bitbake rev: ef9c033b011e68bbfedf7ddf118633c14388aaaf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add debugging for overrides stability issueRichard Purdie2022-11-271-1/+3
| | | | | | | | | | If someone is unfortunate enough to run into override recursion issues they're hard to debug with the existing message. We can at least show the values that OVERRIDES takes to show there is some problem and aid debugging. (Bitbake rev: 43035b75201616e7bfd680d3d15c5c0fc7c04eb6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command.py: cleanup bb.cache.parse_recipeChen Qi2022-11-271-1/+1
| | | | | | | | | | | The parse_recipe has been removed from bb.cache, replace it with the databuilder._parse_recipe. (Bitbake rev: d386fa81848247a3d407debf889db8cbcce03359) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/siggen: Switch to use frozensets and optimizeRichard Purdie2022-11-232-11/+9
| | | | | | | | | | | | | | | | | | | | | Python handles frozensets a little more optimally than normal sets. Once we finish parsing, we don't edit this data so we can convert to them. To do that, we need to stop changing them so process ignore_deps earlier then we can freeze the data and keep it frozen. This has the side effect that we need to be careful to sort the data in some of the variables when calculating the hashes. Overall this does seem to show a decent parsing time speed improvement of 20-25% in a local test but this would be highly setup dependent. Also ensure the sigdata can handle exported frozenset and make it import back to them instead of sets. (Bitbake rev: 19475627c363a52da49ec144422c87448ff2a6c5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix race issues around hash equivalence and sstate reuseRichard Purdie2022-11-221-16/+20
| | | | | | | | | | | | | | | | | | | | | | | | | We identified a use case where a native recipe (autoconf-native) was rebuilt with no change in output yet the sstate for do_package tasks wasn't being used. The issue is that do_package tasks have a hard dependency on pseudo-native:do_populate_sysroot. That task was one of the many tasks being rehashed when autoconf-native's hash was changed. If update_tasks processed a recipe before it had processed pseudo-native, that recipe would be marked as not possible from sstate and would run the full tasks. The fix is to split the processing into two passes, first to handle the existing covered/notcovered updates, then in the second pass, check whether there are "harddep" issues. This defers the do_package tasks until after pseudo-native is installed from sstate as expected and everything works well again. (Bitbake rev: e479d1e418a7d34f0a4663b4a0e22bb11503c8ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add further debug for sstate reuse issuesRichard Purdie2022-11-221-0/+4
| | | | | | | | | | | | Even after enabling all our debugging, we had a reproducible test case where sstate wasn't being reused and it was unclear from the logs why. This patch adds debugging on the possible codepaths that were breaking and allowed the issue to be debugged and fixed. (Bitbake rev: 9233ad685b9b5e9eeb775fc71761712aaf0e876c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Drop support for not saving the cache fileRichard Purdie2022-11-201-27/+1
| | | | | | | | | | | We don't test not using the cache and I'm not aware of anyone using this, it would be hard to with modern bitbake. Drop the conditional code and simply error if CACHE isn't set. (Bitbake rev: 063ffe699bc5fc23174643dfedb66864cacfcff8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: worker/runqueue: Reduce initial data transfer in workerdataRichard Purdie2022-11-201-16/+36
| | | | | | | | | | | | | | | | | When setting up the worker we were transfering large amounts of data which aren't needed until task execution time. Defer the fakeroot and taskdeps data until they're needed for a specific task. This will duplicate some information when executing different tasks for a given recipe but as is is spread over the build run, it shouldn't be an issue overall. Also take the opportunity to clean up the silly length argument lists that were being passed around at the expense of extra dictionary keys. (Bitbake rev: 3a82acdcf40bdccd933c4dcef3d7e480f0d7ad3a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server: Ensure cooker profiling worksRichard Purdie2022-11-202-4/+6
| | | | | | | | | | | | | The previous cleanups meant that when the cooker was started, profiling was always disabled as configuration was sent to the server later and this was too late to profile the main loop. Pass the "profile" option over the server commandline so that we can profile cooker itself again, the setting can now take effect early enough. (Bitbake rev: c97c1f1c127ef3f8fbbd1b4e187ab58bfb0a73e5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Drop unused functionRichard Purdie2022-11-201-12/+0
| | | | | | | | | | At this point users appear to all call add_info directly. Failing to do that means the file dependency tracking code isn't active so would cause problems. Therefore drop the unused function. (Bitbake rev: 6b24efc0f4d19738d96754280e70bc493005167d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Drop broken/unused codeRichard Purdie2022-11-202-39/+19
| | | | | | | | | | | | | | Some parts of functions in Cache() were broken and unused, there was also a totally unused function. This was historical as a result of the cooker parsing process needing to handle cached entries in the main thread but parsing actions in seperate processes. Document the way it works, update the function name to be clear about what it now does and drop the old code which was unused. (Bitbake rev: af83ee32df85c8e4144f022a1f58493eb72cb18e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/cookerdata: Move recipe parsing functions from cache to ↵Richard Purdie2022-11-205-72/+59
| | | | | | | | | | | | | | | databuilder When 'NoCache' was written, databuilder/cookerdata didn't exist. It does now and the recipe parsing functionality contained in NoCache clearly belongs there, it isn't a cache function. Move those functions, renaming to match the style in databuilder but otherwise not changing functionality for now. Fix up the callers to match (which make it clear this is the right move). (Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Add refreshed oe-core and poky fixturesTim Orling2022-11-142-86/+24
| | | | | | | | | | | | After updating gen_fixtures.py, run ./gen_fixtures.py --all This includes the latest stable/supported releases. Currently excludes 'dunfell' as it seems to crash. (Bitbake rev: 58e29fb15cd65795d6cf65a4db2b87f34649cb1f) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fixtures/gen_fixtures.py: update branchesTim Orling2022-11-141-5/+14
| | | | | | | | | | | Drop Honister and Hardknott as they are both EOL Add Langdale as it is the latest stable release Add comment about Dunfell (currently crashing) (Bitbake rev: e060daae28529c25968cacc5536a1034e1e1c030) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fixtures/README: django 1.8 -> 3.2Tim Orling2022-11-141-1/+1
| | | | | | | | | We should reference the docs for the current LTS version we are using. (Bitbake rev: 0dca22a184240d464bbd4ef276a6b13e44e29725) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: process_submodules(): Set nobranch=1 for urlRobert Yang2022-11-141-0/+1
| | | | | | | | | | | | Just like download_submodule() does, fixed warings when run bb.fetch2.Fetch([url], d) in process_submodules' function: WARNING: grpc-native-1.50.0-r0 do_fetch: URL: gitsm://github.com/abseil/abseil-cpp.git;protocol=https;name=third_party/abseil-cpp;subpath=third_party/abseil-cpp does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls. (Bitbake rev: 0ed7c75eb0508a1f699f47d7f22d559501865f61) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix regression in gitsm submodule path parsingPavel Zhukov2022-11-141-1/+1
| | | | | | | | | | | | | | | | Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression in submodules path parsing. As the result gitsm fetcher fails on each submodule which name begins from the name of the parent repo which is totally valid usecase [Yocto #14045] [1] Fix the code to error out only if submodule's name is equal to parent name but not if it's part of it. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4 (Bitbake rev: 3ad27272c18f2bb9edd441f840167a3dabd5407b) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/data_smart/build: Clean up datastore finalize/update_data ↵Richard Purdie2022-11-143-16/+6
| | | | | | | | | | | | | references We dropped the update_data calls a while ago. Clean up the code to match the reality and drop the remaining no-op pieces. Update the comments to reflect the slowest operations and let the cookie monster's spirit live on! (Bitbake rev: 584989ed2b5af4e8799571dece0cf94f995ef14e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: allow python snippets to include a dictionaryMark Asselstine2022-11-102-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #14917] Attempting to use a dictionary in a python code snippet for variable assignment results in an error. For example attempting something such as IDX = "green" VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}" produces the error expansion of VAL threw ExpansionError: Failure expanding variable VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]} which triggered exception SyntaxError: '{' was never closed (Var <VAL>, line 1) The existing __expand_python_regexp__, "\${@.+?}", will match the first close curly bracket encountered, resulting in incomplete and un-parsable code, and thus produce the error. We can correct this by allowing a single depth of nested curly brackets in __expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will match up to and including the matching close curly bracket to the open, '${@', curly bracket, even if there are one or more singly nested curly brackets present. This change allows the usecase described above to function. This change can't be made on its own though. The old regex would, in an obscure way, handle the case where a python snippet contained an unexpandable variable. Since the unexpandable variable is in curly brackets it would cause incomplete/un-parsable python code and thus remain unparsed. So something like VAL = "${@d.getVar('foo') + ${unsetvar}}" would remain unparsed as the close curly bracket in "${unsetvar}" would match and terminate the snippet prematurely. This quirk resulted in the proper handling of python snippets with unexpanded variables. With the change to __expand_python_regexp__ the full snippet will match and be parsed, but to match the old/correct behavior we would not want to parse it until ${unsetvar} can be expanded. To ensure the old/correct behavior for python snippets with unexpanded variables remains in place we add a check for unexpanded variables in the python snippets before running them. This handling of unparsed variables brings two benefits. The first we now have an explicit check visible to all for unexpanded variables instead of a somewhat hidden behavior. The second is that if there are multiple python snippets the old behavior would run the code for each but a single snippet with unexpanded variables would mean all snippets would remain unparsed, meaning more and repeated processing at a later time. For example: "${@2*2},${@d.getVar('foo') ${unsetvar}}" old behavior would give: "${@2*2},${@d.getVar('foo') ${unsetvar}}" new behavior will give: "4,${@d.getVar('foo') ${unsetvar}}" The old behavior would calculate '2*2' but toss the result when the second snippet would fail to parse resulting in future recalculations (or fetching from cache), while the new behavior avoids this. (Bitbake rev: 94e49b9b9e409c29eb04603b1305d96ebe661a4b) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: drop unused __expand_var_regexp__ and __expand_python_regexp__Mark Asselstine2022-11-101-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | First, commit 81efd77987f6decf256967fa16521a40c14d3518 created a copy of __expand_var_regexp__ and __expand_python_regexp__ when creating the initial version of data_smart.py. A while later commit db1c998b31da06d7f3eb09fc6f59a1915b7b549e dropped all references to these variables from data.py. This leaves us today with two versions of these variables in the global scope. However, only those defined in data_smart.py are being used, in data_smart.py. Unfortunately there was no indication in the commit log for commit db1c998b31da indicating why the variables were left in place despite the functions referencing them were being removed. Additionally data.py imports data_smart, thus the versions of __expand_var_regexp__ and __expand_python_regexp__ defined in data_smart.py would be used by all bitbake code, beyond, potentially, some code in data.py which we know has no references to these variables. To remove any potential confusion around these variables drop the old definitions from data.py. (Bitbake rev: 60f43d0428d43c981b44b6c8d125f77440f6c8f9) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: bitbake-layers: checkout layer(s) branch when clone existsMark Asselstine2022-10-291-0/+25
| | | | | | | | | | | | | | | | | | | | [YOCTO #7852] Fixes 'bitbake-layers layerindex-fetch --branch kirkstone meta-arm' not checking out the branch if the repo is already cloned and on a different branch. If a clone of a layer being added already exists check what branch it is on and if necessary attempt to switch to the given branch. If the switch fails to happen the git error will be reported. We also warn if there are uncommitted changes as the changes might go unnoticed and result in unexpected behaviors. (Bitbake rev: d2cb388f58a37db2149fad34e4572d954e6e5441) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: serv: correct closed client socket detectionJustin Bronder2022-10-291-2/+2
| | | | | | | | | | | | | | If the client socket is closed, asyncio.StreamReader.readline() will return an empty bytes object, not None. This prevents multiple tracebacks being logged by bitbake-hashserv each time bitbake is started and performs a connection check. (Bitbake rev: 2d07f252704dff7747fa1f9adf223a452806717f) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: don't set core.fsyncobjectfiles=0Ross Burton2022-10-291-1/+1
| | | | | | | | | | | | | | | | | This git configuration variable is deprecated in 2.36.0 onwards, so git warns in the logs for every git call. Luckily the default value has always been false[1], so we can just remove this. [ YOCTO #14939 ] [1] https://github.com/git/git/commit/aafe9fbaf4f1d1f27a6f6e3eb3e246fff81240ef (Bitbake rev: 8ad310633e0c5d5593631c1196cbdde30147efce) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie2022-10-261-2/+2
| | | | | | | | | CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: bb.tests.fetch.URLHandle: add 2 new testsMark Asselstine2022-10-261-0/+2
| | | | | | | | | | | | | | | | | Add a test for special characters in user and password to qualify decodeurl() inspired by a bug report describing that '=' signs in a password was problematic. Add a second test to qualify decodeurl() as related to the change in commit 628c4bf6c89b [fetch2/__init__: handle @ in package names]. Relates to [YOCTO #14476] (Bitbake rev: ee04cf09c7022168c035affa654773652a49793e) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/ply: Update md5 to better report errors with hashlibMark Hatle2022-10-262-1/+13
| | | | | | | | | | | | | | | | | | | | | | In the case where hashlib is not available, the try would fail and fall through resulting in a backtrace on the usage of the 'sig'. The backtrace itself was confusing and made it difficult to determine what went wrong. Update the import to be in it's own try block with an appropriate message to indicate what went wrong. Note, the current version of ply all of this code has been restructured so this is not applicable upstream. Additionally, some versions of hashlib don't appear to implement the second FIPS related argument. Detect this and support both versions. (Bitbake rev: 484ab42f440070c0369b81f5c69da860fa47a798) Signed-off-by: Mark Hatle <mark.hatle@amd.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix a typoOliver Lang2022-10-251-1/+1
| | | | | | | (Bitbake rev: 82f40261a06d39f0e7748942f480da5b44282fa3) Signed-off-by: Oliver Lang <quantenkeks@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix a typoOliver Lang2022-10-251-1/+1
| | | | | | | (Bitbake rev: cf23612f4e8946b9ed4c9f87b451f32b8c471df2) Signed-off-by: Oliver Lang <quantenkeks@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-14 18:38:11 +0000