summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/tests
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2/svn: Fix mirroring issue with svnKari Sivonen2 hours1-1/+9
| | | | | | | | | | | | Add return false to supports_checksum for svn fetcher which fhis fixes MIRROR usage for svn uris. Also add a testcase. [YOCTO #15473] (Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f) Signed-off-by: Kari Sivonen <kari.sivonen@live.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: add upstream latest version check functionAlexander Kanavin2 hours1-0/+16
| | | | | | | | | | | This is actually rather easy: crate web API provides a json with all the versions, for example: https://crates.io/api/v1/crates/cargo-c/versions (Bitbake rev: f6c2755db9a1f88c8534193b420fa31d135945e6) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle URIs with single-valued query parametersRoss Burton2024-03-221-0/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Whilst typically the URI query is a list of key-value pairs, that's not actually required by the URI specification. For example: http://example.com/foo?bar is a valid query, but this will result in the fetcher raising an exception: File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__ self.query = self._param_str_split(urlp.query, "&") File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]: ValueError: not enough values to unpack (expected 2, got 1) In this case the query is just "bar", but the fetcher is trying to split it into a key-value pair. The URI object exposes the parsed query explicitly as a dictionary of key-value pairs, so we have to be a little creative here: if a value is None then it isn't a key-value pair, but a bare key. Fix this by handling elements without the deliminator in _param_str_split() (by assigning the value to None), and handle a None value when formatting the query in _param_str_join(). (Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure that git LFS objects are availablePhilip Lorenz2024-02-231-4/+47
| | | | | | | | | | | | | | | | | | | | | The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Make latest_versionstring extract tags with slashes ↵Peter Kjellerstedt2024-02-191-0/+2
| | | | | | | | | | | | | | correctly Before, everything up to the last slash was removed when extracting the names of the tags. This would lead to that a tag such as "agent/11.0.0" would be incorrectly identified as "11.0.0", which would then be treated as a correct version matching "^(?P<pver>\d+(\.\d+)+)". (Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Make test_git_latest_versionstring support a max versionPeter Kjellerstedt2024-02-191-11/+14
| | | | | | | | | | | In some cases, the version found by latest_versionstring() may be higher than the real version. Make it possible to specify a maximum version so that this case can be detected. (Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: add test case for using premirror in restricted networkChen Qi2024-02-191-0/+19
| | | | | | | | | | | | We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch' failed even with all contents available in PREMIRRORS. Add a test case to ensure no regression in the future. (Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: add multiple fetches testPavel Zhukov2024-02-191-0/+18
| | | | | | | | | | | Fetch from premirror few times to emulate multiple machines sharing same clonedir or few rebuilds of the package from (pre)mirror Regression test for [Yocto #15369] (Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: support git's safe.bareRepositoryAndré Draszik2024-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the bitbake selftests fail miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: a45e14a7343e36101e45639931322e5649587f57) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: fetch.py: use real subversion repositoryMartin Jansa2024-02-081-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * github no longer provides support for subversion clients: https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients it was shut down on 2024-01-08: https://github.blog/2023-01-20-sunsetting-subversion-support/ and this test was now failing with: ====================================================================== ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn) ---------------------------------------------------------------------- Traceback (most recent call last): File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn fetcher.download() File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download raise FetchError("Unable to fetch URL from any source.", u) bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source. Stdout: Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2 Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output: A trunk/README.md U trunk Checked out revision 2. svn: warning: W205011: Error handling externals definition for 'trunk/bitbake': svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git' svn: E205011: Failure occurred processing one or more externals definitions in the rare cases where subversion was still installed on the host running bitbake-selftest :). to avoid this use still alive repository from https://svn.apache.org/ and pick something rather small and only the trunk subdirectory which is fast to fetch: svn co https://svn.apache.org/repos/asf/serf/trunk takes just 2 sec here adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of "trunk/README" (Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617) Signed-off-by: Martin Jansa <martin.jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast: Fix EXPORT_FUNCTIONS bugRichard Purdie2024-01-101-0/+98
| | | | | | | | | | | | | | | | If you have two classes, both of which set EXPORT_FUNCTIONS for the same funciton and a standard funciton definition for the function that is exported, the export function can sometimes overwrite the standard one. The issue is that the internal flag the code uses isn't ovweritten if the variable is giving a new value. Fix the issue by using a comment in the code that is injected so that we know if it is ours or not. Also add some testing for EXPORT_FUNCTIONS, not perfect but a start. (Bitbake rev: 66306d5151acb0a26a171c338d8f60eb9eb16c6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/event: Add test_lineno_in_eventhandlerRobert Yang2024-01-101-0/+24
| | | | | | | | | Add test_lineno_in_eventhandler to test lineno in eventhandler. (Bitbake rev: 4e5de537bebb68180c5755858c81b095eb9ae2f6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/toaster: Fix assertEquals deprecation warningsRichard Purdie2023-12-061-1/+1
| | | | | | | | | Fix: DeprecationWarning: Please use assertEqual instead (Bitbake rev: dd990ea6843685927954101feb729f3faa3a16d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests: Fix duplicate test_underscore_override testRichard Haar2023-12-021-10/+10
| | | | | | | | | Found a duplicate test, added _2 suffix to one, 74 tests now pass up from 73. (Bitbake rev: ae2a19dadb4f3065b8731a61f45f29e6a70af402) Signed-off-by: Richard Haar <rh@richhaar.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2: git: Update Git-LFS download and testsDesone Burns2023-11-231-26/+35
| | | | | | | | | | | | | | | | | | | | When downloading a Git repository containing an LFS, the Git hooks are not always initialized correctly to perform the download. This change updates the Git downloader to run the "git lfs install" command in order to "smudge" the LFS files. The tests have also been updated to do checks for situations in which git lfs is not installed, as the application was required to be installed for any of the tests to run previously. The Git LFS functionality was working to some degree previously, but this change also updates the fetcher to allow LFS downloads for nobranch SRC_URIs. (Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217) Signed-off-by: Desone Burns II <dburns@seegrid.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: Add tests to cover multiple branch/name parametersPavel Zhukov2023-10-231-6/+69
| | | | | | | | | | | Create repository with few branches and test if fetcher can work with such repository as PREMIRROR (Bitbake rev: a1737610e5d5b61e126ec3632d7f27b337a87818) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: selftest/fetch: Ensure top level directory timestamp doesn't break testRichard Purdie2023-10-061-5/+7
| | | | | | | | | | The infrastructure changes highlighed an issue where the top level directory mtime stamp isn't included in the tarballs. This isn't an issue the test is meant to check for so ignore it. (Bitbake rev: 5a1e5b8dc55d6bb4d93bac3492f8c43ff957e712) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Add missing dependency handling of remove operatorInsu Park2023-09-201-0/+26
| | | | | | | | | | | | | | | | | A recipe variable handles its dependencies even on the "contains" variables within the "inline Python expressions" like bb.utils.filter(). And it also handles those in the append operator correctly, but the problem is that it does not so in the remove operator. Fix it by adding the missing dependencies every time the remove operator has been handled. Also add a test case to check if the override operators handle dependencies correctly. (Bitbake rev: b90520eedb1dbc7f6a3928d089fe74fafb864eb5) Signed-off-by: Insu Park <insu0.park@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Improve runtaskdeps data to fix sstate debuggingRichard Purdie2023-08-091-70/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The runtaskdep data in siginfo files was written out with full paths to the bb files, matching bitbake's internal "unique key" ID for recipes/tasks. When originally implemented this made sense. Over time, the main use for the data in siginfo files has become to match against other siginfo files to debug changes of hash calcuations. The recipename data is not useful for this as the siginfo filenames use PN instead which can often be derived from the recipe filename but not always. It is time to throw away the 'tid' data format and switch over the use a hybrid PN form which includes the multiconfig. That can be easily stripped off in the find_siginfo code in oe-core. The other purpose of having a sortable dependency ID is retained and the multiconfig needs to be included to allow the taskhashes to be processed and calculated correctly. PN is meant to be unique between recipes, only one would ever be built so using PN in this location is fine. The one risk of this change is there isn't any compatibility to the old format. I'm not convinced we should spend time complicating the code with it. This change will change the taskhashes everywhere so the only mixing of old and new siginfo files will be either through hash equivalence or through users using the tool against old and new info files manually which will give some weird output but it should be clear they're in different formats as there would be large paths from the old files not present in the new ones. We have options to add backwards compatibility if some issue is found to need that. (Bitbake rev: 637933e2e5a59228a8d17aae4160551cab5f2f61) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.tests.data: don't require the func flag for context functionsChristopher Larson2023-08-041-1/+0
| | | | | | | | | | | Update test_python_snippet_function_reference to not require the 'func' flag, now that we know the real function will be returned for context functions without the flag. (Bitbake rev: 83f41281ec3d9b4327ffc8e2312e1fb8f53cbf02) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests.data: add test for builtin preferred over metadata valueChristopher Larson2023-08-041-0/+4
| | | | | | | | | This test makes sure that '${@eval()}' calls the eval builtin, even if an 'eval' variable is defined in the metadata. (Bitbake rev: e9150447738a48f772240874b3512b08e982b19b) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests.codeparser: add test for exec of builtin from inline pythonChristopher Larson2023-08-041-0/+6
| | | | | | | | | | This ensures that any change to the presence of builtins in inline python execs will be noticed. (Bitbake rev: ee22d3d51c60db2da97422b2be1e42239b7a2324) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests.data: add test for inline python calling a def'd functionChristopher Larson2023-08-041-0/+9
| | | | | | | | | | | This is a test for an issue seen long ago, to avoid regressions, where a reference to a def'd function in the metadata would return the string value from the metadata rather than the function in inline python. (Bitbake rev: 9f7cb22febd557817c164e25a93f5660e9c06358) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Set maxsplit to match expected variablesDit Kozmaj2023-07-271-0/+1
| | | | | | | | | | | | Set the maxsplit value to match the expected number of variables. This also avoids an unnecessary split as the parameters are in the form 'key=value' and the 'value' could contain the '=' character. (Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19) Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests: Use assertLogs to test logging outputPeter Hoyes2023-06-272-10/+12
| | | | | | | | | | | | | By default, pytest captures all stdout and exposes it using its built-in fixtures (capsys, caplog etc), so stdout does not support getvalue(). To support running tests using both unittest and pytest, use assertLogs to capture logging and assert on the log output instead. (Bitbake rev: 2d28caa01bab9540d2bbaf713ae3e5c563d003f5) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: Set git config if not already setPeter Hoyes2023-06-271-3/+9
| | | | | | | | | | | | | | | | | | | git config returns an error code if user.email or user.name are not set, so bb.process.run raises an ExecutionError, failing the test. Improve the logic by catching the ExectionError and using this to set a default value. Change a direct invocation of 'git init' to use self.git_init so that it follows the code path above. Remove the related comment from the README now that git sets up the user details automatically. (Bitbake rev: 0c7079d0609a13713b890a9675abd9fba032f199) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: Rename assertRaisesRegexp to assertRaisesRegexPeter Hoyes2023-06-271-1/+1
| | | | | | | | | | TestCase.assertRaisesRegexp was renamed to assertRaisesRegex in Python 3.2, so rename to fix a warning during test execution. (Bitbake rev: 9bc2ca3197bff7a5ef6e0ece6bfef0cda68b808e) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: Mark TestTimeout as not a test suitePeter Hoyes2023-06-271-1/+2
| | | | | | | | | | | pytest test collection attempts to collect all classes containing "Test". TestTimeout is not a test class so add the __test__ = False attribute to indicate this. (Bitbake rev: b5455b98093a3eaf122ebe3d29c3bef949fd2440) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/tests/fetch.py: set initial branchBergin, Peter2023-05-251-0/+4
| | | | | | | | | | | | | | If you have a host where the git config for initial branch name is something else than 'master' the unittest will fail as they assume the default branch name is 'master'. Fix this by explicitly set the intial branch name at 'git init'. (Bitbake rev: 5bcd213c23da30a84baba88b775b1740e6bb77d0) Signed-off-by: Peter Bergin <peter.bergin@windriver.com> Signed-off-by: Peter Bergin <peter@berginkonsult.se> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Correct unpack for a crate that matches the recipe namePeter Kjellerstedt2023-05-251-0/+25
| | | | | | | | | | | | | | | | | | | The crate fetcher handles a crate with a name that matches the recipe's name specially by placing the unpacked code in the current directory (which typically is ${S}) rather than together with the sources for the other crates. This broke when the URI names for all crates were changed recently to include the version in the name. Correct the crate fetcher to test against ${BP} instead of ${BPN}. Also add a test case to the selftests to avoid this breaking again. [Yocto #15012] (Bitbake rev: 9af52b748357d0c843ce2507ce4d119fd9c37008) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Default to https git protocol where possibleRichard Purdie2023-05-051-29/+29
| | | | | | | | | | The recommendation from server maintainers is that the https protocol is both faster and more reliable than the dedicated git protocol at this point. Switch to it where possible. (Bitbake rev: 879b830fcc60a022682ef22f58c04e7e07d03c63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: ConfHandler: Allow variable flag name with a single characterKai Kang2023-04-131-0/+2
| | | | | | | | | | | | | | Update regex pattern to allow variable flag name with a single character. Regression tests have also been updated in `bb.parse` and `bin/bitbake-selftest -k ParseTest` has been successfully executed. Eliminate a trailing space as well. (Bitbake rev: bb9e523291a3cad6e1596ee6a1e715b5e5feba8f) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: create versioned 'name' entriesEnrico Scholz2023-04-061-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | It is common for rust packages to depend on different versions of the same crate. E.g. | crate://crates.io/windows_x86_64_msvc/0.42.2 \ | crate://crates.io/windows_x86_64_msvc/0.48.0 \ Identification only by the plain crate name makes the sha256sum ambiguous | SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" | SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a" and requires lot of manual work to fix the SRC_URI list. Making the 'name' property unique by appending the version allows direct copy & paste of reported sha256sum errors to complete the crates list. (Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add checksum capability for crate fetcherFrederic Martinsons2023-03-301-0/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This change brings checksum verification of each crate in a recipe, e.g | SRC_URI += " \ | crate://crates.io/aho-corasick/0.7.20 \ | crate://crates.io/atomic-waker/1.1.0 \ | crate://crates.io/cc/1.0.79 \ | " | | SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac" | SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599" | SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f" That will require to move the checksum initialization after the possible call to urldata_init method in order for the crate fetcher to parse the url. Another way of doing could have been implementing a decodeurl method that would have been specific for each fetcher class. (Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler: Allow the '@' character in variable flag namesJames R T2023-03-231-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch enables the usage of the '@' character in variable flag names. One use case of variable flags is to assign the network namespaces of some systemd services/targets to configure other parts of the build process of some system. The filenames of systemd services/targets might contain the '@' character if they are template unit files that can take in a single parameter/argument and be instanced multiple times, as indicated by systemd's official manual page. The '@' character is disallowed as the first character in a variable flag name. Imposing more restrictions on the first character is a compromise to make parsing easier and to allow for more options in the future to extend the syntax. This patch is successfully verified by creating a custom BitBake recipe that sets and unsets the value of a variable flag with the '@' character in its name and ensuring that no ParseError is being thrown. Regression tests have also been added to `bb.parse`. `bin/bitbake-selftest` has also been successfully executed and all tests passed. (Bitbake rev: 00f9ab2cacfbd2a63b6b4959cf5401babae7e32a) Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEENRichard Purdie2023-03-221-4/+4
| | | | | | | | | For readability of following patches, rename this internal variable to allow for others in a similar format. (Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: crate.py: authorize crate url with parametersFrederic Martinsons2023-03-171-0/+24
| | | | | | | | | | | | | | | | | This allow to have classic fetch parameters (like destsuffix, sha256, name ...) not being considered by crate fetcher itself (and so mess up its download) Moreover, it allow to overload the name of the downloaded crate (maybe usefull if there is a naming clash between two crates coming from different repositories) (Bitbake rev: 278bd2f1758b8af97552af8d23d16ffb5127a131) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/npmsw: add more short forms for git operationsMark Asselstine2023-02-261-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | >From the npm-install documentation [1] the CLI provides a set of short forms when the install fetches from git. These include "github:" example: npm install github:mygithubuser/myproject "gist:" example: npm install gist:101a11beef "gitlab:" example: npm install gitlab:mygitlabuser/myproject "bitbucket:" example: npm install bitbucket:mybitbucketuser/myproject Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] by Stefan Herbrechtsmeier added support for the "github:" but the others would marked as 'Unsupported dependency'. The other prefixes are added in this commit, along with extending the tests to cover some of these. However, there is one more short form for github which npm-install allows which forgoes the prefix altogether. example: npm install mygithubuser/myproject Unfortunately this format is a bit problematic as it lacks any easily identifiable 'marker' to match against, and it could be either the github short form or install from folder format. Experimentation shows that the folder format requires a leading './' or '/', so we use this to rule out the ambiguity. If this approach to folder and github formats disambiguation is incorrect it won't matter anyways as the folder format is unrecognized by the code as-is and thus with this change or without, things would fail. Since we have to be less strict in the check for git operations we move it to be the last install format which we check, such that the less ambiguous formats can be sorted out first. [1] https://docs.npmjs.com/cli/v9/commands/npm-install [Yocto #14236] (Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add real git lfs tests and decoratorPaulo Neves2023-02-221-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. (Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: git-lfs restore _find_git_lfsPaulo Neves2023-02-221-16/+24
| | | | | | | | | | Not restoring the mocked _find_git_lfs leads to other tests failing. (Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Add support for new BB_HASH_CODEPARSER_VALS for cache ↵Richard Purdie2023-01-241-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | optimisation Currently the codeparser cache ends up being extended for every parse run since there are values in the functions such as the result of os.getpid() from LOGFIFO in OE-Core. Digging into that issue, there are also lots of similar but different functions being parsed where the change might just be a path to WORKDIR, a change in PN or PV or something like DATE/TIME. There is no reason we have to use these changing values when computing the dependenies of the functions. Even with a small tweak like: BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn" the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run to around 1.3MB and remaining static for oe-core and meta-oe. In my local build, admittedly heavily experimented with, the cache had grown to 120MB. The benefits of doing this are: * faster load time for bitbake since the cache is smaller to read from disk and load into memory * being able to skip saving the cache upon shutdown * lower memory footprint for bitbake * faster codeparser data lookups (since there is less data to search) We only use these special values when passing code fragments to the codeparser to parse so the real variable values should otherwise be used in the hash data. The overall effect of this change, combined with others to avoid saving unchanged cache files can be ~2s on a ~16s parse on my local system and results in a more responsive feeling bitbake. It also allows parsing performance to be investigated more consistently. (Bitbake rev: f24bbaaddb36f479a59a958e7fc90ef454c19473) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/tests: Add parameter to recipe_uriPavel Zhukov2023-01-201-1/+1
| | | | | | | | | | | | While the parameter is not required it allows testing of possible regression in fetcher code when parameter specified and mirrors are used. (Bitbake rev: 1a21918049091a6d77426dbf8868ffdc14ba1003) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/tests: Add test for MercurialPavel Zhukov2023-01-181-0/+44
| | | | | | | | | | | Basic test for mercurial functionality including PREMIRRORONLY with disabled network (Bitbake rev: c0e349a5fc92e9c90afc53800c7e83495393ff79) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Always use threadlockRichard Purdie2022-12-311-16/+1
| | | | | | | | | | | | | | | | With the move to a server idle thread, we always need threading. The existing accessor functions could end up turning this off! I was going to hold the lock whilst changing it, check if the value was already set, cache the result and also fix the event code to always release the lock with a try/finally. Instead, disable the existing functions and use a with: block to handle the lock, keeping things much simpler. (Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: builtins fix for 'd' deletionRichard Purdie2022-12-292-11/+12
| | | | | | | | | | | | | | | | | | | I've been seeing event handlers where 'd' seems to disappear half way through event handler execution. This is problematic when multiple threads are active since this code assumes single threading. The easiest fix is to change the handler function calls to contain d as a parameter as we do elsewhere for other functions. This will break any non-text handlers but I was only able to spot one of those in runqueue. It will also break handlers than call functions that assume 'd' is in the global namespace but those failures should be obvious and we can fix those to pass d around. This solution avoids manipulating builtins which was always a horrible thing to do anyway and solves the issue without needing locking, thankfully. (Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/data/codeparser: Add dependencies from python module functionsRichard Purdie2022-12-171-7/+7
| | | | | | | | | | | | | | | | | | | | Moving code into python modules is a very effective way to reduce parsing time and overhead in recipes. The downside has always been that any dependency information on which variables those functions access is lost and the hashes can therefore become less reliable. This patch adds parsing of the imported module functions and that dependency information is them injected back into the hash dependency information. Intermodule function references are resolved to the full function call names in our module namespace to ensure interfunction dependencies are correctly handled too. (Bitbake rev: 605c478ce14cdc3c02d6ef6d57146a76d436a83c) (Bitbake rev: 91441e157e495b02db44e19e836afad366ee8924) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Drop non-multiconfig aware siggen supportRichard Purdie2022-12-111-1/+1
| | | | | | | | | All siggens in common use should now support multiconfig, drop the compatibility code. (Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: correctly match versioned directoriesAlexander Kanavin2022-12-114-0/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | When obtaining latest upstream versions, the code needs to check if the existing tarball is in a versioned directory (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and if it is, it needs to first obtain the list of all such versioned directories and then check all of them by going one step up in the directory hierarchy. Existing code was returning a correct match when the component name did not have numbers, e.g. a check on 'source/epiphany/43/' would return 43, but was stopping too soon when the component name itself had numbers ('source/libxml2/2.10/' would return libxml2). This change ensures the last match is taken instead of the first. Also, adjust the fetcher tests to check that versioned directories are correctly traversed in this case (e.g. the step to go one level up is taken and a new tarball is discovered in a different versioned directory). (Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: allow python snippets to include a dictionaryMark Asselstine2022-11-101-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #14917] Attempting to use a dictionary in a python code snippet for variable assignment results in an error. For example attempting something such as IDX = "green" VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}" produces the error expansion of VAL threw ExpansionError: Failure expanding variable VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]} which triggered exception SyntaxError: '{' was never closed (Var <VAL>, line 1) The existing __expand_python_regexp__, "\${@.+?}", will match the first close curly bracket encountered, resulting in incomplete and un-parsable code, and thus produce the error. We can correct this by allowing a single depth of nested curly brackets in __expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will match up to and including the matching close curly bracket to the open, '${@', curly bracket, even if there are one or more singly nested curly brackets present. This change allows the usecase described above to function. This change can't be made on its own though. The old regex would, in an obscure way, handle the case where a python snippet contained an unexpandable variable. Since the unexpandable variable is in curly brackets it would cause incomplete/un-parsable python code and thus remain unparsed. So something like VAL = "${@d.getVar('foo') + ${unsetvar}}" would remain unparsed as the close curly bracket in "${unsetvar}" would match and terminate the snippet prematurely. This quirk resulted in the proper handling of python snippets with unexpanded variables. With the change to __expand_python_regexp__ the full snippet will match and be parsed, but to match the old/correct behavior we would not want to parse it until ${unsetvar} can be expanded. To ensure the old/correct behavior for python snippets with unexpanded variables remains in place we add a check for unexpanded variables in the python snippets before running them. This handling of unparsed variables brings two benefits. The first we now have an explicit check visible to all for unexpanded variables instead of a somewhat hidden behavior. The second is that if there are multiple python snippets the old behavior would run the code for each but a single snippet with unexpanded variables would mean all snippets would remain unparsed, meaning more and repeated processing at a later time. For example: "${@2*2},${@d.getVar('foo') ${unsetvar}}" old behavior would give: "${@2*2},${@d.getVar('foo') ${unsetvar}}" new behavior will give: "4,${@d.getVar('foo') ${unsetvar}}" The old behavior would calculate '2*2' but toss the result when the second snippet would fail to parse resulting in future recalculations (or fetching from cache), while the new behavior avoids this. (Bitbake rev: 94e49b9b9e409c29eb04603b1305d96ebe661a4b) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie2022-10-261-2/+2
| | | | | | | | | CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-08 16:08:03 +0000