summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2/svn: Fix mirroring issue with svnKari Sivonen9 days1-0/+3
| | | | | | | | | | | | Add return false to supports_checksum for svn fetcher which fhis fixes MIRROR usage for svn uris. Also add a testcase. [YOCTO #15473] (Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f) Signed-off-by: Kari Sivonen <kari.sivonen@live.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: add upstream latest version check functionAlexander Kanavin9 days1-0/+9
| | | | | | | | | | | This is actually rather easy: crate web API provides a json with all the versions, for example: https://crates.io/api/v1/crates/cargo-c/versions (Bitbake rev: f6c2755db9a1f88c8534193b420fa31d135945e6) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gcp: Add missing runfetchcmd importSven Schwermer2024-04-201-0/+1
| | | | | | | | | This adds the missing import. This bug was introduced with 1ab1d36c. (Bitbake rev: 97ffe14311407f6e705ec24b70870ab32f0637b9) Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-04-111-1/+4
| | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Install Git LFS in local repository configDerek Erdmann2024-03-221-1/+1
| | | | | | | | | | | | | | | | | Git uses a lock file to prevent concurrent modifications to the global config, so if unpack tasks for different recipes try to run "git lfs install" simultaneously the operation can fail: error: could not lock config file /home/build/.gitconfig: File exists exit status 255 Run `git lfs install --force` to reset Git configuration. Adding "--local" sets the smudge and clean filters in the local repository's config instead of modifying the user's global config. (Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7) Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix misleading "no output" msgViswanath Kraleti2024-03-221-1/+4
| | | | | | | | | | | When a command is run with a non-null log, errors are only output to the log and are not returned in the exception. In that case direct users to that logfile instead of telling the command had no output. (Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a) Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle URIs with single-valued query parametersRoss Burton2024-03-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Whilst typically the URI query is a list of key-value pairs, that's not actually required by the URI specification. For example: http://example.com/foo?bar is a valid query, but this will result in the fetcher raising an exception: File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__ self.query = self._param_str_split(urlp.query, "&") File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]: ValueError: not enough values to unpack (expected 2, got 1) In this case the query is just "bar", but the fetcher is trying to split it into a key-value pair. The URI object exposes the parsed query explicitly as a dictionary of key-value pairs, so we have to be a little creative here: if a value is None then it isn't a key-value pair, but a bare key. Fix this by handling elements without the deliminator in _param_str_split() (by assigning the value to None), and handle a None value when formatting the query in _param_str_join(). (Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/git: Escape parentheses in git src namePhilippe Rivest2024-03-221-1/+1
| | | | | | | | | | | | | | | | | | FIXES [YOCTO #15404] When using git fetcher on a repo with parentheses in its URL, the invocation of the git clone command will fail. The clone directory is not quoted thus the shell will return an error and won't execute the command. (Bitbake rev: b5624ee5643d881afa004571a096a189ab5389b5) Cc: Philippe Rivest <privest@genetec.com> (Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6) Signed-off-by: Philippe Rivest <technophil98@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Avoid clean upon failureRichard Purdie2024-03-032-2/+12
| | | | | | | | | | | | | | Currently when git fetches fail, it destroys all the existing local clone data. For large repositories this can introduce long build delays when for example, you just typo'd the git revision hash. The git fetcher should be able to recover most directories so when the fetch is for a git repo, avoid removing things unless clean is explicitly called (e.g. a -c cleanall task). (Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure that git LFS objects are availablePhilip Lorenz2024-02-231-2/+43
| | | | | | | | | | | | | | | | | | | | | The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "bitbake: wget.py: always use the custom user agent"Alexander Kanavin2024-02-201-4/+3
| | | | | | | | | | | | | | | | | | This reverts commit 987ab2a446aab235d6e86e97c05f25fb800d7acc. There's been a report that this breaks downloads from Jfrog Artifactory as self.user_agent is set to 'Mozilla Firefox', and when Artifactory sees that, it sends a response tailored for showing in an interactive browser (which in my opinion it has every right to). If we're using wget, we should say so via wget's default; handling uncooperative servers should be done on per-recipe basis, and ideally with tickets to admins of those servers. (Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Make latest_versionstring extract tags with slashes ↵Peter Kjellerstedt2024-02-191-2/+7
| | | | | | | | | | | | | | correctly Before, everything up to the last slash was removed when extracting the names of the tags. This would lead to that a tag such as "agent/11.0.0" would be incorrectly identified as "11.0.0", which would then be treated as a correct version matching "^(?P<pver>\d+(\.\d+)+)". (Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: A bit of clean-up of latest_versionstring()Peter Kjellerstedt2024-02-191-11/+10
| | | | | | | | | This is mostly preparations for the next commit. (Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: add comment in try_premirrorsChen Qi2024-02-191-0/+5
| | | | | | | | | | The purpose of ensuring 'incremental fetch' is not easy to see from the codes. So add comments to explain this. (Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: fix a corner case in try_premirrorChen Qi2024-02-191-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For gitsm recipes, it's possible that some URL is used more than once. e.g., A -> B:rev1 (B is a submodule of A) A -> C (C is a submodule of A) C -> B:rev2 (B is a submodule of C) A anc C are both using B as submodules, but on different revs. Now if we have: B:rev1 -> D B:rev2 -> E Then, the mirror will not be fully used. Say we have all repo mirrors for A, B, C, D, E, then in theory it's not necessary to reach out to any network for downloading. But it's not the case. After downloading B(rev1) and its submodule D from mirrors, the fetch process continues to download C, thus B(rev2) and E. Now it finds that B needs an update because its submodule E needs an update. Of course this is true because E is not downloaded yet. Now the problem comes to whether to use mirror or not. The git.py defines try_premirror to return 'False' when the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and try_mirror returns False, resulting in not using mirror and going to upstream directly. We can see that the mirrors are not fully used. This is usually not problem, as the cost is only some network download. But in case the following two settings are there, we get errors. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.allowed.domain" In such case, the gitsm recipe A will fail to fetch. Note that all contents that A needs are in mirrors and now it's failing to fetch. This is unexpected. Note that the different revs of the same repo in gitsm recipe is not the only way to reveal this problem. For example, there might be a recipe call B that uses B:rev3. Check the protobuf and grpc recipes as an example. For now, we can use the following steps to reproduce this issue. To be clear, the grpc recipe in meta-oe is now 1.60.0. 1. Add in local.conf: DL_DIR = "${TOPDIR}/downloads-premirror" bitbake grpc -c fetch 2. Comment out the DL_DIR setting in local.conf and add the following lines: PREMIRRORS:append = " \ git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ " 3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'. This command succeeds and this shows that the premirror holds everything we need. 4. Add the following lines and then 'bitbake grpc -c fetch'. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.domain" After step 4, the error message is as below: ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used This patch fixes this problem by handling this corner case, that is, if the URL is not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should try premirrors because trying to reach upstream is destined to fail. (Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: Fetch mirror into HEADPavel Zhukov2024-02-191-1/+5
| | | | | | | | | | | | | | | Fix the issue with using of (pre)mirror in case if clonedir exists but outdated. Previous version of the code fetched new mirror content into FETCH_HEAD instead of branch which caused refetch from the upstream. Add new remote add fetch from it instead so the ref can be found by "_contains_ref" Fixes [Yocto #15369] (Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git2: support git's safe.bareRepositoryAndré Draszik2024-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the bitbake git fetcher fails miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. While at it, fix one open-coded invocation of git that wasn't using ud.basecmd Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: always use the custom user agentThomas Perrot2024-02-101-3/+4
| | | | | | | | | | | | | | | | | | | Add the "--user-agent" paramater in the wget base command to perform all wget commands with this parameter, because a few HTTP servers block requests with the default wget user agent. For example, "hg.openjdk.org" never send a response to requests have been sent with wget: wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73 Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected. HTTP request sent, awaiting response... (Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c) Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: support temporary AWS credentialsToni Lammi2024-02-091-0/+1
| | | | | | | | | | | | | Support AWS_SESSION_TOKEN which is used in temporary AWS credentials. Fixes [YOCTO #15384]. (Bitbake rev: ae1e4c90bbc2002cb2728c64649c095c00220ceb) Signed-off-by: Toni Lammi <toni.lammi@kone.com> Reported-by: Toni Lammi <toni.lammi@tl-software.fi> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget/checkstatus(): include the URL in debugging output about ↵Alexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | status check failure Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/checkstatus(): do not print the URI twice in FetchError exceptionAlexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | Previously, there was duplicate clutter in the output, particularly if the URI points to sstate cache items: bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work (Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure GCP fetcher checks if file exists before download.Charlie Johnston2023-12-021-2/+5
| | | | | | | | | | | | | | | | | The GCP fetcher was calling bb.fetch2.check_network_access with "gsutil stat" as the command, but then never actually ran that command to check if the file exists. In cases where the file did not exist in a gs:// premirror, this would lead to an unhandled exception from do_fetch when the GCP python API tried to perform the download. This change resolves that issue by adding a runfetchcmd to call gsutil. (Bitbake rev: 1ab1d36c0af6fc58a974106b61ff4d37da6cb229) Signed-off-by: Charlie Johnston <charlie.johnston@loftorbital.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/aws: forward env-vars used in gitlab-ci K8sFelix Moessbauer2023-11-231-0/+2
| | | | | | | | | | | | | | | | | | | This patch adds the following variables to the allow-list, which are used in the "IAM roles for AWS when using the GitLab chart": - AWS_ROLE_ARN - AWS_WEB_IDENTITY_TOKEN_FILE These variables are set in the CI job environment and are needed to access the sstate cache artifacts in a connected S3 bucket. [1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com> (Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: add missing destsuffix and subpath parameters in ↵Julien Stephan2023-11-231-0/+8
| | | | | | | | | | | | docstrings Docstring for git fetcher is missing destsuffix and subpath parameters, so add them (Bitbake rev: 7cea6c2efefbb0f789f654c04a61d9ba770c148b) Signed-off-by: Julien Stephan <jstephan@baylibre.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2: git: Update Git-LFS download and testsDesone Burns2023-11-231-2/+12
| | | | | | | | | | | | | | | | | | | | When downloading a Git repository containing an LFS, the Git hooks are not always initialized correctly to perform the download. This change updates the Git downloader to run the "git lfs install" command in order to "smudge" the LFS files. The tests have also been updated to do checks for situations in which git lfs is not installed, as the application was required to be installed for any of the tests to run previously. The Git LFS functionality was working to some degree previously, but this change also updates the fetcher to allow LFS downloads for nobranch SRC_URIs. (Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217) Signed-off-by: Desone Burns II <dburns@seegrid.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: fix lfs fetch with destsuffix paramRagesh Nair2023-10-231-2/+2
| | | | | | | | | | | | | | | Pre-fetching of LFS blobs now also works if a destsuffix parameter is supplied. This also fixes issues with Yocto Kirkstone (and newer) builds, where fetching the LFS blobs is retried unsuccessfully during do_unpack, as network access is blocked for any task other than do_fetch. (Bitbake rev: e411dc07d032be6811d0393c50a06fc28e669b24) Signed-off-by: Ragesh Nair <ragesh.nair@duagon.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add API for upstream source tracingAlberto Pianon2023-10-157-0/+91
| | | | | | | | | | | | | | | | | This patch adds an API to bb.fetch2 to enable users to plug in an unpack tracer that can trace each source file back to its corresponding upstream source url, even when multiple upstream sources are combined together in the same unpack directory. This may be required for software composition analysis, license compliance, and detailed SBoM generation. This patch provides only the needed hooks in bb.fetch2 code and a dummy abstract class defining the API; users may load their own unpack tracer class by setting the BB_UNPACK_TRACER_CLASS config parameter. (Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f) Signed-off-by: Alberto Pianon <alberto@pianon.eu> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Adds vscode devcontainer supportMatthias Schnelte2023-09-261-0/+1
| | | | | | | | | | | | | | | | Visual studio code is injecting a git credential helper into every dev container. This helper is forwarding the git credentials from the host into the container. In order for this helper to work the REMOTE_CONTAINERS_IPC needs to be available in the environment for all git operations that require authentication. (Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963) Signed-off-by: Matthias Schnelte <develop@schnelte.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: Use path_is_descendant() instead of path for repo checkJoshua Watt2023-09-141-9/+6
| | | | | | | | | | | | | | Using path prefixes to check if the git directory is a descendant of the clone directory can be easily confused with symlinkes and bind mounts, causing directories to be deleted unnecessarily. Instead, use bb.utils.path_is_descendant() which is immune to the these sorts of problems. (Bitbake rev: b4d7a0546630620480b7fee159b84c3506e941a2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: Remove useless try..else clauseJoshua Watt2023-09-021-4/+4
| | | | | | | | | | There is no reason to have the else clause in this try block, as it can be moved into the try block, which is clearer. (Bitbake rev: 5625849e9327fc71a38eea00d4506f80abc11bc6) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: Check if clone directory is a git repoJoshua Watt2023-08-301-1/+29
| | | | | | | | | | | | | | | | | If the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo. (Bitbake rev: 2117db3146ce38bb4a6e2df40b6cd2ab11b514d5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: tolerate git-lfs in submodulesRandolph Sapp2023-08-301-4/+7
| | | | | | | | | | | | | Explicitly pass down the lfs parameter from the source repo URI to submodules. Add smudge skip to final checkout phase to make sure we don't access the network here. Everything should have been fetched and setup from the lfs logic in the git fetcher at this point. (Bitbake rev: 1f8f21fe024b391d3a6670c64b839db0eee083c3) Signed-off-by: Randolph Sapp <rs@ti.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add new srcrev fetcher APIRichard Purdie2023-08-241-5/+24
| | | | | | | | | | | | | | Add new functions to return some of the get_srcrev data in new and different ways. We need two different forms of the data, one is a string to inject into PKGV, the other is the full revisions as a string to include in hash computations so that the hash changes when the input revisions change. This allows us to clean up and simplify the code in OE-Core and move the version information from PV to PKGV. (Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add Google Cloud Platform (GCP) fetcherEmil Ekmečić2023-08-112-1/+101
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fetcher allows BitBake to fetch from a Google Cloud Storage bucket. The fetcher expects a gs:// URI of the following form: SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH" The fetcher uses the Google Cloud Storage Python Client, and expects it to be installed, configured, and authenticated prior to use. If accepted, this patch should merge in with the corresponding oe-core patch titled "Add GCP fetcher to list of supported protocols". Some comments on the patch: There is also documentation for the fetcher added to the User Manual. I'm still not completely sure about the recommends_checksum() being set to True. As I've noted in the mailing list, it will throw warnings if the fetcher is used in recipes without specifying a checksum. Please let me know if this is intended behavior or if it should be modified. Here is how this fetcher conforms to the fetcher expectations described at this link: https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README a) Yes, network fetching only happens in the fetcher b) The fetcher has nothing to do with the unpack phase so there is no network access there c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher only downloads to the DL_DIR in the same way that other fetchers, namely the S3 and Azure fetchers do. d) The fetcher is identical to the S3 and Azure fetchers in this context e) Yes, the fetcher output is deterministic because it is downloading tarballs from a bucket and not modifying them in any way. f) I set up a local proxy using tinyproxy and set the http_proxy variable to test whether the Python API respected the proxy. It appears that it did as I could see traffic passing through the proxy. I also did some searching online and found posts indicating that the Google Cloud Python APIs supported the classic Linux proxy variables, namely: - https://github.com/googleapis/google-api-python-client/issues/1260 g) Access is minimal, only checking if the file exists and downloading it if it does. h) Not applicable, BitBake already knows which version it wants and the version infomation is encoded in the filename. The fetcher has no concept of versions. i) Not applicable j) Not applicable k) No tests were added as part of this change. I didn't see any tests for the S3 or Azure changes either, is that OK? l) I'm not 100% familiar but I don't believe this fetcher is using any tools during parse time. Please correct me if I'm wrong. (Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b) Signed-off-by: Emil Ekmečić <eekmecic@snap.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Check if path is 'None' before calculating checksumsBELOUARGA Mohamed2023-08-041-0/+3
| | | | | | | | | | | Add one more verification that checks if localpath is None, because we can't compute checksum of a None. (Bitbake rev: 47ab9d21171a834cbac3d1ce368d59fd71d14452) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/npmsw: Check if there are dependencies before ↵BELOUARGA Mohamed2023-08-041-7/+9
| | | | | | | | | | | | | trying to fetch them When there are no dependencies, _foreach_proxy_method does not verify that there are dependencies to fetch before fetching them. (Bitbake rev: 48a102e49448656ef25fb689af7b0971fde523e3) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: Document that we won't support propagating user parameterYoann Congal2023-07-291-0/+6
| | | | | | | | | [YOCTO #13550] (Bitbake rev: 5e45b8eab60d651c98a950533043a4c96b9c8b01) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Set maxsplit to match expected variablesDit Kozmaj2023-07-271-1/+1
| | | | | | | | | | | | Set the maxsplit value to match the expected number of variables. This also avoids an unnecessary split as the parameters are in the form 'key=value' and the 'value' could contain the '=' character. (Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19) Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: Support old and new shrinkwrap formatsRichard Purdie2023-06-301-9/+26
| | | | | | | | | | | | | "fetch2/npmsw: Add support for the new format of the shrinkwrap file" added support for the new format shrinkwrap files but this regressed our tests which still use the old format. Similar to how npm handles this, support both for now until we can migrate our tests. (Bitbake rev: 9941b480a0e2a8b57f2ed069cd583f2784394a2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: Remove special caracters that causes recipe tool to failBELOUARGA Mohamed2023-06-291-3/+6
| | | | | | | | | | Packages like @(._.)/execute causes problems because they generate names that are not supported by yocto (Bitbake rev: b2e1be67d2acca27451bed59874bc1c2a7ec44a6) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: Don't fetch dev dependencies when they are not demandedBELOUARGA Mohamed2023-06-291-0/+2
| | | | | | | | | | Dev dependencies should not be fetched only if it is specified in the recipe. (Bitbake rev: 224c2bb1fd3f16ac4063dc0eb8ba43eee34ba782) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: Add support for the new format of the shrinkwrap fileBELOUARGA Mohamed2023-06-291-16/+9
| | | | | | | | | | | | | | Npm is a package manager that has its own manner to handle installation of packages. But it is not yocto friendly, for instance NPM fetch dependencies in the middle of compilation. The shrinkwrap file changed its format over npm versions, but npm does not version this file, so we can use it properly. The actual changes make NPM depencies work with the actual shrinkwrap format. (Bitbake rev: 19b9f7f0f451a636f3fdcdc1bb283ab431ede612) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: evaluate PATH before patching HOMEArno Baumfalk2023-05-251-0/+1
| | | | | | | | | | | | | When the user configures the TMPDIR variable in a way that it contains $HOME (in local.conf), the DataSmart (d) object's PATH variable will also contain $HOME. As the NpmEnvironment patches HOME with a temporary directory, further evaluation of PATH will become invalid an the build breaks with an error (npm not found). As countermeasure PATH is evaluated *before* patching HOME. (Bitbake rev: 893f3b116b628457fe0bfaf4b8394b1cd05a90a5) Signed-off-by: Arno Baumfalk <a.baumfalk@astro-kom.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/local: Add debug to include chosen pathRichard Purdie2023-05-251-0/+2
| | | | | | | | | | | The current debug lists all the potential search paths but not the end result which isn't helpful for debugging. Tweak to include the end result. (Bitbake rev: a98b8dbb5fbd937a9530793485cf7c68fdd75cc2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Clarify different localpaths functionsRichard Purdie2023-05-252-3/+3
| | | | | | | | | | | | To avoid confusion, rename the local fetcher localpaths function to something more specific so it doesn't appear similar to the localpaths top level function. (Bitbake rev: ada5ea712cc42fff8c55bd261ee6d7579efd4a68) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Correct unpack for a crate that matches the recipe namePeter Kjellerstedt2023-05-251-2/+2
| | | | | | | | | | | | | | | | | | | The crate fetcher handles a crate with a name that matches the recipe's name specially by placing the unpacked code in the current directory (which typically is ${S}) rather than together with the sources for the other crates. This broke when the URI names for all crates were changed recently to include the version in the name. Correct the crate fetcher to test against ${BP} instead of ${BPN}. Also add a test case to the selftests to avoid this breaking again. [Yocto #15012] (Bitbake rev: 9af52b748357d0c843ce2507ce4d119fd9c37008) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Simplify extraction of crate names and versions from URIsPeter Kjellerstedt2023-05-251-3/+3
| | | | | | | | (Bitbake rev: 4f2d8bcbaea33425f6de2c3c88a0beabec8a59f9) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npmsw fetcher: Avoid instantiating Fetch class if url list is emptySvend Meyland Nicolaisen2023-04-201-1/+3
| | | | | | | | | | | | | | | | Recipes containing both git and npmsw sources in the SRC_URI fail during fetch from the shrinkwrap. It seems that when the fetcher is fetching from the shrinkwrap, the SRCREV variable has been deleted but it till ends up fetching from the git source resulting in an error because SRCREV is undefined. The root cause of this is that the Fetch class defaults to urls from the SRC_URI when the urls parameter contains an empty list. This patch will ensure that Fetch is not instantiated if the urls list is empty. (Bitbake rev: e602963dfd505eef08702366383358d29ee20c4d) Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: create versioned 'name' entriesEnrico Scholz2023-04-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | It is common for rust packages to depend on different versions of the same crate. E.g. | crate://crates.io/windows_x86_64_msvc/0.42.2 \ | crate://crates.io/windows_x86_64_msvc/0.48.0 \ Identification only by the plain crate name makes the sha256sum ambiguous | SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" | SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a" and requires lot of manual work to fix the SRC_URI list. Making the 'name' property unique by appending the version allows direct copy & paste of reported sha256sum errors to complete the crates list. (Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Display all missing checksum at onceFrederic Martinsons2023-04-041-8/+10
| | | | | | | | | | | | | | | | | | | | | | | | Instead of only display the first and stop. For recipe (crago based) that can contains several artifacts to fetch with their checksum, it will particularly handy to display all of missing one. An example of error message would be NOTE: Executing Tasks ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe: SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4" SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b" SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9" SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b" ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe') ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025 ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1' (Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-17 15:38:53 +0000