summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch/wget/checkstatus(): include the URL in debugging output about ↵Alexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | status check failure Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/checkstatus(): do not print the URI twice in FetchError exceptionAlexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | Previously, there was duplicate clutter in the output, particularly if the URI points to sstate cache items: bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work (Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure GCP fetcher checks if file exists before download.Charlie Johnston2023-12-021-2/+5
| | | | | | | | | | | | | | | | | The GCP fetcher was calling bb.fetch2.check_network_access with "gsutil stat" as the command, but then never actually ran that command to check if the file exists. In cases where the file did not exist in a gs:// premirror, this would lead to an unhandled exception from do_fetch when the GCP python API tried to perform the download. This change resolves that issue by adding a runfetchcmd to call gsutil. (Bitbake rev: 1ab1d36c0af6fc58a974106b61ff4d37da6cb229) Signed-off-by: Charlie Johnston <charlie.johnston@loftorbital.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/aws: forward env-vars used in gitlab-ci K8sFelix Moessbauer2023-11-231-0/+2
| | | | | | | | | | | | | | | | | | | This patch adds the following variables to the allow-list, which are used in the "IAM roles for AWS when using the GitLab chart": - AWS_ROLE_ARN - AWS_WEB_IDENTITY_TOKEN_FILE These variables are set in the CI job environment and are needed to access the sstate cache artifacts in a connected S3 bucket. [1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com> (Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: add missing destsuffix and subpath parameters in ↵Julien Stephan2023-11-231-0/+8
| | | | | | | | | | | | docstrings Docstring for git fetcher is missing destsuffix and subpath parameters, so add them (Bitbake rev: 7cea6c2efefbb0f789f654c04a61d9ba770c148b) Signed-off-by: Julien Stephan <jstephan@baylibre.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2: git: Update Git-LFS download and testsDesone Burns2023-11-231-2/+12
| | | | | | | | | | | | | | | | | | | | When downloading a Git repository containing an LFS, the Git hooks are not always initialized correctly to perform the download. This change updates the Git downloader to run the "git lfs install" command in order to "smudge" the LFS files. The tests have also been updated to do checks for situations in which git lfs is not installed, as the application was required to be installed for any of the tests to run previously. The Git LFS functionality was working to some degree previously, but this change also updates the fetcher to allow LFS downloads for nobranch SRC_URIs. (Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217) Signed-off-by: Desone Burns II <dburns@seegrid.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: fix lfs fetch with destsuffix paramRagesh Nair2023-10-231-2/+2
| | | | | | | | | | | | | | | Pre-fetching of LFS blobs now also works if a destsuffix parameter is supplied. This also fixes issues with Yocto Kirkstone (and newer) builds, where fetching the LFS blobs is retried unsuccessfully during do_unpack, as network access is blocked for any task other than do_fetch. (Bitbake rev: e411dc07d032be6811d0393c50a06fc28e669b24) Signed-off-by: Ragesh Nair <ragesh.nair@duagon.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add API for upstream source tracingAlberto Pianon2023-10-157-0/+91
| | | | | | | | | | | | | | | | | This patch adds an API to bb.fetch2 to enable users to plug in an unpack tracer that can trace each source file back to its corresponding upstream source url, even when multiple upstream sources are combined together in the same unpack directory. This may be required for software composition analysis, license compliance, and detailed SBoM generation. This patch provides only the needed hooks in bb.fetch2 code and a dummy abstract class defining the API; users may load their own unpack tracer class by setting the BB_UNPACK_TRACER_CLASS config parameter. (Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f) Signed-off-by: Alberto Pianon <alberto@pianon.eu> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Adds vscode devcontainer supportMatthias Schnelte2023-09-261-0/+1
| | | | | | | | | | | | | | | | Visual studio code is injecting a git credential helper into every dev container. This helper is forwarding the git credentials from the host into the container. In order for this helper to work the REMOTE_CONTAINERS_IPC needs to be available in the environment for all git operations that require authentication. (Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963) Signed-off-by: Matthias Schnelte <develop@schnelte.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: Use path_is_descendant() instead of path for repo checkJoshua Watt2023-09-141-9/+6
| | | | | | | | | | | | | | Using path prefixes to check if the git directory is a descendant of the clone directory can be easily confused with symlinkes and bind mounts, causing directories to be deleted unnecessarily. Instead, use bb.utils.path_is_descendant() which is immune to the these sorts of problems. (Bitbake rev: b4d7a0546630620480b7fee159b84c3506e941a2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: Remove useless try..else clauseJoshua Watt2023-09-021-4/+4
| | | | | | | | | | There is no reason to have the else clause in this try block, as it can be moved into the try block, which is clearer. (Bitbake rev: 5625849e9327fc71a38eea00d4506f80abc11bc6) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: Check if clone directory is a git repoJoshua Watt2023-08-301-1/+29
| | | | | | | | | | | | | | | | | If the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo. (Bitbake rev: 2117db3146ce38bb4a6e2df40b6cd2ab11b514d5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: tolerate git-lfs in submodulesRandolph Sapp2023-08-301-4/+7
| | | | | | | | | | | | | Explicitly pass down the lfs parameter from the source repo URI to submodules. Add smudge skip to final checkout phase to make sure we don't access the network here. Everything should have been fetched and setup from the lfs logic in the git fetcher at this point. (Bitbake rev: 1f8f21fe024b391d3a6670c64b839db0eee083c3) Signed-off-by: Randolph Sapp <rs@ti.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add new srcrev fetcher APIRichard Purdie2023-08-241-5/+24
| | | | | | | | | | | | | | Add new functions to return some of the get_srcrev data in new and different ways. We need two different forms of the data, one is a string to inject into PKGV, the other is the full revisions as a string to include in hash computations so that the hash changes when the input revisions change. This allows us to clean up and simplify the code in OE-Core and move the version information from PV to PKGV. (Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add Google Cloud Platform (GCP) fetcherEmil Ekmečić2023-08-112-1/+101
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fetcher allows BitBake to fetch from a Google Cloud Storage bucket. The fetcher expects a gs:// URI of the following form: SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH" The fetcher uses the Google Cloud Storage Python Client, and expects it to be installed, configured, and authenticated prior to use. If accepted, this patch should merge in with the corresponding oe-core patch titled "Add GCP fetcher to list of supported protocols". Some comments on the patch: There is also documentation for the fetcher added to the User Manual. I'm still not completely sure about the recommends_checksum() being set to True. As I've noted in the mailing list, it will throw warnings if the fetcher is used in recipes without specifying a checksum. Please let me know if this is intended behavior or if it should be modified. Here is how this fetcher conforms to the fetcher expectations described at this link: https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README a) Yes, network fetching only happens in the fetcher b) The fetcher has nothing to do with the unpack phase so there is no network access there c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher only downloads to the DL_DIR in the same way that other fetchers, namely the S3 and Azure fetchers do. d) The fetcher is identical to the S3 and Azure fetchers in this context e) Yes, the fetcher output is deterministic because it is downloading tarballs from a bucket and not modifying them in any way. f) I set up a local proxy using tinyproxy and set the http_proxy variable to test whether the Python API respected the proxy. It appears that it did as I could see traffic passing through the proxy. I also did some searching online and found posts indicating that the Google Cloud Python APIs supported the classic Linux proxy variables, namely: - https://github.com/googleapis/google-api-python-client/issues/1260 g) Access is minimal, only checking if the file exists and downloading it if it does. h) Not applicable, BitBake already knows which version it wants and the version infomation is encoded in the filename. The fetcher has no concept of versions. i) Not applicable j) Not applicable k) No tests were added as part of this change. I didn't see any tests for the S3 or Azure changes either, is that OK? l) I'm not 100% familiar but I don't believe this fetcher is using any tools during parse time. Please correct me if I'm wrong. (Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b) Signed-off-by: Emil Ekmečić <eekmecic@snap.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Check if path is 'None' before calculating checksumsBELOUARGA Mohamed2023-08-041-0/+3
| | | | | | | | | | | Add one more verification that checks if localpath is None, because we can't compute checksum of a None. (Bitbake rev: 47ab9d21171a834cbac3d1ce368d59fd71d14452) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/npmsw: Check if there are dependencies before ↵BELOUARGA Mohamed2023-08-041-7/+9
| | | | | | | | | | | | | trying to fetch them When there are no dependencies, _foreach_proxy_method does not verify that there are dependencies to fetch before fetching them. (Bitbake rev: 48a102e49448656ef25fb689af7b0971fde523e3) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: Document that we won't support propagating user parameterYoann Congal2023-07-291-0/+6
| | | | | | | | | [YOCTO #13550] (Bitbake rev: 5e45b8eab60d651c98a950533043a4c96b9c8b01) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Set maxsplit to match expected variablesDit Kozmaj2023-07-271-1/+1
| | | | | | | | | | | | Set the maxsplit value to match the expected number of variables. This also avoids an unnecessary split as the parameters are in the form 'key=value' and the 'value' could contain the '=' character. (Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19) Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: Support old and new shrinkwrap formatsRichard Purdie2023-06-301-9/+26
| | | | | | | | | | | | | "fetch2/npmsw: Add support for the new format of the shrinkwrap file" added support for the new format shrinkwrap files but this regressed our tests which still use the old format. Similar to how npm handles this, support both for now until we can migrate our tests. (Bitbake rev: 9941b480a0e2a8b57f2ed069cd583f2784394a2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: Remove special caracters that causes recipe tool to failBELOUARGA Mohamed2023-06-291-3/+6
| | | | | | | | | | Packages like @(._.)/execute causes problems because they generate names that are not supported by yocto (Bitbake rev: b2e1be67d2acca27451bed59874bc1c2a7ec44a6) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: Don't fetch dev dependencies when they are not demandedBELOUARGA Mohamed2023-06-291-0/+2
| | | | | | | | | | Dev dependencies should not be fetched only if it is specified in the recipe. (Bitbake rev: 224c2bb1fd3f16ac4063dc0eb8ba43eee34ba782) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: Add support for the new format of the shrinkwrap fileBELOUARGA Mohamed2023-06-291-16/+9
| | | | | | | | | | | | | | Npm is a package manager that has its own manner to handle installation of packages. But it is not yocto friendly, for instance NPM fetch dependencies in the middle of compilation. The shrinkwrap file changed its format over npm versions, but npm does not version this file, so we can use it properly. The actual changes make NPM depencies work with the actual shrinkwrap format. (Bitbake rev: 19b9f7f0f451a636f3fdcdc1bb283ab431ede612) Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: evaluate PATH before patching HOMEArno Baumfalk2023-05-251-0/+1
| | | | | | | | | | | | | When the user configures the TMPDIR variable in a way that it contains $HOME (in local.conf), the DataSmart (d) object's PATH variable will also contain $HOME. As the NpmEnvironment patches HOME with a temporary directory, further evaluation of PATH will become invalid an the build breaks with an error (npm not found). As countermeasure PATH is evaluated *before* patching HOME. (Bitbake rev: 893f3b116b628457fe0bfaf4b8394b1cd05a90a5) Signed-off-by: Arno Baumfalk <a.baumfalk@astro-kom.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/local: Add debug to include chosen pathRichard Purdie2023-05-251-0/+2
| | | | | | | | | | | The current debug lists all the potential search paths but not the end result which isn't helpful for debugging. Tweak to include the end result. (Bitbake rev: a98b8dbb5fbd937a9530793485cf7c68fdd75cc2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Clarify different localpaths functionsRichard Purdie2023-05-252-3/+3
| | | | | | | | | | | | To avoid confusion, rename the local fetcher localpaths function to something more specific so it doesn't appear similar to the localpaths top level function. (Bitbake rev: ada5ea712cc42fff8c55bd261ee6d7579efd4a68) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Correct unpack for a crate that matches the recipe namePeter Kjellerstedt2023-05-251-2/+2
| | | | | | | | | | | | | | | | | | | The crate fetcher handles a crate with a name that matches the recipe's name specially by placing the unpacked code in the current directory (which typically is ${S}) rather than together with the sources for the other crates. This broke when the URI names for all crates were changed recently to include the version in the name. Correct the crate fetcher to test against ${BP} instead of ${BPN}. Also add a test case to the selftests to avoid this breaking again. [Yocto #15012] (Bitbake rev: 9af52b748357d0c843ce2507ce4d119fd9c37008) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Simplify extraction of crate names and versions from URIsPeter Kjellerstedt2023-05-251-3/+3
| | | | | | | | (Bitbake rev: 4f2d8bcbaea33425f6de2c3c88a0beabec8a59f9) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npmsw fetcher: Avoid instantiating Fetch class if url list is emptySvend Meyland Nicolaisen2023-04-201-1/+3
| | | | | | | | | | | | | | | | Recipes containing both git and npmsw sources in the SRC_URI fail during fetch from the shrinkwrap. It seems that when the fetcher is fetching from the shrinkwrap, the SRCREV variable has been deleted but it till ends up fetching from the git source resulting in an error because SRCREV is undefined. The root cause of this is that the Fetch class defaults to urls from the SRC_URI when the urls parameter contains an empty list. This patch will ensure that Fetch is not instantiated if the urls list is empty. (Bitbake rev: e602963dfd505eef08702366383358d29ee20c4d) Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: create versioned 'name' entriesEnrico Scholz2023-04-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | It is common for rust packages to depend on different versions of the same crate. E.g. | crate://crates.io/windows_x86_64_msvc/0.42.2 \ | crate://crates.io/windows_x86_64_msvc/0.48.0 \ Identification only by the plain crate name makes the sha256sum ambiguous | SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" | SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a" and requires lot of manual work to fix the SRC_URI list. Making the 'name' property unique by appending the version allows direct copy & paste of reported sha256sum errors to complete the crates list. (Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Display all missing checksum at onceFrederic Martinsons2023-04-041-8/+10
| | | | | | | | | | | | | | | | | | | | | | | | Instead of only display the first and stop. For recipe (crago based) that can contains several artifacts to fetch with their checksum, it will particularly handy to display all of missing one. An example of error message would be NOTE: Executing Tasks ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe: SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4" SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b" SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9" SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b" ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe') ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025 ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1' (Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/local: Mention the value of localpath in failure messageRichard Purdie2023-03-301-1/+1
| | | | | | | | | | We have an obtuse rare and intermittent bitbake fetch failure. Understanding the value of ud.localpath at the time of failure would be helpful to debug it further. Tweak the debugging to provide this in the output. (Bitbake rev: 935ab36f64d25c09f83209cd369714c2bc89aa7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-03-301-2/+6
| | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: crate.py: make checksum verification mandatoryFrederic Martinsons2023-03-301-1/+1
| | | | | | | | (Bitbake rev: 9c07e377a21fe1d9660d084da93873acd289b581) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add checksum capability for crate fetcherFrederic Martinsons2023-03-301-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This change brings checksum verification of each crate in a recipe, e.g | SRC_URI += " \ | crate://crates.io/aho-corasick/0.7.20 \ | crate://crates.io/atomic-waker/1.1.0 \ | crate://crates.io/cc/1.0.79 \ | " | | SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac" | SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599" | SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f" That will require to move the checksum initialization after the possible call to urldata_init method in order for the crate fetcher to parse the url. Another way of doing could have been implementing a decodeurl method that would have been specific for each fetcher class. (Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/sftp: Fix fetching URIs with spacesDaniel Ammann2023-03-231-1/+1
| | | | | | | | | | Spaces can be used in URIs with %20, but the URI needs to be quoted. (Bitbake rev: 0f887589c279ce07692b4d8e118954a9d214ca45) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add autorev warning when it is set too lateRichard Purdie2023-03-222-3/+7
| | | | | | | | | | | | | | | | | | | | Bitbake expects a consistent metadata environment when it parses. There are plenty of ways you can set a recipe to autorev at a point where parts of the fetcher have already been triggered leading to obsure bugs which I struggled to debug, let alone anyone not familar with the code. If anyone is running into the message from the commit, the issue is likely one of timing. Keep in mind that the anonymous python code in base.bbclass will expand variables like FILESPATH, WORKDIR and others which contain PV. The recipe needs to be set to AUTOREV before that anonymous python runs. In particular, that means you can't set SRCREV = "${AUTOREV}" in other anonymous python, it needs to happen earlier. (Bitbake rev: 4d9ec332d5bfc8b60b54f8ec2a17d34e35aa903a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEENRichard Purdie2023-03-222-2/+2
| | | | | | | | | For readability of following patches, rename this internal variable to allow for others in a similar format. (Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: crate.py: authorize crate url with parametersFrederic Martinsons2023-03-171-3/+6
| | | | | | | | | | | | | | | | | This allow to have classic fetch parameters (like destsuffix, sha256, name ...) not being considered by crate fetcher itself (and so mess up its download) Moreover, it allow to overload the name of the downloaded crate (maybe usefull if there is a naming clash between two crates coming from different repositories) (Bitbake rev: 278bd2f1758b8af97552af8d23d16ffb5127a131) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Combine urlopener exceptionsMark Hatle2023-03-111-17/+1
| | | | | | | | | | | No reason to have three identical exception handles, refactor to catch any of the exceptions with the same block of code. (Bitbake rev: b29f6e04091b6bfe697dc41c76880de466736fc3) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Add catch TimeoutError exceptionMark Hatle2023-03-111-0/+8
| | | | | | | | | | | | | | We've observed TimeoutError exceptions during the sstate-cache mirror fetch, it appears that due to the number of (invalid) files requested the remote side is eventually dropping the connection (not closing it) which can result in a TimeoutError exception being sent, while rate it is different from the urllib.error.URLError or ConnectionResetError. (Bitbake rev: 6041b34740deee09ea65d705702555456a5e05d8) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/npmsw: add more short forms for git operationsMark Asselstine2023-02-261-7/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | >From the npm-install documentation [1] the CLI provides a set of short forms when the install fetches from git. These include "github:" example: npm install github:mygithubuser/myproject "gist:" example: npm install gist:101a11beef "gitlab:" example: npm install gitlab:mygitlabuser/myproject "bitbucket:" example: npm install bitbucket:mybitbucketuser/myproject Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] by Stefan Herbrechtsmeier added support for the "github:" but the others would marked as 'Unsupported dependency'. The other prefixes are added in this commit, along with extending the tests to cover some of these. However, there is one more short form for github which npm-install allows which forgoes the prefix altogether. example: npm install mygithubuser/myproject Unfortunately this format is a bit problematic as it lacks any easily identifiable 'marker' to match against, and it could be either the github short form or install from folder format. Experimentation shows that the folder format requires a leading './' or '/', so we use this to rule out the ambiguity. If this approach to folder and github formats disambiguation is incorrect it won't matter anyways as the folder format is unrecognized by the code as-is and thus with this change or without, things would fail. Since we have to be less strict in the check for git operations we move it to be the last install format which we check, such that the less ambiguous formats can be sorted out first. [1] https://docs.npmjs.com/cli/v9/commands/npm-install [Yocto #14236] (Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix path construction for relative submodule URIMike Crowe2023-02-261-1/+1
| | | | | | | | | | | | | | | | | | | The submodule repository URI contains a path to something not necessarily on the local filesystem. This means that we can't use realpath to normalise it without risking getting bad results if the path happens to match something on the local filesystem. This situation can cause very confusing errors if that matching local path happens to be a symlink to somewhere else. Using normpath rather than realpath means that the path simplification follows simple rules on the string rather than looking at the local filesystem and avoids problems. (Bitbake rev: 47b271e6c8d96960ebe70f80e58f30cc4cbf42e1) Signed-off-by: Mike Crowe <mac@mcrowe.com> Co-authored-by: Dave Craig Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Drop unused importRichard Purdie2023-02-261-1/+0
| | | | | | | | This import is no longer used anywhere so can be removed. (Bitbake rev: 956128e394581855bf0d03b32a975dc91c2a7e0c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARSCaner Altinbasak2023-02-261-1/+3
| | | | | | | | | These environment variables are needed by gclient and needed to be passed into fetcher. (Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Replace mkdtemp with TemporaryDirectory and avoid ↵Paulo Neves2023-02-221-4/+1
| | | | | | | | | | | | | | exception masking Due to using mkdtemp instead of TemporaryDirectory we needed to manually cleanup the directory in a try finally block. With tempfile.TemporaryDirectory we can handle the cleanup with a "with" statement and not need to manually clean up oursevels. (Bitbake rev: 36a231d2a45e7c87172196538d18ded81a234774) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Removed unused variables in _contains_lfsPaulo Neves2023-02-221-5/+0
| | | | | | | | | | branchname was set but never used in the context of _contains_lfs method. (Bitbake rev: a64ec64c2faf1ac6ce995cdd4e311d8be6046155) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: clean up netrc usageRoss Burton2023-02-171-4/+5
| | | | | | | | | | | Assigning a return value which is potentially None to a tuple and catching TypeError is pretty ugly. Rewrite the code to explicitly check the value for clarity. (Bitbake rev: f4ebb27616ac2df27c29a6052b1526a4c48db607) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add NODE_EXTRA_CA_CERTS to export listGeorge Kelly2023-02-091-0/+1
| | | | | | | | | | | | | | | | If you are behind a corporate proxy, the npm fetcher uses the proxy IP address already passed in the list of exports. However, it will error if the proxy injects its own root CA certificate. Thus, the NODE_EXTRA_CA_CERTS environment variable must be passed so the user can include their company's root CA as a trusted CA inside node's certificate store. (Bitbake rev: 5daecc25a59ce5bf6c31f5085e6caa7cb543517e) Signed-off-by: George Kelly <george.kelly1097@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache ↵Richard Purdie2023-01-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | location Currently the codeparser cache is set from CACHE, which is typically in bitbake.conf which means we can't read/write any cache until it is found/read. We may well have python expressions to parse before that happens. The net result is suboptimal functioning of the codeparser cache since it will often be invalidated by data that is never written. This patch changes the codeparser and filechecksum caches to use BB_CACHE as their setting and defaults it to ${TOPDIR}/cache. The patch doesn't change where the "persistent" data such as prserver and hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing cache resists (still currently CACHE). I've left those for a later patch. The patch does ensure data parsed by the core datastore parsing calls is written back since this is now much more useful after this change. (Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-01-10 12:06:23 +0000