linux/poky.git/bitbake/lib/bb/fetch2/init.py, branch nanbield

bitbake: fetch2: Add API for upstream source tracing

2023-10-15T08:12:43+00:00

This patch adds an API to bb.fetch2 to enable users to plug in an unpack tracer that can trace each source file back to its corresponding upstream source url, even when multiple upstream sources are combined together in the same unpack directory. This may be required for software composition analysis, license compliance, and detailed SBoM generation. This patch provides only the needed hooks in bb.fetch2 code and a dummy abstract class defining the API; users may load their own unpack tracer class by setting the BB_UNPACK_TRACER_CLASS config parameter. (Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f) Signed-off-by: Alberto Pianon Signed-off-by: Richard Purdie

bitbake: fetch2: Adds vscode devcontainer support

2023-09-26T09:38:45+00:00

Visual studio code is injecting a git credential helper into every dev container. This helper is forwarding the git credentials from the host into the container. In order for this helper to work the REMOTE_CONTAINERS_IPC needs to be available in the environment for all git operations that require authentication. (Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963) Signed-off-by: Matthias Schnelte Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2: Add new srcrev fetcher API

2023-08-24T15:49:38+00:00

Add new functions to return some of the get_srcrev data in new and different ways. We need two different forms of the data, one is a string to inject into PKGV, the other is the full revisions as a string to include in hash computations so that the hash changes when the input revisions change. This allows us to clean up and simplify the code in OE-Core and move the version information from PV to PKGV. (Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723) Signed-off-by: Richard Purdie

bitbake: fetch2: add Google Cloud Platform (GCP) fetcher

2023-08-11T15:23:01+00:00

This fetcher allows BitBake to fetch from a Google Cloud Storage bucket. The fetcher expects a gs:// URI of the following form: SSTATE_MIRRORS = "file://.* gs:///PATH" The fetcher uses the Google Cloud Storage Python Client, and expects it to be installed, configured, and authenticated prior to use. If accepted, this patch should merge in with the corresponding oe-core patch titled "Add GCP fetcher to list of supported protocols". Some comments on the patch: There is also documentation for the fetcher added to the User Manual. I'm still not completely sure about the recommends_checksum() being set to True. As I've noted in the mailing list, it will throw warnings if the fetcher is used in recipes without specifying a checksum. Please let me know if this is intended behavior or if it should be modified. Here is how this fetcher conforms to the fetcher expectations described at this link: https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README a) Yes, network fetching only happens in the fetcher b) The fetcher has nothing to do with the unpack phase so there is no network access there c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher only downloads to the DL_DIR in the same way that other fetchers, namely the S3 and Azure fetchers do. d) The fetcher is identical to the S3 and Azure fetchers in this context e) Yes, the fetcher output is deterministic because it is downloading tarballs from a bucket and not modifying them in any way. f) I set up a local proxy using tinyproxy and set the http_proxy variable to test whether the Python API respected the proxy. It appears that it did as I could see traffic passing through the proxy. I also did some searching online and found posts indicating that the Google Cloud Python APIs supported the classic Linux proxy variables, namely: - https://github.com/googleapis/google-api-python-client/issues/1260 g) Access is minimal, only checking if the file exists and downloading it if it does. h) Not applicable, BitBake already knows which version it wants and the version infomation is encoded in the filename. The fetcher has no concept of versions. i) Not applicable j) Not applicable k) No tests were added as part of this change. I didn't see any tests for the S3 or Azure changes either, is that OK? l) I'm not 100% familiar but I don't believe this fetcher is using any tools during parse time. Please correct me if I'm wrong. (Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b) Signed-off-by: Emil Ekmečić Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2: Check if path is 'None' before calculating checksums

2023-08-04T10:48:26+00:00

Add one more verification that checks if localpath is None, because we can't compute checksum of a None. (Bitbake rev: 47ab9d21171a834cbac3d1ce368d59fd71d14452) Signed-off-by: BELOUARGA Mohamed Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2: Set maxsplit to match expected variables

2023-07-27T20:09:14+00:00

Set the maxsplit value to match the expected number of variables. This also avoids an unnecessary split as the parameters are in the form 'key=value' and the 'value' could contain the '=' character. (Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19) Signed-off-by: Dit Kozmaj Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2: Clarify different localpaths functions

2023-05-25T09:33:15+00:00

To avoid confusion, rename the local fetcher localpaths function to something more specific so it doesn't appear similar to the localpaths top level function. (Bitbake rev: ada5ea712cc42fff8c55bd261ee6d7579efd4a68) Signed-off-by: Richard Purdie Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2: Display all missing checksum at once

2023-04-04T12:12:23+00:00

Instead of only display the first and stop. For recipe (crago based) that can contains several artifacts to fetch with their checksum, it will particularly handy to display all of missing one. An example of error message would be NOTE: Executing Tasks ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe: SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4" SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b" SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9" SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b" ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe') ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025 ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1' (Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643) Signed-off-by: Frederic Martinsons Signed-off-by: Richard Purdie

bitbake: fetch2: Add checksum capability for crate fetcher

2023-03-30T09:51:54+00:00

This change brings checksum verification of each crate in a recipe, e.g | SRC_URI += " \ | crate://crates.io/aho-corasick/0.7.20 \ | crate://crates.io/atomic-waker/1.1.0 \ | crate://crates.io/cc/1.0.79 \ | " | | SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac" | SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599" | SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f" That will require to move the checksum initialization after the possible call to urldata_init method in order for the crate fetcher to parse the url. Another way of doing could have been implementing a decodeurl method that would have been specific for each fetcher class. (Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec) Signed-off-by: Frederic Martinsons Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2: Add autorev warning when it is set too late

2023-03-22T13:57:04+00:00

Bitbake expects a consistent metadata environment when it parses. There are plenty of ways you can set a recipe to autorev at a point where parts of the fetcher have already been triggered leading to obsure bugs which I struggled to debug, let alone anyone not familar with the code. If anyone is running into the message from the commit, the issue is likely one of timing. Keep in mind that the anonymous python code in base.bbclass will expand variables like FILESPATH, WORKDIR and others which contain PV. The recipe needs to be set to AUTOREV before that anonymous python runs. In particular, that means you can't set SRCREV = "${AUTOREV}" in other anonymous python, it needs to happen earlier. (Bitbake rev: 4d9ec332d5bfc8b60b54f8ec2a17d34e35aa903a) Signed-off-by: Richard Purdie

linux/poky.git/bitbake/lib/bb/fetch2/__init__.py, branch nanbield