linux/poky.git/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst, branch 5.2_M2

bitbake: doc: bitbake-user-manual: remove reference to SSTATE_MIRRORS variable

2023-09-05T07:13:13+00:00

This variable is implemented in OE-Core, and therefore only documented in the Yocto Project manual. (Bitbake rev: 1019998e2f8682c8f2f13320fdb0de1a9a23e854) Signed-off-by: Michael Opdenacker Signed-off-by: Richard Purdie

bitbake: doc: Document challenges of tags with git fetcher

2023-09-05T07:13:13+00:00

Using tags with the git fetcher may cause surprising behaviour. There are reasons for this, document them. (Bitbake rev: 56224da378ab63526d44fd7a70bcfd2cffe245cc) Signed-off-by: Richard Purdie

bitbake: fetch2: add Google Cloud Platform (GCP) fetcher

2023-08-11T15:23:01+00:00

This fetcher allows BitBake to fetch from a Google Cloud Storage bucket. The fetcher expects a gs:// URI of the following form: SSTATE_MIRRORS = "file://.* gs:///PATH" The fetcher uses the Google Cloud Storage Python Client, and expects it to be installed, configured, and authenticated prior to use. If accepted, this patch should merge in with the corresponding oe-core patch titled "Add GCP fetcher to list of supported protocols". Some comments on the patch: There is also documentation for the fetcher added to the User Manual. I'm still not completely sure about the recommends_checksum() being set to True. As I've noted in the mailing list, it will throw warnings if the fetcher is used in recipes without specifying a checksum. Please let me know if this is intended behavior or if it should be modified. Here is how this fetcher conforms to the fetcher expectations described at this link: https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README a) Yes, network fetching only happens in the fetcher b) The fetcher has nothing to do with the unpack phase so there is no network access there c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher only downloads to the DL_DIR in the same way that other fetchers, namely the S3 and Azure fetchers do. d) The fetcher is identical to the S3 and Azure fetchers in this context e) Yes, the fetcher output is deterministic because it is downloading tarballs from a bucket and not modifying them in any way. f) I set up a local proxy using tinyproxy and set the http_proxy variable to test whether the Python API respected the proxy. It appears that it did as I could see traffic passing through the proxy. I also did some searching online and found posts indicating that the Google Cloud Python APIs supported the classic Linux proxy variables, namely: - https://github.com/googleapis/google-api-python-client/issues/1260 g) Access is minimal, only checking if the file exists and downloading it if it does. h) Not applicable, BitBake already knows which version it wants and the version infomation is encoded in the filename. The fetcher has no concept of versions. i) Not applicable j) Not applicable k) No tests were added as part of this change. I didn't see any tests for the S3 or Azure changes either, is that OK? l) I'm not 100% familiar but I don't believe this fetcher is using any tools during parse time. Please correct me if I'm wrong. (Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b) Signed-off-by: Emil Ekmečić Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: fetch2/git: Clarify the meaning of namespace

2022-12-21T14:15:26+00:00

Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Marek Vasut Signed-off-by: Richard Purdie

bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

2022-12-09T13:22:11+00:00

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt (Bitbake rev: d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Marek Vasut Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: bitbake-user-manual: update references to Yocto Project manual

2022-12-08T10:49:53+00:00

(Bitbake rev: 5d27c555f7e1cc2064a39a1ce862e09a399185f7) Signed-off-by: Michael Opdenacker Signed-off-by: Richard Purdie

bitbake: bitbake-user-manual: npm fetcher: improve description of SRC_URI format

2022-08-04T16:09:51+00:00

Using the term "Parameter" which is consistent with the description of SRC_URI parameters in the following text. (Bitbake rev: 87e42f1202162152c779ccc8bbd06f88f0bdab96) Signed-off-by: Michael Opdenacker Reported-by: Quentin Schulz Reviewed-by: Quentin Schulz Signed-off-by: Luca Ceresoli Signed-off-by: Richard Purdie

bitbake: doc: bitbake-user-manual: document npm and npmsw fetchers

2022-06-22T21:41:58+00:00

This addresses bug [YOCTO #10098] (Bitbake rev: cca7999586317435d79bf53df4359cdd8bfadff4) Signed-off-by: Michael Opdenacker Signed-off-by: Luca Ceresoli Signed-off-by: Richard Purdie

bitbake: doc: bitbake-user-manual: add explicit target for crates fetcher

2022-06-22T21:41:58+00:00

Like in other sections describing fetchers (Bitbake rev: c9bab35f6aecbf85ee1a19a7b70e15a80b42471f) Signed-off-by: Michael Opdenacker Signed-off-by: Luca Ceresoli Signed-off-by: Richard Purdie

bitbake: doc: bitbake-user-manual: branch parameter now mandatory in git SRC_URIs

2022-03-26T09:27:43+00:00

(Bitbake rev: 981d1929937519cd98200a6ef291d97a16d358ad) Signed-off-by: Michael Opdenacker Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie