summaryrefslogtreecommitdiffstats
path: root/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2/git: Clarify the meaning of namespaceMarek Vasut2022-12-211-2/+2
| | | | | | | | | | | | | | Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository ↵Marek Vasut2022-12-091-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | metadata The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> (Bitbake rev: d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: update references to Yocto Project manualMichael Opdenacker2022-12-081-2/+2
| | | | | | | (Bitbake rev: 5d27c555f7e1cc2064a39a1ce862e09a399185f7) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: npm fetcher: improve description of SRC_URI formatMichael Opdenacker2022-08-041-2/+2
| | | | | | | | | | | | | Using the term "Parameter" which is consistent with the description of SRC_URI parameters in the following text. (Bitbake rev: 87e42f1202162152c779ccc8bbd06f88f0bdab96) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reported-by: Quentin Schulz <foss@0leil.net> Reviewed-by: Quentin Schulz <foss+yocto@0leil.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: document npm and npmsw fetchersMichael Opdenacker2022-06-221-2/+74
| | | | | | | | | | This addresses bug [YOCTO #10098] (Bitbake rev: cca7999586317435d79bf53df4359cdd8bfadff4) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: add explicit target for crates fetcherMichael Opdenacker2022-06-221-0/+2
| | | | | | | | | | Like in other sections describing fetchers (Bitbake rev: c9bab35f6aecbf85ee1a19a7b70e15a80b42471f) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: branch parameter now mandatory in git ↵Michael Opdenacker2022-03-261-3/+3
| | | | | | | | | | SRC_URIs (Bitbake rev: 981d1929937519cd98200a6ef291d97a16d358ad) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: add branch parameter to git SRC_URI examplesMichael Opdenacker2022-03-261-4/+4
| | | | | | | | | | This parameter is now required by the git fetcher module (Bitbake rev: d61b349581c219e7f9d50f683177184fa473cb83) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: add "crate" fetcherMichael Opdenacker2022-03-081-0/+16
| | | | | | | | (Bitbake rev: cca1533e9a684bef8b27130c42abcd1125332038) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reviewed-by: Quentin Schulz <foss+yocto@0leil.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: specify the URL to be used with SSH over ↵Quentin Schulz2022-01-171-0/+14
| | | | | | | | | | | | | | | | git is different from `git clone`'s The URL expected by git fetcher when the SSH protocol is used differs from the one given by Git servers to be used with the `git clone` command. Add a note making this specificity known to users. Cc: Quentin Schulz <foss+yocto@0leil.net> (Bitbake rev: 81c09d434aed13fcfc4ba426ae6a0b849447eb5f) Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: add redirectauth parameterJustin Bronder2021-12-081-0/+5
| | | | | | | | | | | | | | | | | | | | Add a parameter that limits sending Basic authentication in the Authorization header to only the first host and not any that we're redirected to. Ignoring potential security concerns, temporary AWS URLs will reject any request that includes authentication details in both the query parameters (from the redirect) and in the Authorization header. Temporary AWS URLs are now being used for release assets from private Github repositories. According to the previous discussion linked below, they're also in use by bitbucket. See also: https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/ (Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Add striplevel unpack parameterStefan Herbrechtsmeier2021-11-261-0/+3
| | | | | | | (Bitbake rev: f0442a30d3a8459195dbf51a778ffb6150688a0a) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Remove newline string in MIRRORS/PREMIRRORSRichard Purdie2021-11-211-10/+10
| | | | | | | | This syntax is obsolete, update to the correct modern version. (Bitbake rev: 744e9a4c2b6f44116435feb62ac64ff256c752e9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: replace http with https for URLsJon Mason2021-10-081-1/+1
| | | | | | | | | | | https has been the preferred way to access websites for many years now. Change all of the URLs with a _working_ https server/certificate to use that URL. (Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual-fetching: S should be set to WORKDIR/git ↵Quentin Schulz2021-09-101-0/+7
| | | | | | | | | | | | | for git fetcher Document that S has to be set to "${WORKDIR}/git" in order for the recipe to work if the git fetcher is fetching the main source of code for the recipe. (Bitbake rev: c394f34437686c2f57a27f0c06d7aeb3268bfd41) Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fetch securely by defaultRoss Burton2021-08-121-0/+4
| | | | | | | | | | | | | The days of broken certificates are behind us now, so instead of always passing --no-check-certificate to wget, don't pass it by default and instead only pass it BB_CHECK_SSL_CERTS = "0". [ YOCTO #14108 ] (Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: replace ``FOO`` by :term:`FOO` where possibleQuentin Schulz2021-07-261-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a variable has a glossary entry and some rST files write about those variables, it's better to point to the glossary entry instead of just highlighting it by surrounding it with two tick quotes. The script that is used to do the replacement of ``FOO`` by :term:`FOO` is the following Python code: import re from pathlib import Path from runpy import run_module import contextlib import io import sys re_term = re.compile(r'variables.html#term-([a-zA-Z_0-9]*)') terms = [] new_terms = set() with contextlib.redirect_stdout(io.StringIO()) as f: run_module('sphinx.ext.intersphinx', run_name='__main__') objects = f.getvalue() match = re_term.search(objects) while match: if match.group(1): terms.append(match.group(1)) match = re_term.search(objects, match.end()) for rst in Path('.').rglob('*.rst'): with open(rst, 'r') as f: content = "".join(f.readlines()) for term in terms: content = re.sub(r'``({})``(?!.*\s+[~=-]{{{:d},}})'.format(term, len(term)), r':term:`\1`', content) with open(rst, 'w') as f: f.write(content) This script takes one argument as input: an objects.inv which can be gotten from doc/_build/html/objetcs.inv after running `make html`. Note that this excludes from replacement terms that appear in section titles as it requires refs to be changed too. This can be automated too if need be but right now it looks a bit confusing to have an anchor link (for sections) also have a term/reference link in it. I am not sure this is desired today. (Bitbake rev: aba88f40c47133ed9bc999e0298aca3bc8490912) Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com> Signed-off-by: Quentin Schulz <foss@0leil.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: code insertion simplification over two linesMichael Opdenacker2021-04-261-6/+3
| | | | | | | | | | | | | | | This simplifies paragraphs ending with a colon and followed by code insertion. Automatically substituted through the command: sed -i -z "s/:\n\s*::/::/g" file.rst This generates identical HTML output. (Bitbake rev: 51c80fc3497eecc8e50194fe1ff8069b59f03eda) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: simplify colon usageMichael Opdenacker2021-04-261-21/+21
| | | | | | | | | | - This replaces instances of ": ::" by "::", which generates identical HTML output (Bitbake rev: fd8ce4dcaff3aae395f9945fb0a3be54905e1727) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/user-manual-fetching: Remove basepath unpack parameter docsRichard Purdie2021-04-201-3/+0
| | | | | | | | | | | | The code for this was removed in 2016 in commit e659a3b0c2771679057ee3e13cd42e6c62383ff2. Nobody seems to have missed it so remove the documentation so we match the code. [YOCTO #13449] (Bitbake rev: 76bf42ea41a28b19d0377c2e548b0a59119fdf67) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: document no support for using passwords in git ↵Paul Eggleton2021-04-151-0/+9
| | | | | | | | | | | | URLs This is based on the comment added in revision aded964eed4ce5a725ed1ab477efabc86b1aa481. (Bitbake rev: 082683da089115d8b6f71f221cabb41ac401f733) Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: docs: Add Az fetcher documentationAlejandro Hernandez Samaniego2021-03-111-0/+28
| | | | | | | (Bitbake rev: e3b480636a3c2716effd619b59cf55e11f9a6db0) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: docs: sphinx: replace special quotes with double quotesNicolas Dechesne2020-10-061-1/+1
| | | | | | | (Bitbake rev: c87cc35a5665afbf67f6dbb3458976c215fd5ee3) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: last manual round of fixes/improvementsNicolas Dechesne2020-09-161-112/+131
| | | | | | | | | | Review all pages, and fix up for formatting which was not covered by pandoc, such as some links and code block sections. (Bitbake rev: d99760cc687cc9c24d6d9a1c49fa094574476e0c) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: bitbake-user-manual: insert additional blank line after titleNicolas Dechesne2020-09-161-0/+2
| | | | | | | (Bitbake rev: 4f94633a68fd44223d6be458fc0d3616dfdfd1a7) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: add SPDX headersNicolas Dechesne2020-09-161-0/+2
| | | | | | | (Bitbake rev: 4e461224c1825ebba6024fb8d962e726598da153) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: fixup for linksNicolas Dechesne2020-09-161-22/+22
| | | | | | | | | | | | | | | | | | | | | | | | Since we converted the list of variables into a Sphinx glossary, the automatic conversion from Pandoc does not produce proper links. We fix them up using a Python regexp. Similarly some http links were not converted correctly, and can also be fixed up with a regexp. This patch was generated by running the following regexp: line = re.sub("` <(https?://.*)>`__", "\\1", line) line = re.sub("`+(\w+)`* <#var-bb-\\1>`__", ":term:`\\1`", line) (Bitbake rev: 57300955beb200c81762880f44f1a499451b7432) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sphinx: initial sphinx supportNicolas Dechesne2020-09-161-0/+629
This commit is autogenerated pandoc to generate an inital set of reST files based on DocBook XML files. A .rst file is generated for each .xml files in all manuals with this command: cd <manual> for i in *.xml; do \ pandoc -f docbook -t rst --shift-heading-level-by=-1 \ $i -o $(basename $i .xml).rst \ done The conversion was done with: pandoc 2.9.2.1-91 (Arch Linux). Also created an initial top level index file for each document, and added all 'books' to the top leve index.rst file. The YP manuals layout is organized as: Book Chapter Section Section Section Sphinx uses section headers to create the document structure. ReStructuredText defines sections headers like that: To break longer text up into sections, you use section headers. These are a single line of text (one or more words) with adornment: an underline alone, or an underline and an overline together, in dashes "-----", equals "======", tildes "~~~~~~" or any of the non-alphanumeric characters = - ` : ' " ~ ^ _ * + # < > that you feel comfortable with. An underline-only adornment is distinct from an overline-and-underline adornment using the same character. The underline/overline must be at least as long as the title text. Be consistent, since all sections marked with the same adornment style are deemed to be at the same level: Let's define the following convention when converting from Docbook: Book => overline === (Title) Chapter => overline *** (1.) Section => ==== (1.1) Section => ---- (1.1.1) Section => ~~~~ (1.1.1.1) Section => ^^^^ (1.1.1.1.1) During the conversion with pandoc, we used --shift-heading-level=-1 to convert most of DocBook headings automatically. However with this setting, the Chapter header was removed, so I added it back manually. Without this setting all headings were off by one, which was more difficult to manually fix. At least with this change, we now have the same TOC with Sphinx and DocBook. (Bitbake rev: 6bf6c8d63787aed7624793c24af3fa603b5ac961) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>