summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-04-111-2/+6
| | | | | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: show SRCREV and git repo in error message about fixed ↵Martin Jansa2023-02-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SRCREV * in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to be one character shorter and because fetcher uses: if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]): to decide which SRCREV values are fixed SRCREVs this one was considered as tag or branch name, because it was only 39 chars long The original error message wasn't very helpful as it doesn't show which repo or which SRCREV was considered missing: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) with SRCPV included in PV as error recomments it's a bit better: bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16 The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T with this change the first error will read: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) (Bitbake rev: 9bbdedc0ba7ca819b898e2a29a151d6a2014ca11) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Clarify the meaning of namespaceMarek Vasut2023-02-041-1/+2
| | | | | | | | | | | | | | | | | Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: 86f2fa5261da959cda706c794a0047e5e89d4d6b) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository ↵Marek Vasut2023-02-041-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | metadata The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> (Bitbake rev: c17fc1468ab84663b919e2809606b1b8ea2bebd9) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix regression in gitsm submodule path parsingPavel Zhukov2022-12-091-1/+1
| | | | | | | | | | | | | | | | | | | Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression in submodules path parsing. As the result gitsm fetcher fails on each submodule which name begins from the name of the parent repo which is totally valid usecase [Yocto #14045] [1] Fix the code to error out only if submodule's name is equal to parent name but not if it's part of it. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4 (Bitbake rev: 7e268c107bb0240d583d2c34e24a71e373382509) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Error out if submodule refers to parent repoPavel Zhukov2022-10-111-0/+3
| | | | | | | | | | | | | | | | | If submodule refers to specific revision of the parent repository it causes deadlock in bitbake locking mechanism (lock is acquired to fetch the parent and cannot be released before all submodules are fetched). raise FetchError in such situation to prevent deadlocking. [Yocto 14045] (Bitbake rev: 987712c4c8fefd86a1f5116c11ee86e296e852ee) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Fix npm to use https rather than httpNeil Horman2022-10-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Hit this error while building nlf-native recently: { "error": { "summary": "URI malformed", "detail": "" } } Some poking about led me to discover that: 1) The npm.py tool replaces npm:// with http://, not https:// 2) Some versions of the npm tool don't handle 301 redirects properly, choosing to display the above error instead when using the default nodejs registry It would be good to go fix npm to handle the redirect properly, but it seems like it would also be good to assume secure http when contacting a registry, hence, this patch (Bitbake rev: bb5c43220f5f1c3d82334c65aff1ce13008db8d9) Signed-off-by: Neil Horman <nhorman@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add copyright headers where missingRichard Purdie2022-10-111-0/+2
| | | | | | | | | | | | Where copyright headers were not present, add them to make things clear. (Bitbake rev: e591325b2bd901c381003deb96a7b32a7148e93e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure directory exists before creating symlinkRichard Purdie2022-09-161-0/+2
| | | | | | | | | | | | | | | If the mirrors code is trying to create a symlink and the parent directory doesn't exist, as might be the case for sstate mirrors where the fetch is into a subdir, it can silently fail. Ensure the directory exists in this case to avoid issues. (Bitbake rev: ff3afb1c1bb236c4a52c62a74f2917071e0af55b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit eff16e474ee7dc49ae433420a4c8d15d3314a618) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: gitsm: fix incorrect handling of git submodule relative urlsGennaro Iorio2022-09-161-1/+1
| | | | | | | | | | | | | | As specified by git submodule manual relative urls can start either with '..' or './', second case was incorrectly managed leading to an interpretation of urls starting with './' as absoulte urls. (Bitbake rev: d828cd2a16ddf4f084e61ffe44471483e132653a) Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/osc: Small fixes for osc fetcherGunjan Gupta2022-05-251-6/+10
| | | | | | | | | | | | | | | | | | | | | | The current fetcher seemed to have some issues that made it difficult when trying to use the same. This patch fixes the following * Make consistent use of the path that needs to be used as oscdir * The path mentioned in os.access in download function was not same as ud.moddir which would result into invoking of fetch command instead of update command even if directory already existed * Before creating oscrc, make sure oscdir exists and create it if it does not exist * Updated the configuration to use apiurl and added a new parameter to control whether http or https needs to be used to connect to apiurl (Bitbake rev: 8ac6e09447d884e658c556388d6014279c50f202) Signed-off-by: Gunjan Gupta <viraniac@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3ec78686f3c0ea2304097b86a965f9be4b0cb879) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git : Use cat as pagerRichard Purdie2022-05-121-1/+1
| | | | | | | | | | | | We don't have less in HOSTTOOLS in OE and this can confuse git. Force the pager to cat to be consistent and minimal everywhere. (Bitbake rev: 59c16ae6c55c607c56efd2287537a1b97ba2bf52) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d3d406e8552fdd865dc58b419a84411736475ad2) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/ssh.py: decode path back for sshRobert Yang2022-05-121-2/+4
| | | | | | | | | | | | | | | | | The path has been encoded by urllib.parse.quote(), so decode it back for ssh. Fixed when fetch from PREMIRRORS via ssh: $ bitbake bonnie++ libsigc++-2.0 -cfetch scp: /path/to/downloads/libsigc%2B%2B-2.10.7.tar.xz: No such file or directory (Bitbake rev: 3969786a787eea34e096b932d52cd02978aacb8e) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c1c8fc678eb4783cea3974328a5fa8d1b79f1266) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/osc: Add missing parameterRichard Purdie2022-05-121-1/+1
| | | | | | | | | | | | This probably means the osc fetcher isn't being used but fix the missing parameter. (Bitbake rev: 73fbb743a2def2037d4053605e77e09d2d8a9fd0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a23c201cb6efc5c0abf763c26f905442f0eebb68) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Drop unused importRichard Purdie2022-05-121-1/+0
| | | | | | | | | (Bitbake rev: 43bb92a2728832eeb1a207809e21404c86ff1710) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 879f17ecd5ba09e217cef74f6a51339b145e8ef5) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: fix logger.debug lineRicardo Salveti2022-04-151-1/+1
| | | | | | | | | | | | | | | | | logger.debug was giving an integer value (2) as event message, causing knotty to crash when running with debug enabled. bitbake/lib/bb/ui/knotty.py", line 685, in main event.msg = taskinfo['title'] + ': ' + event.msg TypeError: can only concatenate str (not "int") to str Same issue also happens in the original code that was taken from oe-core (openembedded-core/meta/lib/crate.py honister) / meta-rust. (Bitbake rev: c212b0f3b542efa19f15782421196b7f4b64b0b9) Signed-off-by: Ricardo Salveti <ricardo@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add GIT_SSH_COMMAND to the list of exportsPavel Zhukov2022-04-141-0/+1
| | | | | | | | | | GIT_SSH_COMMAND is more convinient to use if arguments have to be passed and the user doesn't want to create a wrapper script around ssh. (Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: canonicalize ids in generated tarballsOlaf Mandel2022-04-141-1/+1
| | | | | | | | | | | | | | Change the owner information in the mirror tarballs generated using BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit 0178ab83, which used the original pokybuild:user information, but failed to clean up the numerical user and group ids. Now set the more canonical values of oe:oe and 0:0. (Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c) Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> CC: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: stop generated tarballs from leaking infoOlaf Mandel2022-03-261-1/+4
| | | | | | | | | | | | | | | | | | | | When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs of git repositories, they leaked local information: username, group and time of the last fetch. Remove all these by setting fixed information: * uname = pokybuild * gname = users * mtime = committer time of newest commit in repo The username and group value were taken from the archives available on the downloads.yoctoproject.org mirror. The modification time is chosen so it still retains some relationship to the contents of the archive. (Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896) Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: ssh: support checkstatusDaniel Wagenknecht2022-03-041-0/+40
| | | | | | | | | | This implements support for sstate mirrors using ssh as transport protocol. (Bitbake rev: 0a3b5b3de7bcb1c5c3748cba42d394cc484e966b) Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: ssh: fix path handlingDaniel Wagenknecht2022-03-041-0/+5
| | | | | | | | | | Fix absolute paths and paths containing the ":" character. Both is necessary for supporting sstate mirrors via ssh (not implemented yet). (Bitbake rev: df5505a1ba15524c3a185360d687854300aef342) Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: ssh: username and password are optionalDaniel Wagenknecht2022-03-041-1/+1
| | | | | | | | | | Support URLs like ssh://HOST/PATH. They were previously not recognized due to a missing @ sign. (Bitbake rev: a2aa18bd27dac8902e52b466cb7118f71367d3dc) Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: Replace remaining "blacklist"/"whitelist" usageScott Murray2022-02-211-1/+1
| | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language replace the remaining non-backwards-compatibility related usage of "blacklist"/"whitelist" with "allowed"/"disallowed" in a few places as appropriate. (Bitbake rev: f579fb8c23d2919d25641fa4234f8a1e9c06a922) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Abstract fetcher environment to a functionRichard Purdie2022-02-202-8/+12
| | | | | | | | | | | The changing of the environment inside the wget fetcher can race if threading is used, such as with sstate in OE-Core. Abstract the function so the environment can be correct before the function is called, removing the race since the enviroment is then no longer changed. (Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Fix grammar "can happen"Zygmunt Krynicki2022-02-201-1/+1
| | | | | | | (Bitbake rev: 52630eefb5174e4ca357ac57085093a7f5767bd8) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Remove duplicate "the"Zygmunt Krynicki2022-02-201-1/+1
| | | | | | | (Bitbake rev: 02e29e90eac56e46635e9ae44e5c5da684b70ed2) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix typo "conform" and "processed"Zygmunt Krynicki2022-02-201-2/+2
| | | | | | | (Bitbake rev: e1c13a820f9e3154699ec32d84cce502464e4f6b) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: move loop-invariant load of BB_ORIGENVZygmunt Krynicki2022-02-201-4/+3
| | | | | | | | | | | BB_ORIGENV is used as a fallback environment block. It is repeatedly accessed inside the loop. Since it is a loop invariant, move it out of the loop. (Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/cooker: Fix source revision handling with floating upstreamsRichard Purdie2022-02-172-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | Where a git url uses a tag instead of a full source revision, breakage can currently occur in builds. Issues include: * the revision being looked up in multiple tasks (fetch and unpack) * the risk a different revision may be obtained in those tasks * that some tasks may not be allowed to access the network * that a revision may not be consistent throughout a given build * rerunning a specific task may given inconsistent results To fix this, stop the workers from cleaning out the source revision store. This should only be done in the cooker itself (based on current policy). Also, where the code "sees" an upstream access, mark the recipe as not to be cached. The reparse re-triggers the upstream lookup by the server. Add a test to ensure that if get_srcrev isn't called, the user is told they're using a configuration that is known to break. (Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npm: Put all downloaded files in the npm2 directoryPeter Kjellerstedt2022-02-052-4/+8
| | | | | | | | | | | | | Previously npm files that specify downloadfilename= in the SRC_URI would be downloaded to the root of ${DL_DIR} rather than in the ${DL_DIR}/npm2 directory where all other npm files are downloaded. This should make it simpler when setting up and configuring a premirror with the downloaded npm packages. (Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Correct handling of replacing the basename in URIsPeter Kjellerstedt2022-02-051-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The solution implementated in commit 96c30007 (fetch2: fix downloadfilename issue with premirror) missed two corner cases. The first is if the basename of the original URI also appears somewhere else in the replacement URI, in which case it would also be replaced. The second is if the basename of the original URI partially matches the basename of the replacement URI, in which case the inital part of the basename from the replacement URI would be left behind. The second case caused test_npm_premirrors_with_specified_filename to fail. The solution is to prefix the basename with a slash when matching to avoid partial matches, and only replace the basename at the end of the URI. This also adds two test cases that test for these problems. Before this they would give the following errors: - ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] + ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] ? +++++ ++++++++++ - ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] + ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] ? +++++ (Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add crate fetcherRichard Purdie2022-01-192-0/+139
| | | | | | | | | This imports the crate fetcher from OE-Core to resolve various module issues and adds some very very basic tests of that new fetcher. (Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: npm: Use temporary file for empty user configStefan Herbrechtsmeier2021-12-141-9/+5
| | | | | | | | | | | | Always use a temporary file for the user config 'NPM_CONFIG_USERCONFIG' because npm otherwise failed if configs and npmrc aren't set: double-loading config "/dev/null" as "global", previously loaded as "user" (Bitbake rev: 9f272ad7f76c1559e745e9af686d0a529f917659) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: npm: Quote destdir in run chmod commandStefan Herbrechtsmeier2021-12-141-1/+1
| | | | | | | | | | | Quote destdir in run chmod command to support special characters in package name and to avoid syntax error for packages like '@(._.)/execute'. (Bitbake rev: a701dfce3f0e74b4d7c687eeda83fe9c8e7240b1) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: do not hardcode tarball compressors in version checkAlexander Kanavin2021-12-081-1/+1
| | | | | | | (Bitbake rev: ba3aa8591327d43935f000c6884637997438ecb2) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: add redirectauth parameterJustin Bronder2021-12-081-1/+11
| | | | | | | | | | | | | | | | | | | | Add a parameter that limits sending Basic authentication in the Authorization header to only the first host and not any that we're redirected to. Ignoring potential security concerns, temporary AWS URLs will reject any request that includes authentication details in both the query parameters (from the redirect) and in the Authorization header. Temporary AWS URLs are now being used for release assets from private Github repositories. According to the previous discussion linked below, they're also in use by bitbucket. See also: https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/ (Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add striplevel support to unpackStefan Herbrechtsmeier2021-11-261-0/+2
| | | | | | | | | | | | | | | | | | | | Add a parameter `striplevel` to the SRC_URI to strip NUMBER leading components (levels) from file names on extraction. For example, if the archive `archive.tar.gz` contains `some/file`, the SRC_URI `https://.../archive.tar.gz;subdir=other;striplevel=1` will extract `some/file` to `other/file`. This is useful to extract archives to a specified directory instead of the original root component of the archive. The feature is required for the npm support. The npm package contents should reside in a subfolder inside a npm archive (usually it is called package/). npm strips one directory layer when installing the package. (Bitbake rev: aa4926e5d9c92f33b4434e2da709ff0bf3049f5b) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Unify tar command in unpackStefan Herbrechtsmeier2021-11-261-8/+9
| | | | | | | | | | | The tar command and its arguments are repeated for many archive types in the unpack function. Unify the common parts in a variable to prepare further extension. (Bitbake rev: a08e57c9eaec1d9740a96149bf4843e576da4e5c) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix url remap issue and add testcaseRichard Purdie2021-11-211-1/+1
| | | | | | | | | | Using "" as a target for .replace() is a really bad idea as it duplicates the replacement for every character in the string. Add a testcase which triggered this and correct the code to return the correct result. (Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Add README on fetcher design constraintsRichard Purdie2021-11-111-0/+57
| | | | | | | | | There have been requests to better document the contraints of fetcher design and operation. This README attempts to start that. (Bitbake rev: d9cda7835816ecd5a60f0575f6ce832ec9c6aced) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Add timeout for checkstatus calls (30s)Richard Purdie2021-11-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 15s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] (Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Handle mirror user/password replacements correctlyRichard Purdie2021-11-031-0/+4
| | | | | | | | | | | Username or password replacements in URIs were being appended rather than replaced in mirror url remapping. Fix this and add a test case. [YOCTO #13823] (Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Fix string concatination potential performance issuesRichard Purdie2021-11-031-8/+8
| | | | | | | | | | | | | | | | Python scales badly when concatinating strings in loops. Most of these references aren't problematic but at least one (in data.py) is probably a performance issue as the issue is compounded as strings become large. The way to handle this in python is to create lists which don't reconstruct all the objects when appending to them. We may as well fix all the references since it stops them being copy/pasted into something problematic in the future. This patch was based on issues highligthted by a report from AWS Codeguru. (Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Clean up use of len()Richard Purdie2021-11-032-3/+3
| | | | | | (Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Show warning for invalid github urlsRichard Purdie2021-11-031-0/+1
| | | | | | | | On master, tell the users they need to update their urls for github. (Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Handle github dropping git:// supportRichard Purdie2021-11-031-0/+4
| | | | | | | | | github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. (Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Add a warning asking users to set a branch in git urlsRichard Purdie2021-11-031-1/+4
| | | | | | | | | | | | There is much uncertainty around what tools and hosting providers will do about default git branch naming in the future. To help ensure we can handle the various scenarios, we will make branch names required in SRC_URI. To start that process, show users a warning if it isn't set. This may also allow us to change the default at some point in the future. (Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Allow git fetcher to support subdir paramRichard Purdie2021-11-031-8/+19
| | | | | | | | | | | | | | | | | | The git fetcher is odd in that it supports destsuffix as a parameter but not the default documented subdir parameter. destsuffix is more limited as it can't take absolute paths. Rework the code to correctly support subdir. Also cleanup to use the None default .get() values and be a bit more pythonic and use subpath as the variable name for subpath for code clarity. We could consider dropping destsuffix as a parameter as some future point. Also fix the tests not to pass in a subdir parameter which was never used but now causes errors. (Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for github prefix in npm shrinkwrap versionStefan Herbrechtsmeier2021-11-011-0/+2
| | | | | | | (Bitbake rev: 1d8af6aed0a929f493d2c3e31b8d3ee3a70beb43) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for duplicate dependencies without urlStefan Herbrechtsmeier2021-11-011-1/+5
| | | | | | | (Bitbake rev: 9dcd9a3c671b24982aff9a994e31cb9f491471f5) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-07-16 22:17:37 +0000