linux/poky.git/bitbake/lib/bb/fetch2, branch honister-3.4.1

bitbake: fetch2: Fix url remap issue and add testcase

2021-11-24T10:08:24+00:00

Using "" as a target for .replace() is a really bad idea as it duplicates the replacement for every character in the string. Add a testcase which triggered this and correct the code to return the correct result. (Bitbake rev: 44a83b373e1fc34c93cd4a6c6cf8b73b230c1520) Signed-off-by: Richard Purdie (cherry picked from commit 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: fetch: Handle mirror user/password replacements correctly

2021-11-12T17:27:58+00:00

Username or password replacements in URIs were being appended rather than replaced in mirror url remapping. Fix this and add a test case. [YOCTO #13823] (Bitbake rev: 85e7af227a48faec65838dcb7e73b17344bb2a0d) Signed-off-by: Richard Purdie (cherry picked from commit 66ad58bb87e5158aced572be4f1d5726bc97fcce) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: fetch/wget: Add timeout for checkstatus calls (30s)

2021-11-11T11:04:15+00:00

We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 30s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] (Bitbake rev: ba97caa58efe25bb62d2378fa52d21b6a6aa446c) Signed-off-by: Richard Purdie

bitbake: fetch2: fix downloadfilename issue with premirror

2021-11-03T12:30:51+00:00

The following commit to fix [Yocto #13039] causes regression of the behavior of PREMIRRORS. "bitbake: fetch2: fix premirror URI when downloadfilename defined" Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb as an example. SRC_URI = "\ http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \ ... " With the above commit, it now tries to fetch 1.4.0.tar.gz instead of freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz. The commit above tries to avoid fetching from invalid url such as: https:///1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz. And its solution is to make basename to be 1.4.0.tar.gz, thus causing the regression. This patch fixes the above regression. For Yocto #13039, it now tries to fetch from url: https:///freeDiameter-1.4.0.tar.gz. (Bitbake rev: 78949cf3fd31d8a408e93af7e27bcf26ae7942f4) Signed-off-by: Chen Qi Signed-off-by: Richard Purdie (cherry picked from commit 96c30007dc0b32eee2b15771daec7948bc9bfd97) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: fetch/git: Handle github dropping git:// support

2021-11-03T11:31:21+00:00

github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. (Bitbake rev: f19eefdaa5b43460f00d79d002f96112a6aa3c9a) Signed-off-by: Richard Purdie

bitbake: fetch2/perforce: Fix typo

2021-10-26T12:47:21+00:00

(Bitbake rev: 20eae05fdd6cb7ace87ad005f72c256e2fddb3d0) Signed-off-by: Richard Purdie

bitbake: fetch2: npmsw: Add support for local tarball and link sources

2021-10-08T15:45:06+00:00

(Bitbake rev: 4f983dc419a1a6f635a5d333f253d49244cec374) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: fetch2: npm: Create config npmrc in environment instantiation

2021-10-08T15:45:05+00:00

Create a configuration npmrc per npm environment to avoid repeated creation of the same configuration file. Create the file via python to avoid multiple npm config calls and add the ability to pass a file path instead of a temporary file. Deprecate the npm configs argument of the run function. The configs should be passed to npm environment or as command specific arguments. (Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: fetch2: npm: Support npm archives with missing search directory mode

2021-10-08T15:45:05+00:00

Delay directory restore and set execute/search directory mode bits in unpack to support npm archives with a missing search directory mode. (Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8) Signed-off-by: Stefan Herbrechtsmeier Signed-off-by: Richard Purdie

bitbake: fetch2/gitsm: remove the 'nugget' SRCREV caching

2021-10-08T15:45:05+00:00

The cached revisions which are used to decide if a repository doesn't need to be updated are misleading when used in conjunction with mirror tarballs and can cause partial fetches to happen, resulting in unpack errors as repositories were not fetched. A concrete example: edk2-firmware in meta-arm is at version 202102 (ef91b0). This is built on the autobuilder so the source mirror contains the repository as a mirror tarball. If I build edk2-firmware 202102 the gitsm fetcher will initially download the top-level repository and then iterate into the submodules to also fetch those repositories, including cmocka from cryptomilk.org. edk2-firmware will then unpack and build successfully. I then update edk2-firmware to 202105 (e1999b) and build it. Gitsm.needs_update() starts by calling Git.needs_update() which returns False, as the mirror tarball contains this revision. It then looks at the "nuggets" which are SRCREVs it has fetched before. The mirror tarball itself contains the nugget for e1999b as this has been built on the autobuilder, so needs_update return False, no more fetching is done, and the build proceeds to unpack. However, as part of the 202105 upgrade the URL of the cmocka submodule changed, and this new repository was never fetched. This means that unpack fails as one of the required git repositories isn't available. The nugget codepaths appear to be an attempt at optimising the fetch process, but have demonstratable failure cases. Just removing them entirely solves the edk2-firmware example, and all of the fetcher test cases still pass. (Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie