linux/poky.git/bitbake/lib/bb/fetch2, branch 3.1_M1

bitbake: fetch2/clearcase: Fix warnings from python 3.8

2019-11-21T23:09:03+00:00

bitbake/lib/bb/fetch2/clearcase.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="? if command is 'mkview': bitbake/lib/bb/fetch2/clearcase.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'rmview': bitbake/lib/bb/fetch2/clearcase.py:159: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'setcs': Python 3.8 is quite correct and we so mean "==" here, fix it to avoid the warnings. (Bitbake rev: 1fab03f6e10eaa13b8a89ce0b2f9fe8ce5157189) Signed-off-by: Richard Purdie

bitbake: fetch2/hg: Fix various runtime issues

2019-11-07T15:12:29+00:00

Fix mercurial fetching after breakage from changes to the core fetcher. Fix username and password usage and setting moddir needed by setup_revisions. (Bitbake rev: 8962b27abc68427eae085624f26300108ae88aa1) Signed-off-by: Volker Vogelhuber Signed-off-by: Richard Purdie

bitbake: fetch2: Ensure cached url data is matched to a datastore

2019-11-05T10:37:12+00:00

There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 0540abd338ed2ffd822edbd2947cab2f18873422) Signed-off-by: Richard Purdie

bitbake: fetch2/git: fetch shallow revs when needed

2019-10-28T05:31:53+00:00

When bitbake determines if a git clone needs updating, it only checks for the needed srcrevs, not the revs listed in BB_GIT_SHALLOW_REVS, which will fail if using shallow and the needed rev was added to the upstream git repo after a previous fetch. Ensure that we also check for shallow revs. [YOCTO #13586] (Bitbake rev: 93b65e3c6ca64f644946953980595c44fbbcc748) Signed-off-by: Christopher Larson Signed-off-by: Richard Purdie

bitbake: fetch2/git: refactor check for git-lfs command

2019-09-27T12:02:19+00:00

Refactor the git-lfs checking: this means both clearer code in the download() function and allows unit testing to monkeypatch the functionality. (Bitbake rev: 33cf9172ded50a869f7201ba463ab9ecc69b8252) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: fetch2/git: add git-lfs toggle option

2019-09-19T19:30:35+00:00

Add a new 'lfs' option to the git fetcher so that we can optionally not fetch git-lfs content, for repositories that contain LFS data that we don't actually need for building. By default lfs is set to 1, so if the repository has LFS content then git-lfs is required. Setting lfs to 0 will mean that git-lfs won't be required to fetch, and some files will be missing. (Bitbake rev: be0b78ccfc5ede98041bc0545a15092494b12b26) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: svn fetcher: allow "svn propget svn:externals" to fail

2019-09-18T16:52:01+00:00

Not all servers and repositories have this property set which results in failures like this when actual svn checkout command succeeded: svn: warning: W200017: Property 'svn:externals' not found on '' svn: E200000: A problem occurred; see other errors for details (Bitbake rev: 238636f033cbf18e5741f0ea0e64db40e84f5838) Signed-off-by: Mikko Rapeli Signed-off-by: Richard Purdie

bitbake: fetch2: show warning when renaming the archive with bad checksum failed

2019-09-03T09:18:58+00:00

* noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: 9a1bf4ba9ec00c2a222d820f8f83d1f056b021d6) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: fetch2/wget: avoid 'maximum recursion depth' RuntimeErrors when handling 403 codes

2019-07-27T21:46:09+00:00

The code says that some servers respond with 403 codes when they really mean 405 codes. But we still need to account for legitimate 403 codes. Before this change, I noticed that sstate mirror checking was taking a very long time when I purposely entered incorrect credentials into my .netrc file for our sstate mirror. Instrumenting the code, I discovered tracebacks like the following for every mirror access attempt: File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus fetcher.checkstatus() File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus ret = try_mirrors(self, self.d, ud, mirrors, True) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors ret = try_mirror_url(fetch, origud, uds[index], ld, check) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url found = ud.method.checkstatus(fetch, ud, ld) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus opener.open(r) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) ... (repeats until recursion depth is reached) Solution is to make sure we only attempt the GET request once when handling 403/405 error codes. (Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7) Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie

bitbake: fetch2/npm: Use npm pack to download node modules instead of wget

2019-07-27T21:46:09+00:00

Using npm pack to download the main node module and its dependencies allow for the use of private npm modules and access to them via .npmrc (Bitbake rev: e5eda3871893e4eadeb311aeb997e183675598f4) Signed-off-by: Mads Andreasen Signed-off-by: Richard Purdie