summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2/s3: allow to use credentials from environment variablesAdam Romanek2021-07-111-1/+4
| | | | | | | | | | | | | | | | | Previously access to AWS S3 was expected to be preconfigured and credentials to be stored in ~/.aws/credentials. With this change one can use Bitbake s3 fetcher without AWS credentials stored permanently as above, just with them exported as the following environment variables: - AWS_ACCESS_KEY_ID, - AWS_SECRET_ACCESS_KEY. - AWS_DEFAULT_REGION. (Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48) Signed-off-by: Adam Romanek <romanek.adam@gmail.com> Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: run gc in foreground to avoid race with tarAdam Romanek2021-07-101-1/+1
| | | | | | | | | | | | | | | | | | | | It looks like git gc can interrupts the package creation when BB_GENERATE_MIRRORS_TARBALL is in use. Log excerpts: tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output: tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. (Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877) Signed-off-by: Adam Romanek <romanek.adam@gmail.com> Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add check for empty SRC_URI hash stringScott Weaver2021-06-061-1/+4
| | | | | | | | | | | | | | | | | | | | | | No error was being reported when the hash string was set to empty. For example: SRC_URI[md5sum] = "" On a related note (not a bug): Because whitespace in the string will result in a checksum mismatch, the error message was updated to make it a little clearer why the error was thrown. For example: SRC_URI[md5sum] = " " or SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1" Now creates a message like this: File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5 checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected [YOCTO #14232] (Bitbake rev: a13510d0028e234ea2f4744b0d0c38558395c70f) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: when checking latest versions, consider all numerical ↵Alexander Kanavin2021-05-141-1/+1
| | | | | | | | | | | | | | directories Previously the regex was maching x.y, but wasn't matching x, which is a problem e.g. here: https://download.gnome.org/sources/epiphany/ (the new gnome version scheme adds 40-series at the end). (Bitbake rev: c03101576f447263ea38e8464210d3a3a2c27226) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/svn: Fix parsing revision of SVN repos with redirectsHarald Brinkmann2021-05-061-1/+1
| | | | | | | | | | svn was printing a message when encountering HTTP redirects. This confused the revision parser. (Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146) Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/s3: Add progress handler for S3 cp commandPrzemyslaw Gorszkowski2021-05-011-1/+40
| | | | | | | | Adds progress support for fetchers from S3. (Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/gitsm: Fix crash when using git LFS and submodulesNiels Avonds2021-04-201-1/+1
| | | | | | | | | | | | | | | | Gitsm fetcher crashes when cloning a repository that contains LFS files. This happens because the unpack method is called during download, but the submodules have not been downloaded yet at this point. This issue was introduced in this commit: 977b7268bf4fd425cb86d4a57500350c9b829162 [YOCTO #14283] (Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e) Signed-off-by: Niels Avonds <niels@codebits.be> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: add support for disabling shared clones on unpackHenning Schild2021-04-181-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | By default the unpacker will create a "shared" clone when cloning from the DL_DIR to the WORKDIR. This patch introduces an option to control that behaviour. Imagine some recipe steps are executed in a namespace that is different from the one your downloader and unpacker ran in. (chroot) Because a "shared" clone has an absolute reference to its "alternate" you now have to make that "alternate" visible in that new namespace (chroot) at the exact place. With this patch you can unpack "noshared" and get a stand-alone copy. This copy will also work if the "alternate" is not visible or existant. The switch is a global bitbake switch and will affect all git urls. Build systems that need "noshared" most likely need it for everything they do with git. (Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102) Signed-off-by: Henning Schild <henning.schild@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add Azure Storage fetcher implementationAlejandro Hernandez Samaniego2021-03-112-1/+96
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allows bitbake to fetch from an Azure Storage account. The fetcher submodule is compatible with the az:// URI protocol, its functionality is based on bitbakes wget fetcher, superior in performance to using a propietary tool like azcopy which can handle cloud storage account operations with more functionality (that we dont need in a fetcher) but less compatibility. A sample URI uses can be defined in the following way: SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz" This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.: SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n" PREMIRRORS_prepend = "\ git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ " Can also be used with non-public access Azure Storage accounts/containers via a Shared Access Signature by declaring the AZ_SAS variable which will be automatically used by the fetcher: AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature" (Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatusAlejandro Hernandez Samaniego2021-02-261-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | If an exception is raised when running host python code, the fetcher immediately crashes, this might be temporary depending on the servers reliability. Catch the exception when the connection was reset and try once again to fetch the data. File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto 0665: if self._timeout_occurred: 0666: raise OSError("cannot read from timed out object") 0667: while True: 0668: try: *** 0669: return self._sock.recv_into(b) 0670: except timeout: 0671: self._timeout_occurred = True 0672: raise 0673: except error as e: Exception: ConnectionResetError: [Errno 104] Connection reset by peer (Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-1012-53/+53
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: download LFS content too during do_fetchMatt Hoosier2021-01-231-3/+41
| | | | | | | | | | | | | | Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] (Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle empty elements in _param_str_splitRoss Burton2021-01-231-1/+1
| | | | | | | | | | | | | _param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws an exception if a lone semicolon is used as the value doesn't split into two items. Fix by checking that the result of the first split has content. (Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/perforce: Fix localfile to include ud.moduleRichard Purdie2021-01-231-1/+2
| | | | | | | | | As reported by Katu Txakur, the output depends on the module path set so this needs to be accounted for in localfile. (Bitbake rev: 9861ed37bb1c5d09c3b4852d2a252e3f3c86ab14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Update user-agentChristophe Priouzeau2021-01-231-3/+8
| | | | | | | | | | | | With the usage of enterprise proxy, the user-agent defined are too old and refused by proxy configuration. Updating to something more modern is desirable. (Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400) Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com> Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Fix usehead for non-default namesJoey Degges2021-01-081-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch2: use relative symlinks for anything pulled from ↵Richard Purdie2020-12-091-5/+2
| | | | | | | | | | | | | PREMIRRORS" This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb. Unfortunately this caused problems where incorrect paths were found in paths and files were not being found. This was reported on the mailing list but no invetigation or resolution was found. Revert until we can figure out what was wrong. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: use relative symlinks for anything pulled from PREMIRRORSHombourger, Cedric2020-11-241-2/+5
| | | | | | | | | | | | | | | | | | try_mirror_url() creates a symlink named as the original file to make everything look like files specified in SRC_URI were downloaded from their original location. The link is however created as an absolute reference, this makes DL_DIR non-relocatable. This also causes issues with the Isar project since it bind mounts DL_DIR to /downloads to perform some of its build tasks in a chrooted environment (rendering all symbolic links from DL_DIR invalid). Modify ensure_symlink() to take an optional "relative" argument and have that function use os.path.relpath() to produce a relative symlink. (Bitbake rev: 481e66ea8fc2fc91903127d66b0f1b0fe86baedb) Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> Cc: Chris Larson <Chris_Larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/fetch2/__init__.py: drop _PYTHON_SYSCONFIGDATA_NAME unsettingAlexander Kanavin2020-11-201-5/+0
| | | | | | | | | | With introduction of python3targetconfig class in core this is no longer needed. (Bitbake rev: 47b64cfacd7c498ef9ed5486d117f2d69a39f225) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Document that we won't support passwords in git urlsRichard Purdie2020-11-031-0/+5
| | | | | | | | This keeps coming up, document why we don't do this and the alternatives. (Bitbake rev: aded964eed4ce5a725ed1ab477efabc86b1aa481) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add zstd support to unpackMert Kirpici2020-11-031-0/+4
| | | | | | | | | | This commit adds unpack support to fetched files with extensions: '.tar.zst', '.tzst' and '.zst' (Bitbake rev: 4348743a7e36f3a495e2dc48f83989d9481060f2) Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI urlCharlie Davies2020-10-171-6/+7
| | | | | | | | | | | This commit replaces the instances where escaped double quotes are used to support SRC_URI url containing spaces with the more pythonic shlex.quote(). (Bitbake rev: 4f9ba9c794de55bea0343267467bddea99844374) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix handling of `\` in file:// SRC_URILeif Middelschulte2020-09-251-1/+1
| | | | | | | | | | | | | Using backslashes in file:// URIs was broken. Either the resolver would fail or the subsequent `cp` command. Try to avoid this by putting the filenames into quotes. Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=8161 (Bitbake rev: aa857fa2e9cf3b0e43a9049b04ec4b0b3c779b11) Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch/git: add support for SRC_URI containing spaces in urlCharlie Davies2020-09-231-6/+6
| | | | | | | | | | | | | | | | | | | Microsoft's TFS VCS system allows for spaces in a git repository url. An example of a valid url is: ssh://tfs-my-company.org:22/tfs/My Projects/FooBar This commit adds support for such urls by implementing two changes. Firstly, when bitbake makes a git command line call the url is surrounded by quotes so that the url, regardless of spaces, is treated as one argument. Secondly, additional parsing of various filepath variables, which are based off of the url, are now completed with any spaces in the url replaced with underscores. (Bitbake rev: eb38b6f0935763f7ba19e5618f376fcae1dac41a) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Drop globbing supprt in file:// SRC_URIsRichard Purdie2020-08-262-24/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Globbing in file:// urls is terminally broken. Currently when its used, the file checksum code is basically bypassed. This means changes to the source files don't change the task checksum, the task doesn't rebuild when the inputs change and things generally break. To make globbing work generically, we'd have to scan the file system for all possible matches to the glob and log whether they exist or not. We can't simply log the files which exist, we have to also know which files could later exist and influence the choice of file so we know when to reparse. For a simple file://xxx/*, this could be done but for bigger patterns, it becomes much more problemtic. We already support file://xxx/ in urls. So, lets decide we'll not support globs in file://urls. Worse case users can put files in a directory and reference that, moving files into place if needed. Remove all the glob special cases (see the comments if anyone doesn't believe this is terminally broken) and error to the user if they have such urls. (Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: fix most undefined code picked up by pylintFrazer Clews2020-08-252-4/+6
| | | | | | | | | | Correctly import, and inherit functions, and variables. Also fix some typos and remove some Python 2 code that isn't recognised. (Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5) Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Remove buffering parameterRichard Purdie2020-08-241-4/+1
| | | | | | | | | | The buffering parameter was removed in python 3.1 and made default so we can clean up the code. This removes weird looking double exceptions when connections fail. (Bitbake rev: 06b7bafbd18a47c8db2f7b943dc535c65df176bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Change git fetcher not to destroy old referencesRichard Purdie2020-07-121-1/+1
| | | | | | | | | | | | | | | | It looks like we're about to see a lot of changes in branch names in repos. If we have the prune option here, those old names are lost, the changes propagate to our source mirrors and our old releases break. We have the force option so any replaced references should be replaced, its only orphaned branches which will now be preserved. I believe this behaviour will cause us fewer problems given the changes that look likely to happen. (Bitbake rev: 820ab886e79eea516560c0c008e4cf059c6e11a3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: perforce: add local path handling SRC_URI optionsAlexandru N. Onea2020-06-251-2/+45
| | | | | | | | | | | | | | | | | | | | This patch implements three new SRC_URI options for the perforce fetcher, namely: * module * remotepath The options are intended to provide the user more control over the downloaded file paths by allowing the user to specify how much of the remote path should be preserved locally. The changes in this patch are backwards compatible, i.e. if none of the introduced options is specified, the default (old) behavior is enforced. (Bitbake rev: aab228822d2f221c01337dd57d7582c51ce9a505) Signed-off-by: Alexandru N. Onea <onea.alex@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: perforce: add basic progress handler for perforceAlexandru N. Onea2020-06-251-1/+33
| | | | | | | | | | | | This patch adds a basic implementation of a progress handler for the perforce fetcher, based on the number of files to be downloaded and the output behavior of the p4 print command used in the fetcher implementation. (Bitbake rev: f0582292bf79b0988048683dfd086aa3b9787344) Signed-off-by: Alexandru N. Onea <onea.alex@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: Make need_update() process submodulesPaul Barker2020-06-151-4/+35
| | | | | | | | | | | | If the bitbake.srcrev nugget is not present for the commit we're interested in we should not just bail out and say that an update is needed. Instead we can recursively walk through the submodules and check for the presence of the required commits. (Bitbake rev: cfc78316309556bec487ef0a5a9205e41f1be86f) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: Mark srcrev as fetched once all submodules are processedPaul Barker2020-06-151-3/+3
| | | | | | | | | | | | This prevents multiple bitbake.srcrev entries being created when a repository has more than one submodule. It also ensures that the bitbake.srcrev entry is not added unless all submodules are correctly fetched. (Bitbake rev: 860ff1193fe53f04696d41635a720c2d1f29fa7f) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add the ability to list expanded URL dataPaul Barker2020-06-042-0/+46
| | | | | | | | | | | | | | | | | | Some fetchers may download additional sources along with those explicitly listed in SRC_URI. These "implicit URLs" will be needed by the archiver to ensure that all sources can be archived. We can't just return a list of URL strings since each URL may need its own SRCREV data so we return a list of FetchData objects. Each fetcher can override the implicit_urldata() function to provide the additional FetchData objects. For now this is just needed in the gitsm fetcher to walk git submodules recursively. (Bitbake rev: 1350f241b7d991bd191ce9e44f6662e4376c6e24) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git.py: Use the correct branch to check if the repository has LFS ↵Mauro Queirós2020-06-041-2/+9
| | | | | | | | | | | | | objects. Function "contains_lfs" was only looking at the master branch when searching for LFS content. LFS may be configured in specific branches only, so we need to use the correct branch. (Bitbake rev: 4fa67c2af830035a1ddedc14592ee25a15ebff22) Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git.py: LFS bitbake note should not be printed if need_lfs is not set.Mauro Queirós2020-06-041-1/+1
| | | | | | | | | | | The message "Repository %s has LFS content but it is not being fetched" was being printed, even when Git-LFS was available and "lfs=1" was set. In those situations, we want to fetch LFS content, so that message would not make sense. (Bitbake rev: 45028dfda5a29a34ab408cb3f11d72ae17963340) Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git.py: skip smudging if lfs=0 is setMauro Queirós2020-06-041-0/+3
| | | | | | | | | | | Git-LFS objects were being fetched even when lfs=0 was not set. This patch disables LFS smudging when lfs=0. That way, only the LFS pointers are downloaded during checkout. (Bitbake rev: 646d86df7de774255246a3d7051c308e43eb257d) Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/cvs: Fix CVS fetcher clean methodVyacheslav Yurkov2020-05-271-11/+9
| | | | | | | | | | | Clean method assumes that download directory should be determined from CVSDIR variable, but this is handled differently in download method. Now we set download directory for the whole class in urldata_init (Bitbake rev: 6d7ad9b06b82612f0dd17ffccea7ab98f9077198) Signed-off-by: Vyacheslav Yurkov <Vyacheslav.Yurkov@bruker.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Set User-Agent when checking status of a URLPaul Barker2020-04-241-0/+1
| | | | | | | | | | | | | | | | | | When a website is behind a CDN like Cloudflare there may be a "Browser Integrity Check" or other test applied to requests before they are allowed through to the server. Downloading via wget passes these tests as headers are set appropriately, however the Python urllib module may fail these tests unless additional headers are set. This causes Wget.checkstatus() to fail where Wget.download() would actually succeed. For Cloudflare in particular a valid User-Agent is needed, it's easy to add this to the headers in Wget.checkstatus(). The user agent string is copied from Wget._fetch_index(). (Bitbake rev: 4679d3cdb9cdf23f3962aa61c599ad7474591f9f) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix urldata_cache key issuesRichard Purdie2020-03-301-4/+7
| | | | | | | | | Upon inspection its clear the way the keys for this cache were being handled would break it and cause the cache to never be used. Fix this. (Bitbake rev: 9a5dd1be63395c76d3fac2c3c7ba6557fe47b442) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY ↵Khem Raj2020-03-241-14/+4
| | | | | | | | | | | | | | is "cache"" As per mailing list discussion, the cache policy was behaving correctly before and wouldn't expect to update after the initial fetch even for AUTOREV. This reverts commit ba093a38539960e645e994a66ed7872a604c00a9. (Bitbake rev: 51f827911b7202de3e855e683fdbd732d7a84e09) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: Unpack shallow mirror tarballsPaul Barker2020-03-111-5/+16
| | | | | | | | | | When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked temporarily so that the .gitmodules file can be examined. (Bitbake rev: 3987db953e414255ce278bc25a5f6cec0f2a30c7) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Limit shown checksums to sha256Richard Purdie2020-03-021-1/+4
| | | | | | | | | | | | | | | | Currently bitbake will list many checksums for a recipe when none are present, encouraging users to add them all to a recipe. We don't need/want them all. We used to show md5 and sha256 but given the concerns about md5, switch to showing just sha256 going forward which seems like the sensible one to standardise upon. There will be no change to existing recipe functionality. (Bitbake rev: 47f0c849ed13ba554d9523b926d92405e8251702) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Avoid UnboundLocalError exceptionRichard Purdie2020-02-201-9/+9
| | | | | | | | | | | | | The update codepath would trigger: Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment Fix this so the code functions as intended in both fetch and update cases. [YOCTO #13798] (Bitbake rev: 16c4e930ff37ea6eac2ac0cb2197908ce3a1cc53) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: svn: care for path_specJens Rehsack2020-02-191-1/+1
| | | | | | | | | | | | | Documentation says: "path_spec": A specific directory in which to checkout the specified svn module. but existing svn fetcher uses "module" always as path of checked out svn-module, regardless whether path_spec was given or not. (Bitbake rev: 75223644ab9bc94fc268f1bab775e66c4188f279) Signed-off-by: Jens Rehsack <sno@NetBSD.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY is "cache"Peter Kjellerstedt2020-02-191-4/+14
| | | | | | | | | | | | Mark any keys used to cache the srcrevs for a recipe as "dontcache" if BB_DONT_CACHE is set for the recipe. Remove any such keys upon the next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will make sure the srcrev is updated as expected if ${AUTOREV} is used. (Bitbake rev: ba093a38539960e645e994a66ed7872a604c00a9) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Make fetcher_compare_revisions() workPeter Kjellerstedt2020-02-191-19/+14
| | | | | | | | | | This seems to have been broken for a very long time. Now it also works regardless of BB_SRCREV_POLICY. (Bitbake rev: ffd663a8e07e2e39e8ca2d2493f4f98037c5f9e4) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add the npmsw fetcherJean-Marie LEMETAYER2020-01-272-0/+257
| | | | | | | | | | | | | | | | | | | | | | | | This commit adds a new npmsw fetcher that fetches every npm dependencies described in a npm shrinkwrap file: https://docs.npmjs.com/files/shrinkwrap.json.html The main package must be fetched separately: SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \ npmsw://${THISDIR}/npm-shrinkwrap.json" Since a separation has been created between the package and its dependencies, the package can also be fetched with a non npm fetcher without impacting the general behavior: SRC_URI = "git://github.com/foo/bar.git;protocol=https \ npmsw://${THISDIR}/npm-shrinkwrap.json" (Bitbake rev: f5223be54450bf20e0bfbd53b372a7748a44b475) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: refactor the npm fetcherJean-Marie LEMETAYER2020-01-271-271/+267
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit refactors the npm fetcher to improve some points and fix others: - The big change is that the fetcher is only fetching the package source and no more the dependencies. Thus the npm fetcher act as the other fetchers e.g git, wget. The dependencies will be handled later. - The fetcher only resolves the url of the package using 'npm view' and then forwards it to a proxy fetcher. - This commit also fixes a lot of issues with the package names (exotic characters, scoped packages) which were badly handled. - The validation files - lockdown.json and npm-shrinkwrap.json - are no longer used by the fetcher. Instead, the downloaded tarball is verified with the 'integrity' and 'shasum' provided in the 'npm view' of the package [1][2]. 1: https://docs.npmjs.com/files/package-lock.json#integrity 2: https://www.w3.org/TR/SRI (Bitbake rev: 0f451cdc43130d503ada53ed1b4fc5a24943f6ef) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fix downloadfilename parameterJean-Marie LEMETAYER2020-01-271-3/+4
| | | | | | | | | | | When using a download filename with characters which can be interpreted by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the filename fixes the issue. (Bitbake rev: ed652dce5200161068eccdbfaaaefde33136eb09) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow fetchers to forward the done conditionJean-Marie LEMETAYER2020-01-271-10/+20
| | | | | | | | | | | | | This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the done condition to a proxy fetcher. (Bitbake rev: ee3a2545e99e6e99559a72bcda64797ae674ec71) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-07-02 19:03:13 +0000