linux/poky.git/bitbake/lib/bb/fetch2/init.py, branch uninative-3.2

bitbake: bitbake: Add Azure Storage fetcher implementation

2021-03-11T14:04:45+00:00

Allows bitbake to fetch from an Azure Storage account. The fetcher submodule is compatible with the az:// URI protocol, its functionality is based on bitbakes wget fetcher, superior in performance to using a propietary tool like azcopy which can handle cloud storage account operations with more functionality (that we dont need in a fetcher) but less compatibility. A sample URI uses can be defined in the following way: SRC_URI = "az://.blob.core.windows.net//foo.tar.xz" This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.: SSTATE_MIRRORS = "file://.* az://.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n" PREMIRRORS_prepend = "\ git://.*/.* az://.blob.core.windows.net/downloads/ \n \ ftp://.*/.* az://.blob.core.windows.net/downloads/ \n \ http://.*/.* az://.blob.core.windows.net/downloads/ \n \ https://.*/.* az://.blob.core.windows.net/downloads/ \n \ " Can also be used with non-public access Azure Storage accounts/containers via a Shared Access Signature by declaring the AZ_SAS variable which will be automatically used by the fetcher: AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature" (Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4) Signed-off-by: Alejandro Enedino Hernandez Samaniego Signed-off-by: Richard Purdie

bitbake: logging: Make bitbake logger compatible with python logger

2021-02-10T23:48:16+00:00

The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: fetch2: handle empty elements in _param_str_split

2021-01-23T17:09:26+00:00

_param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws an exception if a lone semicolon is used as the value doesn't split into two items. Fix by checking that the result of the first split has content. (Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: Revert "fetch2: use relative symlinks for anything pulled from PREMIRRORS"

2020-12-09T12:36:22+00:00

This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb. Unfortunately this caused problems where incorrect paths were found in paths and files were not being found. This was reported on the mailing list but no invetigation or resolution was found. Revert until we can figure out what was wrong. Signed-off-by: Richard Purdie

bitbake: fetch2: use relative symlinks for anything pulled from PREMIRRORS

2020-11-24T15:26:12+00:00

try_mirror_url() creates a symlink named as the original file to make everything look like files specified in SRC_URI were downloaded from their original location. The link is however created as an absolute reference, this makes DL_DIR non-relocatable. This also causes issues with the Isar project since it bind mounts DL_DIR to /downloads to perform some of its build tasks in a chrooted environment (rendering all symbolic links from DL_DIR invalid). Modify ensure_symlink() to take an optional "relative" argument and have that function use os.path.relpath() to produce a relative symlink. (Bitbake rev: 481e66ea8fc2fc91903127d66b0f1b0fe86baedb) Signed-off-by: Cedric Hombourger Cc: Chris Larson Signed-off-by: Richard Purdie

bitbake: lib/bb/fetch2/init.py: drop _PYTHON_SYSCONFIGDATA_NAME unsetting

2020-11-20T10:50:18+00:00

With introduction of python3targetconfig class in core this is no longer needed. (Bitbake rev: 47b64cfacd7c498ef9ed5486d117f2d69a39f225) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: fetch2: add zstd support to unpack

2020-11-03T08:21:10+00:00

This commit adds unpack support to fetched files with extensions: '.tar.zst', '.tzst' and '.zst' (Bitbake rev: 4348743a7e36f3a495e2dc48f83989d9481060f2) Signed-off-by: Mert Kirpici Signed-off-by: Richard Purdie

bitbake: fetch2: fix handling of `\` in file:// SRC_URI

2020-09-25T07:32:02+00:00

Using backslashes in file:// URIs was broken. Either the resolver would fail or the subsequent `cp` command. Try to avoid this by putting the filenames into quotes. Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=8161 (Bitbake rev: aa857fa2e9cf3b0e43a9049b04ec4b0b3c779b11) Signed-off-by: Leif Middelschulte Signed-off-by: Richard Purdie

bitbake: fetch2: Drop globbing supprt in file:// SRC_URIs

2020-08-26T08:05:38+00:00

Globbing in file:// urls is terminally broken. Currently when its used, the file checksum code is basically bypassed. This means changes to the source files don't change the task checksum, the task doesn't rebuild when the inputs change and things generally break. To make globbing work generically, we'd have to scan the file system for all possible matches to the glob and log whether they exist or not. We can't simply log the files which exist, we have to also know which files could later exist and influence the choice of file so we know when to reparse. For a simple file://xxx/*, this could be done but for bigger patterns, it becomes much more problemtic. We already support file://xxx/ in urls. So, lets decide we'll not support globs in file://urls. Worse case users can put files in a directory and reference that, moving files into place if needed. Remove all the glob special cases (see the comments if anyone doesn't believe this is terminally broken) and error to the user if they have such urls. (Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7) Signed-off-by: Richard Purdie

bitbake: fetch2: Add the ability to list expanded URL data

2020-06-04T12:28:16+00:00

Some fetchers may download additional sources along with those explicitly listed in SRC_URI. These "implicit URLs" will be needed by the archiver to ensure that all sources can be archived. We can't just return a list of URL strings since each URL may need its own SRCREV data so we return a list of FetchData objects. Each fetcher can override the implicit_urldata() function to provide the additional FetchData objects. For now this is just needed in the gitsm fetcher to walk git submodules recursively. (Bitbake rev: 1350f241b7d991bd191ce9e44f6662e4376c6e24) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

linux/poky.git/bitbake/lib/bb/fetch2/__init__.py, branch uninative-3.2