summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: event: Fix multiconfig event handler change performance regressionsRichard Purdie2021-03-111-11/+11
| | | | | | | | | | | | | | There were two issues in this code, firstly the code could stack duplicates in the variable, secondly, calling "if data" caused the datastore to compute len(data) which is comparitively expensive. Checking "if data is not None" is much much faster/cheaper. The issue was clear from "bitbake -p -P" output where the time in register() showed large amounts of time in the __len__ function of the datastore. (Bitbake rev: 9298d08b7dcf9d79f54536f87686d65c4ad7deb9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: Honor quiet when parsing recipesManuel Leonhardt2021-03-111-1/+1
| | | | | | | | | | | When using parse_recipes, honor quiet so that scripts and custom plugins for recipetool are able to mute progress bars from bitbake that would otherwise print to STDOUT. (Bitbake rev: ab9d26c4847a062cadaae5fb8caac0ead5f958db) Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Print pseudo.log if fakeroot task failedTomasz Dziendzielski2021-03-112-9/+40
| | | | | | | | | | | | | | | | | | | | | | | | Currently if pseudo fails we can only see the path to pseudo.log. If we have no access to server and can only rely on bitbake log then debugging becomes impossible. This printing needs to be added in runqueue level, not inside task execution, because in some cases task fails with pseudo abort really early and we don't even see any log. In this change I'm adding pseudo log printing in every fakeroot task failure that logged `mismatch`, `error` or `fatal` to logfile, because we have no other way to communicate with pseudo if it failed or not. Only lines from last pseudo server execution will be printed. (Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Adrian Walag Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com> Signed-off-by: Wiktor Baura <wbaura@gmail.com> Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event.py: fix regression about INVALIDCONFChen Qi2021-03-111-0/+4
| | | | | | | | | | | | | A previous commit uses __BBHANDLERS_MC to record all handlers, but it does not take into consideration of INVALIDCONF, thus causing regression. We need to record the name before returned AlreadyRegistered, otherwise, when reparsing due to INVALIDCONF, bascially all handlers are not called. (Bitbake rev: f6fb6f22fa6ba1ffe121948d7f53ac724358491e) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: docs: Add AZ_SAS definition to glossaryAlejandro Hernandez Samaniego2021-03-111-0/+13
| | | | | | | (Bitbake rev: 134df841b273d94e767426876459da348d96dd23) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: docs: Add Az fetcher documentationAlejandro Hernandez Samaniego2021-03-112-0/+30
| | | | | | | (Bitbake rev: e3b480636a3c2716effd619b59cf55e11f9a6db0) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add Azure Storage fetcher implementationAlejandro Hernandez Samaniego2021-03-112-1/+96
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allows bitbake to fetch from an Azure Storage account. The fetcher submodule is compatible with the az:// URI protocol, its functionality is based on bitbakes wget fetcher, superior in performance to using a propietary tool like azcopy which can handle cloud storage account operations with more functionality (that we dont need in a fetcher) but less compatibility. A sample URI uses can be defined in the following way: SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz" This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.: SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n" PREMIRRORS_prepend = "\ git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ " Can also be used with non-public access Azure Storage accounts/containers via a Shared Access Signature by declaring the AZ_SAS variable which will be automatically used by the fetcher: AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature" (Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: __init__.py: Fix bitbake debug log handlingRichard Purdie2021-03-091-1/+1
| | | | | | | | | | | | For a while I've been puzzled as to why debug logging from runqueue wouldn't appear on the console with -DD. The logic in the bbdebug handling is inverted so fix it and now we see the expected messages from runqueue with -D and -DD. This should then let us debug other issues using those log messages. (Bitbake rev: 34145b950be03aff8f9b88207cf843abf002ab13) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add setscene task overlap sanity checkRichard Purdie2021-03-091-0/+4
| | | | | | | | | | We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. (Bitbake rev: 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task execution corruption issueRichard Purdie2021-03-091-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. (Bitbake rev: f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatusAlejandro Hernandez Samaniego2021-02-261-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | If an exception is raised when running host python code, the fetcher immediately crashes, this might be temporary depending on the servers reliability. Catch the exception when the connection was reset and try once again to fetch the data. File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto 0665: if self._timeout_occurred: 0666: raise OSError("cannot read from timed out object") 0667: while True: 0668: try: *** 0669: return self._sock.recv_into(b) 0670: except timeout: 0671: self._timeout_occurred = True 0672: raise 0673: except error as e: Exception: ConnectionResetError: [Errno 104] Connection reset by peer (Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Fix broken builds when multiconfig has a hyphen in the nameJoshua Watt2021-02-217-26/+26
| | | | | | | | | | | | | | | 5f7fdf7b2d ("bitbake: event: Prevent bitbake from executing event handler for wrong multiconfig target") broke multiconfig builds contain a hyphen, since it's attempt to use the multiconfig as part of a function name and python functions are not allowed to contain a hyphen. Rework the bitbake multiconfig test to test a multiconfig with a hyphen and one with an underscore to validate this doesn't break in the future. (Bitbake rev: c3168df330a4563cbd03ba74de55a22217d823ed) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Force parser shutdown after catching an exceptionJan Brzezanski2021-02-211-5/+5
| | | | | | | | | | | | | | | | Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. (Bitbake rev: 915330e1dbae1ee8fd9a0358decf2c294f771961) Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: providers: use pythonic empty list checkCharlie Davies2021-02-211-1/+1
| | | | | | | (Bitbake rev: 591542734988856afa53b99e5eef6e5ca15630b8) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: providers: fix incorrect return type bugCharlie Davies2021-02-211-1/+1
| | | | | | | | | | | | | | | | | | If no eligible providers are found then an integer zero is returned. This causes the following error, in two possible places in taskdata.py, when the return value is used in a list comprehension: [snip] eligible = [p for p in eligible if not p in self.failed_fns] TypeError: 'int' object is not iterable [\snip] Fix by returning the variable eligible itself, of type list. (Bitbake rev: 217c4b436b588a6a47aeaddf61531711ad3fca67) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: contrib: Add Dockerfile for building hash serverJoshua Watt2021-02-161-0/+19
| | | | | | | | | | Adds a Dockerfile for build the reference hash equivalence server in a container (Bitbake rev: d9a0a88db5888039bfbb6ce5129b43350f79b1c1) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump version to 1.49.2Richard Purdie2021-02-162-2/+2
| | | | | | | | This allows metadata to rely upon BB_DEFAULT_UMASK. (Bitbake rev: 969ac64adab236ce2d5196bcc294005a497913ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker/runqueue: Add support for BB_DEFAULT_UMASKRichard Purdie2021-02-163-2/+11
| | | | | | | | | | | | | Currently each task has to have a umask specified individually. This is leading to determinism issues since it is easy to miss specifying this for an extra task. Add support for specifing the default task umask globally which simplifies the problem. (Bitbake rev: 3e664599fd54a8a37ce587022fcbce5ca26f2ed3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Prevent bitbake from executing event handler for wrong ↵Tomasz Dziendzielski2021-02-164-6/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | multiconfig target When multiconfig is used bitbake might try to run events that don't exist for specific mc target. In cooker.py we pass `self.databuilder.mcdata[mc]` data that contains names of events' handlers per mc target, but fire_class_handlers uses global _handlers variable that is created during parsing of all the targets. This leads to a problem where bitbake runs event handler that don't exist for a target or even overrides them - if multiple targets use event handler with the same name but different code then only one version will be executed for all targets. See [YOCTO #13071] for detailed bug information. Add mc target name as a prefix to event handler name so there won't be two different handlers with the same name. Add internal __BBHANDLERS_MC variable to have the handlers lists per machine. (Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump release to 1.49.1Richard Purdie2021-02-102-2/+2
| | | | | | (Bitbake rev: 9f23fa605c542a705d00c6c263491899d55bb0d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-1034-228/+237
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bblayers/action: When adding layers, catch BBHandledExceptionRichard Purdie2021-02-101-2/+2
| | | | | | | | | | | When adding a layer, parse error can occur, raising BBHandledException. Catch this and error, aborting the layer add to meet user expectations. [YOCTO #14054] (Bitbake rev: ceddb5b3d229b83c172656053cd29aeb521fcce0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure reparsing is handled correctlyRichard Purdie2021-02-101-0/+2
| | | | | | | | | | | | | | | From tinfoil, if you edit bblayers.conf and break it, then call parseConfiguration (e.g. by adding a bad layer with bitbake-layers), the system doens't show any parse error yet it should. Add in a call to the updateCache function so that things really are reparsed when requested. Partially fixes [YOCTO #14054] (Bitbake rev: e655f9361b9c3b77906b8e06b5cc76bc5180640e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Don't classify shell functions that names start with ↵Tomasz Dziendzielski2021-02-101-1/+1
| | | | | | | | | | | | | | | | | | "python*" as python function If shell function name starts with 'python' or 'fakeroot' parser wrongly assumes it's python/fakeroot function. [YOCTO #14204] Use regex lookahead assertions to check if 'python' expression is followed by whitespace or '(' and if 'fakeroot' is followed by whitespace. (Bitbake rev: b07b226d5d1b3acd3f76d8365bc8002293365999) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add get-outhash messagePaul Barker2021-02-102-14/+38
| | | | | | | | | | | | | | | The get-outhash message can be sent via the get_outhash client method. This works in a similar way to the get message but looks up a db entry by outhash rather than by taskhash. It is intended to be used as a read-only form of the report message. As both handle_get_outhash and handle_report use the same query string we can factor this out. (Bitbake rev: dc19606ada29a4d8afde4fcecd8ec986b47b867e) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: server: Support searching upstream for outhashPaul Barker2021-02-102-0/+32
| | | | | | | | | | | | | Use the new get-outhash message to perform a read-only query against an upstream server (if present) when a reported taskhash/outhash combination is not found in the current database. If a matching entry is found upstream it is copied into the current database so it can be found by future queries. (Bitbake rev: 2be4f7f0d2ccb09917398289e8140e1467e84bb2) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add short forms of remaining command line argumentsPaul Barker2021-02-101-3/+3
| | | | | | | | | Short form arguments are added for convenience. (Bitbake rev: 921199a4923ce383b27e23c9b7e34eb785c8bae3) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Support upstream command line argumentPaul Barker2021-02-101-1/+2
| | | | | | | | | | | The hashserv server already implements support for pulling hash data from another "upstream" server. Add the -u/--upstream argument to the bitbake-hashserv app to expose this functionality to users. (Bitbake rev: 8de510f1de35e581bcd5858edf23619c6a4cf923) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Support read-only serverPaul Barker2021-02-104-12/+53
| | | | | | | | | | | | | | | | | | The -r/--readonly argument is added to the bitbake-hashserv app. If this argument is given then clients may only perform read operations against the server. The read-only mode is implemented by simply not installing handlers for write operations, this keeps the permission model simple and reduces the risk of accidentally allowing write operations. As a sqlite database can be safely opened by multiple processes in parallel, it's possible to start two hashserv instances against a single database if you wish to export both a read-only port and a read-write port. (Bitbake rev: 492bb02eb0e071c792407ac3113f92492da1a9cc) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Try and avoid potential short write events issuesRichard Purdie2021-02-061-2/+6
| | | | | | | | | | | | | | | | | We're seeing occasional issues where builds fail as events were written from the worker children in the form <event>partial data<event>full event</event>. This causes failures as bitbake server can't parse that and exits. This could be due to short writes to the worker event pipe which we weren't checking. Check this and loop accordingly. Also add some asserts to detect other potential causes. Thanks to Joshua Watt for help in spotting the issue. [YOCTO #14181] (Bitbake rev: a9451746a4bd7ccedf4c72cd03ad4ff0ab0143aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: client: Fix handling of null responsesPaul Barker2021-02-061-1/+1
| | | | | | | | | | If the server returns an empty response ("null" in json), this cannot be iterated to check for the presence of the "chunk-stream" key. (Bitbake rev: bf75370bcd6d02ed08cd959eec6190196b792515) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-hashclient: Remove obsolete call to client.connectPaul Barker2021-02-061-3/+0
| | | | | | | | | | | | The connect function was previously removed from the hashserv Client class but the bitbake-hashclient app was not updated. The client is connected during hashserv.create_client() anyway so not separate connect call is needed. (Bitbake rev: 99bdb236bceeffa0083a0fa529280b217c1d310d) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Don't treat mc recipe (Midnight Commander) as a multiconfig ↵Tomasz Dziendzielski2021-02-064-8/+8
| | | | | | | | | | | | | | | | target When we run `devtool build mc` recipe's task dependencies are expanded to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig and "do_package_sysroot" as multiconfigname. | ERROR: Multiconfig dependency mc:do_populate_sysroot depends on | nonexistent multiconfig configuration named do_populate_sysroot (Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: download LFS content too during do_fetchMatt Hoosier2021-01-232-11/+61
| | | | | | | | | | | | | | Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] (Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: add test for empty query parametersRoss Burton2021-01-231-0/+19
| | | | | | | | | To exercise the previous patch, add a test case. (Bitbake rev: efdf6d5c4cd8155bc54dc667346f58dccb138e07) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle empty elements in _param_str_splitRoss Burton2021-01-231-1/+1
| | | | | | | | | | | | | _param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws an exception if a lone semicolon is used as the value doesn't split into two items. Fix by checking that the result of the first split has content. (Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/perforce: Fix localfile to include ud.moduleRichard Purdie2021-01-231-1/+2
| | | | | | | | | As reported by Katu Txakur, the output depends on the module path set so this needs to be accounted for in localfile. (Bitbake rev: 9861ed37bb1c5d09c3b4852d2a252e3f3c86ab14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Update user-agentChristophe Priouzeau2021-01-231-3/+8
| | | | | | | | | | | | With the usage of enterprise proxy, the user-agent defined are too old and refused by proxy configuration. Updating to something more modern is desirable. (Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400) Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com> Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Fix usehead for non-default namesJoey Degges2021-01-082-8/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Test usehead with a non-default nameJoey Degges2021-01-081-0/+41
| | | | | | | | | | | | Add tests for fetching a URL with the usehead parameter set and a non-default name set. We currently expect the local version of this test to fail since there is a bug in the usehead implementation that breaks for non-default names. (Bitbake rev: a2345110f217fac429f6ec15f699c87c39531e7c) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Document behavior of test_gitfetch_useheadJoey Degges2021-01-081-0/+4
| | | | | | | | | | | | The test `test_gitfetch_usehead' exercises a way to override the usehead feature by setting SRCREV. It may not be obvious that this is what is being exercised here so let's add some comments to document the expected behavior. (Bitbake rev: 1cd998c19101e3b093e81c126b3048c5d56058b0) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Organize usehead tests by net requirementsJoey Degges2021-01-081-28/+27
| | | | | | | | | | | | | Move the local only usehead test to the FetcherLocalTest class so it will be run when BB_SKIP_NETTESTS=yes since it does not require network access. Rename the usehead tests to better match the new organization. (Bitbake rev: 137cfa13d5319bc91c3e5fe6c7062cb8c8484d64) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add docstrings to functionsMilan Shah2021-01-081-4/+34
| | | | | | | | | | | | | | | | | | A list of functions that now has a docstring. * vercmp_string * explode_dep_versions * prunedir * prune_suffix * to_boolean * contains_any * export_proxies See [YOCTO #9725] for details. (Bitbake rev: b61ba4a18693a9e553d2a93161feb0bcc1c82384) Signed-off-by: Milan Shah <mshah@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Include all packages a recipe provides in ↵Peter Kjellerstedt2020-12-211-1/+3
| | | | | | | | | | | | | | | | | | | | | | | SkippedPackage.rprovides The provided packages by a skipped recipe are supposed to be listed in SkippedPackage.rprovides, which is used when generating a meaningful error message when a build fails because of a skipped package. Previously this variable only contained the contents of ${RPROVIDES}. However, most recipes don't define RPROVIDES, they define RPROVIDES_<pkg> for each package they provide. Additionally, the recipe provides the packages in PACKAGES without them being included in ${RPROVIDES}. Before this change, having a runtime dependency on a skipped non-recipe package would result in a build error stating that the build failed because the package was skipped, but without providing any reason for why it was skipped. (Bitbake rev: efd026c26a377b826a49b945a8212bf7de8a480a) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Make CoreRecipeInfo include rprovides_pkg for skipped recipesPeter Kjellerstedt2020-12-211-2/+2
| | | | | | | | | This will be needed by SkippedPackage in the cooker. (Bitbake rev: 93d01614565bd540d05fbc1791dd66e46723d683) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Ensure hash reflects vardepvalue flags correctlyRichard Purdie2020-12-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | The get_hash() function is used to decide if the base configuration has changed and hence whether a reparse is required. The vardepvalue flag's value was not expanded but it is often used in contexts like: METADATA_REVISION = "${@base_detect_revision(d)}" METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" which in it's unexpanded form means reparsing doesn't happen when it should as the data appears unchanged. Update get_hash to expand the values of vardepvalue so reparsing works as expected. This avoids basehash mismatch errors such as the one recently caused by using METADATA_REVISION in poky.conf's DISTRO_VERSION variable. The issue there could be exposed by a recipe using DISTRO_VERSION with the sequence: bitbake os-release <change the revision of the metadata with a dummy commit> bitbake os-release -C install which was caused because METADATA_REVISION changed but the metadata didn't reparse. (Bitbake rev: 26ccf1575aef2d6e2d7717d3bd10b1ed0d5a777d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: msg: use safe YAML loaderRoss Burton2020-12-101-1/+1
| | | | | | | | | | | | | | | | | If using a YAML file in BB_LOGCONFIG, the unsafe loader is used and this causes a runtime warning: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. As log configuration YAML is relatively simple we can just use safe_load(). (Bitbake rev: 71aaac9efa69abbf6c27d174e0862644cbf674ef) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Fix broken AF_UNIX path length limitJoshua Watt2020-12-102-5/+26
| | | | | | | | | | | | Fixes the bug were long paths would break Unix domain socket clients (for real this time; the previous attempt was missing os.path.basename). Adds some tests to prevent regressions (Bitbake rev: 77790e3656048eff5cb1a086c727d86d32773b68) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch2: use relative symlinks for anything pulled from ↵Richard Purdie2020-12-091-5/+2
| | | | | | | | | | | | | PREMIRRORS" This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb. Unfortunately this caused problems where incorrect paths were found in paths and files were not being found. This was reported on the mailing list but no invetigation or resolution was found. Revert until we can figure out what was wrong. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: client: Fix AF_UNIX path length limitsJoshua Watt2020-12-091-3/+12
| | | | | | | | | | | | | | | | Restores a fix for unix domain socket path length limits when using the synchronous hash equivalence client that was accidentally removed when the async client was added. Unfortunately, it's much more difficult to fix the same problem when using the async client directly due to the interaction of chdir() and async code, but this will at least restore the old behavior in the synchronous case. (Bitbake rev: 53e85022a8b1c8f407c9418260c59beffb96f0f9) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-29 02:47:15 +0000