summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-04-141-2/+6
| | | | | | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: 5b105e76dd7de3b9a25b17b397f2c12c80048894) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: toaster: Add refreshed oe-core and poky fixturesTim Orling2023-03-152-31/+31
| | | | | | | | | | | After updating gen_fixtures.py, run ./gen_fixtures.py --all This includes the latest stable/supported releases. (Bitbake rev: 88a9cc0318c6ef01976d8b09dfc1a92cf0a1e498) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fixtures/gen_fixtures.py: update branchesTim Orling2023-03-151-5/+6
| | | | | | | | | | | Re-introduce Dunfell since it is supported until April 2024 Drop Honister and Hardknott as they are both EOL Add Langdale as it is the latest stable release (Bitbake rev: 2dda597de3fcf6911e4022a8c279ae00413ec747) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fixtures/README: django 1.8 -> 3.2Tim Orling2023-03-151-1/+1
| | | | | | | | | We should reference the docs for the current LTS version we are using. (Bitbake rev: add4569195f8f64000555679e282b0b12d1ca3f8) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Drop dubious exception handling codeRichard Purdie2023-03-141-5/+0
| | | | | | | | | | | | | This code appears to be dangerous, it swallows exceptions, turning them into "handled" versions which then show no errors to the user. This is a pretty poor user experience and I can't see why this code should be swallowing such things. Drop the worst bits of code. (Bitbake rev: 2b239555f76e4e98ca704e7ef60e796d0f19463c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Improve early exception handlingRichard Purdie2023-03-141-9/+1
| | | | | | | | | | | | | | | | | | Martin Jansa reported that if you put a syntax error into an imported module such as qa.py in OE, no error is shown. Part of the issue appears to be that the catch_parse_error() decorator only catches certain exceptions and SyntaxError isn't one of them. As far as I can tell we should remove all the special cases and use the more advanced code in all cases, not just expansion errors. I confirmed this now prints a proper error message for a qa.py syntax error. (Bitbake rev: 4b922345a40f7cc803eb46c4906269691d408940) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Remove incorrect SystemExit usageRichard Purdie2023-03-141-1/+1
| | | | | | | | | | | | | Calling SystemExit doesn't work well with server/client usage since the string isn't printed to the right place. Use bb.fatal() instead which prints the right log output and raises and handled exception which then shows correctly on the UI. (Bitbake rev: 8001c9b5d3a3111f1134557f221325fe2593c2d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Allow to_boolean to support int valuesRichard Purdie2023-03-141-0/+3
| | | | | | | | | | | | | | | | | | Some variables may be set as: X = 1 as well the more usual X = "1" so add support to to_boolean to handle this case. (Bitbake rev: 3cc9fe911f764e4553078dbeed9497f6f08336ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the systemFrank de Brabander2023-03-141-0/+16
| | | | | | | | | | | | | | | | | Get rid of the duplicate code and add extra check that the locale en_US.UTF-8 is available on the system. This new helper method is now located right above the method filter_environment() which sets LC_ALL environment variable to 'en_US.UTF-8'. [YOCTO #10165] (Bitbake rev: 2fe0c90da89b0a4e2e133f8ffa7a93d71097bb32) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a4ce040a6fd540a1cac52f808f909f9fcf8c961c) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix inefficient string concatenationEtienne Cordonnier2023-02-171-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython has an optimization for statements in the form "a = a + b" or "a += b". It seems that this line does not get optimized, because it has a form a = a + b + c: data = data + "./" + f.split("/./")[1] For that reason, it does a copy of data for each iteration, potentially copying megabytes of data for each iteration. Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds instead of 45 seconds on my test setup where SRC_URI points to a big directory. Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython: "do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b" However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup: data = ''.join((data, "./", f.split("/./")[1])) I have changed the other lines to also use += for consistency only, however those were in the form a = a + b and were optimized already. Co-authored-by: JJ Robertson <jrobertson@snap.com> (Bitbake rev: 592ee222a1c6da42925fb56801f226884b6724ec) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Clarify the meaning of namespaceMarek Vasut2023-02-041-1/+2
| | | | | | | | | | | | | | | | | Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: a40fc6a3f774bcb28cf72701ac146ceb7ae8061a) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository ↵Marek Vasut2023-02-041-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | metadata The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> (Bitbake rev: 2b4583922de184a2d96324cd4e3f85993341cd14) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: include SSL certificate paths in export_proxiesRoss Burton2023-01-301-12/+9
| | | | | | | | | | | | | | | | | | | | | | | | bb.utils.export_proxies() is a poor-man's alternative for the environment setup code in bb/fetch2, but it's used in several places where recipes want to download manually (such as cve-update-db-native). Notably, export_proxies() doesn't pass on the SSL certificate paths from the original environment, so if SSL_CERT_FILE needs to be set (for example, in a buildtools environment) then proxies work but SSL doesn't. In an ideal world export_proxies and the same logic in fetch2 would merge, but until then we can add the SSL_CERT_ variables and duplicate the basic logic: check the datastore first and then the original environment for variables. Also remove the return value as nothing ever checked it. [ YOCTO #15000 ] (Bitbake rev: 28ba566c25af72afe62ea3cb62b0bafeffc47de8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Add bitbake.sock race handlingRichard Purdie2023-01-161-1/+11
| | | | | | | | | | | | | | | | | | | We've seen cases where the bitbake.sock file appears to disappear but the server continues to hold bitbake.lock. The most likely explaination is that some previous build directory was moved out the way, a server there kept running, eventually exited and removed the sock file from the wrong directory. To guard against this, save the inode information for the sock file and check it before deleting the file. The new code isn't entirely race free but should guard against what is a rare but annoying potential issue. (Bitbake rev: 52b6b099a47555811b8d0b311f62af712dd6eb8e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b02ebbffdae27e564450446bf84c4e98d094ee4a) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: log odd unlink events with bitbake.sockFrank de Brabander2023-01-161-2/+3
| | | | | | | | | | | | | | | Log when the socket file already exists and is removed before recreating a new socket. Log when unlinking the socket file failed. (Bitbake rev: 9779fad4d9e2540b24bb91cfa38fc1984402bef2) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cfd7c9899f988bab6d9fe7bbfbdb60603fb5ed34) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix regression in gitsm submodule path parsingPavel Zhukov2023-01-061-1/+1
| | | | | | | | | | | | | | | | | | | Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression in submodules path parsing. As the result gitsm fetcher fails on each submodule which name begins from the name of the parent repo which is totally valid usecase [Yocto #14045] [1] Fix the code to error out only if submodule's name is equal to parent name but not if it's part of it. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4 (Bitbake rev: f0f166aee766b4bb1f8cf8b35dfc7d406c75e6a4) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix race issues around hash equivalence and sstate reuseRichard Purdie2022-11-241-16/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | We identified a use case where a native recipe (autoconf-native) was rebuilt with no change in output yet the sstate for do_package tasks wasn't being used. The issue is that do_package tasks have a hard dependency on pseudo-native:do_populate_sysroot. That task was one of the many tasks being rehashed when autoconf-native's hash was changed. If update_tasks processed a recipe before it had processed pseudo-native, that recipe would be marked as not possible from sstate and would run the full tasks. The fix is to split the processing into two passes, first to handle the existing covered/notcovered updates, then in the second pass, check whether there are "harddep" issues. This defers the do_package tasks until after pseudo-native is installed from sstate as expected and everything works well again. (Bitbake rev: 3b2d1331487a74863e74cc0c5564004cbb1b5a4a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e479d1e418a7d34f0a4663b4a0e22bb11503c8ab) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: bitbake-layers: checkout layer(s) branch when clone existsMark Asselstine2022-11-101-0/+25
| | | | | | | | | | | | | | | | | | | | | | | [YOCTO #7852] Fixes 'bitbake-layers layerindex-fetch --branch kirkstone meta-arm' not checking out the branch if the repo is already cloned and on a different branch. If a clone of a layer being added already exists check what branch it is on and if necessary attempt to switch to the given branch. If the switch fails to happen the git error will be reported. We also warn if there are uncommitted changes as the changes might go unnoticed and result in unexpected behaviors. (Bitbake rev: 138dd7883ee2c521900b29985b6d24a23d96563c) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d2cb388f58a37db2149fad34e4572d954e6e5441) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: serv: correct closed client socket detectionJustin Bronder2022-11-101-2/+2
| | | | | | | | | | | | | | | | | If the client socket is closed, asyncio.StreamReader.readline() will return an empty bytes object, not None. This prevents multiple tracebacks being logged by bitbake-hashserv each time bitbake is started and performs a connection check. (Bitbake rev: 4bdd9ba43f34a1473db31a6a3b10bd33e358fe3a) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2d07f252704dff7747fa1f9adf223a452806717f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: don't set core.fsyncobjectfiles=0Ross Burton2022-11-101-1/+1
| | | | | | | | | | | | | | | | | | | | This git configuration variable is deprecated in 2.36.0 onwards, so git warns in the logs for every git call. Luckily the default value has always been false[1], so we can just remove this. [ YOCTO #14939 ] [1] https://github.com/git/git/commit/aafe9fbaf4f1d1f27a6f6e3eb3e246fff81240ef (Bitbake rev: 13f86aeb53cd73c03bfb2f00fe923b51ec8d1c73) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8ad310633e0c5d5593631c1196cbdde30147efce) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie2022-10-261-2/+2
| | | | | | | | | CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: bb.tests.fetch.URLHandle: add 2 new testsMark Asselstine2022-10-261-0/+2
| | | | | | | | | | | | | | | | | Add a test for special characters in user and password to qualify decodeurl() inspired by a bug report describing that '=' signs in a password was problematic. Add a second test to qualify decodeurl() as related to the change in commit 628c4bf6c89b [fetch2/__init__: handle @ in package names]. Relates to [YOCTO #14476] (Bitbake rev: ee04cf09c7022168c035affa654773652a49793e) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/ply: Update md5 to better report errors with hashlibMark Hatle2022-10-262-1/+13
| | | | | | | | | | | | | | | | | | | | | | In the case where hashlib is not available, the try would fail and fall through resulting in a backtrace on the usage of the 'sig'. The backtrace itself was confusing and made it difficult to determine what went wrong. Update the import to be in it's own try block with an appropriate message to indicate what went wrong. Note, the current version of ply all of this code has been restructured so this is not applicable upstream. Additionally, some versions of hashlib don't appear to implement the second FIPS related argument. Detect this and support both versions. (Bitbake rev: 484ab42f440070c0369b81f5c69da860fa47a798) Signed-off-by: Mark Hatle <mark.hatle@amd.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.2.0Richard Purdie2022-09-291-1/+1
| | | | | | (Bitbake rev: 074da4c469d1f4177a1c5be72b9f3ccdfd379d67) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add enable_loopback_networking()Mattias Jernberg2022-09-291-1/+41
| | | | | | | | | | | | | It can be used to enable the loopback interface, typically after calling disable_network(). Also correct a typo in a debug message. (Bitbake rev: 0d317209d4234c5f05a9fcdc13c52f502f104018) Signed-off-by: Mattias Jernberg <mattias.jernberg@axis.com> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/ssh.py: fix checkstatusPascal Bach2022-09-291-4/+2
| | | | | | | | | | | | | | | | | The output of runfetchcmd is always empty in this case, as the test doesn't produce any output. SSH either returns 0 or 1, which is handled via exceptions. This means the current check is not only unnecessary but prevents the function from working. We can just assume that if we reach the end of the function that the file exists and return True. (Bitbake rev: d599af48635fab587e5b913591b95daf87b40080) Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate fixup c212b0f3 loglevelJose Quaresma2022-09-211-1/+1
| | | | | | | | | | c212b0f3 change the debug log level unintentional when tryng to fix a knotty issue. This will maintain the same debug log level 2 as before. (Bitbake rev: 19f8265023281f3b1d5d0a02e47f8d7d08cfcc16) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: warning when the debug message is invalidJose Quaresma2022-09-211-0/+4
| | | | | | | | | | | | | There are many messed up calls for the debug log, so is better to warm about this as they will not work as expected. The level need to be an integer and the msg a string. (Bitbake rev: c1d9c1d25ce36848040dc0ce182835e497ccbb82) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Fix a few more logger debug() callsJoshua Watt2022-09-212-14/+14
| | | | | | | | | | | f68682a7 ("logging: Make bitbake logger compatible with python logger") replaced several .debug() calls to make them comply with the standard python logging API, but a few were missed. (Bitbake rev: eb25cd4d64b9a4e8e2b2ce7fbccc123d00b2fc2b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Remove unneeded conditionalPaulo Neves2022-09-211-7/+6
| | | | | | | | | | The condition will always evaluate to true and thus is redundant. (Bitbake rev: be1ee681e8a566564549068dcf90c95c36544815) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure mirror tarballs don't enforce checksumRichard Purdie2022-09-071-0/+1
| | | | | | | | | local file fetches now validate checksums. The checksums for mirror tarballs of repositories will not match so ignore these checksums. (Bitbake rev: 6424f4b7e9c1ba8db81346e8b3a806dd035d4551) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fix kirkstone versionOtavio Salvador2022-09-071-1/+1
| | | | | | | (Bitbake rev: 04014b8b2c3d7bb80d7d8dca97b7472f0e6b4ebb) Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc/client: Fix unix domain socket chdir race issuesRichard Purdie2022-09-051-9/+13
| | | | | | | | | | | | | | | | | | | | The connect_unix() call had a bug where if a relative path to a socket was passed (which the non-async client always does), and the current working directory was changed after the initial call, it would fail to reconnect if it became disconnected, since the socket couldn't be found relative to the new current working directory. To work around this, change the socket connection for UNIX domain sockets to be synchronous and change current working before connecting. This isn't ideal since the connection could block the entire event loop, but in practice this shouldn't happen since the socket are local files anyway. Help debugging and resolving from Joshua Watt. (Bitbake rev: 5964bb67bb20df7f411ee0650cf189504a05cf25) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Avoid bad checksum race issuesRichard Purdie2022-09-052-4/+12
| | | | | | | | | | | | | | | | | If two recipes have conflicting checksums for a file, the code will currently remove the existing file when a mismatch is downloaded, even if another task successfully fetched it. This changes the code to verify the checksum (if possible) before replacing the file. This removes a potential race window and stops builds failing everywhere from one incorrect checksum. To make this work, we need to be able to override localpath and avoid NoChecksum errors being logged. (Bitbake rev: 4b8de2e7d12667d69d86ffe6e9f85a7932c4c9a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch: use BPN instead"Richard Purdie2022-09-011-1/+1
| | | | | | | | | | PN is correct here, bitbake has no knowledge of BPN. This reverts commit d613e48c07d4b12219270c1359cbf2f390b848dd. (Bitbake rev: cffcfacb747d41304c857b17bfea646e220b2389) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: use BPN insteadMingli Yu2022-09-011-1/+1
| | | | | | | | | | | | When checking for the non-existing file, BPN is actually the acutal recipe name. And we should use BPN for the error message and it also fix the below test when multilib is enabled. $ oe-selftest -r bbtests.BitbakeTests.test_invalid_recipe_src_uri (Bitbake rev: d613e48c07d4b12219270c1359cbf2f390b848dd) Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: Add test for possible gitsm deadlockPavel Zhukov2022-09-011-0/+9
| | | | | | | (Bitbake rev: b0506480baa9bcf3ef645b0aed5a07ad9950245c) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: Add Timeout classPavel Zhukov2022-09-011-0/+19
| | | | | | | | | | | | | The class and exception aim to test rare cases there deadlocks are possible. Can be used in context managers: with Timeout(<value>): do_deadlock() (Bitbake rev: c5fcdd804d422f959a189b270d72123a50e74da6) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Error out if submodule refers to parent repoPavel Zhukov2022-09-011-0/+3
| | | | | | | | | | | | | | If submodule refers to specific revision of the parent repository it causes deadlock in bitbake locking mechanism (lock is acquired to fetch the parent and cannot be released before all submodules are fetched). raise FetchError in such situation to prevent deadlocking. [Yocto 14045] (Bitbake rev: 0361ecf7eb82c386a9842cf1f3cb706c0a112e77) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Fix npm to use https rather than httpNeil Horman2022-09-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Hit this error while building nlf-native recently: { "error": { "summary": "URI malformed", "detail": "" } } Some poking about led me to discover that: 1) The npm.py tool replaces npm:// with http://, not https:// 2) Some versions of the npm tool don't handle 301 redirects properly, choosing to display the above error instead when using the default nodejs registry It would be good to go fix npm to handle the redirect properly, but it seems like it would also be good to assume secure http when contacting a registry, hence, this patch (Bitbake rev: 2cd76e8aabe4e803c760e60f06cfe1f470714ec7) Signed-off-by: Neil Horman <nhorman@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Pass lock argument in fileslockedJoshua Watt2022-08-311-2/+4
| | | | | | | | | | | | Pass additional arguments in the fileslocked() context manager to the underlying lockfile() function. This allows the context manager to be used for any types of locks (non-blocking, shared, etc.) that the lockfile() function supports. (Bitbake rev: 7a8eb8da8e8495051e174721062da08e06168024) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change pressure file warning to a noteRichard Purdie2022-08-241-1/+1
| | | | | | | | | The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: 9bdedc8074990e613c9567e2cd8072f8d885f07f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler/cooker: Implement recipe and global classesRichard Purdie2022-08-126-22/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | We have some confusion for users since some classes are meant to work in the configuration space (or "globally") and some are meant to be selected by recipes individually. The cleanest way I could find to clarify this is to create "classes-global" and "classes-recipe" directories which contain the approproate classes and have bitbake switch scope between them at the appropriate point during parsing. The existing "classes" directory is always searched as a fallback. Once a class is moved to a specific directory, it will no longer be found in the incorrect context. A good example from OE is that INHERIT += "testimage" will no longer work but IMAGE_CLASSES += "testimage" will, which makes the global scope cleaner by only including it where it is useful and intended to be used (images). (Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add copyright headers where missingRichard Purdie2022-08-1225-0/+50
| | | | | | | | | Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Make inherit calls more directlyRichard Purdie2022-08-121-1/+4
| | | | | | | | | | | Rather than recursing into the conf handler code, simply call into the parse code directly when inheriting files as we've already resolved the paths and don't need anything the other codepath brings. This makes the codepath clearer at the expense of some slight duplication. (Bitbake rev: 0f4f3af6d93a0018df58b8a1d8d423c78ba6526d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Allow earlier exit for classes not foundRichard Purdie2022-08-121-0/+3
| | | | | | | | | Rather than relying on later code to error if the class isn't found, exit earlier and more clearly from a code perspective. (Bitbake rev: f7c55c8147329670fd5bc55b1ae3f47f25b89bab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: prefix the tasks with a timestamp in the log task_orderJose Quaresma2022-08-121-1/+3
| | | | | | | | | | This is useful when debugging as it helps understand possible race conditions between tasks of diferent recipes. (Bitbake rev: 950a2ea4c91d6e13d7587104367fa85cc7efe01c) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add memory pressure regulationAryaman Gupta2022-08-121-5/+22
| | | | | | | | | | | | | | | | | | | Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix insufficent entropy in sigtask file namesJoshua Watt2022-08-092-1/+22
| | | | | | | | | | | | | | | | | | | | | | | | | Signature generation uses mkstemp() to get a file descriptor to a unique file and then write the signature into it. However, the unique file name generation in glibc is based on the system timestamp, which means that with highly parallel builds it is more likely than one might expect expected that a conflict will occur between two different builder nodes. When operating over NFS (such as a shared sstate cache), this can cause race conditions and rare failures (particularly with NFS servers that may not correctly implement O_EXCL). The signature generation code is particularly susceptible to races since a single "sigtask." prefix used for all signatures from all tasks, which makes collision even more likely. To work around this, add an internal implementation of mkstemp() that adds additional truly random entropy to the file name to eliminate conflicts. (Bitbake rev: 97955f3c1c738aa4b4478a6ec10a08094ffc689d) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop deadlock breaking force failRichard Purdie2022-08-081-3/+0
| | | | | | | | | | | | | | I'm 99% certain this failing of a scenequeue task corrupts runqueue and causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted and did weird things so drop this code. We've seen builds where the deadlock triggers and it then tries to run tasks where the SQ task already ran with very confusing failures. It is likely it is this code causing it. (Bitbake rev: 8efced47fcb47851a370fd6786df6fb377f99963) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-10 19:09:09 +0000