summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: event/utils: Avoid deadlock from lock_timeout() and recursive eventsRichard Purdie2025-03-132-1/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7f0e797d3100> which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. (Bitbake rev: 82b9f42126983579da03bdbb4e3ebf07346118a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: utils: Add signal blocking for lock_timeoutRichard Purdie2025-03-131-0/+2
| | | | | | | | | | | | We never want to exit whilst holding these locks as it deadlocks all python threads. Add signal blocking around the lock critical part so a signal shouldn't cause such an exit. (Bitbake rev: de552cf58fcbdea64b359a0fdcdf3673da3a780d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a097755c671e2b530dea6200a94b39fa9dca246c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: utils: Tweak lock_timeout logicRichard Purdie2025-03-131-1/+1
| | | | | | | | | | | | We should really try and take the lock in the try/finally block so that in some rare cases such as badly timed interrupt/signal, we always release the lock. (Bitbake rev: f0f7bed1291afd63a27f1863901445e6d40c4a09) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: utils: Print information about lock issue before exitingRichard Purdie2025-03-131-0/+1
| | | | | | | | (Bitbake rev: bde34a174886161def6f9f2c5194870a4edfefb2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cdf6c51a064f8f335c3262b7f102618996f1a229) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bblayers/query: Fix using "removeprefix" string methodJoerg Schmidt2025-02-211-1/+2
| | | | | | | | | | | | | | | | | | The minimum Python version required for Yocto 5.0 is 3.8 which causes failure in poky/bitbake/lib/bblayers/query.py when listing layers by using command "bitbake-layers show-recipes -f --bare --mc MC" for the given multiconfig MC. The reason for that failure is the use of "removeprefix" string method which got introduced in Python 3.9. This patch replaces the "removeprefix" method with an equivalent solution supported by Python 3.8. (Bitbake rev: 58e5c70a0572ff5994dc181694e05cd5d3ddaf66) Signed-off-by: Joerg Schmidt <joerg.schmidt@garmin.com> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: cooker: Make cooker 'skiplist' per-multiconfig/mcChris Laplante2025-02-124-19/+43
| | | | | | | | | | | | | | | | | | | | Previously, the cooker skiplist was shared across multiconfigs (including default ''). If you had a recipe that was incompatible with several multiconfigs for different reasons, then the displayed reason (i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might vary across invocations of bitbake. This was caused by the random order in which recipes are parsed under different multiconfig contexts, with each skip reason overwriting the previously assigned reason. I hit this specificially when using COMPATIBLE_MACHINE, but COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too. (Bitbake rev: a8578d12dfe5a3c92119bfee224595a392d6b0dd) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c51f01a35ed9a928402eab0899598b5c59602eef) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Fix git shallow test failure with git >= 2.48Richard Purdie2025-01-291-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | >From git 2.48 release notes: """ When "git fetch $remote" notices that refs/remotes/$remote/HEAD is missing and discovers what branch the other side points with its HEAD, refs/remotes/$remote/HEAD is updated to point to it. """ This means with git 2.48 onwards, there is a mystery "HEAD" revision appearing in some of our shallow clone tests. We can avoid this by using the same canonicalization as used for the reference revisions. This resolves autobuilder failures on the Fedora 40 workers. (Bitbake rev: f4b2ae97a8dd745b463e5e10fbc8d30d29d223c5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c83444d1210740e27b1744d3aa7c5cad4e28db2f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix scenetask processing performance issueRichard Purdie2024-12-091-1/+5
| | | | | | | | | | | | | Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. (Bitbake rev: 1452e74cc4ddfaadc6537a049877a66cec253c8d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 923c19b9713e398d8e66e6d4422dfd4c18a03486) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Optimise setscene loop processingRichard Purdie2024-12-091-2/+8
| | | | | | | | | | | | Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. (Bitbake rev: 5465f9b6e3b4748e563efc53657af96f02a41c7a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 00f4d932e3af0eeb333339cbe942010fc76dee0f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix performance of multiconfigs with large overlapRichard Purdie2024-12-091-2/+4
| | | | | | | | | | | | | | | | | | | | | There have been complaints about the performance of large multiconfig builds for a while. The key missing data point was that the builds needed to have large overlaps in sstate objects. This can be simulated by building the same things with just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of deferred tasks. The issue is that the expensive checks in the setscene loop were hit every time through runqueue's execute function before the check on deferred tasks. This leads to task execution starvation as that only happens once per iteration. Move the skip check earlier in the function which speeds things up enormously and should improve performance of such builds for users. (Bitbake rev: 5392a2bf6d82626e11eec5578eb4de53ba09f1fe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9c6c506757f2b3e28c8b20513b45da6b4659c95f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2: use persist_data context managersChris Laplante2024-11-301-31/+31
| | | | | | | | | | | | | | | | | | Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s. See https://docs.python.org/3/whatsnew/3.13.html#sqlite3 The previous commit fixed persist_data's context manager to close the connection, but we were never actually using `with` in the first place. This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: 9602a684568910fd333ffce907fa020ad3661c26) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> (cherry picked from commit 61f803c7d92a012b62837b0cdae4789a394b260e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: persist_data: close connection in SQLTable __exit__Chris Laplante2024-11-301-0/+1
| | | | | | | | | | | | | | | A Connection's __exit__ does not close the connection, but rather deals with transactions. See https://docs.python.org/3/library/sqlite3.html#how-to-use-the-connection-context-manager This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: a5640a2527cf1995e2afb80c95f303dea52c73b3) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> (cherry picked from commit b0fba0f5af555ff970406b5cf3bf8fb113d953d1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/wget: Increase timeout to 100s from 30sRichard Purdie2024-11-301-2/+2
| | | | | | | | | | | Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: 70b09f62bc5622fa64bb668cdad610bd48bf7bb9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: Remove custom exception backtrace formattingJoshua Watt2024-11-305-121/+25
| | | | | | | | | | | | | | | | | | | | | | | | Removes the code in bitbake to show custom backtrace formatting for exceptions. In particular, the bitbake exception code prints function arguments, which while helpful is a security problem when passwords and other secrets can be passed as function arguments. As it turns out, the handling of the custom serialized exception stack frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass formatted exceptions"), which changed the events to pass a preformatted stacktrack list of strings, but the passing of the serialized data was never removed. Change all the code to use the python traceback API to format exceptions instead of the custom code; conveniently traceback.format_exception() also returns a list of stack trace strings, so it can be used as a drop in replacement for bb.exception.format_exception() (Bitbake rev: 1f7889c2f5e192fb108b51883aac8685ccb187f2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of mobile-broadband-provider to ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | decouple from gnome gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 44c4c487e09a383bd7c1d195cb63b7f06c17ac2e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 91e268b11ed683bd197026f9b36001f6d54ee05c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of sysprof to decouple from gnome ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: d6b76a928ed4ffe8bb1b425f02077ce8554d0fbc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 008808755ed6cfeb6c41273e69ce718f0833c26c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Remove downloads/tmpdir when failedRobert Yang2024-11-021-3/+5
| | | | | | | | | | | The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 03387d4f60cbf2ae93bb48d53688fa71197f5179) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Add call_process_submodules() to remove duplicated codeRobert Yang2024-11-021-28/+14
| | | | | | | | | | | There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: a6a5de754494128512ccbc8ce62ec24364fd1bb8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/git: Use quote from shlex, not pipesRichard Purdie2024-10-011-2/+1
| | | | | | | | | | | The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 9ef4f2a30127455562e38e869793a723eed6c07c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: don't try to preserve all attributes when unpacking filesRoss Burton2024-10-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When copying files as part of the unpack we currently use cp -p, which is a shortcut for --preserve=mode,ownership,timestamps. We do want to preserve timestamps, because some fetchers set these explicitly. We don't care about ownership. If the files are owned by us then they ill remain owned by us, and if they're not then the attempt to change ownership will be silently ignored. In a shared DL_DIR where files have group ownership this group access isn't relevant in the single-user build tree. We do want to preserve executable bits in the mode, but cp always does this. The difference between --preserve=mode and no --preserve is that the mode isn't preserved exactly (no sticky bits, no suid, umask is applied) but this also isn't a relevant difference in a build tree. Also expand the arguments to be clearer about what options are being passed. The impetus for this is that coreutils 9.4 includes a change in gnulib[1] and will now try to preserve permission-based xattrs if asked to preserve the mode. This can result in cp failing when copying a file from a NFSv4 server with ACLs stored in xattrs to a non-NFS directory where those xattrs cannot be written: cp: preserving permissions for ‘./jquery-3.7.1.js’: Operation not supported The error comes from the kernel refusing to write a system.nfs4_acl xattr to a file on ext4. This situation doesn't appear on all systems with coreutils 9.4, at the time of writing it fails on Ubuntu 24.04 onwards but not Fedora 40. This is because /etc/xattr.conf is used to determine which xattrs describe permissions, and Fedora 40 has removed the NFSv4 attributes[2]. Also, use long-form options to make the cp command clearer. [1] https://github.com/coreutils/gnulib/commit/eb6a8a4dfb [2] https://src.fedoraproject.org/rpms/attr/blob/rawhide/f/0003-attr-2.4.48-xattr-conf-nfs4-acls.patch [ YOCTO #15596 ] (Bitbake rev: 2f35dac0c821ab231459922ed98e1b2cc599ca9a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update GoModTest and GoModGitTestChristian Lindeberg2024-10-011-0/+55
| | | | | | | | | | | Add test cases for a module path consisting of only a hostname and add checksum verification of the unpacked go.mod files. (Bitbake rev: 9380859b59923dee17469348f472a22e11be1779) Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Update fixtures for styheadRichard Purdie2024-09-173-12/+75
| | | | | | | | Add styhead. Also fix a typo in the usage instructions. (Bitbake rev: d3c84198771b7f79aa84dc73061d8ca071fe18f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gomod: Support URIs with only a hostnamePeter Kjellerstedt2024-09-171-2/+6
| | | | | | | | | | | | | | | | When calculating the module name for a gomod URI with only a hostname, e.g.: gomod://go.opencensus.io;version=v0.24.0;sha256sum=203a767d7f8e7c1ebe5588220ad168d1e15b14ae70a636de7ca9a4a88a7e0d0c the non-existing path would actually be treated as "/", which resulted in a trailing slash being added to the module name preventing the unpack method from correctly locating the go.mod file. (Bitbake rev: f0e02e1de4d649e647e4ab61341042dd38d0eeb0) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch2/gitsm: use configparser to parse .gitmodules"Martin Jansa2024-09-161-13/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit a4f42e396e2942fde94b8b4944487c1c45f7a295. Unfortunately configparser is a bit more strict and fails to parse e.g.: https://github.com/espressif/esp-idf/blob/e104dd7f27d2e73ab0e9b614dd7b9295099069bf/.gitmodules [submodule "components/bt/controller/lib_esp32"] path = components/bt/controller/lib_esp32 url = ../../espressif/esp32-bt-lib.git The url is using 8 spaces while path 1 tab causing: Exception: configparser.NoOptionError: No option 'url' in section: 'submodule "components/bt/controller/lib_esp32"' It was fixed in: https://github.com/espressif/esp-idf/commit/62ca8e2fb478a1cdc0e47003025265cd0d840395#diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584 but mcuboot is using a bit older esp-idf revision in: https://github.com/ATmobica/mcuboot/blame/main/.gitmodules and mcuboot is then used as submodule in: https://github.com/project-chip/connectedhomeip/blob/master/.gitmodules so it might take a while for the fix to be propagated everywhere. Not sure how common these issues are, but configparser parses "url = ../../espressif/esp32-bt-lib.git" as 2nd line of "path" value (because it's indented differently) while git submodule and old gitsm implementation parses it as separate path and url keys. (Bitbake rev: d9eb2650256292a524a0c0bb1c0562d87bedcbc4) Signed-off-by: Martin Jansa <martin.jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: increment 9 to 10 correctlyDan McGregor2024-09-122-1/+3
| | | | | | | | | | | | | | | | | | | Previously, incrementing "0.9" would result in "0.1.0", which generally gets recognised as a lower version number. Even more surprising, incrementing "0.99" returned "0.1.0.0". This is due to the behaviour of the list function on a string object; it adds each character as an element in a new list, causing the new string '10' to become the list [ '1', '0' ]. Instead of converting a string to a list, add the string to a new list, and concatenate it with the existing list slice. And provide test cases for "0.9" -> "0.10" and related edge cases. (Bitbake rev: 96ddeefa88ff4c37e9ea096726a7cdca5b5b4572) Signed-off-by: Dan McGregor <dan.mcgregor@usask.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: use configparser to parse .gitmodulesRoss Burton2024-09-121-11/+13
| | | | | | | | | | .gitmodules is basically ini-style, so use configparser instead of manually parsing by hand. (Bitbake rev: a4f42e396e2942fde94b8b4944487c1c45f7a295) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add gomodgit fetcherChristian Lindeberg2024-09-103-2/+228
| | | | | | | | | | | | | | | Add a go module fetcher for downloading module dependencies to the module cache directly from a git repository. The fetcher can be used with the go-mod class in OE-Core. A module dependency can be specified with: SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;srcrev=..." (Bitbake rev: 29ff38ccf0d5389a5bee81e252a78548361a9d7c) Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add gomod fetcherChristian Lindeberg2024-09-103-1/+196
| | | | | | | | | | | | | | | Add a go module fetcher for downloading module dependencies to the module cache from a module proxy. The fetcher can be used with the go-mod class in OE-Core. A module dependency can be specified with: SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..." (Bitbake rev: 5ff4694bf305e266ebf0abab5d9745c6b6d07d67) Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix rare file-checksum hash issueRichard Purdie2024-09-051-1/+1
| | | | | | | | | | | | | | There is a subtle issue with full pathnames being included in the file checksums since the sorting might be different depending upon how layers are being setup causing hash mismatches for recipes appeneded from other layers with differing directory layouts. Avoid this by filtering out to the path basename which is what is written into the sig data anyway later in the code. (Bitbake rev: 83acc21cdfdb410082c0871ac7693d29a7c5627d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Allow code visitor expressions to be declared in metadataRichard Purdie2024-08-292-5/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allow the metadata to define code visitor expressions which mean that custom dependencies can be handled in function libraries. An example is the qa.handle_error function in OE which can set something like: """ def handle_error_visitorcode(name, args): execs = set() contains = {} warn = None if isinstance(args[0], ast.Constant) and isinstance(args[0].value, str): for i in ["ERROR_QA", "WARN_QA"]: if i not in contains: contains[i] = set() contains[i].add(args[0].value) else: warn = args[0] execs.add(name) return contains, execs, warn handle_error.visitorcode = handle_error_visitorcode """ Meaning that it can have contains optimisations on ERROR and WARN_QA instead of hard dependencies. One drawback to this solution is the parsing order. Functions with visitorcode need to be defined before anything else references them or the visitor code will not function for the earlier references. (Bitbake rev: 5bd0c65c217394cde4c8e382eba6cf7f4b909c97) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: allow the '@' character in package namesLeonard Göhrs2024-08-251-3/+4
| | | | | | | | | | | | | | | | | | | The '@types/ramda' [1] npm package has recently gained a dependency on the 'types-ramda' [2] npm package. Both have the same version number. The name mangling results in the tarballs of both packages sharing the same name, but different contents. Fix that by accepting '@' as valid character in the package name, resulting in one package named @types-ramda and one called types-ramda. [1]: https://www.npmjs.com/package/@types/ramda [2]: https://www.npmjs.com/package/types-ramda (Bitbake rev: 7c9573cb6ea2081bc585eb65267f3124fd4d7e43) Signed-off-by: Leonard Göhrs <l.goehrs@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gcp.py: remove slow calls to gsutil statEtienne Cordonnier2024-08-251-7/+7
| | | | | | | | | | | | | | | The changes of 1ab1d36c0af6fc58a974106b61ff4d37da6cb229 added calls to "gsutil stat" to avoid unhandled exceptions, however: - in the case of checkstatus() this is redundant with the call to self.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False and does not throw an exception in case the file does not exist. - Also the call to gsutil stat is much slower than using the python client to call exists() so we should not replace the call to exists() with a call to gsutil stat. - I think the intent of calling check_network_access in checkstatus() was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable. - add a try/except block in download() instead of the extra call to gsutil [RP: Tweak to avoid import until needed so google module isn't required for everyone] (Bitbake rev: dd120f630e9ddadad95fe83728418335a14d3c3b) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: allow packages not declaring a registry versionEnguerrand de Ribaucourt2024-08-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We fetch npm dependencies from the npm-shrinkwrap.json file. They can point to a package on the NPM registry with a version field, or to a git/http/file URL with the resolved field. Such packages are allowed not to declare a registry version field because they may not have been published to the NPM registry. The previous implementation refuses to fetch such packages and throws an error. The resolved field contains the exact source, including the revision, wich we can use to pass as SRC_URI to the git/http/file fetcher. The integrity field is also mandatory for HTTP tarballs which will ensure reproducibility. So even if the version field is not present, we are still fetching a precise revision of the package. Another commit published along this stack is also required in the npm class to support these packages. v5: - improve commit message v3: - Split bitbake npmsw.py modification in another commit Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> (Bitbake rev: 209982b5a3efc8081e65b4326bf9b64eef7f0ba0) Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: fix fetching git revisions not on masterEnguerrand de Ribaucourt2024-08-231-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The NPM package.json documentation[1] states that git URLs may contain a commit-ish suffix to specify a specific revision. When running `npm install`, this revision will be looked for on any branch of the repository. The bitbake implementation however translates the URL stored in package.json into a git URL to be fetch by the bitbake git fetcher. The bitbake fetcher git.py, enforces the branch to be master by default. If the revision specified in the package.json is not on the master branch, the fetch will fail while the package.json is valid. To fix this, append the ";nobranch=1" suffix to the revision in the git URL to be fetched. This will make the bitbake git fetcher ignore the branch and respect the behavior of `npm install``. This can be tested with the following command: $ devtool add --npm-dev https://github.com/seapath/cockpit-cluster-dashboard.git -B version Which points to a project which has a package.json with a git URL: ```json "devDependencies": { "cockpit-repo": "git+https://github.com/cockpit-project/cockpit.git#d34cabacb8e5e1e028c7eea3d6e3b606d862b8ac" } ``` In this repo, the specified revision is on the "main" branch, which would fail without this fix. [1] https://docs.npmjs.com/cli/v10/configuring-npm/package-json#git-urls-as-dependencies Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> (Bitbake rev: 37a35adf7882f231c13643dbf9168497c6a242a1) Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/knotty: respect NO_COLOR & check for tty; rename print_hyperlink ↵Chris Laplante2024-08-231-5/+9
| | | | | | | | => format_hyperlink (Bitbake rev: 3f6de25a8a4d73dfba864aa6a543c5eafa9b7c7c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/knotty: print log paths for failed tasks in summaryChris Laplante2024-08-231-3/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | When tasks fail, it's very frustrating to have to scroll up to find the log path(s). Many of us have the muscle memory to navigate to the 'temp' directories under tmp/work/, but new users do not. This change enhances the final summary to include log paths (reported via bb.build.TaskFailed events). Here's an example: NOTE: Tasks Summary: Attempted 856 tasks of which 853 didn't need to be rerun and 3 failed. Summary: 3 tasks failed: virtual:native:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/x86_64-linux/ncurses-native/6.5/temp/log.do_fetch.1253462 /home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/core2-64-poky-linux/ncurses/6.5/temp/log.do_fetch.1253466 virtual:nativesdk:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-ncurses/6.5/temp/log.do_fetch.1253467 Summary: There were 3 WARNING messages. Summary: There were 6 ERROR messages, returning a non-zero exit code. Each log is rendered as a clickable hyperlink in the terminal. See https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda (Bitbake rev: 2852a478ab03a482989c3a7e247860ab4f0e9f3e) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/exec_task: Log str() instead of repr() for exceptions in buildTroels Dalsgaard Hoffmeyer2024-08-151-2/+2
| | | | | | | | | | | | | | | | When getting errors during build, they would be printed using repr(), which doesnt have a lot of context in some cases. For example FileNotFoundError(2, "file or directory not found"), would be printed, without the path of the file not found. This changes the build logging to use str() instead, which according to the spec is fore human readable strings, whereas repr() is for string representations that can be be used as valid python. (Bitbake rev: 2a97024b8b9245ec47deace011a7560a25491207) Signed-off-by: Troels Dalsgaard Hoffmeyer <tdah@bang-olufsen.dk> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Ensure addtask before/after tasknames have prefix appliedRichard Purdie2024-08-132-5/+14
| | | | | | | | | | | | | "addtask do_XXX before YYY after ZZZ " where YYY or ZZZ is missing the "do_" prefix don't work as expected. Ajust the code so that it doesn't just silently do the wrong thing but works as expected. Expand a test case to cover this. (Bitbake rev: 21670b9bb8936ec44aedff26163948bbc2ceb44a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler/ast: Improve addtask handlingRichard Purdie2024-08-133-44/+38
| | | | | | | | | | | | | | | | | | The recent addtask improvement to handle comments complicated the regex significantly and there are already a number of corner cases in that code which aren't handled well. Instead of trying to complicate the regex further, switch to code logic instead. This means the following cases are now handled: * addtask with multiple task names * addtask with multiple before constraints * addtask with multiple after constraints The testcase is updated to match the improvements. (Bitbake rev: 417016b83c21fca7616b2ee768d5d08e1edd1e06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Separate out data_hash and hook to tinfoilRichard Purdie2024-08-132-7/+16
| | | | | | | | | | | | | | | | | Within tinfoil, the user can write to the configuration data but it won't cause the data_hash checksum to be re-written, meaning cached parsing data can be reused when it would now be incorrect. Abstract out the data_hash code and add it to the invalidateCaches command, called by tinfoil.modified_files() meaning that tinfoil can instruct bitbake to update the caches and re-parse if necessary. Also move the data_hash entirely into databuilder and drop the copy in cooker as obsolete and not needed. (Bitbake rev: d9ee77829f693ce75348fa64f406fcecfe4989aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Drop unused functionRichard Purdie2024-08-131-19/+0
| | | | | | | | | I can't spot any users of this function and it is poking at variables inside cooker that could and are about to change so drop it. (Bitbake rev: 52491808706e9e58b5e6b59d2d792353d77c8b66) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Handle comments in addtask/deltaskRichard Purdie2024-08-132-2/+12
| | | | | | | | | | | | | | | Technically our syntax would allow for comments after an addtask/deltask. Currently these get silently processed in various ways by the code which is bad. Tweak the regex to drop any comments and add test cases to ensure this continues to work in the future. [YOCTO #15250] (Bitbake rev: 64f8796e55a8826ffec0d76b993c8256713f67a3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Improve performance for VariableHistoryRobert Yang2024-08-031-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS $ bitbake -p -P Check profile.log.processed, the record() cost more than 20 seconds, it is less than 1 second when multiconfig is not enabled, and there would be the following error when more muticonfigs are enabled: Timeout while waiting for a reply from the bitbake server Don't change the type of loginfo['detail'] or re-assign it can make record() back to less than 1 second, this won't affect COW since loginfo is a mutable type. The time mainly affected by two factors: 1) The number of enabled layers, nearly 1 second added per layer when the number is larger than 50. 2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer when the layers number is larger than 50. (Bitbake rev: 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Enforce default remote name to "origin"Yuri D'Elia2024-07-131-1/+1
| | | | | | | | | | | | Enforce the default remote name to "origin", as assumed in numerous places. This prevents build failures in case the system/user configuration sets this to a different value. (Bitbake rev: 1d7360031164f04887c792fb0b2dd86c6ccfcc23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()Robert Yang2024-07-131-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Update the test cases since the implementation is changed: * test_shallow_multi_one_uri() The a_branch and v0.0 had the same revision, and it required fetch a_branch and remove histories of v0.0 which were conflicted, and bitbake reported: fatal: no commits selected for shallow requests Make a_branch and v0.0 have different revs to fix the problem. And now the 'rev^' is not needed, so update self.assertRevCount() as well. * test_shallow_multi_one_uri_depths() Update self.assertRevCount(), now git only fetches the required revs. * test_shallow_fetch_missing_revs() The command is: $ git fetch --shallow-exclude=v0.0 master But master and v0.0 uses the same revision, so there is no commit to fetch. * test_shallow_fetch_missing_revs_fails() Two unneeded committs are not fetched now: - rev^ - One not specified or required tag. So update self.assertRevCount() (Bitbake rev: 48eff9d9a660ad6b9bd8b53a7dcec600ef42b1d1) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use git shallow fetch to implement clone_shallow_local()Robert Yang2024-07-131-26/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch can make the following settings much more faster: BB_GIT_SHALLOW = "1" BB_GENERATE_MIRROR_TARBALLS = "1" * The previous implementation was: - Make a full clone for the repo from local ud.clonedir - Use git-make-shallow to remove unneeded revs It was very slow for recipes which have a lot of SRC_URIs, for example vulkan-samples and docker-compose, the docker-compose can't be done after 5 hours. $ bitbake vulkan-samples -cfetch Before: 12 minutes Now: 2 minutes $ bitbake docker-compose -cfetch Before: More than 300 minutes Now: 15 minutes * The patch uses git shallow fetch to fetch the repo from local ud.clonedir: - For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev Then the git repo will be shallow, and git-make-shallow is not needed any more. And git shallow fetch will download less commits than before since it doesn't need "rev^" to parse the dependencies, the previous code always need 'rev^'. (Bitbake rev: a5a569c075224fe41707cfa9123c442d1fda2fbf) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: checkstatus: drop shared connecton when catch Timeout errory75zhang2024-07-041-1/+6
| | | | | | | | | | | | | | | | * to avoid wrong http response in checkstatus function: in wget checkstatus() we are using 'HTTPConnectionCache' to share connections 1. state_file1(exists on http server) use shared connection <shared1> to send request 2. http_server recieved request of state_file1, but delayed by some reason to sent respone 3. state_file1 checkstatus() failed by timeout and drop shared connection <shared1> 4. state_file2(not exists on http server) get shared connection <shared1> and send request 5. http_server finally send 200 response for state_file1 6. state_file2 recived 200 response and thought it was exists on http_server (Bitbake rev: bf6d0282ab88b4edc4b9e58184cd76cce965abbd) Signed-off-by: y75zhang <yang-mark.zhang@nokia-sbell.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/clearcase: remove True option to getVar calls in clearcase modulePeter Marko2024-07-021-2/+2
| | | | | | | | | | | | | Layer cleanup similar to https://git.openembedded.org/openembedded-core/commit/?id=26c74fd10614582e177437608908eb43688ab510 It was probably not found before beacause of the extra "d" parameter. That seem to be a bug as getVar does not support that. (Bitbake rev: 720189b810995c5737853458b7eb3779ca0df37e) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Skip non-local functions for module dependenciesRichard Purdie2024-07-011-0/+5
| | | | | | | | | | | | | | If modules do something like "from glob import glob" then we end up checksumming the glob code. That leads to bugs as the code can change between different python versions for example, leading to checksum instability. We should ignore functions not from the current file as implemented by this change. (Bitbake rev: 1e6f862864539d6f6a0bea3e4479e0dd40ff3091) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser/data: Ensure module function contents changing is ↵Richard Purdie2024-07-012-12/+18
| | | | | | | | | | | | | | | | | | | | accounted for Currently, if a pylib function changes contents, the taskhash remains unchanged since we assume the functions have stable output. This is probably a poor assumption so take the code of the function into account in the taskhashes. This avoids certain frustrating build failures we've been seeing in automated testing. To make this work we have to add an extra entry to the python code parsing cache so that we can store the hashed function contents for efficiency as in the python module case, that isn't used as the key to the cache. The cache version changes since we're adding data to the cache. (Bitbake rev: b2c3438ebe62793ebabe2c282534893908d520b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-06 16:46:34 +0000