summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2: Ensure that git LFS objects are availablePhilip Lorenz2024-02-232-6/+90
| | | | | | | | | | | | | | | | | | | | | The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: progressbar: accept value over initial maxvalEnguerrand de Ribaucourt2024-02-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a very rare case where the maxval is improperly computed initially for cache loading progress, and the value will go over. Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:progress() # we might have calculated incorrect total size because a file # might've been written out just after we checked its size In that case, progressbar will receive a value over the initial maxval. This results in a ValueError stack trace as well as bitbake returning 1. Traceback (most recent call last): File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main cacheprogress.update(event.current) File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in update raise ValueError('Value out of range') ValueError: Value out of range This fix mirrors the behavior of MulticonfigCache and accepts the new value as the new maxval. This is also what the percentage printout is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:percentage() I encountered this issue randomly while working on a project with VSCode saving files while commands where fired. Note: This file is a fork from python-progressbar. It hasn't been refreshed in 8 years. We did only two commits, 5 years ago with minor modifications. This new change is also not how the upstream project is behaving. (Bitbake rev: 7cea7f7a87da041fc1ad370c5c3d15aabad3a0d4) Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add support for BB_LOADFACTOR_MAXRichard Purdie2024-02-231-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Some ditros don't enable /proc/pressure and it tends to be those which we see bitbake timeout issues on, seemingly as load gets too high and the bitbake processes don't get scheduled in for minutes at a time. Add support for stopping running extra tasks if the system load average goes above a certain threshold by setting BB_LOADFACTOR_MAX. The value used is scaled by CPU number, so a value of 1 would be when the load average equals the number of cpu cores of the system, under one only starts tasks when the load average is below the number of cores. This means you can centrally set a value such as 1.5 which will then scale correctly to different sized machines with differing numbers of CPUs. The pressure regulation is probably more accurate and responsive, however our graphs do show singificant load spikes on some workers and this patch is aimed at trying to avoid those. Pressure regulation is used where available in preference to this load factor regulation when both are set. (Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Re-enable connection pooling with psycopg 3 driverTobias Hagelborn2024-02-231-1/+5
| | | | | | | | | | | | Re-enable connection pooling in case `postgresql+psygopg` driver is used. Async connection pooling is supported in psycopg 3 [psycopg] driver in SQLAlchemy. Allow the connection pool to grow to arbitrary size. (Bitbake rev: 4fe05513b5314c201725e3f8ad54f58d70c56258) Signed-off-by: Tobias Hagelborn <tobiasha@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "bitbake: wget.py: always use the custom user agent"Alexander Kanavin2024-02-201-4/+3
| | | | | | | | | | | | | | | | | | This reverts commit 987ab2a446aab235d6e86e97c05f25fb800d7acc. There's been a report that this breaks downloads from Jfrog Artifactory as self.user_agent is set to 'Mozilla Firefox', and when Artifactory sees that, it sends a response tailored for showing in an interactive browser (which in my opinion it has every right to). If we're using wget, we should say so via wget's default; handling uncooperative servers should be done on per-recipe basis, and ideally with tickets to admins of those servers. (Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Make latest_versionstring extract tags with slashes ↵Peter Kjellerstedt2024-02-192-2/+9
| | | | | | | | | | | | | | correctly Before, everything up to the last slash was removed when extracting the names of the tags. This would lead to that a tag such as "agent/11.0.0" would be incorrectly identified as "11.0.0", which would then be treated as a correct version matching "^(?P<pver>\d+(\.\d+)+)". (Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: A bit of clean-up of latest_versionstring()Peter Kjellerstedt2024-02-191-11/+10
| | | | | | | | | This is mostly preparations for the next commit. (Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Make test_git_latest_versionstring support a max versionPeter Kjellerstedt2024-02-191-11/+14
| | | | | | | | | | | In some cases, the version found by latest_versionstring() may be higher than the real version. Make it possible to specify a maximum version so that this case can be detected. (Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: add comment in try_premirrorsChen Qi2024-02-191-0/+5
| | | | | | | | | | The purpose of ensuring 'incremental fetch' is not easy to see from the codes. So add comments to explain this. (Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: add test case for using premirror in restricted networkChen Qi2024-02-191-0/+19
| | | | | | | | | | | | We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch' failed even with all contents available in PREMIRRORS. Add a test case to ensure no regression in the future. (Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: fix a corner case in try_premirrorChen Qi2024-02-191-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For gitsm recipes, it's possible that some URL is used more than once. e.g., A -> B:rev1 (B is a submodule of A) A -> C (C is a submodule of A) C -> B:rev2 (B is a submodule of C) A anc C are both using B as submodules, but on different revs. Now if we have: B:rev1 -> D B:rev2 -> E Then, the mirror will not be fully used. Say we have all repo mirrors for A, B, C, D, E, then in theory it's not necessary to reach out to any network for downloading. But it's not the case. After downloading B(rev1) and its submodule D from mirrors, the fetch process continues to download C, thus B(rev2) and E. Now it finds that B needs an update because its submodule E needs an update. Of course this is true because E is not downloaded yet. Now the problem comes to whether to use mirror or not. The git.py defines try_premirror to return 'False' when the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and try_mirror returns False, resulting in not using mirror and going to upstream directly. We can see that the mirrors are not fully used. This is usually not problem, as the cost is only some network download. But in case the following two settings are there, we get errors. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.allowed.domain" In such case, the gitsm recipe A will fail to fetch. Note that all contents that A needs are in mirrors and now it's failing to fetch. This is unexpected. Note that the different revs of the same repo in gitsm recipe is not the only way to reveal this problem. For example, there might be a recipe call B that uses B:rev3. Check the protobuf and grpc recipes as an example. For now, we can use the following steps to reproduce this issue. To be clear, the grpc recipe in meta-oe is now 1.60.0. 1. Add in local.conf: DL_DIR = "${TOPDIR}/downloads-premirror" bitbake grpc -c fetch 2. Comment out the DL_DIR setting in local.conf and add the following lines: PREMIRRORS:append = " \ git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ " 3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'. This command succeeds and this shows that the premirror holds everything we need. 4. Add the following lines and then 'bitbake grpc -c fetch'. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.domain" After step 4, the error message is as below: ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used This patch fixes this problem by handling this corner case, that is, if the URL is not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should try premirrors because trying to reach upstream is destined to fail. (Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: add multiple fetches testPavel Zhukov2024-02-191-0/+18
| | | | | | | | | | | Fetch from premirror few times to emulate multiple machines sharing same clonedir or few rebuilds of the package from (pre)mirror Regression test for [Yocto #15369] (Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: Fetch mirror into HEADPavel Zhukov2024-02-191-1/+5
| | | | | | | | | | | | | | | Fix the issue with using of (pre)mirror in case if clonedir exists but outdated. Previous version of the code fetched new mirror content into FETCH_HEAD instead of branch which caused refetch from the upstream. Add new remote add fetch from it instead so the ref can be found by "_contains_ref" Fixes [Yocto #15369] (Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git-make-shallow: support git's safe.bareRepositoryAndré Draszik2024-02-191-16/+18
| | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the git-make-shallow fails miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: 7c63989db4590564516ed150930f4e2fa503e98f) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: support git's safe.bareRepositoryAndré Draszik2024-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the bitbake selftests fail miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: a45e14a7343e36101e45639931322e5649587f57) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git2: support git's safe.bareRepositoryAndré Draszik2024-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the bitbake git fetcher fails miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. While at it, fix one open-coded invocation of git that wasn't using ud.basecmd Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump version to 2.7.3 for hashserv changesRichard Purdie2024-02-192-2/+2
| | | | | | (Bitbake rev: c1e0a0b6ddc9667c9d62319bd9ccd4eb8c64c2a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: hashserv: Postgres adaptations for ignoring duplicate insertsTobias Hagelborn2024-02-191-15/+38
| | | | | | | | | | | | | | | | Hash Equivalence server performs unconditional insert also of duplicate hash entries. This causes excessive error log entries in Postgres. Rather ignore the duplicate inserts. The alternate behavior should be isolated to the postgres engine type. (Bitbake rev: e8d2d178d0fe96f9d6031c97328e8be17d752716) Signed-off-by: Tobias Hagelborn <tobiasha@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add parallel unihash exist APIJoshua Watt2024-02-191-0/+32
| | | | | | | | | Adds API to query if unihashes are known to the server in parallel (Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add parallel query APIJoshua Watt2024-02-191-34/+87
| | | | | | | | | | | | | | | | Implements a new API called get_unihashes() that allows for querying multiple unihashes in parallel. The API is also reworked to make it easier for derived classes to interface with the new API in a consistent manner. Instead of overriding get_unihash() to add custom handling for local hash calculating (e.g. caches) derived classes should now override get_cached_unihash(), and return the local unihash or None if there isn't one. (Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add Client PoolJoshua Watt2024-02-192-0/+163
| | | | | | | | | | Implements a Client Pool derived from the AsyncRPC client pool that allows querying for multiple equivalent hashes in parallel (Bitbake rev: ba4c764d8061c7b88cd4985ca493d6ea6e317106) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add Client Pool objectJoshua Watt2024-02-192-1/+78
| | | | | | | | | | | | Adds an abstract base class that can be used to implement a pool of client connections. The class implements a thread that runs an async event loop, and allows derived classes to schedule work on the loop and wait for the work to be finished. (Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add unihash-exists APIJoshua Watt2024-02-196-33/+151
| | | | | | | | | | | | | | | Adds API to check if the server is aware of the existence of a given unihash. This can be used as an optimization for sstate where a client can query the hash equivalence server to check if a unihash exists before querying the sstate cache. If the hash server isn't aware of the existence of a unihash, then there is very likely not a matching sstate object, so this should be able to significantly cut down on the number of negative hits on the sstate cache. (Bitbake rev: cfe0ac071cfb998e4a1dd263f8860b140843361a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: sqlalchemy: Use _execute() helperJoshua Watt2024-02-191-157/+140
| | | | | | | | | | Use the _execute() helper to execute queries. This helper does the logging of the statement that was being done manually everywhere. (Bitbake rev: 0409a00d62f45afb1b172acbcea17bf17942e846) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add Unihash Garbage CollectionJoshua Watt2024-02-196-116/+684
| | | | | | | | | | | | Adds support for removing unused unihashes from the database. This is done using a "mark and sweep" style of garbage collection where a collection is started by marking which unihashes should be kept in the database, then performing a sweep to remove any unmarked hashes. (Bitbake rev: 433d4a075a1acfbd2a2913061739353a84bb01ed) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: improve the loglevel error message to be more helpfulPaul Gortmaker2024-02-191-1/+1
| | | | | | | | | | | | | | | | | | Coming from a kernel background, I was thinking along the lines of dmesg -n <integer> for loglevel adjustments. So I tried various large and small and even zero number values with no luck before getting frustrated and opening up the python. Let us save others the frustration and give a hint what the args it expects should look like. (Bitbake rev: df184b2a4e80fca847cfe90644110b74a1af613e) Signed-off-by: Paul Gortmaker <paulg@kernel.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskexp_ncurses: ncurses version of taskexp.pyDavid Reyna2024-02-181-0/+1511
| | | | | | | | | | | | | | | | * Create an ncurses version of the GTK app "taskexp.py". * Add these additional features: - Sort tasks in recipes by their dependency order - Print individual and/or recipe-wide dependencies to a file - Add a wild card filter - Show the target recipes on BOLD * Provide a GUI self test * Provide a non-ncurses self test for ptest (Bitbake rev: f49bec66ad51c8cddeceafbbe2445c46e396ee8b) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve setcene performance when encoutering many 'hard' ↵Richard Purdie2024-02-131-1/+11
| | | | | | | | | | | | | | | | | | dependencies "bitbake world -n --setscene-only" shows poor performance as the numbers of tasks increases. Analysys shows this is due to the "deferred" hard dependencies being repeatedly processed which doesn't allow much forward progress in overall task execution. To avoid this, mark when it has been done and don't reprocess until dependencies are updated. We have to be careful as we've seen bugs where these dependencies aren't processed at the right time. They also have to be reproceseed during task migration/rehashing so the code has to "self heal" the data structures. (Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise taskname lookups in next_buildable_taskRichard Purdie2024-02-131-3/+3
| | | | | | | | | | A quick profile of bitbake world showed 147 million calls to taskname_from_tid(). The next_buildable_task function is performance senstive so move the call inside the if block to reduce the number of calls and speed the code up. (Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve performance for executing tasksRichard Purdie2024-02-131-10/+21
| | | | | | | | | | | | | Now that runqueue performance profiling works again we can see a lot of time is lost in build_taskdepdata. Whilst we can't compute that in advance, we can compute the individual entries. Therefore put a cache in place to compute those and save overhead in starting up tasks. (Bitbake rev: c4519b542702ba25023e53d77b275a6fa571ec50) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/lib/bs4/tests/test_tree.py: python 3.12 regexAdrian Freihofer2024-02-131-1/+1
| | | | | | | | | | | | Python 3 interprets string literals as Unicode strings, and therefore \s is treated as an escaped Unicode character which is not correct. Declaring the RegEx pattern as a raw string instead of unicode is required for Python 3. (Bitbake rev: a5f1bc69ef2e456bd163303a07cfb73825b01576) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process/server: Fix typoRichard Purdie2024-02-101-1/+1
| | | | | | | | Ensure the message matches the filenames the code actually uses. (Bitbake rev: deb7db2e2b125c6a6732db4f185f4de5926494fd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: README: simpler link to contributor guideMichael Opdenacker2024-02-101-1/+1
| | | | | | | (Bitbake rev: 57d2f54e00374fe7452e123ec3c6e7ac27afb024) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: always use the custom user agentThomas Perrot2024-02-101-3/+4
| | | | | | | | | | | | | | | | | | | Add the "--user-agent" paramater in the wget base command to perform all wget commands with this parameter, because a few HTTP servers block requests with the default wget user agent. For example, "hg.openjdk.org" never send a response to requests have been sent with wget: wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73 Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected. HTTP request sent, awaiting response... (Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c) Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Add profile logging for main loopRichard Purdie2024-02-101-0/+16
| | | | | | | | | When the idle/main loop was added, we didn't include profiling information for it. There is a performance issue in there, add logging for it. (Bitbake rev: d8d5cd43a60560f67e86f4f625113b0f73b944c0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: support temporary AWS credentialsToni Lammi2024-02-091-0/+1
| | | | | | | | | | | | | Support AWS_SESSION_TOKEN which is used in temporary AWS credentials. Fixes [YOCTO #15384]. (Bitbake rev: ae1e4c90bbc2002cb2728c64649c095c00220ceb) Signed-off-by: Toni Lammi <toni.lammi@kone.com> Reported-by: Toni Lammi <toni.lammi@tl-software.fi> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: fetch.py: use real subversion repositoryMartin Jansa2024-02-081-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * github no longer provides support for subversion clients: https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients it was shut down on 2024-01-08: https://github.blog/2023-01-20-sunsetting-subversion-support/ and this test was now failing with: ====================================================================== ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn) ---------------------------------------------------------------------- Traceback (most recent call last): File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn fetcher.download() File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download raise FetchError("Unable to fetch URL from any source.", u) bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source. Stdout: Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2 Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output: A trunk/README.md U trunk Checked out revision 2. svn: warning: W205011: Error handling externals definition for 'trunk/bitbake': svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git' svn: E205011: Failure occurred processing one or more externals definitions in the rare cases where subversion was still installed on the host running bitbake-selftest :). to avoid this use still alive repository from https://svn.apache.org/ and pick something rather small and only the trunk subdirectory which is fast to fetch: svn co https://svn.apache.org/repos/asf/serf/trunk takes just 2 sec here adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of "trunk/README" (Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617) Signed-off-by: Martin Jansa <martin.jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: fix walking the task dependencies and show better ↵Martin Jansa2024-02-082-7/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | error * when comparing 2 tmp/stamps/*do_configure.sigdata* I got TypeError("filename must be a str or bytes object, or a file") but both files I was comparing were Zstandard compressed data (v0.8+), Dictionary ID: None according to "file" with TypeError catched to show which file it failed to open I got better error which shows it was trying to read "do_prepare_recipe_sysroot.sigdata" file now and after a while you might notice that it's not just the expected file, but a dict with 'path', 'sstate', 'time'. Fix that in bitbake-diffsigs but keep the TypeError and add OSError in case it will eventually walk on file which isn't zstd compressed pipecompress throws CompressionError. ERROR: Failed to open {'path': '5.15.do_prepare_recipe_sysroot.sigdata.99b12a401341a0df7c3553cb00c87a7674295496bd5c25ed71764ee0d0fb8eb8', 'sstate': False, 'time': 1707136354.991718}: filename must be a str or bytes object, or a file Traceback (most recent call last): File "bitbake/bin/bitbake-diffsigs", line 192, in <module> output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/siggen.py", line 1039, in compare_sigfiles recout = recursecb(dep, a[dep], b[dep]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/bin/bitbake-diffsigs", line 102, in recursecb out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/siggen.py", line 857, in compare_sigfiles raise err File "bitbake/lib/bb/siggen.py", line 853, in compare_sigfiles with bb.compress.zstd.open(a, "rt", encoding="utf-8", num_threads=1) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/compress/zstd.py", line 12, in open return bb.compress._pipecompress.open_wrap(ZstdFile, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/compress/_pipecompress.py", line 59, in open_wrap raise TypeError("filename must be a str or bytes object, or a file") TypeError: filename must be a str or bytes object, or a file * if I replace zstd file with just plaintext it fails with: $ echo foo > foo $ echo foo > bar $ bitbake-diffsigs foo bar zstd: /*stdin*\: unsupported format ERROR: Process died with 1 sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'> with this change it shows the name of the file which it failed to uncompress: $ bitbake-diffsigs foo bar zstd: /*stdin*\: unsupported format ERROR: Failed to open sigdata file 'foo': Process died with 1 ERROR: Process died with 1 sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'> (Bitbake rev: f3f843a4fd6e0a9e8f6edef5dd3cf1fce29c50ba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Version bump for inherit_defer additionRoss Burton2024-01-262-2/+2
| | | | | | | | | | We've added a new statement, inherit_defer, so bump the version so this can be checked. (Bitbake rev: 191e6eb2bceb467c97e315301f1f64722cf0e976) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/BBHandler: Add inherit_defer supportRichard Purdie2024-01-182-1/+31
| | | | | | | | | | | | | | | | | | | | | | Add support for an inherit_defer statement which works as inherit does but is only evaulated at the end of parsing. This allows conditional expressions to be evaulated 'late' meaning changes to PACKAGECONFIG from bbappends can be applied for example. This addresses a usability/confusion issue users have with the current conditional inherit mechanism since they don't realise the condition has to be expanded immediately with the current model. There is a commented out warning we could enable in future which warns about the use of a conditional inherit that isn't deferred. There is a behaviour difference in the placement of the inherit, particularly around variables set using ?= which means wen swapping from one to the other, caution must be used as values can change. (Bitbake rev: 5c2e840eafeba1f0f754c226b87bfb674f7bea29) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall all bugRichard Purdie2024-01-121-18/+24
| | | | | | | | | | | | | Where chains of RDEPENDS are multiple levels deep, the runall code was not accounting for this and recursing deeply enough to gather all dependencies. Fix this by iterating over the result until no more dependencies are found. Tested-by: Jonas Gorski <jonas.gorski@bisdn.de> Reported-by: Jonas Gorski <jonas.gorski@bisdn.de> (Bitbake rev: 966f25dfc23a6d17b2b6d3e0100e9ae264f99025) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: rework 'bitbake -S printdiff' logicAlexander Kanavin2024-01-101-15/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously printdiff code would iterate over tasks that were reported as invalid or absent, trying to follow dependency chains that would reach the most basic invalid items in the tree. While this works in tightly controlled local builds, it can lead to bizarre reports against industrial-sized sstate caches, as the code would not consider whether the overall target can be fulfilled from valid sstate objects, and instead report missing sstate signature files that perhaps were never even created due to hash equivalency providing shortcuts in builds. This commit reworks the logic in two ways: - start the iteration over final targets rather than missing objects and try to recursively arrive at the root of the invalid object dependency. A previous version of this patch relied relies on finding the most 'recent' signature in stamps or sstate in a different function later, and recursively comparing that to the current signature, which is unreliable on real world caches. - if a given object can be fulfilled from sstate, recurse only into its setscene dependencies; bitbake wouldn't care if dependencies for the actual task are absent, and neither should printdiff I wrote a recursive function for following dependencies, as doing recursive algorithms non-recursively can result in write-only code, as was the case here. [YOCTO #15289] (Bitbake rev: aadeca63da5d96160ce4d6d71da556e2e033f9b7) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/knotty: properly handle exceptions when calling runCommand()Mark Asselstine2024-01-101-10/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | In runCommand() the send() and recv() can fail and raise BrokenPipeError and EOFError exceptions when the bitbake-server is unexpectedly terminated. In these cases a python traceback is currently dumped. Similarly updateFromServer() which calls runCommand() can also raise these and other exceptions, and currently lacks proper exception handling resulting in python traceback. We wrap calls to runCommand() and updateFromServer() in a try/except block as well as improve the exception handling for updateToServer(). This along with the earlier commit which added text to the BrokenPipeError and EOFError exceptions in runCommand() to indicate a bitbake-server termination may have occurred, should improve the user's ability to understand and handle these errors. An easy way to trigger each of the runCommand() exceptions is to 'kill -9' bitbake-server before (causes EOFError) or after (causes BrokenPipeError) the "Loading Cache" stage. (Bitbake rev: 804d366ee3ddc0f37f0a6c712c8d42db45b119bc) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: catch and expand multiprocessing connection exceptionsMark Asselstine2024-01-101-2/+8
| | | | | | | | | | | | | | | | | | | | | | | Doing builds on systems with limited resources, or with high demand package builds such as chromium it isn't uncommon for the OOM Killer to be triggered and for bitbake-server to be selected as the process to be killed. When the bitbake-server does terminate unexpectedly due to the OOM Killer or otherwise, this currently results in a generic python traceback with little indication as to what has failed. Here we trap and raise the exceptions while extending the exception text in runCommand() to make it clear that this is most likely caused by the bitbake-server unexpectedly terminating. Callers of runCommand() should be updated to properly handle the BrokenPipeError and EOFError exceptions to avoid printing a python traceback, but even if they don't, the added text in the exceptions should provide some hints as to what might have caused the failure. (Bitbake rev: 5ff62b802f79acc86bbd6a99484f08501ff5dc2d) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast: Fix EXPORT_FUNCTIONS bugRichard Purdie2024-01-102-4/+103
| | | | | | | | | | | | | | | | If you have two classes, both of which set EXPORT_FUNCTIONS for the same funciton and a standard funciton definition for the function that is exported, the export function can sometimes overwrite the standard one. The issue is that the internal flag the code uses isn't ovweritten if the variable is giving a new value. Fix the issue by using a comment in the code that is injected so that we know if it is ours or not. Also add some testing for EXPORT_FUNCTIONS, not perfect but a start. (Bitbake rev: 66306d5151acb0a26a171c338d8f60eb9eb16c6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget/checkstatus(): include the URL in debugging output about ↵Alexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | status check failure Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/checkstatus(): do not print the URI twice in FetchError exceptionAlexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | Previously, there was duplicate clutter in the output, particularly if the URI points to sstate cache items: bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work (Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/event: Add test_lineno_in_eventhandlerRobert Yang2024-01-101-0/+24
| | | | | | | | | Add test_lineno_in_eventhandler to test lineno in eventhandler. (Bitbake rev: 4e5de537bebb68180c5755858c81b095eb9ae2f6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: event: Inject empty lines to make code match lineno in ↵Robert Yang2024-01-101-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | filename So that we can get the correct error messages. * In python 3.10.9, the error message was: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 4 # SPDX-License-Identifier: MIT ^^^^^ SyntaxError: invalid syntax This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4. * Now the error message and lineno is correct: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 256 an error line ^^^^^ SyntaxError: invalid syntax And no impact on parsing time: * Before: $ rm -fr cache tmp; time bitbake -p real 0m27.254s * Now: $ rm -fr cache tmp; time bitbake -p real 0m27.200s (Bitbake rev: c212933d9c786806852c87f188250a4f0a14c048) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster/tests: Bug-fix ToasterTable show_rows testcasesAlassane Yattara2024-01-104-6/+9
| | | | | | | | | | | | Test if some rows are visible in table instead of compare row to row_to_show, because sometime full avaiblable content did not display Failed: https://autobuilder.yoctoproject.org/typhoon/#/builders/161/builds/147/steps/12/logs/stdio (Bitbake rev: 5b0a48265aafa62259c575707c3afa6dd56f8008) Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-01-10 09:16:11 +0000