summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: data_smart.py: clear expand_cache in _setvar_update_overridevarsChen Qi2025-03-081-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At the end of this function, self.overrides is cleared, which means we'll construct a new self.overrides after this call. And a new self.overrides will give out different expand_cache, so the old expand_cache should also be cleared to avoid any wrong value. Currently, there's a problem revealed by recent recipe specific virtual provider patch. If we enable multilib and set "OVERRIDES:prepend" in local.conf, things don't work. Here's the error message: ERROR: Nothing PROVIDES 'lib32-gcc-cross-x86_64' Below are reproduce steps: 1. Add in local.conf the following lines: MACHINE ?= "qemux86-64" require conf/multilib.conf MULTILIBS ?= "multilib:lib32" DEFAULTTUNE:virtclass-multilib-lib32 ?= "core2-32" OVERRIDES:prepend = "some-override:" (Note that using :append and :remove also reproduces the issue.) 2. bitbake -n lib32-sysstat (bitbake -n core-image-minimal also reproduces the issue) The expandWithRefs calls getVar, which fills expand_cache. So when setting OVERRIDES:prepend, this will fill the expand_cache. When overridevars are updated, if we don't clear expand_cache, we'll retrieve wrong values. Previously, things happened to work because there's a call to expand PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc, which in turn expands TARGET_VENDOR. Now what we expand is PREFERRED_PROVIDER_ virtual/cross-cc, so the problem is revealed. (Bitbake rev: 7375d32e8c1af20c51abec4eb3b072b4ca58b239) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart.py: simple clean upChen Qi2025-03-081-4/+1
| | | | | | | | | | | | | When var matches __setvar_regexp__, which has the value as below, __setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$') the keyword will not be __doc__ or __module__ and var will always contain ":". (Bitbake rev: 9cac695951d0fd587063b250bfc8ef0f87acad18) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart.py: remove unnecessary ? from __expand_var_regexp__Chen Qi2025-03-081-1/+1
| | | | | | | | | | | The non-greedy modifier suffix ? is not necessary here because } is not in the character set [a-zA-Z0-9\-_+./~:]. (Bitbake rev: aae570b7f051fc5deee7a1712a02ed92498a4461) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Fix git shallow test failure with git >= 2.48Richard Purdie2025-01-251-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | >From git 2.48 release notes: """ When "git fetch $remote" notices that refs/remotes/$remote/HEAD is missing and discovers what branch the other side points with its HEAD, refs/remotes/$remote/HEAD is updated to point to it. """ This means with git 2.48 onwards, there is a mystery "HEAD" revision appearing in some of our shallow clone tests. We can avoid this by using the same canonicalization as used for the reference revisions. This resolves autobuilder failures on the Fedora 40 workers. (Bitbake rev: aa0e540fc31a1c26839efd2c7785a751ce24ebfb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c83444d1210740e27b1744d3aa7c5cad4e28db2f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: cooker: Make cooker 'skiplist' per-multiconfig/mcChris Laplante2025-01-243-12/+36
| | | | | | | | | | | | | | | | | | | Previously, the cooker skiplist was shared across multiconfigs (including default ''). If you had a recipe that was incompatible with several multiconfigs for different reasons, then the displayed reason (i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might vary across invocations of bitbake. This was caused by the random order in which recipes are parsed under different multiconfig contexts, with each skip reason overwriting the previously assigned reason. I hit this specificially when using COMPATIBLE_MACHINE, but COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too. (Bitbake rev: 7dde14582bfd104c6da26e3f5ecf2ef37a1494ce) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: ui/knotty: respect NO_COLOR & check for tty; rename print_hyperlink ↵Chris Laplante2025-01-241-5/+9
| | | | | | | | | => format_hyperlink (Bitbake rev: 72a334d5d5763476d5421a902fabb303f3f84bd8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: ui/knotty: print log paths for failed tasks in summaryChris Laplante2025-01-241-3/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When tasks fail, it's very frustrating to have to scroll up to find the log path(s). Many of us have the muscle memory to navigate to the 'temp' directories under tmp/work/, but new users do not. This change enhances the final summary to include log paths (reported via bb.build.TaskFailed events). Here's an example: NOTE: Tasks Summary: Attempted 856 tasks of which 853 didn't need to be rerun and 3 failed. Summary: 3 tasks failed: virtual:native:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/x86_64-linux/ncurses-native/6.5/temp/log.do_fetch.1253462 /home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/core2-64-poky-linux/ncurses/6.5/temp/log.do_fetch.1253466 virtual:nativesdk:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-ncurses/6.5/temp/log.do_fetch.1253467 Summary: There were 3 WARNING messages. Summary: There were 6 ERROR messages, returning a non-zero exit code. Each log is rendered as a clickable hyperlink in the terminal. See https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda (Bitbake rev: 9c020cc314bfd0702bb1d457d94925c6e9613268) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix scenetask processing performance issueRichard Purdie2024-12-091-1/+5
| | | | | | | | | | | | | Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. (Bitbake rev: f40a3a477d5241b697bf2fb030dd804c1ff5839f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 923c19b9713e398d8e66e6d4422dfd4c18a03486) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Optimise setscene loop processingRichard Purdie2024-12-091-2/+8
| | | | | | | | | | | | Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. (Bitbake rev: 28569e9796d4b34d7b77b4f79074ab7854850386) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 00f4d932e3af0eeb333339cbe942010fc76dee0f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix performance of multiconfigs with large overlapRichard Purdie2024-12-091-2/+4
| | | | | | | | | | | | | | | | | | | | | There have been complaints about the performance of large multiconfig builds for a while. The key missing data point was that the builds needed to have large overlaps in sstate objects. This can be simulated by building the same things with just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of deferred tasks. The issue is that the expensive checks in the setscene loop were hit every time through runqueue's execute function before the check on deferred tasks. This leads to task execution starvation as that only happens once per iteration. Move the skip check earlier in the function which speeds things up enormously and should improve performance of such builds for users. (Bitbake rev: db083cfe9e33c9fd7ffeead7b8c6023a5d581976) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9c6c506757f2b3e28c8b20513b45da6b4659c95f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2: use persist_data context managersChris Laplante2024-12-061-31/+31
| | | | | | | | | | | | | | | | | Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s. See https://docs.python.org/3/whatsnew/3.13.html#sqlite3 The previous commit fixed persist_data's context manager to close the connection, but we were never actually using `with` in the first place. This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: 6c2641f7a9e92c1b82e306f59ddd3c1249c52cbf) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: persist_data: close connection in SQLTable __exit__Chris Laplante2024-12-061-0/+1
| | | | | | | | | | | | | | A Connection's __exit__ does not close the connection, but rather deals with transactions. See https://docs.python.org/3/library/sqlite3.html#how-to-use-the-connection-context-manager This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: c9ea354c2df3d94b9ebe7810e83df9efb6007013) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/wget: Increase timeout to 100s from 30sRichard Purdie2024-12-061-2/+2
| | | | | | | | | | | Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: c7f282cd27edfd78830b61db586ed669808893a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: Remove custom exception backtrace formattingJoshua Watt2024-12-065-121/+25
| | | | | | | | | | | | | | | | | | | | | | | | Removes the code in bitbake to show custom backtrace formatting for exceptions. In particular, the bitbake exception code prints function arguments, which while helpful is a security problem when passwords and other secrets can be passed as function arguments. As it turns out, the handling of the custom serialized exception stack frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass formatted exceptions"), which changed the events to pass a preformatted stacktrack list of strings, but the passing of the serialized data was never removed. Change all the code to use the python traceback API to format exceptions instead of the custom code; conveniently traceback.format_exception() also returns a list of stack trace strings, so it can be used as a drop in replacement for bb.exception.format_exception() (Bitbake rev: c25e7ed128b9fd5b53d28d678238e2f3af52ef8b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/git: Use quote from shlex, not pipesRichard Purdie2024-11-281-2/+1
| | | | | | | | | | | | The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 70bd343fb273ad174e56d08c8b80c5594501e030) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of mobile-broadband-provider to ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | decouple from gnome gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 377eba2361850adfb8ce7e761ef9c76be287f88c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 91e268b11ed683bd197026f9b36001f6d54ee05c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of sysprof to decouple from gnome ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 12e2b11e4038138ad1a02b9d4d36d0904fbc820d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 008808755ed6cfeb6c41273e69ce718f0833c26c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Remove downloads/tmpdir when failedRobert Yang2024-11-021-3/+5
| | | | | | | | | | | The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 6894f68204130713d3651fceacadc7f8061174a8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Add call_process_submodules() to remove duplicated codeRobert Yang2024-11-021-28/+14
| | | | | | | | | | | There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: 76de94d46ce270c4e485f8178725973adf0f42eb) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: asyncrpc: Use client timeout for websocket open timeoutJoshua Watt2024-10-301-1/+5
| | | | | | | | | | | | The default connection timeout for websockets is 10 seconds, so use the provided client timeout instead (which defaults to 30 seconds). (Bitbake rev: 270c0c8bdc802c04fd3021c082d58cf8e8d68d82) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 23681775e5941e54ebead469addf708fca1e6beb) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart: Improve performance for VariableHistoryRobert Yang2024-08-261-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS $ bitbake -p -P Check profile.log.processed, the record() cost more than 20 seconds, it is less than 1 second when multiconfig is not enabled, and there would be the following error when more muticonfigs are enabled: Timeout while waiting for a reply from the bitbake server Don't change the type of loginfo['detail'] or re-assign it can make record() back to less than 1 second, this won't affect COW since loginfo is a mutable type. The time mainly affected by two factors: 1) The number of enabled layers, nearly 1 second added per layer when the number is larger than 50. 2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer when the layers number is larger than 50. (Bitbake rev: d251668d9a7a8dd25bd8767efb30d6d9ff8b1ad3) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: codeparser: Skip non-local functions for module dependenciesRichard Purdie2024-07-261-0/+5
| | | | | | | | | | | | | | | | If modules do something like "from glob import glob" then we end up checksumming the glob code. That leads to bugs as the code can change between different python versions for example, leading to checksum instability. We should ignore functions not from the current file as implemented by this change. (Bitbake rev: 11d83170922a2c6b9db1f6e8c23e533526984b2c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e6f862864539d6f6a0bea3e4479e0dd40ff3091) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: codeparser/data: Ensure module function contents changing is ↵Richard Purdie2024-07-262-12/+18
| | | | | | | | | | | | | | | | | | | | | | accounted for Currently, if a pylib function changes contents, the taskhash remains unchanged since we assume the functions have stable output. This is probably a poor assumption so take the code of the function into account in the taskhashes. This avoids certain frustrating build failures we've been seeing in automated testing. To make this work we have to add an extra entry to the python code parsing cache so that we can store the hashed function contents for efficiency as in the python module case, that isn't used as the key to the cache. The cache version changes since we're adding data to the cache. (Bitbake rev: 4bf332ccac283ca3440e81d8c781fcc23fe10b98) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b2c3438ebe62793ebabe2c282534893908d520b4) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/wget: Fix failure path for files that are empty or don't existRichard Purdie2024-06-241-9/+9
| | | | | | | | | | | | | | | When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: 8714a02e13477a9d97858b3642e05f28247454b5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak to work on Fedora40Richard Purdie2024-06-241-3/+3
| | | | | | | | | | | | | | On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1 and python only binds to one of the addresses, leading to test failures. Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying to test things other than the host networking. (Bitbake rev: 5fa596b3f2e49fbd144347b9800c76de72a29e28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibilityRudolf J Streif2024-06-241-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '<path>') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak test to match upstream repo url changeRichard Purdie2024-06-141-1/+1
| | | | | | | | | | Upstream changed their urls, update our test to match. (Bitbake rev: 0791e66a47185d6f202af6be0d39e36a8e41850a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit dc391b86540ec5e0a0f1d811c776a22d443b1c06) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: siggen: Enable batching of unihash queriesJoshua Watt2024-06-061-3/+6
| | | | | | | | | | | Uses the batching API of the client to reduce the effect of latency when making multiple queries to the server (Bitbake rev: fdac31655fb5441139f70e50292dbb5774cfdafe) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bb: Use namedtuple for Task dataJoshua Watt2024-06-062-18/+39
| | | | | | | | | | | | | Task dependency data is becoming unwieldy with the number of indices it contains. Convert it to use a named tuple instead, which allows members to be indexed by a named property or an index (which allows it to retain backward compatibility). (Bitbake rev: bc7c44affe8e3fa94e92c3bcb8ad85bf11963779) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Improve rehash get_unihash parallelismRichard Purdie2024-06-061-2/+10
| | | | | | | | | | Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. (Bitbake rev: 682fb42420eebf18f8a98bd3992baf9034dac5ac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Process unihashes in parallel at initRichard Purdie2024-06-062-9/+10
| | | | | | | | | | | | | | | Improve the runqueue init code to call unihash queries in parallel since this is faster and more efficient, particularly on slower links with longer round trip times. The call to the function from cooker is unneeded since that function calls prepare() and hence this functionality will already have run, so drop that obsolete call. (Bitbake rev: e0486054c7a4c637446c18608e9983cc8dc4d7fe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: siggen/runqueue: Report which dependencies affect the taskhashjoshua Watt2024-06-062-4/+6
| | | | | | | | | | | | | | Report which task dependencies in BB_TASKDEPDATA are included in the taskhash. This allows tasks to identify which tasks dependencies may change without the task re-running. Knowing this information is important for tasks that want to transfer information from dependencies (such as SPDX) (Bitbake rev: 853423661779023763a87462b623b6e9ff2798b2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Allow rehash loop to exit in case of interruptsRichard Purdie2024-06-061-1/+3
| | | | | | | | | | | The initial hash serve loop exits in the case where interrupts are present but probably checks a bit too often. Tweak that and also allow the slow rehash loop to break on interrupt, improving bitbake Ctrl+C response. (Bitbake rev: ca2f63d84b90cbd99936c9d7815d912d9019a594) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Add timing warnings around slow loopsRichard Purdie2024-06-061-0/+22
| | | | | | | | | | | | | | With hashserve enabled, there are two slow paths/loops, one at initial runqueue generation and also during the rehash process when new outhashes are found. Add timing information at the hashserve log level for when these loops take longer than 30s or 60s overall. This will leave evidence in the logs when things are running particularly slowly. (Bitbake rev: 9ee503c79936b13f1d45f9e43211f77a528cdbfa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/gcp: Add missing runfetchcmd importSven Schwermer2024-05-291-0/+1
| | | | | | | | | | This adds the missing import. This bug was introduced with 1ab1d36c. (Bitbake rev: b5159c0373e2e7d403aed16e096ad655f38b1fa7) Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: parse: Improve/fix cache invalidation via mtimeRichard Purdie2024-05-291-4/+8
| | | | | | | | | | | | | | | | | | | | We have been seeing obscure failures in devtool, particularly on newer autobuilder workers where it appears the cache is assumed to be valid when it shouldn't be. We're using the 'seconds' granulation mtime field which is not really a good way of telling if a file has changed. We can switch to the "ns" version which is better however also add in inode number and size as precautions. We already have all this data and tuples are fast so there isn't really any cost to do so. This hopefully fixes [YOCTO #15318]. (Bitbake rev: 2aa519fec30ecf0e55202d6421fbba2f0320e211) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d9e5d313c79500e3c70ab9c3239b6b2180194f67) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: cooker: Handle ImportError for websocketsjoshua Watt2024-05-291-1/+2
| | | | | | | | | | | | | | Handles ImportError when creating a hash equivalence to ping the server. This notifies user earlier with a more precise error if websockets can't be used, and also prevents passing a known bad upstream value to the local server (Bitbake rev: 93190565fd1251e8f47d9a6291739f8b8ed5ec87) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: cooker: Use hash client to ping upstream serverjoshua Watt2024-05-021-4/+3
| | | | | | | | | | | | | | The cooker attempts to connect to the upstream hash equivalent server to warn the user early if it is misconfigured. However, this was making the assumption that it was a raw TCP connection and failed when attempting to use a websocket upstream server. Fix this by creating an hash client and using the ping API to check the server instead of using a raw socket. (Bitbake rev: 8f90d10f9efc9a32e13f6bd031992aece79fe7cc) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.8.0Richard Purdie2024-04-161-1/+1
| | | | | | (Bitbake rev: c86466d51e8ff14e57a734c1eec5bb651fdc73ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Capture SSL environment for hashserverJoshua Watt2024-04-161-24/+70
| | | | | | | | | | | | | | Now that the bitbake hash server supports SSL connections, we need to capture a few environment variables which can affect the ability to connect via SSL. Note that the variables are only put in place to affect the environment while actually invoking the server [RP: Tweak to use BB_ORIGENV as well] [RP: Tweak to handle os.environ restore correctly] (Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Handle unclosed functions correctlyRichard Purdie2024-04-141-1/+7
| | | | | | | | | | | | | | | | | A function accidentally defined as: somefunction() { : } which is unclosed due to the space at the end, would currently silently cause breakage. Have the parser throw and error for this. [YOCTO #15470] (Bitbake rev: a7dce72da6be626734486808f1b731247697e638) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: include parse_address from hashservMichael Opdenacker2024-04-141-0/+23
| | | | | | | | | | | | | | Moving the code and related definitions from hashserv/__init__.py to asyncrpc/client.py, allowing this function to be used in other asyncrpc clients. (Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Suggested-by: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-04-111-1/+4
| | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add support for hashserve credentialsJoshua Watt2024-03-231-5/+16
| | | | | | | | | | Adds support for hashserver credentials to be specified in the SignatureGenerator (Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Install Git LFS in local repository configDerek Erdmann2024-03-221-1/+1
| | | | | | | | | | | | | | | | | Git uses a lock file to prevent concurrent modifications to the global config, so if unpack tasks for different recipes try to run "git lfs install" simultaneously the operation can fail: error: could not lock config file /home/build/.gitconfig: File exists exit status 255 Run `git lfs install --force` to reset Git configuration. Adding "--local" sets the smudge and clean filters in the local repository's config instead of modifying the user's global config. (Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7) Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: better estimate number of available cpusFelix Moessbauer2024-03-221-1/+4
| | | | | | | | | | | | | When running in a cgroup which is limited to a subset of cpus (via cpuset.cpus), cpu_count() should return the number of cpus that can be used instead of the number of cpus the system has. This also aligns the semantics with nproc. (Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix misleading "no output" msgViswanath Kraleti2024-03-221-1/+4
| | | | | | | | | | | When a command is run with a non-null log, errors are only output to the log and are not returned in the exception. In that case direct users to that logfile instead of telling the command had no output. (Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a) Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle URIs with single-valued query parametersRoss Burton2024-03-222-2/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Whilst typically the URI query is a list of key-value pairs, that's not actually required by the URI specification. For example: http://example.com/foo?bar is a valid query, but this will result in the fetcher raising an exception: File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__ self.query = self._param_str_split(urlp.query, "&") File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]: ValueError: not enough values to unpack (expected 2, got 1) In this case the query is just "bar", but the fetcher is trying to split it into a key-value pair. The URI object exposes the parsed query explicitly as a dictionary of key-value pairs, so we have to be a little creative here: if a value is None then it isn't a key-value pair, but a bare key. Fix this by handling elements without the deliminator in _param_str_split() (by assigning the value to None), and handle a None value when formatting the query in _param_str_join(). (Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: support NO_COLORPeter A. Bigot2024-03-222-2/+2
| | | | | | | | | | | | | Red text on a black background can make it difficult for people with visual impairments to read the text of error messages. Respect the presence of a non-empty NO_COLOR environment variable as an indication the user doesn't want colorization to be automatically enabled. See: https://no-color.org/ (Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85) Signed-off-by: Peter A. Bigot <pab@pabigot.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/git: Escape parentheses in git src namePhilippe Rivest2024-03-221-1/+1
| | | | | | | | | | | | | | | | | | FIXES [YOCTO #15404] When using git fetcher on a repo with parentheses in its URL, the invocation of the git clone command will fail. The clone directory is not quoted thus the shell will return an error and won't execute the command. (Bitbake rev: b5624ee5643d881afa004571a096a189ab5389b5) Cc: Philippe Rivest <privest@genetec.com> (Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6) Signed-off-by: Philippe Rivest <technophil98@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-01-23 19:52:34 +0000