summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: ast: Fix fragment behaviour with overridesRichard Purdie4 hours1-1/+2
| | | | | | | | | | | | | | | | | | | | | Imagine a machine fragment machine/A and a configuration which sets: MACHINE = "B" MACHINE:forcevariable = "C" As I understand it, the fragment behaviour was intended to replace the MACHINE = "B", so the override would still be active. The current code replaces all variable overrides. parsing=True, switches to the other behaviour, which I believe was the design intent and the behaviour users would expect. This is useful to allow test configurations to override a MACHINE setting without change the fragments which would complicate the test code. (Bitbake rev: 6b4882a55567d440713ee34d81810167cd958d56) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Make the "Initialising tasks" progress bar finish properlyPeter Kjellerstedt4 hours1-1/+4
| | | | | | | | | | The "Initialising tasks" progress bar needs to be finished before update_scenequeue_data() creates another progress bar. (Bitbake rev: 620d53d7b61fbe98547047b07cb69e98afeb3fc5) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add support for skipping a fragment selectionAlexander Kanavin4 hours1-2/+8
| | | | | | | | | | | | | | | | | In autobuilder testing a use case arised where - the available choices in configuration file for choosing a machine are incomplete - putting every possible machine choice into that configuration is undesirable/not possible - autobuilder code can write a machine selection into the bitbake config after the fact. So this --skip-selection option is intended for advanced users that know what they're doing and is generally not recommended as it requires manually tweaking the bitbake config to make it usable. (Bitbake rev: fa81154cd8a204b394afa698daa0162e6337e7fa) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: registry: make a separate configuration for poky with ↵Alexander Kanavin4 hours1-3/+11
| | | | | | | | | | | | | | | | | sstate mirror Using sstate mirror places much higher requirements on both network robustness and network bandwidth than just fetching source code. When the network doesn't meet those, the user experience can be very frustrating as errors can be cryptic and intermittent, or bitbake would just seemingly do nothing for a long time. Let's make sstate an opt-in, and provide a caution for using it. When it does work, it works wonderfully and does accelerate builds significantly. (Bitbake rev: 9ac11955b8ee5a0cc964ef62d881ec6564b24a48) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: clarify that default answer to prompts is noPaul Gortmaker4 hours1-3/+3
| | | | | | | | | | | | | | | | It is common practice to put the default choice in upper case for yes/no interactive prompts, so that when people just hit enter, they know what they are getting. An example that linux users are probably familiar with is "sensors-detect" from the "lm-sensors" package. Unify all the prompts to be the same and indicate that the default answer from hitting enter is a no with an upper case N. No functional changes. (Bitbake rev: c7b1a34a25a386426218d63d2aeba0e7c6ecf5b0) Signed-off-by: Paul Gortmaker <paulg@kernel.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: allow using {THISDIR}/my-layerYoann Congal4 hours2-5/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This implement the ability to use "{THISDIR}/my-layer" in the "bb-layers" list. "{THISDIR}" is remplaced by the directory containing the configuration file. In small projects, we try to keep the setup a simple as possible: a single git repo containing both the build confguration (e.g. a bitbake-setup configuration file) and the meta layer with project recipes/machine/distro. This change allows this kind of setup: ├── meta-my-project/ # the project layer └── my-project.conf.json # the bb-setup configuration file by writing, in my-project.conf.json: "bitbake-setup": { "configurations": [{ "bb-layers": [ "{THISDIR}/meta-my-project" Note: in this case meta-my-project is not present as a "source", so, not handled by bb-setup update/status. It is expected of the user to handle this on their own (is our case, a simple git workflow). (Bitbake rev: 61fbdeb358979bb13c9e50d807f43e554497dc9c) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/tests/setup.py: unset BBPATH to ensure isolation from the ↵Alexander Kanavin4 hours1-0/+5
| | | | | | | | | | | | existing bitbake environment bitbake-setup deduces top directory from BBPATH, which, if set, interferes with the tests' own setup. (Bitbake rev: ca21545bb847138d75d985d6d7eb0dc9b14550be) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/tests/setup.py: define test parameters in a single dictionaryAlexander Kanavin4 hours1-3/+7
| | | | | | | | | This makes maintaining and extending them easier. (Bitbake rev: d4379bb791b8961538a4d445d5f956fafffbb0e5) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: dash support for init-build-env scriptYoann Congal4 hours1-7/+8
| | | | | | | | | | | | | | | | | | | | | | Being minimalist, dash does not support the (non-POSIX) feature of passing an argument while sourcing a script. Like in . <some path>/oe-init-build-env <build dir> With dash, one must use: set <build dir> # puts <build dir> in $1 cd <some path> . ./oe-init-build-env # can only be called from its directory in dash To do this: * Instead of a symlink to oe-init-build-env, keep a symlink to the directory containing it (called "oe-init-build-env-dir") * Generate a init-build-env script that dash can source using the above snippet. (Bitbake rev: d77386e6ab155081ff3bef2aab413ff8061e83a5) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: suggest "." instead of "source"Yoann Congal4 hours1-1/+1
| | | | | | | | | | | | "." is in POSIX standard[0], whereas "source" is only supported in more feature-full shells (bash, zsh, ...) [0]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#dot (Bitbake rev: 3c5f04343b30efcf53d13f13203c5b0f3a518b99) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: improve robustness of loading/writing settingsAlexander Kanavin4 hours1-3/+3
| | | | | | | | | | | | | | | Particularly: - ensure global settings command line argument is always expanded to full path - ensure any errors that happen when loading settings are reported at that point, otherwise we get an empty dictionary and cryptic key errors later (Bitbake rev: b3f3ea52e90fb2036074cc88fd00496eb995346d) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add support for specifying branches in repo checkoutsAlexander Kanavin4 hours4-7/+23
| | | | | | | | | | | | | | | | | | | | Previously bitbake-setup was checking out 'detached commits' using fetcher's nobranch feature, as that is the only option when only a revision is in the config. Branches are optional, but beneficial, as - checkout directory will be on a branch, making it easier for users to understand where they are if they need to make changes (also bitbake will print branch information instead of saying 'HEAD:sha'). - supply chain security! Enforcing a branch means any specified revision has to be on it, and no one can sneak in (accidentally or deliberately!) some dangling commit, or something from their private branch in the same repo. (Bitbake rev: 14dd5f53913419a6c26c0b809a592bd19ae8495a) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: Improve the already initialized testRichard Purdie4 hours1-2/+2
| | | | | | | | | If the directory already exists but hasn't been setup, the current test can fail so improve it. (Bitbake rev: f8cc590403212886cc5ea19fe1e78dcd3438b612) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: Switch to internal default registry filesRichard Purdie4 hours1-1/+1
| | | | | | | | | Switch the url to be the default internal registry rather than a private repo which was intended for testing. (Bitbake rev: dd1841a7c3edc2749924370c307ff40ec65f0fe1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add default registry config for OpenEmbedded's nodistro and Yocto ↵Richard Purdie4 hours2-0/+124
| | | | | | | | | | | | | | | Project's poky Rather than forcing every user to have to access a separate repository for data for some key defaults, add those for nodistro and poky. This gives us some commonly used setups and also something to test against too. We will need to come up with a criteria for adding these, most likely based on community usage/demand with the OE TSC governing that policy. (Bitbake rev: ce6561608904e971a5cf9214a68391ea7fccc6b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: Allow local registry pathsRichard Purdie4 hours1-4/+9
| | | | | | | | | | It is useful for bitbake-setup to support local paths without access through the fetcher so that internal data to the bitbake repository can be used as a default. (Bitbake rev: 5b8cd9d2156ef6bdd4c6e61130fadc0ff85f7f1f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: adjust the error message about missing websockets module to ↵Alexander Kanavin4 hours1-1/+4
| | | | | | | | | mention bitbake-setup (Bitbake rev: d857ed2044c8153c69bd3e275cfc8a9b6a8cdc5e) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add 'install-buildtools' commandAlexander Kanavin4 hours2-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This basically calls install-buildtools from oe-core/poky, but it ensures via command line parameters that the installation location is stable and the downloads are preserved for reproducibility: $ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf ====== Buildtools archive is downloaded into /home/alex/bitbake-builds/yocto-master-testing/buildtools-downloads/20250319141333 and its content installed into /home/alex/bitbake-builds/yocto-master-testing/buildtools ... (output from install-buildtools script) ====== It also detects when buildtools are already installed, and will direct users what to do: ====== alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf Buildtools are already installed in /home/alex/bitbake-builds/yocto-master-testing/buildtools. If you wish to use them, you need to source the the environment setup script e.g. $ . /home/alex/bitbake-builds/yocto-master-testing/buildtools/environment-setup-x86_64-pokysdk-linux You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation ====== This commits includes fixes by Gyorgy Sarvari <skandigraun@gmail.com> https://github.com/kanavin/bitbake/pull/2 (Bitbake rev: 9fb564b720ff5d29a245ec900c8996da7f5f24a6) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add tests to bitbake-selftestAlexander Kanavin4 hours2-0/+267
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Run like this: alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok ---------------------------------------------------------------------- Ran 1 test in 9.223s OK The test does a basic run-through of init, then status/update on an unchanged configuration, then status/update on a configuration changed via new commits to the test layer, then status/update on configuration changed via the top level json config file. Note that nothing whatsoever is fetched from the network; the test relies entirely on synthetic data contained inside itself, including minimal stubs for oe-setup-build and bitbake-config-build. This data is used to create temporary git repositories then clone them via local filesystem URIs. Later on this can be supplemented by an oe-selftest that tests bitbake-setup against real config files in the official configuration repository and real layers, templates and fragments. (Bitbake rev: 59ec7e2ac56e364bae39ccb5e884c447a5379531) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add the initial implementationAlexander Kanavin4 hours1-0/+809
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Preamble ======== The latest iteration of this patchset is available at https://github.com/kanavin/bitbake I recommend taking the patches from there to ensure that you are not trying out outdated code. For the rationale and design guidelines please see this message: https://lists.openembedded.org/g/openembedded-architecture/message/1913 Left out for now but will be done later: - official configuration repository - documentation Amble *scratch* HOWTO ===================== 1. If you don't know where to start, run 'bitbake-setup init'. Bitbake-setup will ask a few questions about available configuration choices and set up a build. Note: 'init' sub-command can also take a path or a URL with a configuration file directly. You can see how those files look like here: https://github.com/kanavin/bitbake-setup-configurations 2. You can then source the bitbake environment and run bitbake to perform builds as usual: $ . /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env Also, subsequent status/update commands will not require a separate --build-dir argument telling bitbake-setup where the build is. 3. To check if the build configuration needs to be updated, run: === $ bin/bitbake-setup status ... Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed. === If the configuration has changed, you will see the difference as a diff. ... - "rev": "akanavin/sstate-for-all" + "rev": "akanavin/bitbake-setup-testing" ... If the configuration has not changed, but layer revisions referred to it have (for example if the configuration specifies a tip of a branch), you will see that too: === ... Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de === 4. If the configuration has changed, you can bring it in sync with: $ bin/bitbake-setup update Note that it will also rename/preserve the existing build/conf directory, and print changes in bitbake configuration (diff of content of build/conf/) if that has changed. I can't at the moment think of anything more clever that is also not much more brittle or complex to implement, but open to suggestions. Terminology =========== - 'top directory' means the place under which bitbake-setup reads and writes everything. bitbake-setup makes a promise to not touch anything outside of that, unless otherwise directed to by entries in settings (currently there is one such setting for fetcher downloads for layers and config registries). Top directory can be selected by an environment variable, a command line option, or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment (e.g. we are in a bitbake environment), then the top directory is deduced from that and doesn't need to be specified by hand. - 'settings' means bitbake-setup operational parameters that are global to all builds under a top directory. E.g. the location of configuration registry, or where the bitbake fetcher should place the downloads (DL_DIR setting). Settings are stored in a .conf file in ini format just under the top directory. - 'build' means a tree structure set up by 'bitbake-setup init', consisting of, at least, a layers checkout, and a bitbake build. It maps 1:1 to the json data it was constructed from, which is called 'build configuration'. Build configurations are constructed from generic configurations that may involve making one or more choices about available options in them. Generic configurations are files, URLs or are obtained from git repositories called 'config registries', in which case they can be listed with 'bitbake-setup list'. There can be multiple 'builds' under a top directory. Here are two example generic configurations that showcase this: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-options.conf.json https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-nested-configs.conf.json - 'bitbake-setup status' will tell if a build is in sync with the generic configuration it was made from. 'bitbake-setup update' will bring a build in sync with a configuration if needed. - 'bitbake build' means a particular sub-tree inside a build that bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. Bitbake configurations are constructed from templates and fragments, with existing mechanisms provided by oe-core. The configuration file format is specified such that other mechanisms to set up a bitbake build can be added; there was a mention of ability to specify local.conf content and a set of layers directly in a configuration. I think that scales poorly compared to templates and fragments, but I made sure alternative ways to configure a bitbake build are possible to add in the future :) - 'source override' is a json file that can be used to modify revisions and origins of layers that need to be checkout into a build (e.g. when master branches need to be changed to master-next for purposes of testing). Such a file is specified with a command-line option to 'init' and an example can be seen here: https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-next.override.json This commit includes fixes by Ryan Eatmon <reatmon@ti.com> https://github.com/kanavin/bitbake/pull/1 Gyorgy Sarvari <skandigraun@gmail.com> https://github.com/kanavin/bitbake/pull/2 Johannes Schneider <johannes.schneider@leica-geosystems.com> https://github.com/kanavin/bitbake/pull/3 https://github.com/kanavin/bitbake/pull/5 (Bitbake rev: bf3542076cdd751da9de8745aa3f351f5c6b0c5f) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: Make include_all raise a parse error for absolute pathsPeter Kjellerstedt4 hours2-0/+4
| | | | | | | | | It makes no sense to use an absolute path with include_all. (Bitbake rev: a39d1d5df740824d40c4d8e7f4199ccc518592e0) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: Make include_all support empty variable expansionsPeter Kjellerstedt4 hours3-5/+9
| | | | | | | | | | | | | include and require support empty variable expansions, typically used with conditional expansions. However, include_all does not, and instead reports an error for the first path in BBPATH. Rewrite include_all so that its behavior matches include and require. (Bitbake rev: 010928ceac48b20e6ab530f8e256b72201b778ec) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: extend classes and include/require documentationAntonin Godard11 days3-29/+131
| | | | | | | | | | | | | | | | | | | | | | | | The current documentation is lacking details on the different kinds of directories for putting classes as well as the order in which BitBake includes files (classes or include files). This patch does the following changes: - Mention the missing classes-recipe and classes-global when applicable. - Add a Class Types section detailed the three different directories for putting classes. - Move the existing section on locating classes/include files below the documentation on the different directives (include/require/inherit). - Extend the documentation on locating files with include/require and inherit to give proper examples, hopefully demystifying this mechanism a bit. [YOCTO #15724] (Bitbake rev: 7bd36f6c6d33211bb2a6b6fc6d40bdbd83b8b7c3) Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Add tests for include, require and include_allPeter Kjellerstedt11 days1-0/+60
| | | | | | | | | | Simple tests for various ways to include and require configuration files. (Bitbake rev: bdcb67824172ed5cd76fa0274dc3d27aeb52e77c) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Remove unnecessary calls to file.flush()Peter Kjellerstedt11 days1-8/+0
| | | | | | | | | | | There is no reason to call file.flush() as the last call within a file context manager since ending the context will close the file and thus flush it. (Bitbake rev: 90d5ce926d434d60d6df28b7d77b3cbc9b62ce14) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Use with statement to avoid ResourceWarningPeter Kjellerstedt11 days1-51/+44
| | | | | | | (Bitbake rev: 46a5e31713fc526a1807b1617fba8a01c1564641) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update tests after bitbake tag removalMathieu Dubois-Briand11 days1-12/+12
| | | | | | | | | | Tags for bitbake 2.8.6 and 2.8.7 have been removed from the git: use some other ones. (Bitbake rev: dcf12947954f3184d99fbf9813b3f3f667dacc5f) Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: verify if local clone contains tagGyorgy Sarvari2025-08-192-3/+51
| | | | | | | | | | | | | | | | | | | | In case a recipe specifies a git SRC_URI along with revision and tag, but only the revision is present in the local clone without the tag (because it was tagged after it was cloned), then unpacking fails with the following error: ... rev-list -n 1 1.0 failed with exit code 128, output:\nfatal: ambiguous argument \'1.0\': unknown revision or path not in the working tree This happens because the during the download step only the revision's presence is verified to decide if the repository needs to be updated. To avoid this, check also if the tag is present in the local repository, when the "tag" tag is specified. (Bitbake rev: 546b347b4d3d82c01ecc99f45296f66e44638adc) Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/bitbake-user-manual-ref-variables: update BB_TASK_IONICE_LEVELPeter Marko2025-08-191-2/+2
| | | | | | | | | | | | | | | | | Provided example does not work due to two problems. CFQ scheduler was removed in kernel 4.20 (per [1]) Replace it with BFQ scheduler. Also fix typo "queu" -> "queue". [1] https://en.wikipedia.org/wiki/I/O_scheduling (Bitbake rev: 6716853e35ebc2e1523210a83b483cdacb600295) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: print() was eliminated from all loops for better performance.Benjamin Szőke2025-08-191-7/+7
| | | | | | | | | | | Refactoring prints, print() functions were eliminated from all loops and it uses "\n".join(...) in a single print() call for better performance. (Bitbake rev: c32c3d9b83818661e12f3e437563ab4e1fa05e15) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Use 40 Hz refresh rate (FPS) for footer update.Benjamin Szőke2025-08-191-4/+20
| | | | | | | | | | Refresh footer in 40 Hz to avoid heavy print() flooding but keep it fluent for human eyes. (Bitbake rev: c36efdf642d858c6997819744d00a3c1965c6417) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Use a StringIO buffer for update footer.Benjamin Szőke2025-08-191-10/+26
| | | | | | | | | | | Optimize printing in footer update with use a StringIO buffer and it prints content to terminal in a single call in every cycle. (Bitbake rev: 32ba622d78f20b231f30f848379b4bbc3d7414da) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: progressbar: Add self._fd_console to use for self._handle_resize()Benjamin Szőke2025-08-191-2/+9
| | | | | | | | | | Introduce self._fd_console as a dedicated attribute of self._handle_resize(). (Bitbake rev: f8c76eb89d52b1c28407f0b52dfe4318faa47cd2) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update version to 2.15.2Richard Purdie2025-08-122-2/+2
| | | | | | (Bitbake rev: 7ad404fd1aa0a48978c0351c5f52f17b8992c8b8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: use enum to represent runQueue state to improve readabilityChris Laplante2025-08-121-44/+46
| | | | | | | (Bitbake rev: e61265570f5415d647666891d806836437330219) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/runqueue: remove unused 'd' arg from check_for_interruptsChris Laplante2025-08-122-17/+17
| | | | | | | (Bitbake rev: e0c5e94cc14dd6b1cbde491073c1f9407c97e428) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: remove some redundant control flowChris Laplante2025-08-121-4/+1
| | | | | | | | | | | Prior to commit aa84a900e ("cooker: Ensure delays are accurately transfered to the idle loops from runqueue"), this was necessary. But now retval is returned directly. (Bitbake rev: c2eb4336fe10b1bf8bbc6291c32ca362840f39e1) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add bb.utils.explode_deps as a filter functionJoshua Watt2025-08-121-0/+2
| | | | | | | | | Marks bb.utils.explode_deps as callable from filter functions (Bitbake rev: bb07003641e76de994482f7835a432f20297af96) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add setVarFilter function to implement variabl filteringRichard Purdie2025-08-121-0/+19
| | | | | | | | | | | | | | | | Adds a new setVarFilter() API to the data store allowing filters to be applied to variables. Note that filters are applied to the non-override part of the variable name so a filter set against RDEPENDS would apply against RDEPENDS:${PN} and friends. The filter function is applied before returning the final variable value. (Bitbake rev: a9471c10d1de039474ddb4738abd286b928d82f4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add filter supportJoshua Watt2025-08-123-0/+231
| | | | | | | | | | | | | | Add the python API for applying filters to a string and being able to register functions as filters. Filter functions are pure functions where an input is translated into an output and there are no external data accesses. This means translations can be cached as they won't change. (Bitbake rev: 7d25d7511ca14213eea78ee739d260295cfa4045) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ignore KeyErrors during shutdownRichard Purdie2025-08-041-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When parsing failures occur, trap KeyError to avoid these kinds of tracebacks (from a parsing error in meta-ti). bb.parse.ParseError: ParseError at /srv/pokybuild/yocto-worker/check-layer-nightly/build/meta-ti/meta-ti-bsp/recipes-kernel/linux/linux-ti-mainline_git.bb:6: Could not inherit file classes/${KERNEL_BASE_CLASS}.bbclass During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 1639, in updateCache if not self.parser.parse_next(): File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2314, in parse_next self.shutdown(clean=False, eventmsg=str(exc)) File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2209, in shutdown read_results() File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2195, in read_results self.result_queue.get(timeout=0.25) File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get return _ForkingPickler.loads(res) File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cache.py", line 338, in __setstate__ setattr(self, key, self._restore(state[key], pid)) File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cache.py", line 318, in _restore ret[dep] = map[mapnum] KeyError: 156 (Bitbake rev: 750c68ee7ee3f3d4518348e4c948243504880770) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: ignore comments in .gitattributes when detecting LFSGyorgy Sarvari2025-07-311-1/+1
| | | | | | | | | | | | | | | | | | | | Fixes [YOCTO 15917] When fetching a git repository the .gitattributes file is scanned, checking if LFS support is required for the repository. This scan checks if the word "lfs" is present in the file, however the used regex doesn't account for comments, which makes some repositories[1] be to misidentified as requiring LFS support (which fails fetching, in case lfs support isn't installed on the build host). To avoid it, change the used regex to ignore lines starting with "#". [1]: e.g. https://github.com/MicrochipTech/cryptoauthlib (Bitbake rev: 7917a758fc328747116c7899e689171bd0efc883) Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/bitbake-user-manual-ref-variables: document BB_DEFER_BBCLASSESAntonin Godard2025-07-312-0/+22
| | | | | | | | | | | Document the BB_DEFER_BBCLASSES variable added in 8e741b2e885a ("ast/BBHandler: Add support for BB_DEFER_BBCLASSES"). (Bitbake rev: 3f9eba26b29db526fed263a7e70b824988e17656) Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/bitbake-user-manual-intro: Mention existence of three supported ↵Robert P. J. Day2025-07-311-0/+12
| | | | | | | | | | | | | "classes" directories Early in Section 1, mention that BitBake supports three related classes directories. (Bitbake rev: 0d5201e904b0e6f4dd9f7bb116dda508ec09b78a) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/bitbake-user-manual-intro: Add examples to show general wildcardingRobert P. J. Day2025-07-311-0/+2
| | | | | | | | | | | Add a couple more examples to the wildcarding section to make sure readers understand it's not just a single character wildcard. (Bitbake rev: 572062ba1f0a2953a62ef1974e35134fcb462f5e) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/bitbake-user-manual-ref-variables: fixupsAntonin Godard2025-07-311-11/+11
| | | | | | | | | | | | Fix typos reported by Quentin here: https://lore.kernel.org/r/cee995db-4c3d-4bf0-813d-ce3f5f8f92e3@cherry.de https://lore.kernel.org/r/73497b5e-721e-4e5f-beb4-c56ab9178c9f@cherry.de Cc: Quentin Schulz <quentin.schulz@cherry.de> (Bitbake rev: add8a7ac093d60e731a638c25ec15efcc3988781) Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: clarify significance of "include_all" directiveRobert P. J. Day2025-07-281-30/+39
| | | | | | | | | | | | Rewrite the include and include_all sections to drive home the fact that the include_all directive is relevant in only very specific cases, and not something developers should expect to use in normal operation. (Bitbake rev: 4b3bfe70d02cc1c11972357e2dc595acc75056e5) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Keep query parameters in URL during checkstatusPhilippe-Alexandre Mathieu2025-07-281-1/+4
| | | | | | | | | | | | | | | | | | | | When recreating the uri in wget's checkstatus method, we only use the scheme, netloc and path. This completely strips the query parameters from the final URI and potentially breaks the checking functionality from certain fetchers. This is the case for the Azure storage fetcher, as it requires a SAS token that is formatted as a series of query parameters. The error manifests itself when using a private storage account as a PREMIRROR or SSTATE_MIRROR (since regular SRC_URI won't run the checkstatus). This problem is present in scarthgap, but wasn't in kirkstone. CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> (Bitbake rev: 096301250455e2a83bdd818a56317c62436c9981) Signed-off-by: Philippe-Alexandre Mathieu <pamathieu@poum.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sigen: Avoid bitbake abort with EOFerror from an incomplete ↵Pedro Ferreira2025-07-231-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | hashserv communication The issue itself is sporadic but aborts the build with an uncaught exception. The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_report_unihash(d) 0003: File: '(...)/poky/meta/classes/sstate.bbclass', lineno: 882, function: sstate_report_unihash 0878: report_unihash = getattr(bb.parse.siggen, 'report_unihash', None) 0879: 0880: if report_unihash: 0881: ss = sstate_state_fromvars(d) *** 0882: report_unihash(os.getcwd(), ss['task'], d) 0883:} 0884: 0885:# 0886:# Shell function to decompress and prepare a package for installation File: '(...)/poky/lib/bb/siggen.py', lineno: 651, function: report_unihash 0647: method = self.method 0648: if tid in self.extramethod: 0649: method = method + self.extramethod[tid] 0650: *** 0651: data = self.client().report_unihash(taskhash, method, outhash, unihash, extra_data) 0652: new_unihash = data['unihash'] 0653: 0654: if new_unihash != unihash: 0655: hashequiv_logger.debug('Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 139, function: wrapper 0135: pass 0136: 0137: def _get_downcall_wrapper(self, downcall): 0138: def wrapper(*args, **kwargs): *** 0139: return self.loop.run_until_complete(downcall(*args, **kwargs)) 0140: 0141: return wrapper 0142: 0143: def _add_methods(self, *methods): File: '/usr/lib/python3.9/asyncio/base_events.py', lineno: 642, function: run_until_complete 0638: future.remove_done_callback(_run_until_complete_cb) 0639: if not future.done(): 0640: raise RuntimeError('Event loop stopped before Future completed.') 0641: *** 0642: return future.result() 0643: 0644: def stop(self): 0645: """Stop running the event loop. 0646: File: '(...)/poky/lib/hashserv/client.py', lineno: 70, function: report_unihash 0066: m["taskhash"] = taskhash 0067: m["method"] = method 0068: m["outhash"] = outhash 0069: m["unihash"] = unihash *** 0070: return await self.invoke({"report": m}) 0071: 0072: async def report_unihash_equiv(self, taskhash, method, unihash, extra={}): 0073: await self._set_mode(self.MODE_NORMAL) 0074: m = extra.copy() File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 104, function: invoke 0100: async def proc(): 0101: await self.socket.send_message(msg) 0102: return await self.socket.recv_message() 0103: *** 0104: return await self._send_wrapper(proc) 0105: 0106: async def ping(self): 0107: return await self.invoke({"ping": {}}) 0108: File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 82, function: _send_wrapper 0078: async def _send_wrapper(self, proc): 0079: count = 0 0080: while True: 0081: try: *** 0082: await self.connect() 0083: return await proc() 0084: except ( 0085: OSError, 0086: ConnectionError, File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 70, function: connect 0066: await self.socket.send("") 0067: 0068: async def connect(self): 0069: if self.socket is None: *** 0070: self.socket = await self._connect_sock() 0071: await self.setup_connection() 0072: 0073: async def close(self): 0074: if self.socket is not None: File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 55, function: connect_sock 0051: import websockets 0052: 0053: async def connect_sock(): 0054: try: *** 0055: websocket = await websockets.connect(uri, ping_interval=None) 0056: except (OSError, asyncio.exceptions.TimeoutError, websockets.InvalidHandshake, websockets.InvalidURI) as exc: 0057: raise ConnectionError("Could not connect to websocket: %s" % exc) from exc 0058: return WebsocketConnection(websocket, self.timeout) 0059: File: '/usr/local/lib/python3.9/dist-packages/websockets/asyncio/client.py', lineno: 444, function: __await_impl__ 0440: async with asyncio_timeout(self.open_timeout): 0441: for _ in range(MAX_REDIRECTS): 0442: self.connection = await self.create_connection() 0443: try: *** 0444: await self.connection.handshake(*self.handshake_args) 0445: except asyncio.CancelledError: 0446: self.connection.close_transport() 0447: raise 0448: except Exception as exc: File: '/usr/local/lib/python3.9/dist-packages/websockets/asyncio/client.py', lineno: 104, function: handshake 0100: # receiving a response, when the response cannot be parsed, or when the 0101: # response fails the handshake. 0102: 0103: if self.protocol.handshake_exc is not None: *** 0104: raise self.protocol.handshake_exc 0105: 0106: def process_event(self, event: Event) -> None: 0107: """ 0108: Process one incoming event. File: '/usr/local/lib/python3.9/dist-packages/websockets/client.py', lineno: 315, function: parse 0311: 0312: def parse(self) -> Generator[None]: 0313: if self.state is CONNECTING: 0314: try: *** 0315: response = yield from Response.parse( 0316: self.reader.read_line, 0317: self.reader.read_exact, 0318: self.reader.read_to_eof, 0319: ) File: '/usr/local/lib/python3.9/dist-packages/websockets/http11.py', lineno: 238, function: parse 0234: 0235: try: 0236: status_line = yield from parse_line(read_line) 0237: except EOFError as exc: *** 0238: raise EOFError("connection closed while reading HTTP status line") from exc 0239: 0240: try: 0241: protocol, raw_status_code, raw_reason = status_line.split(b" ", 2) 0242: except ValueError: # not enough values to unpack (expected 3, got 1-2) Exception: EOFError: connection closed while reading HTTP status line (Bitbake rev: 5ba7c2f0797a72536a81f57276d4e5c75f23011c) Signed-off-by: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Remove multiprocessingpoolJoshua Watt2025-07-231-16/+0
| | | | | | | | | | This API is no longer used and bitbake has moved beyond Python 2.7.3 as the minimum version, so remove it. (Bitbake rev: 0eb7b5dd512ed8d8b77b5779858b9fbd99edb4a4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-10-04 18:11:36 +0000