summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
Commit message (Collapse)AuthorAgeFilesLines
* The poky repository master branch is no longer being updated.Richard Purdie2025-11-07538-136343/+0
| | | | | | | | | | | | | | | | | | | | | You can either: a) switch to individual clones of bitbake, openembedded-core, meta-yocto and yocto-docs b) use the new bitbake-setup You can find information about either approach in our documentation: https://docs.yoctoproject.org/ Note that "poky" the distro setting is still available in meta-yocto as before and we continue to use and maintain that. Long live Poky! Some further information on the background of this change can be found in: https://lists.openembedded.org/g/openembedded-architecture/message/2179 Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: Rename bb-layers-relative to bb-layers-file-relativeRichard Purdie2025-11-071-7/+7
| | | | | | | | | | | | | | | The difference between bb-layers and bb-layers-relative is unclear as both are relative paths. Rename one to "file-relative" which makes it clear it is relative to the current file, without becomming a long name. https://lists.openembedded.org/g/bitbake-devel/message/18296 Based on a patch from Alexander Kanavin <alex@linutronix.de> but with different naming. (Bitbake rev: dcb17758b99767ab6da4172cf60eabc9269082dd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: replace {THISDIR} token with a keyword: ↵Alexander Kanavin2025-11-061-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | bb-layers-relative {THISDIR} is a special value token that can be used in the list of enabled layers to specify the layer location relative to the confguration file: https://git.openembedded.org/bitbake/commit/?id=b3153be29de8b8570b0c184369bd41f4c646cf92 This replaces the token with an explicit separate keyword for such layers: so that special processing to determine the final value can be avoided, and the feature can be formalized in the json schema: instead of "bb-layers": [ "{THISDIR}/meta-my-project" ] this allows "bb-layers-relative": [ "meta-my-project" Going forward I think we should strive to avoid any further special value tokens. (Bitbake rev: 90da82bd2bfcfd5590c9ae06015737b616074b56) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: capture revisions while checking out layersJohannes Schneider2025-11-061-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | When initializing a build setup from a conf.json that only sets 'rev' to a tag or branch, the actual revision would not be captured or logged. To capture the current layer state after an 'init' or 'update', the checkout_layers function is extended to store the revision the bb.fetch.Fetch pulled, and write that information into a sources-fixed-revisions.json file. This file can then be fed back into bitbake-setup init as: --sources-overrides This new 'sources-fixed-revisions.json' is written during 'update_build' and stored alongside the 'config-upstream.json' in the config dir. And put with the later under version control by calling 'commit_config" after 'update_build'. The use of 'deepcopy' is necessary to not modify the original input data - which python passes around as reference. (Bitbake rev: 95866ff03f78e987ae7e47daad053bc0f353eea4) Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: tests: move duplicate code into 'check_setupdir_files'Johannes Schneider2025-11-061-12/+8
| | | | | | | | | | | | | | | | | | All places that called into 'check_setupdir_files' did the same preparation step to load the upstream-config.json and then pass it into the function. Since the 'setuppath' is already passed into the function, and the name and relative location of the upstream-config.json is fixed, constructing the file path and loading the json could be done in the function. De-duplicate code by loading the json inside the function instead. (Bitbake rev: 16d77c83ae3ce92ddab84d714a93fd3bb7def5e2) Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: support bitbake-setupDavid Reyna2025-11-041-24/+75
| | | | | | | | | | | | | | | | | | | | | | | | This adds support for the new bitbake-setup: * Maintain backwards compatibility for poky-classic releases * Assume bitbake-setup for all releases 'master' and forward * When no ".templateconf" assume Poky and not OE * Generate and use "init-build-env" script instead of 'oe-init-build-env' directly, but "cd" to the clone and not the local installation. * Include "bblayers.conf" because "cookerdata.py" now explicitly required it, and let it be over-ridden by the usual "toaster-bblayers.conf". * Force the value "TOPDIR" in "toaster.conf" to the local project's directory. Normally "cookerdata.py" sets TOPDIR from getcwd(), but somehow in bitbake-setup mode the CWD at this point has become ".../layers/openembedded-core", so "bitbake.conf and all other files cannot be found. This non-obvious problem was hard to discover. Until this is understood, the forced value will make things work. [YOCTO #16012] (Bitbake rev: 22b3fd418efc10da7e4a8c1725de285714d2396b) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: rename 'build' -> 'setup'Alexander Kanavin2025-11-041-30/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | After a terminology review by Antonin we brainstormed something less confusing than 'build' and 'build directory' for the place where bitbake-setup clones layers and creates a *bitbake* build directory in. People are bound to get these two confused and mix them up, and 'setup' is much more distinct and aligns nicely with 'bitbake-setup'. It's also not claimed by anything else in OE/Yocto. So before: top-dir -> build-dir (can be several) -> bitbake build dir, layer dir, config dir Now: top-dir -> setup-dir (can be several) -> bitbake build dir, layer dir, config dir This also updates the respective command line options, I understand it's a breaking change, but as before the tweaks are simple and we need to get the terminology right for the users, and now is the time to do it. (Bitbake rev: eeb81a35bf0304451f7612950d5156ea7ff18bad) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: when raising ParameterError for an invalid proto, ↵Chris Laplante2025-11-031-1/+1
| | | | | | | | | | include the proto in the message (Bitbake rev: 671314811ac1c023efacc6a2f148d8d84ba1adb2) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: commandline: use subsubparser for settings ↵Johannes Schneider2025-10-141-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | {list,set,unset} Previously the sub-command 'settings' would take any number of arguments and then silently do nothing if the number wasn't three. The help text was also not clear about this, marking the positionals separately as optional: usage: bitbake-setup settings [-h] [--global] [--unset UNSET UNSET] [-l] [section] [key] [value] The '--unset SECTION SETTING' also did not integrate too well, as it had its own positional arguments for section+setting. For a bit more consistency and a explorable help, a sub-subparser is added, that provides the commands: bitbake-setup settings list bitbake-setup settings set foo bar baz bitbake-setup settings unset foo bar with a '--global' that is added from a stand-alone parent parser, so that it shows up in all sub-command help texts. The new help text now reads: usage: bitbake-setup settings [-h] [--global] {list,set,unset} ... and the respective sub commands: usage: bitbake-setup settings list [-h] [--global] usage: bitbake-setup settings set [-h] [--global] <section> <setting> <value> usage: bitbake-setup settings unset [-h] [--global] <section> <setting> (Bitbake rev: 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c) Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: further rework the settings handlingAlexander Kanavin2025-10-141-4/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After some further feedback, additional changes are made: 1. 'setting' command is renamed to 'settings' to better reflect that it is an interface to various ways of managing settings. 2. This command now has a -l/--list option to list all settings with their values (same as 'git config -l'). 3. A new level of settings (built-in defaults) is added, and used as a last resort after command line options, top dir settings file and global settings file. 4. This means bitbake-setup does not have to write and use a global settings file, and it no longer does so when initializing a build, avoiding default 'pollution' of ~/.config/bitbake-setup/ which can be problematic or unwelcome. A global settings file is still created if a setting is explicitly requested to be placed into it. 5. 'install-global-settins' is removed as the use case for it (tweak default settings before using them to initialize a build) can be achieved by setting the settings individually. 5. Similarly, a top dir settings file is no longer created by default and only appears if a setting needs to be written into it. 6. Default dl-dir is again created inside a top directory and not in ~/.cache/ to make default builds fully contained in the top directory (which was also asked about). (Bitbake rev: 664f8ec48d42d2ddc5f234c4f7d590fa597f489a) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: rework the settings handlingAlexander Kanavin2025-10-141-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is the outcome of various discussions, suggestions and pull requests on github. What has specifically changed? 1. The sources for the settings are no longer separated, but are stacked and given priorities, from highest to lowest: a. '--setting section key value' on the command line b. a settings file in the top directory c. a global settings file in ~/.config/bitbake-setup/ (or in a file pointed to by --global-settings) Any setting can be in any of these three locations (other than top dir name and prefix which do not make sense in the settings file in the top directory). 2. A global settings file must contain all of the needed settings, while a settings file in the top directory can be empty (and this is how they are written out if they do not exist). Specifically, both dl-dir and registry settings have been relocated to the global file, and dl-dir defaults to ~/.cache/bitbake-setup/downloads, rather than somewhere in top dir. 3. The file name for both global and top dir settings is now 'settings.conf'. 4. --top-dir-prefix and --top-dir-name options have been removed and superseded by a generic, universal --setting option. 5. 'install-settings' command has been removed, as it is no longer does anything useful, and is superseded by the 'setting' command (see below). 'install-global-settings' has been retained, to be able to have a set of global defaults that can be changed without initializing a build. 6. 'change-setting', 'change-global-setting' and 'install-settings' have all been replaced by a single 'setting' command that mimics 'git config' in its parameters: a. Changing a setting: bitbake-setup setting [--global] default dl-dir /path/to/downloads b. Removing a setting: bitbake-setup setting [--global] --unset default dl-dir (Bitbake rev: 713e7f213c6d4a620be9ce34d5f4396af48e1d69) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: tests: add environment-passthroughJohannes Schneider2025-10-141-0/+23
| | | | | | | | | | Add a test configuration to cover the 'bb-env-passthrough-additions' conf.json key, and add it to the test routine. (Bitbake rev: 24f12b68692f9ebb5d3813bc3b1771e43298f640) Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: tests: add a template-only test configurationJohannes Schneider2025-10-141-8/+27
| | | | | | | | | | | | | bitbake-setup conf.json can be purely template driven, to setup older yocto LTS (e.g. scarthgap) based layer-collections - where fragment support was not yet present in oe-core+bitbake. Add a configuration for that scenario and add it to the test routine. (Bitbake rev: 23e121befa0779fbb1f342984b583c04ccc637a0) Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: allow using {THISDIR}/my-layerYoann Congal2025-10-141-2/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This implement the ability to use "{THISDIR}/my-layer" in the "bb-layers" list. "{THISDIR}" is remplaced by the directory containing the configuration file. In small projects, we try to keep the setup a simple as possible: a single git repo containing both the build confguration (e.g. a bitbake-setup configuration file) and the meta layer with project recipes/machine/distro. This change allows this kind of setup: ├── meta-my-project/ # the project layer └── my-project.conf.json # the bb-setup configuration file by writing, in my-project.conf.json: "bitbake-setup": { "configurations": [{ "bb-layers": [ "{THISDIR}/meta-my-project" Note: in this case meta-my-project is not present as a "source", so, not handled by bb-setup update/status. It is expected of the user to handle this on their own (is our case, a simple git workflow). (Bitbake rev: b3153be29de8b8570b0c184369bd41f4c646cf92) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/tests/setup.py: unset BBPATH to ensure isolation from the ↵Alexander Kanavin2025-10-141-0/+5
| | | | | | | | | | | | existing bitbake environment bitbake-setup deduces top directory from BBPATH, which, if set, interferes with the tests' own setup. (Bitbake rev: e974d42eb5229f755e14b46a00ad06b23b53e143) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/tests/setup.py: define test parameters in a single dictionaryAlexander Kanavin2025-10-141-3/+7
| | | | | | | | | This makes maintaining and extending them easier. (Bitbake rev: 16dc8e3dad7dde7e7651cce13549e61574cafba1) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add support for specifying branches in repo checkoutsAlexander Kanavin2025-10-141-5/+6
| | | | | | | | | | | | | | | | | | | | Previously bitbake-setup was checking out 'detached commits' using fetcher's nobranch feature, as that is the only option when only a revision is in the config. Branches are optional, but beneficial, as - checkout directory will be on a branch, making it easier for users to understand where they are if they need to make changes (also bitbake will print branch information instead of saying 'HEAD:sha'). - supply chain security! Enforcing a branch means any specified revision has to be on it, and no one can sneak in (accidentally or deliberately!) some dangling commit, or something from their private branch in the same repo. (Bitbake rev: 45ed9b9faebdaa8cb7cc8dd2a6d51ec8eea06e73) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: adjust the error message about missing websockets module to ↵Alexander Kanavin2025-10-141-1/+4
| | | | | | | | | mention bitbake-setup (Bitbake rev: 2d15e6fc93f6a73c9c84e399613f931f1c783922) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add 'install-buildtools' commandAlexander Kanavin2025-10-141-0/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This basically calls install-buildtools from oe-core/poky, but it ensures via command line parameters that the installation location is stable and the downloads are preserved for reproducibility: $ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf ====== Buildtools archive is downloaded into /home/alex/bitbake-builds/yocto-master-testing/buildtools-downloads/20250319141333 and its content installed into /home/alex/bitbake-builds/yocto-master-testing/buildtools ... (output from install-buildtools script) ====== It also detects when buildtools are already installed, and will direct users what to do: ====== alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup install-buildtools Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf Buildtools are already installed in /home/alex/bitbake-builds/yocto-master-testing/buildtools. If you wish to use them, you need to source the the environment setup script e.g. $ . /home/alex/bitbake-builds/yocto-master-testing/buildtools/environment-setup-x86_64-pokysdk-linux You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation ====== This commits includes fixes by Gyorgy Sarvari <skandigraun@gmail.com> https://github.com/kanavin/bitbake/pull/2 (Bitbake rev: 3fe3096847046110c72b23fce37fb4a459b1d748) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-setup: add tests to bitbake-selftestAlexander Kanavin2025-10-141-0/+266
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Run like this: alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok ---------------------------------------------------------------------- Ran 1 test in 9.223s OK The test does a basic run-through of init, then status/update on an unchanged configuration, then status/update on a configuration changed via new commits to the test layer, then status/update on configuration changed via the top level json config file. Note that nothing whatsoever is fetched from the network; the test relies entirely on synthetic data contained inside itself, including minimal stubs for oe-setup-build and bitbake-config-build. This data is used to create temporary git repositories then clone them via local filesystem URIs. Later on this can be supplemented by an oe-selftest that tests bitbake-setup against real config files in the official configuration repository and real layers, templates and fragments. (Bitbake rev: e3aa3eb46bd3196fa5415fa36e3737636fd6a1c0) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty, uihelper: Remove running_pids and lastpidsPeter Kjellerstedt2025-10-132-7/+1
| | | | | | | | | | | | | * lastpids has not been used for almost 10 years. * There is no longer any need to use running_pids to keep track of the order the pids in running_tasks were added as dicts are guaranteed to remember the insertion order since Python 3.7. (Bitbake rev: 562b5bf0be2883144391e7030a9dce6a233802ac) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast: Warn on multiple builtin config fragments for the same variableYoann Congal2025-10-131-0/+21
| | | | | | | | | | | | | | | | Having multiple builtin config fragments for the same variable (eg OE_FRAGMENTS = "... machine/A ... machine/B") is not supported. Warn the user to make them fix this but continue with the normal variable evaluation : the last affectation "wins". Added warning looks like: WARNING: Multiple builtin fragments are enabled for machine via variable OE_FRAGMENTS: machine/qemux86-64 machine/test machine/qemux86-64. This likely points to a mis-configuration in the metadata, as only one of them should be set. The build will use the last value. (Bitbake rev: 1c12aa23f6678dc289fc0e0d8b4dad311bd39c35) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast: Fix fragment behaviour with overridesRichard Purdie2025-10-091-1/+2
| | | | | | | | | | | | | | | | | | | | | Imagine a machine fragment machine/A and a configuration which sets: MACHINE = "B" MACHINE:forcevariable = "C" As I understand it, the fragment behaviour was intended to replace the MACHINE = "B", so the override would still be active. The current code replaces all variable overrides. parsing=True, switches to the other behaviour, which I believe was the design intent and the behaviour users would expect. This is useful to allow test configurations to override a MACHINE setting without change the fragments which would complicate the test code. (Bitbake rev: f65bc6aaf4c11bc7e566c895209c093627a3015b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Add tests for include, require and include_allPeter Kjellerstedt2025-09-241-0/+60
| | | | | | | | | | Simple tests for various ways to include and require configuration files. (Bitbake rev: bdcb67824172ed5cd76fa0274dc3d27aeb52e77c) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Remove unnecessary calls to file.flush()Peter Kjellerstedt2025-09-241-8/+0
| | | | | | | | | | | There is no reason to call file.flush() as the last call within a file context manager since ending the context will close the file and thus flush it. (Bitbake rev: 90d5ce926d434d60d6df28b7d77b3cbc9b62ce14) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Use with statement to avoid ResourceWarningPeter Kjellerstedt2025-09-241-51/+44
| | | | | | | (Bitbake rev: 46a5e31713fc526a1807b1617fba8a01c1564641) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update tests after bitbake tag removalMathieu Dubois-Briand2025-09-231-12/+12
| | | | | | | | | | Tags for bitbake 2.8.6 and 2.8.7 have been removed from the git: use some other ones. (Bitbake rev: dcf12947954f3184d99fbf9813b3f3f667dacc5f) Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: verify if local clone contains tagGyorgy Sarvari2025-08-192-3/+51
| | | | | | | | | | | | | | | | | | | | In case a recipe specifies a git SRC_URI along with revision and tag, but only the revision is present in the local clone without the tag (because it was tagged after it was cloned), then unpacking fails with the following error: ... rev-list -n 1 1.0 failed with exit code 128, output:\nfatal: ambiguous argument \'1.0\': unknown revision or path not in the working tree This happens because the during the download step only the revision's presence is verified to decide if the repository needs to be updated. To avoid this, check also if the tag is present in the local repository, when the "tag" tag is specified. (Bitbake rev: 546b347b4d3d82c01ecc99f45296f66e44638adc) Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: print() was eliminated from all loops for better performance.Benjamin Szőke2025-08-191-7/+7
| | | | | | | | | | | Refactoring prints, print() functions were eliminated from all loops and it uses "\n".join(...) in a single print() call for better performance. (Bitbake rev: c32c3d9b83818661e12f3e437563ab4e1fa05e15) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Use 40 Hz refresh rate (FPS) for footer update.Benjamin Szőke2025-08-191-4/+20
| | | | | | | | | | Refresh footer in 40 Hz to avoid heavy print() flooding but keep it fluent for human eyes. (Bitbake rev: c36efdf642d858c6997819744d00a3c1965c6417) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Use a StringIO buffer for update footer.Benjamin Szőke2025-08-191-10/+26
| | | | | | | | | | | Optimize printing in footer update with use a StringIO buffer and it prints content to terminal in a single call in every cycle. (Bitbake rev: 32ba622d78f20b231f30f848379b4bbc3d7414da) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: progressbar: Add self._fd_console to use for self._handle_resize()Benjamin Szőke2025-08-191-2/+9
| | | | | | | | | | Introduce self._fd_console as a dedicated attribute of self._handle_resize(). (Bitbake rev: f8c76eb89d52b1c28407f0b52dfe4318faa47cd2) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update version to 2.15.2Richard Purdie2025-08-121-1/+1
| | | | | | (Bitbake rev: 7ad404fd1aa0a48978c0351c5f52f17b8992c8b8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: use enum to represent runQueue state to improve readabilityChris Laplante2025-08-121-44/+46
| | | | | | | (Bitbake rev: e61265570f5415d647666891d806836437330219) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/runqueue: remove unused 'd' arg from check_for_interruptsChris Laplante2025-08-122-17/+17
| | | | | | | (Bitbake rev: e0c5e94cc14dd6b1cbde491073c1f9407c97e428) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: remove some redundant control flowChris Laplante2025-08-121-4/+1
| | | | | | | | | | | Prior to commit aa84a900e ("cooker: Ensure delays are accurately transfered to the idle loops from runqueue"), this was necessary. But now retval is returned directly. (Bitbake rev: c2eb4336fe10b1bf8bbc6291c32ca362840f39e1) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add bb.utils.explode_deps as a filter functionJoshua Watt2025-08-121-0/+2
| | | | | | | | | Marks bb.utils.explode_deps as callable from filter functions (Bitbake rev: bb07003641e76de994482f7835a432f20297af96) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add setVarFilter function to implement variabl filteringRichard Purdie2025-08-121-0/+19
| | | | | | | | | | | | | | | | Adds a new setVarFilter() API to the data store allowing filters to be applied to variables. Note that filters are applied to the non-override part of the variable name so a filter set against RDEPENDS would apply against RDEPENDS:${PN} and friends. The filter function is applied before returning the final variable value. (Bitbake rev: a9471c10d1de039474ddb4738abd286b928d82f4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add filter supportJoshua Watt2025-08-122-0/+230
| | | | | | | | | | | | | | Add the python API for applying filters to a string and being able to register functions as filters. Filter functions are pure functions where an input is translated into an output and there are no external data accesses. This means translations can be cached as they won't change. (Bitbake rev: 7d25d7511ca14213eea78ee739d260295cfa4045) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ignore KeyErrors during shutdownRichard Purdie2025-08-041-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When parsing failures occur, trap KeyError to avoid these kinds of tracebacks (from a parsing error in meta-ti). bb.parse.ParseError: ParseError at /srv/pokybuild/yocto-worker/check-layer-nightly/build/meta-ti/meta-ti-bsp/recipes-kernel/linux/linux-ti-mainline_git.bb:6: Could not inherit file classes/${KERNEL_BASE_CLASS}.bbclass During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 1639, in updateCache if not self.parser.parse_next(): File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2314, in parse_next self.shutdown(clean=False, eventmsg=str(exc)) File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2209, in shutdown read_results() File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2195, in read_results self.result_queue.get(timeout=0.25) File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get return _ForkingPickler.loads(res) File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cache.py", line 338, in __setstate__ setattr(self, key, self._restore(state[key], pid)) File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cache.py", line 318, in _restore ret[dep] = map[mapnum] KeyError: 156 (Bitbake rev: 750c68ee7ee3f3d4518348e4c948243504880770) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: ignore comments in .gitattributes when detecting LFSGyorgy Sarvari2025-07-311-1/+1
| | | | | | | | | | | | | | | | | | | | Fixes [YOCTO 15917] When fetching a git repository the .gitattributes file is scanned, checking if LFS support is required for the repository. This scan checks if the word "lfs" is present in the file, however the used regex doesn't account for comments, which makes some repositories[1] be to misidentified as requiring LFS support (which fails fetching, in case lfs support isn't installed on the build host). To avoid it, change the used regex to ignore lines starting with "#". [1]: e.g. https://github.com/MicrochipTech/cryptoauthlib (Bitbake rev: 7917a758fc328747116c7899e689171bd0efc883) Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Keep query parameters in URL during checkstatusPhilippe-Alexandre Mathieu2025-07-281-1/+4
| | | | | | | | | | | | | | | | | | | | When recreating the uri in wget's checkstatus method, we only use the scheme, netloc and path. This completely strips the query parameters from the final URI and potentially breaks the checking functionality from certain fetchers. This is the case for the Azure storage fetcher, as it requires a SAS token that is formatted as a series of query parameters. The error manifests itself when using a private storage account as a PREMIRROR or SSTATE_MIRROR (since regular SRC_URI won't run the checkstatus). This problem is present in scarthgap, but wasn't in kirkstone. CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> (Bitbake rev: 096301250455e2a83bdd818a56317c62436c9981) Signed-off-by: Philippe-Alexandre Mathieu <pamathieu@poum.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: sigen: Avoid bitbake abort with EOFerror from an incomplete ↵Pedro Ferreira2025-07-231-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | hashserv communication The issue itself is sporadic but aborts the build with an uncaught exception. The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_report_unihash(d) 0003: File: '(...)/poky/meta/classes/sstate.bbclass', lineno: 882, function: sstate_report_unihash 0878: report_unihash = getattr(bb.parse.siggen, 'report_unihash', None) 0879: 0880: if report_unihash: 0881: ss = sstate_state_fromvars(d) *** 0882: report_unihash(os.getcwd(), ss['task'], d) 0883:} 0884: 0885:# 0886:# Shell function to decompress and prepare a package for installation File: '(...)/poky/lib/bb/siggen.py', lineno: 651, function: report_unihash 0647: method = self.method 0648: if tid in self.extramethod: 0649: method = method + self.extramethod[tid] 0650: *** 0651: data = self.client().report_unihash(taskhash, method, outhash, unihash, extra_data) 0652: new_unihash = data['unihash'] 0653: 0654: if new_unihash != unihash: 0655: hashequiv_logger.debug('Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 139, function: wrapper 0135: pass 0136: 0137: def _get_downcall_wrapper(self, downcall): 0138: def wrapper(*args, **kwargs): *** 0139: return self.loop.run_until_complete(downcall(*args, **kwargs)) 0140: 0141: return wrapper 0142: 0143: def _add_methods(self, *methods): File: '/usr/lib/python3.9/asyncio/base_events.py', lineno: 642, function: run_until_complete 0638: future.remove_done_callback(_run_until_complete_cb) 0639: if not future.done(): 0640: raise RuntimeError('Event loop stopped before Future completed.') 0641: *** 0642: return future.result() 0643: 0644: def stop(self): 0645: """Stop running the event loop. 0646: File: '(...)/poky/lib/hashserv/client.py', lineno: 70, function: report_unihash 0066: m["taskhash"] = taskhash 0067: m["method"] = method 0068: m["outhash"] = outhash 0069: m["unihash"] = unihash *** 0070: return await self.invoke({"report": m}) 0071: 0072: async def report_unihash_equiv(self, taskhash, method, unihash, extra={}): 0073: await self._set_mode(self.MODE_NORMAL) 0074: m = extra.copy() File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 104, function: invoke 0100: async def proc(): 0101: await self.socket.send_message(msg) 0102: return await self.socket.recv_message() 0103: *** 0104: return await self._send_wrapper(proc) 0105: 0106: async def ping(self): 0107: return await self.invoke({"ping": {}}) 0108: File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 82, function: _send_wrapper 0078: async def _send_wrapper(self, proc): 0079: count = 0 0080: while True: 0081: try: *** 0082: await self.connect() 0083: return await proc() 0084: except ( 0085: OSError, 0086: ConnectionError, File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 70, function: connect 0066: await self.socket.send("") 0067: 0068: async def connect(self): 0069: if self.socket is None: *** 0070: self.socket = await self._connect_sock() 0071: await self.setup_connection() 0072: 0073: async def close(self): 0074: if self.socket is not None: File: '(...)/poky/lib/bb/asyncrpc/client.py', lineno: 55, function: connect_sock 0051: import websockets 0052: 0053: async def connect_sock(): 0054: try: *** 0055: websocket = await websockets.connect(uri, ping_interval=None) 0056: except (OSError, asyncio.exceptions.TimeoutError, websockets.InvalidHandshake, websockets.InvalidURI) as exc: 0057: raise ConnectionError("Could not connect to websocket: %s" % exc) from exc 0058: return WebsocketConnection(websocket, self.timeout) 0059: File: '/usr/local/lib/python3.9/dist-packages/websockets/asyncio/client.py', lineno: 444, function: __await_impl__ 0440: async with asyncio_timeout(self.open_timeout): 0441: for _ in range(MAX_REDIRECTS): 0442: self.connection = await self.create_connection() 0443: try: *** 0444: await self.connection.handshake(*self.handshake_args) 0445: except asyncio.CancelledError: 0446: self.connection.close_transport() 0447: raise 0448: except Exception as exc: File: '/usr/local/lib/python3.9/dist-packages/websockets/asyncio/client.py', lineno: 104, function: handshake 0100: # receiving a response, when the response cannot be parsed, or when the 0101: # response fails the handshake. 0102: 0103: if self.protocol.handshake_exc is not None: *** 0104: raise self.protocol.handshake_exc 0105: 0106: def process_event(self, event: Event) -> None: 0107: """ 0108: Process one incoming event. File: '/usr/local/lib/python3.9/dist-packages/websockets/client.py', lineno: 315, function: parse 0311: 0312: def parse(self) -> Generator[None]: 0313: if self.state is CONNECTING: 0314: try: *** 0315: response = yield from Response.parse( 0316: self.reader.read_line, 0317: self.reader.read_exact, 0318: self.reader.read_to_eof, 0319: ) File: '/usr/local/lib/python3.9/dist-packages/websockets/http11.py', lineno: 238, function: parse 0234: 0235: try: 0236: status_line = yield from parse_line(read_line) 0237: except EOFError as exc: *** 0238: raise EOFError("connection closed while reading HTTP status line") from exc 0239: 0240: try: 0241: protocol, raw_status_code, raw_reason = status_line.split(b" ", 2) 0242: except ValueError: # not enough values to unpack (expected 3, got 1-2) Exception: EOFError: connection closed while reading HTTP status line (Bitbake rev: 5ba7c2f0797a72536a81f57276d4e5c75f23011c) Signed-off-by: Pedro Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Remove multiprocessingpoolJoshua Watt2025-07-231-16/+0
| | | | | | | | | | This API is no longer used and bitbake has moved beyond Python 2.7.3 as the minimum version, so remove it. (Bitbake rev: 0eb7b5dd512ed8d8b77b5779858b9fbd99edb4a4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Use a "fork" multiprocessing contextJoshua Watt2025-07-237-9/+35
| | | | | | | | | | | | | | | | | Python 3.14 changes the default multiprocessing context from "fork" to "forkserver"; however bitbake heavily relies on "fork" to efficiently pass data to the child processes. As such, make "fork" context in the bb namespace and use it in place of the normal multiprocessing module. Note that multiprocessing contexts were added in Python 3.4, so this should be safe to use even before Python 3.14 [YOCTO #15858] (Bitbake rev: 62be9113d98fccb347c6aa0a10d5c4ee2857f8b6) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: test/fetch: Switch u-boot based test to use our own mirrorRichard Purdie2025-07-221-1/+1
| | | | | | | | | The upstream servers are having issues so switch to our own shadow copy of the repo. (Bitbake rev: e910c7cd24fd366d6756641cd599c4efeb492e2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Split profile reports into separate filesRichard Purdie2025-07-212-13/+27
| | | | | | | | | Use a more logical name for the profile reports and put each report into a separate file since people struggle to discover them currently. (Bitbake rev: a8145c84e0899285a5e6a809f1515118b002b106) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Optimise signal/sigmask performanceRichard Purdie2025-07-211-4/+13
| | | | | | | | | | | | | Running "time bitbake -pP idle" with a valid cache shows around 800,000 calls to enum creation from python's signal.py. We don't care about this overhead and it adversely affects cache load time quite badly. Try and use _signal directly, falling back to signal, which avoids this overhead we don't need and makes cache loading much faster. (Bitbake rev: ee5fce67ce35b025c68aa61e2e758903269ee346) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Add an option to specify what to profileRichard Purdie2025-07-173-6/+8
| | | | | | | | | | | | | | | | | | Starting with python 3.12, profiling now stays enabled over threads yet you can't extract the profile data in the threads themselves, which makes it difficult to use for our use case. Our main loop starts the idle loop which starts the parsing threads and this means we can't profile in the main loop and the parsing threads or the idle loop at the same time due to this. Add options to the commandline so you can specify which piece of bitbake you want to enable profiling for. This allows some profiling with python 3.12 onwards rather than crashing. (Bitbake rev: 09f29a4968841ee5070f70277ba8c253bb14f017) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Fix an event duplication raceRichard Purdie2025-07-171-2/+5
| | | | | | | | | | | | It is possible for multple bitbake threads to empty ui_queue in parallel leading to duplicate console messages and much confusion when debuging. Use the lock to extract the queue data which means only one thread will processing, removing the duplicate out of order messages. (Bitbake rev: 945095602e40d54efb8de494218f4a2b25c9969f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-01-18 17:46:07 +0000