summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Fix deferred task/multiconfig race issueRichard Purdie2023-06-281-4/+12
| | | | | | | | | | | | | | | If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests: Use assertLogs to test logging outputPeter Hoyes2023-06-272-10/+12
| | | | | | | | | | | | | By default, pytest captures all stdout and exposes it using its built-in fixtures (capsys, caplog etc), so stdout does not support getvalue(). To support running tests using both unittest and pytest, use assertLogs to capture logging and assert on the log output instead. (Bitbake rev: 2d28caa01bab9540d2bbaf713ae3e5c563d003f5) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: Set git config if not already setPeter Hoyes2023-06-271-3/+9
| | | | | | | | | | | | | | | | | | | git config returns an error code if user.email or user.name are not set, so bb.process.run raises an ExecutionError, failing the test. Improve the logic by catching the ExectionError and using this to set a default value. Change a direct invocation of 'git init' to use self.git_init so that it follows the code path above. Remove the related comment from the README now that git sets up the user details automatically. (Bitbake rev: 0c7079d0609a13713b890a9675abd9fba032f199) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: Rename assertRaisesRegexp to assertRaisesRegexPeter Hoyes2023-06-271-1/+1
| | | | | | | | | | TestCase.assertRaisesRegexp was renamed to assertRaisesRegex in Python 3.2, so rename to fix a warning during test execution. (Bitbake rev: 9bc2ca3197bff7a5ef6e0ece6bfef0cda68b808e) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: Mark TestTimeout as not a test suitePeter Hoyes2023-06-271-1/+2
| | | | | | | | | | | pytest test collection attempts to collect all classes containing "Test". TestTimeout is not a test class so add the __test__ = False attribute to indicate this. (Bitbake rev: b5455b98093a3eaf122ebe3d29c3bef949fd2440) Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix handling of virtual files in layername calculationRichard Purdie2023-06-201-2/+4
| | | | | | | | | | I'd tested target configurations but in real world use, it became clear the layername functionality in worker context was failing for virtual class extensions. Fix this. (Bitbake rev: 888778ebfa85677ff36065dfcd0f8a7684edfc80) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Pass hashfn in taskdep dataJoshua Watt2023-06-011-2/+4
| | | | | | | | | | | Include the hashfn (the value of BB_HASHFILENAME) in the task dependency data. This allows tasks to get a specific unique hash for dependent tasks when one is available. (Bitbake rev: 4dbecf6059e495246267b09d0f43086d51e6df2c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server: Fix crash when checking lock fileJoshua Watt2023-06-011-1/+1
| | | | | | | | | | | Fixes a crash when the server process attempts to check the PID of the lock file that resulted because an integer (os.getpid()) was attempting to be concatenated to a string (Bitbake rev: 5d499682a0a739b5269247a8f6dbb874e3eec456) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: evaluate PATH before patching HOMEArno Baumfalk2023-05-251-0/+1
| | | | | | | | | | | | | When the user configures the TMPDIR variable in a way that it contains $HOME (in local.conf), the DataSmart (d) object's PATH variable will also contain $HOME. As the NpmEnvironment patches HOME with a temporary directory, further evaluation of PATH will become invalid an the build breaks with an error (npm not found). As countermeasure PATH is evaluated *before* patching HOME. (Bitbake rev: 893f3b116b628457fe0bfaf4b8394b1cd05a90a5) Signed-off-by: Arno Baumfalk <a.baumfalk@astro-kom.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a ↵Richard Purdie2023-05-255-26/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | recipe There are times when it would be useful for code to know which layer (or collection in old bitbake terms) it is contained within. Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe so that it is possible to determine this. To do it, we need to pass data from the cooker into the recipe endpoints, since only the top level cooker information knows about the layer structure which makes the patch a bit painful. The idea is that this would make layer overrides possible: OVERRIDES .= ":layer-${FILE_LAYERNAME}" which then opens possibilities like: WARN_QA:append:layer-core = " patch-fuzz" as an example where OE-Core could enable specific QA tests only for that specific layer. (Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/tests/fetch.py: set initial branchBergin, Peter2023-05-251-0/+4
| | | | | | | | | | | | | | If you have a host where the git config for initial branch name is something else than 'master' the unittest will fail as they assume the default branch name is 'master'. Fix this by explicitly set the intial branch name at 'git init'. (Bitbake rev: 5bcd213c23da30a84baba88b775b1740e6bb77d0) Signed-off-by: Peter Bergin <peter.bergin@windriver.com> Signed-off-by: Peter Bergin <peter@berginkonsult.se> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/local: Add debug to include chosen pathRichard Purdie2023-05-251-0/+2
| | | | | | | | | | | The current debug lists all the potential search paths but not the end result which isn't helpful for debugging. Tweak to include the end result. (Bitbake rev: a98b8dbb5fbd937a9530793485cf7c68fdd75cc2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Clarify different localpaths functionsRichard Purdie2023-05-252-3/+3
| | | | | | | | | | | | To avoid confusion, rename the local fetcher localpaths function to something more specific so it doesn't appear similar to the localpaths top level function. (Bitbake rev: ada5ea712cc42fff8c55bd261ee6d7579efd4a68) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix/improve collections handlingRichard Purdie2023-05-251-5/+5
| | | | | | | | | | | | | | | | Code changes for FILE_LAYERNAME exposed lifecycle issues around the collections object in Cooker which only appeared in devtool usage in eSDK. Move the collections setup to an earlier stage after parsing completes to avoid any kind of race around it. Also stop overwriting the code variable in MatchFiles. Ultimately we need to combine these codepaths but that is for another patch. (Bitbake rev: 27b872ed4fbe73b3b61e14cb885bb7c16c039cdb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Correct unpack for a crate that matches the recipe namePeter Kjellerstedt2023-05-252-2/+27
| | | | | | | | | | | | | | | | | | | The crate fetcher handles a crate with a name that matches the recipe's name specially by placing the unpacked code in the current directory (which typically is ${S}) rather than together with the sources for the other crates. This broke when the URI names for all crates were changed recently to include the version in the name. Correct the crate fetcher to test against ${BP} instead of ${BPN}. Also add a test case to the selftests to avoid this breaking again. [Yocto #15012] (Bitbake rev: 9af52b748357d0c843ce2507ce4d119fd9c37008) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: Simplify extraction of crate names and versions from URIsPeter Kjellerstedt2023-05-251-3/+3
| | | | | | | | (Bitbake rev: 4f2d8bcbaea33425f6de2c3c88a0beabec8a59f9) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Default to https git protocol where possibleRichard Purdie2023-05-051-29/+29
| | | | | | | | | | The recommendation from server maintainers is that the https protocol is both faster and more reliable than the dedicated git protocol at this point. Switch to it where possible. (Bitbake rev: 879b830fcc60a022682ef22f58c04e7e07d03c63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Log config and parse cache status changesRichard Purdie2023-05-021-12/+27
| | | | | | | | | | It can be hard to tell why bitbake does some things slowly. Log the changes to the base configuration and parse cache status so that it becomes clear from the logs when the cache invalidation causes a slowdown. (Bitbake rev: 6e99d89f3c00a5f53c24d687eaef24f52fe0ef99) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: do not abort on single ctrl-cEnrico Scholz2023-04-201-1/+1
| | | | | | | | | | | | | | | | After b7ed7e9a815c4e10447fd499508be3dbb47f06e8 bitbake aborts immediately when a single ctrl-c is pressed. Patch restores the previous behavior where a single ctrl-c waits for active tasks to finish. https://bugzilla.yoctoproject.org/show_bug.cgi?id=15094 (Bitbake rev: 66131fa6a3e12c28710d09e1dbf3c03f2981280d) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npmsw fetcher: Avoid instantiating Fetch class if url list is emptySvend Meyland Nicolaisen2023-04-201-1/+3
| | | | | | | | | | | | | | | | Recipes containing both git and npmsw sources in the SRC_URI fail during fetch from the shrinkwrap. It seems that when the fetcher is fetching from the shrinkwrap, the SRCREV variable has been deleted but it till ends up fetching from the git source resulting in an error because SRCREV is undefined. The root cause of this is that the Fetch class defaults to urls from the SRC_URI when the urls parameter contains an empty list. This patch will ensure that Fetch is not instantiated if the urls list is empty. (Bitbake rev: e602963dfd505eef08702366383358d29ee20c4d) Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix PSI check calculationChen Qi2023-04-131-4/+9
| | | | | | | | | | | | | | | The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. (Bitbake rev: b4763c2c93e7494e0a27f5970c19c1aac66c228b) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: ConfHandler: Allow variable flag name with a single characterKai Kang2023-04-132-2/+4
| | | | | | | | | | | | | | Update regex pattern to allow variable flag name with a single character. Regression tests have also been updated in `bb.parse` and `bin/bitbake-selftest -k ParseTest` has been successfully executed. Eliminate a trailing space as well. (Bitbake rev: bb9e523291a3cad6e1596ee6a1e715b5e5feba8f) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: add bb.event.ParseErrorMingli Yu2023-04-132-2/+11
| | | | | | | | | Add bb.event.ParseError to let error-report can catch this kind of error. (Bitbake rev: 316524ab59a5e738c25e062923ee5717d88ae5c7) Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: create versioned 'name' entriesEnrico Scholz2023-04-062-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | It is common for rust packages to depend on different versions of the same crate. E.g. | crate://crates.io/windows_x86_64_msvc/0.42.2 \ | crate://crates.io/windows_x86_64_msvc/0.48.0 \ Identification only by the plain crate name makes the sha256sum ambiguous | SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" | SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a" and requires lot of manual work to fix the SRC_URI list. Making the 'name' property unique by appending the version allows direct copy & paste of reported sha256sum errors to complete the crates list. (Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.4.0Richard Purdie2023-04-051-1/+1
| | | | | | (Bitbake rev: 46e1ea2e9a203992bb4de48ea21a8e736419ada2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Display all missing checksum at onceFrederic Martinsons2023-04-041-8/+10
| | | | | | | | | | | | | | | | | | | | | | | | Instead of only display the first and stop. For recipe (crago based) that can contains several artifacts to fetch with their checksum, it will particularly handy to display all of missing one. An example of error message would be NOTE: Executing Tasks ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe: SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4" SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b" SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9" SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b" ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe') ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025 ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1' (Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Make EXCLUDE_FROM_WORLD booleanRobert Yang2023-03-301-1/+1
| | | | | | | | | | | | | | | Fixed: Set EXCLUDE_FROM_WORLD = "0" in recipe_A: $ bitbake world -g Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem: EXCLUDE_FROM_WORLD = "1": Not build in world EXCLUDE_FROM_WORLD = "0": Add back to world build (Bitbake rev: 3f4ede2d67a2d75d3faf8887f90371bd7554d492) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/local: Mention the value of localpath in failure messageRichard Purdie2023-03-301-1/+1
| | | | | | | | | | We have an obtuse rare and intermittent bitbake fetch failure. Understanding the value of ud.localpath at the time of failure would be helpful to debug it further. Tweak the debugging to provide this in the output. (Bitbake rev: 935ab36f64d25c09f83209cd369714c2bc89aa7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-03-301-2/+6
| | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: crate.py: make checksum verification mandatoryFrederic Martinsons2023-03-301-1/+1
| | | | | | | | (Bitbake rev: 9c07e377a21fe1d9660d084da93873acd289b581) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add checksum capability for crate fetcherFrederic Martinsons2023-03-302-6/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This change brings checksum verification of each crate in a recipe, e.g | SRC_URI += " \ | crate://crates.io/aho-corasick/0.7.20 \ | crate://crates.io/atomic-waker/1.1.0 \ | crate://crates.io/cc/1.0.79 \ | " | | SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac" | SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599" | SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f" That will require to move the checksum initialization after the possible call to urldata_init method in order for the crate fetcher to parse the url. Another way of doing could have been implementing a decodeurl method that would have been specific for each fetcher class. (Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/sftp: Fix fetching URIs with spacesDaniel Ammann2023-03-231-1/+1
| | | | | | | | | | Spaces can be used in URIs with %20, but the URI needs to be quoted. (Bitbake rev: 0f887589c279ce07692b4d8e118954a9d214ca45) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler: Allow the '@' character in variable flag namesJames R T2023-03-232-2/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch enables the usage of the '@' character in variable flag names. One use case of variable flags is to assign the network namespaces of some systemd services/targets to configure other parts of the build process of some system. The filenames of systemd services/targets might contain the '@' character if they are template unit files that can take in a single parameter/argument and be instanced multiple times, as indicated by systemd's official manual page. The '@' character is disallowed as the first character in a variable flag name. Imposing more restrictions on the first character is a compromise to make parsing easier and to allow for more options in the future to extend the syntax. This patch is successfully verified by creating a custom BitBake recipe that sets and unsets the value of a variable flag with the '@' character in its name and ensuring that no ParseError is being thrown. Regression tests have also been added to `bb.parse`. `bin/bitbake-selftest` has also been successfully executed and all tests passed. (Bitbake rev: 00f9ab2cacfbd2a63b6b4959cf5401babae7e32a) Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add autorev warning when it is set too lateRichard Purdie2023-03-223-3/+10
| | | | | | | | | | | | | | | | | | | | Bitbake expects a consistent metadata environment when it parses. There are plenty of ways you can set a recipe to autorev at a point where parts of the fetcher have already been triggered leading to obsure bugs which I struggled to debug, let alone anyone not familar with the code. If anyone is running into the message from the commit, the issue is likely one of timing. Keep in mind that the anonymous python code in base.bbclass will expand variables like FILESPATH, WORKDIR and others which contain PV. The recipe needs to be set to AUTOREV before that anonymous python runs. In particular, that means you can't set SRCREV = "${AUTOREV}" in other anonymous python, it needs to happen earlier. (Bitbake rev: 4d9ec332d5bfc8b60b54f8ec2a17d34e35aa903a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEENRichard Purdie2023-03-223-6/+6
| | | | | | | | | For readability of following patches, rename this internal variable to allow for others in a similar format. (Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Make python output print to stdout when running with -v ↵Mark Asselstine2023-03-211-2/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (verbose) When tasks are run with -v (verbose) on the bitbake commandline, shell tasks print their stdout, python tasks do not. This change redirects the python task's print output to an in memory buffer. After the task is executed the output is printed to stdout via the logger. This makes the python task behavior match the shell task behavior when running with -v. The contents of the task's log files remain unchanged after this change. This approach should keep the correct order in most cases, however, if the python task accesses the logger directly, that content will appear before other output. On the other hand, this change should negate the need for python tasks to access the logger directly. Special care is taken to save/restore the existing stdout and stderr and preventing sending output directly to the logger when there are "recursive" calls, for instance when a python function calls a shell function, avoiding printing things potentially out of order and/or multiple times. The logging-test.bb in meta-selftest can be used to review this change. This has been tested with the full bblogging oeqa tests. [Yocto #14544] (Bitbake rev: 81a58647b2f4fc0a2589b2978fc9d81b2bfe6aec) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: crate.py: authorize crate url with parametersFrederic Martinsons2023-03-172-3/+30
| | | | | | | | | | | | | | | | | This allow to have classic fetch parameters (like destsuffix, sha256, name ...) not being considered by crate fetcher itself (and so mess up its download) Moreover, it allow to overload the name of the downloaded crate (maybe usefull if there is a naming clash between two crates coming from different repositories) (Bitbake rev: 278bd2f1758b8af97552af8d23d16ffb5127a131) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Combine urlopener exceptionsMark Hatle2023-03-111-17/+1
| | | | | | | | | | | No reason to have three identical exception handles, refactor to catch any of the exceptions with the same block of code. (Bitbake rev: b29f6e04091b6bfe697dc41c76880de466736fc3) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Add catch TimeoutError exceptionMark Hatle2023-03-111-0/+8
| | | | | | | | | | | | | | We've observed TimeoutError exceptions during the sstate-cache mirror fetch, it appears that due to the number of (invalid) files requested the remote side is eventually dropping the connection (not closing it) which can result in a TimeoutError exception being sent, while rate it is different from the urllib.error.URLError or ConnectionResetError. (Bitbake rev: 6041b34740deee09ea65d705702555456a5e05d8) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc: Fix after currentAsyncCommand locking changesRichard Purdie2023-03-071-1/+1
| | | | | | | | | | | | | | | | | After changes in bitbake b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3, "process/cooker/command: Fix currentAsyncCommand locking/races", command.py assumes it has access to the process server but the xmlrpc backend was passing in the xmlrpc server object leading to errors like: xmlrpc.client.Fault: <Fault 1: "<class 'AttributeError'>:'BitBakeXMLRPCServer' object has no attribute 'set_async_cmd'"> Fixing to pass the process server to command.py resolves this issue. [YOCTO #15008] (Bitbake rev: ce5b65d5fada474ef21ac28440af6ad45287650a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/npmsw: add more short forms for git operationsMark Asselstine2023-02-262-7/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | >From the npm-install documentation [1] the CLI provides a set of short forms when the install fetches from git. These include "github:" example: npm install github:mygithubuser/myproject "gist:" example: npm install gist:101a11beef "gitlab:" example: npm install gitlab:mygitlabuser/myproject "bitbucket:" example: npm install bitbucket:mybitbucketuser/myproject Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] by Stefan Herbrechtsmeier added support for the "github:" but the others would marked as 'Unsupported dependency'. The other prefixes are added in this commit, along with extending the tests to cover some of these. However, there is one more short form for github which npm-install allows which forgoes the prefix altogether. example: npm install mygithubuser/myproject Unfortunately this format is a bit problematic as it lacks any easily identifiable 'marker' to match against, and it could be either the github short form or install from folder format. Experimentation shows that the folder format requires a leading './' or '/', so we use this to rule out the ambiguity. If this approach to folder and github formats disambiguation is incorrect it won't matter anyways as the folder format is unrecognized by the code as-is and thus with this change or without, things would fail. Since we have to be less strict in the check for git operations we move it to be the last install format which we check, such that the less ambiguous formats can be sorted out first. [1] https://docs.npmjs.com/cli/v9/commands/npm-install [Yocto #14236] (Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix path construction for relative submodule URIMike Crowe2023-02-261-1/+1
| | | | | | | | | | | | | | | | | | | The submodule repository URI contains a path to something not necessarily on the local filesystem. This means that we can't use realpath to normalise it without risking getting bad results if the path happens to match something on the local filesystem. This situation can cause very confusing errors if that matching local path happens to be a symlink to somewhere else. Using normpath rather than realpath means that the path simplification follows simple rules on the string rather than looking at the local filesystem and avoids problems. (Bitbake rev: 47b271e6c8d96960ebe70f80e58f30cc4cbf42e1) Signed-off-by: Mike Crowe <mac@mcrowe.com> Co-authored-by: Dave Craig Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Use internal fetcher function to avoid duplicationRichard Purdie2023-02-261-15/+4
| | | | | | | | | | We currently have two lists of "proxy" or "fetcher" environment exports. Make the one in utils match the one on the fetcher which has a more complete list of variables now. (Bitbake rev: f9165a798a307a6f0fee120d5c3de660d3a44ae8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Drop unused importRichard Purdie2023-02-261-1/+0
| | | | | | | | This import is no longer used anywhere so can be removed. (Bitbake rev: 956128e394581855bf0d03b32a975dc91c2a7e0c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARSCaner Altinbasak2023-02-261-1/+3
| | | | | | | | | These environment variables are needed by gclient and needed to be passed into fetcher. (Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix memory resident cache invalidation issueRichard Purdie2023-02-241-6/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've been seeing weird PRServ failures on the autobuilder. These had one failure always followed by a second. Whilst I can't reproduce the first, if I made that test fail, I could reproduce the second with memory resident bitbake. This was with the tests: prservice.BitbakePrTests.test_import_export_replace_db and then prservice.BitbakePrTests.test_pr_service_deb_arch_dep which was giving a strange looking error: NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa) ERROR: No such task: do_package_write_rpm ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1' where the issue is that selftest.inc written by the test framework and containing PACKAGE_CLASSES = "package_deb" was being ignored. The issue is the cached_statements{} within BBHandler() is not being invalidated at the right time. This patch changes the code to ensure base configuration is not parsed until inotify updates have been processed. (Bitbake rev: cada37c6b9e5862ca2c5a54ad6fd1e1f1939cd9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/siggen: fix debug() callRoss Burton2023-02-241-1/+1
| | | | | | | | | | | | | | | Bitbake f68682 changed the logger's debug() method to be compatible with logging.debug(), but this caller was still using the old API where you can pass an integer as the first argument: WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]') Instead, call bbdebug() which has the priority argument. (Bitbake rev: 18d4f9e8387f7994cf6d46300e25dda1c3a593b2) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.3.1Richard Purdie2023-02-231-1/+1
| | | | | | | | | So that OE-Core can depend on bb.event.check_for_interrupts(), bump our verison number to a development series version. (Bitbake rev: dea1b2f3fc31f28daed5da16e62da8895d6e5716) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/cooker/runqueue: Add ability to interrupt longer running codeRichard Purdie2023-02-233-0/+30
| | | | | | | | | | | | | | | | | | | | Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Replace mkdtemp with TemporaryDirectory and avoid ↵Paulo Neves2023-02-221-4/+1
| | | | | | | | | | | | | | exception masking Due to using mkdtemp instead of TemporaryDirectory we needed to manually cleanup the directory in a try finally block. With tempfile.TemporaryDirectory we can handle the cleanup with a "with" statement and not need to manually clean up oursevels. (Bitbake rev: 36a231d2a45e7c87172196538d18ded81a234774) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-16 11:17:17 +0000