summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Fix deferred task/multiconfig race issueRichard Purdie2023-07-071-4/+12
| | | | | | | | | | | | | | | | | If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: 08033b63ae442c774bd3fce62844eac23e6882d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-user-manual: add BB_HASH_CODEPARSER_VALSPaul Eggleton2023-04-221-0/+11
| | | | | | | | | New variable for 2.4. (Bitbake rev: d97d62e2cbe4bae17f0886f3b4759e8f9ba6d38c) Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: add addpylib and BB_GLOBAL_PYMODULESPaul Eggleton2023-04-222-0/+26
| | | | | | | | | New directive and variable for 2.4. (Bitbake rev: 1b74a7b71e6c8a034163b57ef4e13b4a58c1f844) Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: document BB_CACHEDIRPaul Eggleton2023-04-221-0/+5
| | | | | | | | | New variable for 2.4. (Bitbake rev: fa91e1a02240be1f7b78ef70953ff4a0e0106cc2) Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: do not abort on single ctrl-cEnrico Scholz2023-04-221-1/+1
| | | | | | | | | | | | | | | | After b7ed7e9a815c4e10447fd499508be3dbb47f06e8 bitbake aborts immediately when a single ctrl-c is pressed. Patch restores the previous behavior where a single ctrl-c waits for active tasks to finish. https://bugzilla.yoctoproject.org/show_bug.cgi?id=15094 (Bitbake rev: 66131fa6a3e12c28710d09e1dbf3c03f2981280d) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npmsw fetcher: Avoid instantiating Fetch class if url list is emptySvend Meyland Nicolaisen2023-04-221-1/+3
| | | | | | | | | | | | | | | | Recipes containing both git and npmsw sources in the SRC_URI fail during fetch from the shrinkwrap. It seems that when the fetcher is fetching from the shrinkwrap, the SRCREV variable has been deleted but it till ends up fetching from the git source resulting in an error because SRCREV is undefined. The root cause of this is that the Fetch class defaults to urls from the SRC_URI when the urls parameter contains an empty list. This patch will ensure that Fetch is not instantiated if the urls list is empty. (Bitbake rev: e602963dfd505eef08702366383358d29ee20c4d) Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix PSI check calculationChen Qi2023-04-221-4/+9
| | | | | | | | | | | | | | | The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. (Bitbake rev: b4763c2c93e7494e0a27f5970c19c1aac66c228b) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: ConfHandler: Allow variable flag name with a single characterKai Kang2023-04-222-2/+4
| | | | | | | | | | | | | | Update regex pattern to allow variable flag name with a single character. Regression tests have also been updated in `bb.parse` and `bin/bitbake-selftest -k ParseTest` has been successfully executed. Eliminate a trailing space as well. (Bitbake rev: bb9e523291a3cad6e1596ee6a1e715b5e5feba8f) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: add bb.event.ParseErrorMingli Yu2023-04-222-2/+11
| | | | | | | | | Add bb.event.ParseError to let error-report can catch this kind of error. (Bitbake rev: 316524ab59a5e738c25e062923ee5717d88ae5c7) Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: create versioned 'name' entriesEnrico Scholz2023-04-062-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | It is common for rust packages to depend on different versions of the same crate. E.g. | crate://crates.io/windows_x86_64_msvc/0.42.2 \ | crate://crates.io/windows_x86_64_msvc/0.48.0 \ Identification only by the plain crate name makes the sha256sum ambiguous | SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" | SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a" and requires lot of manual work to fix the SRC_URI list. Making the 'name' property unique by appending the version allows direct copy & paste of reported sha256sum errors to complete the crates list. (Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.4.0Richard Purdie2023-04-052-2/+2
| | | | | | (Bitbake rev: 46e1ea2e9a203992bb4de48ea21a8e736419ada2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Display all missing checksum at onceFrederic Martinsons2023-04-041-8/+10
| | | | | | | | | | | | | | | | | | | | | | | | Instead of only display the first and stop. For recipe (crago based) that can contains several artifacts to fetch with their checksum, it will particularly handy to display all of missing one. An example of error message would be NOTE: Executing Tasks ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe: SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4" SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b" SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9" SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b" ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe') ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025 ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1' (Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: bitbake-user-manual: Update EXCLUDE_FROM_WORLDRobert Yang2023-03-301-1/+1
| | | | | | | (Bitbake rev: ccb2878516d6ac2ee348ad67d0db6dda40e12018) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Make EXCLUDE_FROM_WORLD booleanRobert Yang2023-03-301-1/+1
| | | | | | | | | | | | | | | Fixed: Set EXCLUDE_FROM_WORLD = "0" in recipe_A: $ bitbake world -g Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem: EXCLUDE_FROM_WORLD = "1": Not build in world EXCLUDE_FROM_WORLD = "0": Add back to world build (Bitbake rev: 3f4ede2d67a2d75d3faf8887f90371bd7554d492) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/local: Mention the value of localpath in failure messageRichard Purdie2023-03-301-1/+1
| | | | | | | | | | We have an obtuse rare and intermittent bitbake fetch failure. Understanding the value of ud.localpath at the time of failure would be helpful to debug it further. Tweak the debugging to provide this in the output. (Bitbake rev: 935ab36f64d25c09f83209cd369714c2bc89aa7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-03-301-2/+6
| | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: crate.py: make checksum verification mandatoryFrederic Martinsons2023-03-301-1/+1
| | | | | | | | (Bitbake rev: 9c07e377a21fe1d9660d084da93873acd289b581) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add checksum capability for crate fetcherFrederic Martinsons2023-03-302-6/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This change brings checksum verification of each crate in a recipe, e.g | SRC_URI += " \ | crate://crates.io/aho-corasick/0.7.20 \ | crate://crates.io/atomic-waker/1.1.0 \ | crate://crates.io/cc/1.0.79 \ | " | | SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac" | SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599" | SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f" That will require to move the checksum initialization after the possible call to urldata_init method in order for the crate fetcher to parse the url. Another way of doing could have been implementing a decodeurl method that would have been specific for each fetcher class. (Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/sftp: Fix fetching URIs with spacesDaniel Ammann2023-03-231-1/+1
| | | | | | | | | | Spaces can be used in URIs with %20, but the URI needs to be quoted. (Bitbake rev: 0f887589c279ce07692b4d8e118954a9d214ca45) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler: Allow the '@' character in variable flag namesJames R T2023-03-232-2/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch enables the usage of the '@' character in variable flag names. One use case of variable flags is to assign the network namespaces of some systemd services/targets to configure other parts of the build process of some system. The filenames of systemd services/targets might contain the '@' character if they are template unit files that can take in a single parameter/argument and be instanced multiple times, as indicated by systemd's official manual page. The '@' character is disallowed as the first character in a variable flag name. Imposing more restrictions on the first character is a compromise to make parsing easier and to allow for more options in the future to extend the syntax. This patch is successfully verified by creating a custom BitBake recipe that sets and unsets the value of a variable flag with the '@' character in its name and ensuring that no ParseError is being thrown. Regression tests have also been added to `bb.parse`. `bin/bitbake-selftest` has also been successfully executed and all tests passed. (Bitbake rev: 00f9ab2cacfbd2a63b6b4959cf5401babae7e32a) Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: fix links to supported release manualsMichael Opdenacker2023-03-231-40/+36
| | | | | | | | | | | Links to manuals for supported releases were all broken. Only the pre-sphinx ones are still valid. (Bitbake rev: 887804de05d704644328a105e2add6861adb4cab) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add autorev warning when it is set too lateRichard Purdie2023-03-223-3/+10
| | | | | | | | | | | | | | | | | | | | Bitbake expects a consistent metadata environment when it parses. There are plenty of ways you can set a recipe to autorev at a point where parts of the fetcher have already been triggered leading to obsure bugs which I struggled to debug, let alone anyone not familar with the code. If anyone is running into the message from the commit, the issue is likely one of timing. Keep in mind that the anonymous python code in base.bbclass will expand variables like FILESPATH, WORKDIR and others which contain PV. The recipe needs to be set to AUTOREV before that anonymous python runs. In particular, that means you can't set SRCREV = "${AUTOREV}" in other anonymous python, it needs to happen earlier. (Bitbake rev: 4d9ec332d5bfc8b60b54f8ec2a17d34e35aa903a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEENRichard Purdie2023-03-223-6/+6
| | | | | | | | | For readability of following patches, rename this internal variable to allow for others in a similar format. (Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: update fixtures for mickledoreTim Orling2023-03-212-15/+15
| | | | | | | | | | | | | After updating gen_fixtures.py for 'mickledore' 4.2, run './gen_fixutures.py --all' to update oe-core.xml and poky.xml. Drop langdale as support is ending soon. (Bitbake rev: 282a3d618b565a60b8734ca0fb710db93023b5c8) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: update gen_fixtures.py for mickledoreTim Orling2023-03-211-3/+4
| | | | | | | | | | | Update for 'mickledore' 4.2.0 release and bitbake 2.4 branching. Drop 'langdale' as support will be ending soon. Update stable branches to latest patch release. (Bitbake rev: b01d369f6d6cccf77b23017ff343820d800d5621) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Make python output print to stdout when running with -v ↵Mark Asselstine2023-03-211-2/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (verbose) When tasks are run with -v (verbose) on the bitbake commandline, shell tasks print their stdout, python tasks do not. This change redirects the python task's print output to an in memory buffer. After the task is executed the output is printed to stdout via the logger. This makes the python task behavior match the shell task behavior when running with -v. The contents of the task's log files remain unchanged after this change. This approach should keep the correct order in most cases, however, if the python task accesses the logger directly, that content will appear before other output. On the other hand, this change should negate the need for python tasks to access the logger directly. Special care is taken to save/restore the existing stdout and stderr and preventing sending output directly to the logger when there are "recursive" calls, for instance when a python function calls a shell function, avoiding printing things potentially out of order and/or multiple times. The logging-test.bb in meta-selftest can be used to review this change. This has been tested with the full bblogging oeqa tests. [Yocto #14544] (Bitbake rev: 81a58647b2f4fc0a2589b2978fc9d81b2bfe6aec) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: crate.py: authorize crate url with parametersFrederic Martinsons2023-03-172-3/+30
| | | | | | | | | | | | | | | | | This allow to have classic fetch parameters (like destsuffix, sha256, name ...) not being considered by crate fetcher itself (and so mess up its download) Moreover, it allow to overload the name of the downloaded crate (maybe usefull if there is a naming clash between two crates coming from different repositories) (Bitbake rev: 278bd2f1758b8af97552af8d23d16ffb5127a131) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Combine urlopener exceptionsMark Hatle2023-03-111-17/+1
| | | | | | | | | | | No reason to have three identical exception handles, refactor to catch any of the exceptions with the same block of code. (Bitbake rev: b29f6e04091b6bfe697dc41c76880de466736fc3) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: Add catch TimeoutError exceptionMark Hatle2023-03-111-0/+8
| | | | | | | | | | | | | | We've observed TimeoutError exceptions during the sstate-cache mirror fetch, it appears that due to the number of (invalid) files requested the remote side is eventually dropping the connection (not closing it) which can result in a TimeoutError exception being sent, while rate it is different from the urllib.error.URLError or ConnectionResetError. (Bitbake rev: 6041b34740deee09ea65d705702555456a5e05d8) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: update Hello World exampleFawzi KHABER2023-03-071-67/+60
| | | | | | | | | | | | | | | | Fix broken link "Obtaining bitbake". Update documentation for the bitbake hello world example, the output was outdated. Fix LAYERSERIES_COMPAT warning by adding dunfell as default compatible release. Add proper formating for base.bbclass command. Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reviewed-by: Ever ATILANO <ever.atilano@smile.fr> Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr> (Bitbake rev: e859ce61a7422425960342a47785468cef49caa1) Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: ref-variables: add LAYERSERIES_COMPAT to term glossaryFawzi KHABER2023-03-071-0/+23
| | | | | | | | | | | | | | Bitbake throws a warning if the layer compatibility is not defined since cca81e33b58c390dcf5cc3a31555a43b79177166. The description of this variable comes from the Yocto Project manual. Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reviewed-by: Ever ATILANO <ever.atilano@smile.fr> Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr> (Bitbake rev: cbad8cb28a8134d6b2053ef5c2e04bb455fc798b) Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc: Fix after currentAsyncCommand locking changesRichard Purdie2023-03-071-1/+1
| | | | | | | | | | | | | | | | | After changes in bitbake b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3, "process/cooker/command: Fix currentAsyncCommand locking/races", command.py assumes it has access to the process server but the xmlrpc backend was passing in the xmlrpc server object leading to errors like: xmlrpc.client.Fault: <Fault 1: "<class 'AttributeError'>:'BitBakeXMLRPCServer' object has no attribute 'set_async_cmd'"> Fixing to pass the process server to command.py resolves this issue. [YOCTO #15008] (Bitbake rev: ce5b65d5fada474ef21ac28440af6ad45287650a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: documentation: bitbake: add file-checksums to varflags sectionRichard Elberger2023-02-261-0/+17
| | | | | | | | | | | | | Fixes [YOCTO #11605] by: - Adding definition of file-checksums to Variable Flags section. - Describe data to add to list which adds external file dependencies. - Write example on usage to prepend a value to file-checksums list. (Bitbake rev: 6bc65e6402a74c9a65e29333a7e0f1f85dcbcf58) Signed-off-by: Richard Elberger <rich@richelberger.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/npmsw: add more short forms for git operationsMark Asselstine2023-02-262-7/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | >From the npm-install documentation [1] the CLI provides a set of short forms when the install fetches from git. These include "github:" example: npm install github:mygithubuser/myproject "gist:" example: npm install gist:101a11beef "gitlab:" example: npm install gitlab:mygitlabuser/myproject "bitbucket:" example: npm install bitbucket:mybitbucketuser/myproject Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] by Stefan Herbrechtsmeier added support for the "github:" but the others would marked as 'Unsupported dependency'. The other prefixes are added in this commit, along with extending the tests to cover some of these. However, there is one more short form for github which npm-install allows which forgoes the prefix altogether. example: npm install mygithubuser/myproject Unfortunately this format is a bit problematic as it lacks any easily identifiable 'marker' to match against, and it could be either the github short form or install from folder format. Experimentation shows that the folder format requires a leading './' or '/', so we use this to rule out the ambiguity. If this approach to folder and github formats disambiguation is incorrect it won't matter anyways as the folder format is unrecognized by the code as-is and thus with this change or without, things would fail. Since we have to be less strict in the check for git operations we move it to be the last install format which we check, such that the less ambiguous formats can be sorted out first. [1] https://docs.npmjs.com/cli/v9/commands/npm-install [Yocto #14236] (Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix path construction for relative submodule URIMike Crowe2023-02-261-1/+1
| | | | | | | | | | | | | | | | | | | The submodule repository URI contains a path to something not necessarily on the local filesystem. This means that we can't use realpath to normalise it without risking getting bad results if the path happens to match something on the local filesystem. This situation can cause very confusing errors if that matching local path happens to be a symlink to somewhere else. Using normpath rather than realpath means that the path simplification follows simple rules on the string rather than looking at the local filesystem and avoids problems. (Bitbake rev: 47b271e6c8d96960ebe70f80e58f30cc4cbf42e1) Signed-off-by: Mike Crowe <mac@mcrowe.com> Co-authored-by: Dave Craig Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Use internal fetcher function to avoid duplicationRichard Purdie2023-02-261-15/+4
| | | | | | | | | | We currently have two lists of "proxy" or "fetcher" environment exports. Make the one in utils match the one on the fetcher which has a more complete list of variables now. (Bitbake rev: f9165a798a307a6f0fee120d5c3de660d3a44ae8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Drop unused importRichard Purdie2023-02-261-1/+0
| | | | | | | | This import is no longer used anywhere so can be removed. (Bitbake rev: 956128e394581855bf0d03b32a975dc91c2a7e0c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARSCaner Altinbasak2023-02-261-1/+3
| | | | | | | | | These environment variables are needed by gclient and needed to be passed into fetcher. (Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix memory resident cache invalidation issueRichard Purdie2023-02-241-6/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've been seeing weird PRServ failures on the autobuilder. These had one failure always followed by a second. Whilst I can't reproduce the first, if I made that test fail, I could reproduce the second with memory resident bitbake. This was with the tests: prservice.BitbakePrTests.test_import_export_replace_db and then prservice.BitbakePrTests.test_pr_service_deb_arch_dep which was giving a strange looking error: NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa) ERROR: No such task: do_package_write_rpm ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1' where the issue is that selftest.inc written by the test framework and containing PACKAGE_CLASSES = "package_deb" was being ignored. The issue is the cached_statements{} within BBHandler() is not being invalidated at the right time. This patch changes the code to ensure base configuration is not parsed until inotify updates have been processed. (Bitbake rev: cada37c6b9e5862ca2c5a54ad6fd1e1f1939cd9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/siggen: fix debug() callRoss Burton2023-02-241-1/+1
| | | | | | | | | | | | | | | Bitbake f68682 changed the logger's debug() method to be compatible with logging.debug(), but this caller was still using the old API where you can pass an integer as the first argument: WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]') Instead, call bbdebug() which has the priority argument. (Bitbake rev: 18d4f9e8387f7994cf6d46300e25dda1c3a593b2) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.3.1Richard Purdie2023-02-232-2/+2
| | | | | | | | | So that OE-Core can depend on bb.event.check_for_interrupts(), bump our verison number to a development series version. (Bitbake rev: dea1b2f3fc31f28daed5da16e62da8895d6e5716) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/cooker/runqueue: Add ability to interrupt longer running codeRichard Purdie2023-02-233-0/+30
| | | | | | | | | | | | | | | | | | | | Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Replace mkdtemp with TemporaryDirectory and avoid ↵Paulo Neves2023-02-221-4/+1
| | | | | | | | | | | | | | exception masking Due to using mkdtemp instead of TemporaryDirectory we needed to manually cleanup the directory in a try finally block. With tempfile.TemporaryDirectory we can handle the cleanup with a "with" statement and not need to manually clean up oursevels. (Bitbake rev: 36a231d2a45e7c87172196538d18ded81a234774) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Removed unused variables in _contains_lfsPaulo Neves2023-02-221-5/+0
| | | | | | | | | | branchname was set but never used in the context of _contains_lfs method. (Bitbake rev: a64ec64c2faf1ac6ce995cdd4e311d8be6046155) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add real git lfs tests and decoratorPaulo Neves2023-02-221-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. (Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: git-lfs restore _find_git_lfsPaulo Neves2023-02-221-16/+24
| | | | | | | | | | Not restoring the mocked _find_git_lfs leads to other tests failing. (Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Improve idle thread exception handlingRichard Purdie2023-02-201-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | If the inotifier code has an exception, bitbake currently hangs. Catch any exception and exit if seen. Also check the idle thread is alive and exit if it disappears. This should stop bitbake hanging if such a situation arises in future such as this example: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) (Bitbake rev: 358b5b02d5de1ab0f98104c4ec4953e46999b9a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure lock is held with changing notifierRichard Purdie2023-02-201-16/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | We've seen a couple of cases which bitbake hangs due to an inotifer exception such as: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) Ensure we don't destory the inotifier when the idle thread is reading is by holding the lock during setup/teardown. (Bitbake rev: 8fc5c50c2e23017833f93bcd514d708a14fa4266) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop SystemExit usageRichard Purdie2023-02-191-2/+1
| | | | | | | | | Using bb.fatal for a fatal error message is the best practise, switch the code to match other call sites. (Bitbake rev: c27e48fa81c2327a4a355a028884ab457cde3ae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Drop dubious exception handling codeRichard Purdie2023-02-191-5/+0
| | | | | | | | | | | This code appears to be dangerous, it swallows exceptions, turning them into "handled" versions which then show no errors to the user. This is a pretty poor user experience and I can't see why this code should be swallowing such things. Drop the worst bits of code. (Bitbake rev: 13279044f16f2cf2502ebf39d277415f99bb6c18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-13 06:46:45 +0000