summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-04-111-2/+6
| | | | | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Drop dubious exception handling codeRichard Purdie2023-03-141-5/+0
| | | | | | | | | | | | | This code appears to be dangerous, it swallows exceptions, turning them into "handled" versions which then show no errors to the user. This is a pretty poor user experience and I can't see why this code should be swallowing such things. Drop the worst bits of code. (Bitbake rev: a29ffcc12df24737e702e198aeb8f6892884932b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Improve early exception handlingRichard Purdie2023-03-141-9/+1
| | | | | | | | | | | | | | | | | | Martin Jansa reported that if you put a syntax error into an imported module such as qa.py in OE, no error is shown. Part of the issue appears to be that the catch_parse_error() decorator only catches certain exceptions and SyntaxError isn't one of them. As far as I can tell we should remove all the special cases and use the more advanced code in all cases, not just expansion errors. I confirmed this now prints a proper error message for a qa.py syntax error. (Bitbake rev: e66012bfda60ffe1658473e25879aa67909ae65f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Remove incorrect SystemExit usageRichard Purdie2023-03-141-1/+1
| | | | | | | | | | | | | Calling SystemExit doesn't work well with server/client usage since the string isn't printed to the right place. Use bb.fatal() instead which prints the right log output and raises and handled exception which then shows correctly on the UI. (Bitbake rev: b9ae7164d9e744e8eb9aaff79218f57233a449b7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Allow to_boolean to support int valuesRichard Purdie2023-03-141-0/+3
| | | | | | | | | | | | | | | | | | Some variables may be set as: X = 1 as well the more usual X = "1" so add support to to_boolean to handle this case. (Bitbake rev: e7df13a61911b7431802af2b4d7472b2aaf346fa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: show SRCREV and git repo in error message about fixed ↵Martin Jansa2023-02-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SRCREV * in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to be one character shorter and because fetcher uses: if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]): to decide which SRCREV values are fixed SRCREVs this one was considered as tag or branch name, because it was only 39 chars long The original error message wasn't very helpful as it doesn't show which repo or which SRCREV was considered missing: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) with SRCPV included in PV as error recomments it's a bit better: bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16 The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T with this change the first error will read: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) (Bitbake rev: 9bbdedc0ba7ca819b898e2a29a151d6a2014ca11) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: break on first dependent task differenceSchmidt, Adriaan2023-02-171-0/+1
| | | | | | | | | | | | | | | | | compare_sigfiles() recursively calculates differences on all dependent tasks with changed hashes. This is done in arbitrary/alphabetical order, and only the last of those results is returned, while everything else is discarded. This changes the behavior to instead return the first difference and not calculate any more, which significantly speeds up diffs of tasks with many dependencies. (Bitbake rev: 89f13cd4a927a73de98998c27082c63b07671525) Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ea6a676c9aa2864c2eff40eea41ba09ce903a651) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/ply: Update md5 to better report errors with hashlibMark Hatle2023-02-172-1/+13
| | | | | | | | | | | | | | | | | | | | | | | | In the case where hashlib is not available, the try would fail and fall through resulting in a backtrace on the usage of the 'sig'. The backtrace itself was confusing and made it difficult to determine what went wrong. Update the import to be in it's own try block with an appropriate message to indicate what went wrong. Note, the current version of ply all of this code has been restructured so this is not applicable upstream. Additionally, some versions of hashlib don't appear to implement the second FIPS related argument. Detect this and support both versions. (Bitbake rev: d26ed38c233583b35ad1451e521f07d9c2329f4e) Signed-off-by: Mark Hatle <mark.hatle@amd.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> (cherry picked from commit 484ab42f440070c0369b81f5c69da860fa47a798) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix inefficient string concatenationEtienne Cordonnier2023-02-171-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython has an optimization for statements in the form "a = a + b" or "a += b". It seems that this line does not get optimized, because it has a form a = a + b + c: data = data + "./" + f.split("/./")[1] For that reason, it does a copy of data for each iteration, potentially copying megabytes of data for each iteration. Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds instead of 45 seconds on my test setup where SRC_URI points to a big directory. Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython: "do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b" However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup: data = ''.join((data, "./", f.split("/./")[1])) I have changed the other lines to also use += for consistency only, however those were in the form a = a + b and were optimized already. Co-authored-by: JJ Robertson <jrobertson@snap.com> (Bitbake rev: 590ae6fde9da75db3a368e5c0d47920696c33ebf) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Clarify the meaning of namespaceMarek Vasut2023-02-041-1/+2
| | | | | | | | | | | | | | | | | Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: 86f2fa5261da959cda706c794a0047e5e89d4d6b) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository ↵Marek Vasut2023-02-041-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | metadata The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> (Bitbake rev: c17fc1468ab84663b919e2809606b1b8ea2bebd9) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: include SSL certificate paths in export_proxiesRoss Burton2023-01-301-12/+9
| | | | | | | | | | | | | | | | | | | | | | | | bb.utils.export_proxies() is a poor-man's alternative for the environment setup code in bb/fetch2, but it's used in several places where recipes want to download manually (such as cve-update-db-native). Notably, export_proxies() doesn't pass on the SSL certificate paths from the original environment, so if SSL_CERT_FILE needs to be set (for example, in a buildtools environment) then proxies work but SSL doesn't. In an ideal world export_proxies and the same logic in fetch2 would merge, but until then we can add the SSL_CERT_ variables and duplicate the basic logic: check the datastore first and then the original environment for variables. Also remove the return value as nothing ever checked it. [ YOCTO #15000 ] (Bitbake rev: ed0dcc40f80c48839bac20298013d70043858a4e) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix regression in gitsm submodule path parsingPavel Zhukov2022-12-091-1/+1
| | | | | | | | | | | | | | | | | | | Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression in submodules path parsing. As the result gitsm fetcher fails on each submodule which name begins from the name of the parent repo which is totally valid usecase [Yocto #14045] [1] Fix the code to error out only if submodule's name is equal to parent name but not if it's part of it. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4 (Bitbake rev: 7e268c107bb0240d583d2c34e24a71e373382509) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix race issues around hash equivalence and sstate reuseRichard Purdie2022-11-241-16/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | We identified a use case where a native recipe (autoconf-native) was rebuilt with no change in output yet the sstate for do_package tasks wasn't being used. The issue is that do_package tasks have a hard dependency on pseudo-native:do_populate_sysroot. That task was one of the many tasks being rehashed when autoconf-native's hash was changed. If update_tasks processed a recipe before it had processed pseudo-native, that recipe would be marked as not possible from sstate and would run the full tasks. The fix is to split the processing into two passes, first to handle the existing covered/notcovered updates, then in the second pass, check whether there are "harddep" issues. This defers the do_package tasks until after pseudo-native is installed from sstate as expected and everything works well again. (Bitbake rev: 72a3afd99e8b785cb2a2f687e71a58e08cdd9c74) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e479d1e418a7d34f0a4663b4a0e22bb11503c8ab) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie2022-10-271-2/+2
| | | | | | | | | CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: c90d57497b9bcd237c3ae810ee8edb5b0d2d575a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc/client: Fix unix domain socket chdir race issuesRichard Purdie2022-10-111-9/+13
| | | | | | | | | | | | | | | | | | | | | | | The connect_unix() call had a bug where if a relative path to a socket was passed (which the non-async client always does), and the current working directory was changed after the initial call, it would fail to reconnect if it became disconnected, since the socket couldn't be found relative to the new current working directory. To work around this, change the socket connection for UNIX domain sockets to be synchronous and change current working before connecting. This isn't ideal since the connection could block the entire event loop, but in practice this shouldn't happen since the socket are local files anyway. Help debugging and resolving from Joshua Watt. (Bitbake rev: 6603c3e39f1cf746669ec6c9f0be8c6e6ece426e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5964bb67bb20df7f411ee0650cf189504a05cf25) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Error out if submodule refers to parent repoPavel Zhukov2022-10-111-0/+3
| | | | | | | | | | | | | | | | | If submodule refers to specific revision of the parent repository it causes deadlock in bitbake locking mechanism (lock is acquired to fetch the parent and cannot be released before all submodules are fetched). raise FetchError in such situation to prevent deadlocking. [Yocto 14045] (Bitbake rev: 987712c4c8fefd86a1f5116c11ee86e296e852ee) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Fix npm to use https rather than httpNeil Horman2022-10-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Hit this error while building nlf-native recently: { "error": { "summary": "URI malformed", "detail": "" } } Some poking about led me to discover that: 1) The npm.py tool replaces npm:// with http://, not https:// 2) Some versions of the npm tool don't handle 301 redirects properly, choosing to display the above error instead when using the default nodejs registry It would be good to go fix npm to handle the redirect properly, but it seems like it would also be good to assume secure http when contacting a registry, hence, this patch (Bitbake rev: bb5c43220f5f1c3d82334c65aff1ce13008db8d9) Signed-off-by: Neil Horman <nhorman@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add copyright headers where missingRichard Purdie2022-10-1125-0/+50
| | | | | | | | | | | | Where copyright headers were not present, add them to make things clear. (Bitbake rev: e591325b2bd901c381003deb96a7b32a7148e93e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix insufficent entropy in sigtask file namesJoshua Watt2022-10-112-1/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | Signature generation uses mkstemp() to get a file descriptor to a unique file and then write the signature into it. However, the unique file name generation in glibc is based on the system timestamp, which means that with highly parallel builds it is more likely than one might expect expected that a conflict will occur between two different builder nodes. When operating over NFS (such as a shared sstate cache), this can cause race conditions and rare failures (particularly with NFS servers that may not correctly implement O_EXCL). The signature generation code is particularly susceptible to races since a single "sigtask." prefix used for all signatures from all tasks, which makes collision even more likely. To work around this, add an internal implementation of mkstemp() that adds additional truly random entropy to the file name to eliminate conflicts. (Bitbake rev: 63bb5591e833de0e7b552963ad9bc4b39e56fda9) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> (cherry picked from commit 97955f3c1c738aa4b4478a6ec10a08094ffc689d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop deadlock breaking force failRichard Purdie2022-10-111-3/+0
| | | | | | | | | | | | | | | | | I'm 99% certain this failing of a scenequeue task corrupts runqueue and causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted and did weird things so drop this code. We've seen builds where the deadlock triggers and it then tries to run tasks where the SQ task already ran with very confusing failures. It is likely it is this code causing it. (Bitbake rev: f386298fc056ef130c2eb6dabf25eafbd55f55ca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8efced47fcb47851a370fd6786df6fb377f99963) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve deadlock warning messagesRichard Purdie2022-10-111-4/+6
| | | | | | | | | | | | | Tweak the deadlock breaking messages to be explict about which task is blocked on which other task. The messages currently imply it is "freeing" the blocking task which is confusing. (Bitbake rev: d1b84e3cfe9fb8d282d4b700a9fe31891e00d837) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cf7f60b83adaded180f6717cb4681edc1d65b66d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure deferred tasks are sorted by multiconfigRichard Purdie2022-10-111-5/+8
| | | | | | | | | | | | | | | | We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. (Bitbake rev: 513bfd771d9095fcb6a8bf93806673dbf988a4de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: movefile: use the logger for printingJose Quaresma2022-09-161-6/+6
| | | | | | | | | | (Bitbake rev: a5fff68c8c83fed52dae0bdcf2c1713566a5a53e) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit 274b2d7a2fa0b43b0b542cb5471ff832e692ea93) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: remove: check the path again the expand python globJose Quaresma2022-09-161-2/+2
| | | | | | | | | | | | | | | When we call the remove with recurse=True we first check if the remove operation is safe in _check_unsafe_delete_path. But the check is been done on the path instaed of the expanded python glog. (Bitbake rev: 280ea5a776436eab7e664fccea2df2e7ce47e586) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit 7236488b898309ec5f1880936ddae22a28ccf5d3) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure directory exists before creating symlinkRichard Purdie2022-09-161-0/+2
| | | | | | | | | | | | | | | If the mirrors code is trying to create a symlink and the parent directory doesn't exist, as might be the case for sstate mirrors where the fetch is into a subdir, it can silently fail. Ensure the directory exists in this case to avoid issues. (Bitbake rev: ff3afb1c1bb236c4a52c62a74f2917071e0af55b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit eff16e474ee7dc49ae433420a4c8d15d3314a618) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler/BBHandler: Improve comment error messages and add testsRichard Purdie2022-09-163-4/+32
| | | | | | | | | | | | | | | | | | | | | | Currently if you trigger one of the comment errors, the newline characters are stripped and the line numbers are incorrect. In one case it prints the empty line which is also unhelpful. Rework the code around these errors so the line numbers are correct and the lines in question are more clearly displayed complete with newlines so the user can more clearly see the error. I also added a couple of simplistic test cases to ensure that errors are raised by the two known comment format errors. [YOCTO #11904] (Bitbake rev: 01d27562c11d4b05eb30c7f9fefd58b6599fdd15) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 712da71b24445c814d79a206ce26188def8fce0a) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: gitsm: fix incorrect handling of git submodule relative urlsGennaro Iorio2022-09-161-1/+1
| | | | | | | | | | | | | | As specified by git submodule manual relative urls can start either with '..' or './', second case was incorrectly managed leading to an interpretation of urls starting with './' as absoulte urls. (Bitbake rev: d828cd2a16ddf4f084e61ffe44471483e132653a) Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler: Remove lingering closeOla x Nilsson2022-09-161-2/+0
| | | | | | | | | | | | | | | The f.close() statement should have been removed in 459ad524756a3f9b50feeedf31e33502dceae8d5. (Bitbake rev: 23221378ff0d8c6908d75d1be3219aae0beee406) Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9fc1bab6b7e3c0fca3ddec4bc8c7763d2aff8bab) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event.py: ignore exceptions from stdout and sterr operations in atexitMikko Rapeli2022-09-161-2/+8
| | | | | | | | | | | | | | | | | | | | When atexit functions run, stdout and stderr operations may fail, e.g. when output is piped to less but has been exited by the user. This removes error print from output of "bitbake -e sqlite3 | less" if user presses "q" before bitbake has finished processing: [Errno 32] Broken pipeError in atexit._run_exitfuncs: Traceback (most recent call last): File "/home/builder/src/poky/bitbake/lib/bb/event.py", line 135, in print_ui_queue sys.stdout.flush() (Bitbake rev: 65cee11967f60c74fa89bb6d72f32135968a6b87) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 35167536c163eb6b7653cbcaad9f65b834d3e2f8) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Drop sre_constants usageRichard Purdie2022-09-161-3/+2
| | | | | | | | | | | | | | | | | | | | As reported by Martin Jansa <Martin.Jansa@gmail.com>: bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated import sre_constants it's deprecated since 3.11 with: https://github.com/python/cpython/issues/91308 The correct replacement for our usage is re.error so use that instead. (Bitbake rev: c98007217b8e40f1abfdcba709185dc5ddbcd0c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3c0cd401472ffee06d5a93bdba566cb033851fcf) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix unihash cache mismatch issuesRichard Purdie2022-09-161-0/+3
| | | | | | | | | | | | | | | | | | Very occasionally we see errors in eSDK testing on the autobuilder where the task hashes in the eSDK don't match what was just built. I was able to inspect one of these build directories and noticed that the bb_unihashes.dat file in the eSDK was zero sized. Whilst inspecting the code to understand the cause, I noticed that updated hashes are not saved out in subsequent updates of the values in the rehash process. Add a missing sync call to ensure this happens. (Bitbake rev: 81a6f490dd1f5f669c75cd2ceb1105ce7a09c6e4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7912dabbcf444a3c3d971cca4a944a8b931e301b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fix kirkstone versionOtavio Salvador2022-09-071-1/+1
| | | | | | | (Bitbake rev: 7d3b766ef26361c44abc2b8175bc5c44e29c3955) Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Pass lock argument in fileslockedJoshua Watt2022-09-021-2/+4
| | | | | | | | | | | | Pass additional arguments in the fileslocked() context manager to the underlying lockfile() function. This allows the context manager to be used for any types of locks (non-blocking, shared, etc.) that the lockfile() function supports. (Bitbake rev: ce9fe70156e8f909a3a81da017b89ea61bc6fe38) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change pressure file warning to a noteRichard Purdie2022-08-241-1/+1
| | | | | | | | | The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: bc85c044ec250001855f2f9f0717ac031feab7c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: runqueue: add memory pressure regulationAryaman Gupta2022-08-231-5/+22
| | | | | | | | | | | | | | | | | | | | | | | Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) (Bitbake rev: 4ada86cb6b05e6e3aabc8015a6e73aacb14a3388) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: runqueue: add cpu/io pressure regulationAryaman Gupta2022-08-231-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) (Bitbake rev: f4954b878d404a72156fe98609387aa9c5747af0) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <randy.macleod@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix logging issues where only the first message was ↵Richard Purdie2022-06-261-2/+5
| | | | | | | | | | | | | | | | | | displayed I realised only the first logging message was being displayed in a given parsing process. The reason turned out to be the UI handler failing with a "pop from empty list". The default handler was then lost and no further messages were processed. Fix this by catching the exception correctly in the connection writer code. (Bitbake rev: b8fd6f5d9959d27176ea016c249cf6d35ac8ba03) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d3e64f64525187f1409531a0bd99df576e627f7f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Add clean_stamp API function to allow removal of task stampsRichard Purdie2022-05-251-5/+9
| | | | | | | | | | | | | | | | | | We currently have no API to be able to remove all the potential stamps of a task. It is unusual to need to do this, particularly as you could race against other things happening in the system but we do have a use case for this in cleaning up sysroots in OE-Core. The alternative is to mess with CLEANMASK in OE-Core but that is just going to add potential for errors. We need the first part of the make_stamp() function so separate that out so it can be called seperately. (Bitbake rev: 494fcfcb7c4469915c67f544997104d81c417266) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4d671504a25863018ac51c21c005cef0a4d8f05c) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Do not depend on vardepvalueexclude flagTomasz Dziendzielski2022-05-251-0/+2
| | | | | | | | | | | | | | | | | If SRC_URI contains python function that extends vardepvalueexclude its value is being tracked by sstate-cache, which can lead to rebuilds if value is set dynamically (for example gerrit replicas). Return empty string if vardepvalueexclude is checked to fix this behaviour. (Bitbake rev: 943701ee8cb55307996545f0237721413edfb168) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f5f9a7b89a7d8321f03184e61ad6d5ed8d0f840e) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/osc: Small fixes for osc fetcherGunjan Gupta2022-05-251-6/+10
| | | | | | | | | | | | | | | | | | | | | | The current fetcher seemed to have some issues that made it difficult when trying to use the same. This patch fixes the following * Make consistent use of the path that needs to be used as oscdir * The path mentioned in os.access in download function was not same as ud.moddir which would result into invoking of fetch command instead of update command even if directory already existed * Before creating oscrc, make sure oscdir exists and create it if it does not exist * Updated the configuration to use apiurl and added a new parameter to control whether http or https needs to be used to connect to apiurl (Bitbake rev: 8ac6e09447d884e658c556388d6014279c50f202) Signed-off-by: Gunjan Gupta <viraniac@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3ec78686f3c0ea2304097b86a965f9be4b0cb879) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git : Use cat as pagerRichard Purdie2022-05-121-1/+1
| | | | | | | | | | | | We don't have less in HOSTTOOLS in OE and this can confuse git. Force the pager to cat to be consistent and minimal everywhere. (Bitbake rev: 59c16ae6c55c607c56efd2287537a1b97ba2bf52) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d3d406e8552fdd865dc58b419a84411736475ad2) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: correctly handle file names containing colonsRoland Hieber2022-05-121-1/+1
| | | | | | | | | | | | | | | | | | File names containing colons cause split() to return a list with more than two elements, which will lead to a stack trace ending in: ValueError: too many values to unpack (expected 2) Split only once at the last colon, thereby making sure that only two elements are returned. (Bitbake rev: e14ed7dd866334ab40cd335d9c006e5d7b447abd) Signed-off-by: Roland Hieber <rhi@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a70a7376a8708bde07959deb5d5842d7f84ee5f8) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sig file location when using multiconfigRichard Purdie2022-05-121-1/+1
| | | | | | | | | | | | | | | We're using the wrong data store when trying to locate siginfo files, fix this. Thanks to Gregory Lumen <gregorylumen@microsoft.com> for spotting. [YOCTO #14774] (Bitbake rev: 7eb0ef75fd08b6e4ca1e9dca9c96a7b590e5147b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0ed800e19a3197f8e622c8d3b630aae384e60aba) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/ssh.py: decode path back for sshRobert Yang2022-05-121-2/+4
| | | | | | | | | | | | | | | | | The path has been encoded by urllib.parse.quote(), so decode it back for ssh. Fixed when fetch from PREMIRRORS via ssh: $ bitbake bonnie++ libsigc++-2.0 -cfetch scp: /path/to/downloads/libsigc%2B%2B-2.10.7.tar.xz: No such file or directory (Bitbake rev: 3969786a787eea34e096b932d52cd02978aacb8e) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c1c8fc678eb4783cea3974328a5fa8d1b79f1266) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/osc: Add missing parameterRichard Purdie2022-05-121-1/+1
| | | | | | | | | | | | This probably means the osc fetcher isn't being used but fix the missing parameter. (Bitbake rev: 73fbb743a2def2037d4053605e77e09d2d8a9fd0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a23c201cb6efc5c0abf763c26f905442f0eebb68) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: Drop unused variablesRichard Purdie2022-05-121-2/+2
| | | | | | | | | (Bitbake rev: 97e150550a3807c60cfa685abda8dccafc0a1268) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d720dfa40620e64a557edef527148d58fcb1d858) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop pointless variable assignmentRichard Purdie2022-05-121-1/+0
| | | | | | | | | | | | This is set at the start of the loop anyway so it does nothing. Drop the pointless code. (Bitbake rev: dcf78788daa177bf5c438f33b3c9f7ced7aea8ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e6a3173c9cdf349ccbd4cf612868f92cce8717c8) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Use a valid exception for missing implementationRichard Purdie2022-05-121-1/+1
| | | | | | | | | (Bitbake rev: 1e5c5efa6fe9175b27b21cd7a5b833bd3af55238) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7254eb6b3e8ef504ef2274541dcc55f1d42238c6) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/knotty: Drop pointless pass statementRichard Purdie2022-05-121-1/+0
| | | | | | | | | (Bitbake rev: 00f10fe95393728e94339f3512171ebab96f1900) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 625565087d8c9e7a6a79b0b4f3e5be2d77d5f100) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-07 19:09:29 +0000