summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: doc/user-manual: Add BB_LOADFACTOR_MAXRichard Purdie2024-04-121-0/+9
| | | | | | | | Document BB_LOADFACTOR_MAX which was recently added. (Bitbake rev: 833b76e9333e317cab5f17d6f7daaecc89c69547) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: Add section for variable contextSimone Weiß2024-04-112-0/+92
| | | | | | | | | | | | | | This is inspired by the same section in the yocto-docs. It aims to provide information in what contexts(recipes, .conf, bbclass,...) a variable is usually used. For that I tried to group similar variables, so that a short overview is given. This was inspired by [YOCTO #14072], but of course does not implement a warning if a variable is used in an unintended context. (Bitbake rev: 5ced476685376b1a32b7fdd9546f9b61c5962aa0) Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-04-111-1/+4
| | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-hashclient: Warn on bad .netrcJoshua Watt2024-03-261-0/+2
| | | | | | | | | If there is an error parsing .netrc, warn the user on stderr (Bitbake rev: 6366ea8d9c284d10bb8f4129004b55239d9022c0) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add support for hashserve credentialsJoshua Watt2024-03-231-5/+16
| | | | | | | | | | Adds support for hashserver credentials to be specified in the SignatureGenerator (Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Install Git LFS in local repository configDerek Erdmann2024-03-221-1/+1
| | | | | | | | | | | | | | | | | Git uses a lock file to prevent concurrent modifications to the global config, so if unpack tasks for different recipes try to run "git lfs install" simultaneously the operation can fail: error: could not lock config file /home/build/.gitconfig: File exists exit status 255 Run `git lfs install --force` to reset Git configuration. Adding "--local" sets the smudge and clean filters in the local repository's config instead of modifying the user's global config. (Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7) Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Fix silent hang issue caused by unexpected stdout ↵Yang Xu2024-03-221-3/+9
| | | | | | | | | | | | | | | | | | | | | content This patch addresses an issue in bitbake-worker where stdout, reserved for status reporting, is improperly accessed by child processes. The problem occurs during the execution of parseRecipe, which calls anonymous functions. If these functions use print-like operations, they can inadvertently output data to stdout. This unexpected data can cause the runqueue to hang silently, if the stdout buffer is flushed before exec_task is executed. To prevent this, the patch redirects stdout to /dev/null and ensures it is flushed prior to the execution of exec_task. (Bitbake rev: 08f3e677d6af27a41a918aaa9da9c1c9b20a0b95) Signed-off-by: Yang Xu <yang.xu@mediatek.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: better estimate number of available cpusFelix Moessbauer2024-03-221-1/+4
| | | | | | | | | | | | | When running in a cgroup which is limited to a subset of cpus (via cpuset.cpus), cpu_count() should return the number of cpus that can be used instead of the number of cpus the system has. This also aligns the semantics with nproc. (Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix misleading "no output" msgViswanath Kraleti2024-03-221-1/+4
| | | | | | | | | | | When a command is run with a non-null log, errors are only output to the log and are not returned in the exception. In that case direct users to that logfile instead of telling the command had no output. (Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a) Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle URIs with single-valued query parametersRoss Burton2024-03-222-2/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Whilst typically the URI query is a list of key-value pairs, that's not actually required by the URI specification. For example: http://example.com/foo?bar is a valid query, but this will result in the fetcher raising an exception: File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__ self.query = self._param_str_split(urlp.query, "&") File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]: ValueError: not enough values to unpack (expected 2, got 1) In this case the query is just "bar", but the fetcher is trying to split it into a key-value pair. The URI object exposes the parsed query explicitly as a dictionary of key-value pairs, so we have to be a little creative here: if a value is None then it isn't a key-value pair, but a bare key. Fix this by handling elements without the deliminator in _param_str_split() (by assigning the value to None), and handle a None value when formatting the query in _param_str_join(). (Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: support NO_COLORPeter A. Bigot2024-03-222-2/+2
| | | | | | | | | | | | | Red text on a black background can make it difficult for people with visual impairments to read the text of error messages. Respect the presence of a non-empty NO_COLOR environment variable as an indication the user doesn't want colorization to be automatically enabled. See: https://no-color.org/ (Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85) Signed-off-by: Peter A. Bigot <pab@pabigot.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/git: Escape parentheses in git src namePhilippe Rivest2024-03-221-1/+1
| | | | | | | | | | | | | | | | | | FIXES [YOCTO #15404] When using git fetcher on a repo with parentheses in its URL, the invocation of the git clone command will fail. The clone directory is not quoted thus the shell will return an error and won't execute the command. (Bitbake rev: b5624ee5643d881afa004571a096a189ab5389b5) Cc: Philippe Rivest <privest@genetec.com> (Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6) Signed-off-by: Philippe Rivest <technophil98@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: allow '=' in environment variable valuesdavid d zuhn2024-03-222-3/+3
| | | | | | | | | | | | Limit the split to key & value (2 items) instead of the n items one can get if there are '=' characters in the value. Fixes [YOCTO #15447] (Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb) Signed-off-by: david d zuhn <david.zuhn@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Fix bug where umask 0 was not being applied to a taskSava Jakovljev2024-03-221-1/+1
| | | | | | | | | | | | | | | | | | * In the current implementation, "umask" variable is initially set to None and overwritten with user-specified value. However, in the worker implementation, a faulty if clause would only check whether the variable contains a value that evaluates to True, and not whether the variable is defined, so the value of 0 would lead to umask not being changed. This bug makes it impossible to have a task set its umask to value 0, for any possible reason it may want to. Fix this bug by extending the condition checked in the worker implementation. (Bitbake rev: 19f9df6c750c592316a0fa18165b68636281fe3e) Signed-off-by: Sava Jakovljev <sjakovljev@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: improve descriptions of '-S printdiff'Alexander Kanavin2024-03-222-4/+7
| | | | | | | (Bitbake rev: becf88c2250a47102c8d36ad8b40839e0bfa9137) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: remove BB_ENV_PASSTHROUGH from preserved_envvars()Michael Opdenacker2024-03-051-1/+0
| | | | | | | | | | | | | preserved_envvars() is used when the BB_ENV_PASSTHROUGH environment variable is not set. Therefore, its code shouldn't return this variable. (Bitbake rev: 0a33b560233b983456178541603ab96fea22238b) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reported-by: Robert P. J. Day <rpjday@crashcourse.ca> Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Fix a PID file removal race on prserv stopYoann Congal2024-03-031-1/+4
| | | | | | | | | | | | | | | | A race condition has happened where the exiting server removed the PID file between the existence check and the removal, resulting in a FileNotFoundError exception. The fix is to ignore the FileNotFoundError exception, the existence check is now redundant so remove it to simplify. Fixes [YOCTO #14341] (Bitbake rev: 40d00bf9308e0bf73a00134a99a012a292daa1c5) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Avoid clean upon failureRichard Purdie2024-03-032-2/+12
| | | | | | | | | | | | | | Currently when git fetches fail, it destroys all the existing local clone data. For large repositories this can introduce long build delays when for example, you just typo'd the git revision hash. The git fetcher should be able to recover most directories so when the fetch is for a git repo, avoid removing things unless clean is explicitly called (e.g. a -c cleanall task). (Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskexp_ncurses: fix execution example in introductory commentUlrich Ölmann2024-02-281-1/+1
| | | | | | | | | | | | | | Drop the ".py" file extension from the "-u" option's argument that has been overlooked while applying the original patch (see [1]) to make the example work. While at it sort the recipes' names consistently with respect to what is found in the self-test examples below. [1] https://lore.kernel.org/bitbake-devel/6f2645a7c4db2ae149d387544d2b94209cfed3f4.camel@linuxfoundation.org/ (Bitbake rev: 1f4d517b7a0389f78d1f791135f8dc9120e9912b) Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add support for server headersJoshua Watt2024-02-272-3/+46
| | | | | | | | | | | | Adds support for asyncrpc servers to send connection headers to clients on connection. Since this is a breaking protocol change, clients must opt-in to expect headers from the server, corresponding to a version bump in the client protocol. (Bitbake rev: 1cb2b8be6cc5269553f549285592e47b7d29db03) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: layerindexlib: fix missing layer branch backtraceTim Orling2024-02-271-4/+11
| | | | | | | | | | | | | | | | | | | | | When a LayerBranch (a specific layer at a specific release) does not exist in the layerindex database ("Layerindex Metadata"), the dependency would throw a backtrace. Instead fail early and provide an error message. Since layerindexlib will also check the local layers, inform the user that a local checkout might resolve the situation. Recommend that they reach out to the layer maintainers and layer index admins to properly fix it for everyone. While we are here, remove some trailing whitespace. [YOCTO #15365] [YOCTO #13954] (Bitbake rev: 96cbe8f87209a927c157ebcf469f8b9d54fcf92e) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure that git LFS objects are availablePhilip Lorenz2024-02-232-6/+90
| | | | | | | | | | | | | | | | | | | | | The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: progressbar: accept value over initial maxvalEnguerrand de Ribaucourt2024-02-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a very rare case where the maxval is improperly computed initially for cache loading progress, and the value will go over. Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:progress() # we might have calculated incorrect total size because a file # might've been written out just after we checked its size In that case, progressbar will receive a value over the initial maxval. This results in a ValueError stack trace as well as bitbake returning 1. Traceback (most recent call last): File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main cacheprogress.update(event.current) File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in update raise ValueError('Value out of range') ValueError: Value out of range This fix mirrors the behavior of MulticonfigCache and accepts the new value as the new maxval. This is also what the percentage printout is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:percentage() I encountered this issue randomly while working on a project with VSCode saving files while commands where fired. Note: This file is a fork from python-progressbar. It hasn't been refreshed in 8 years. We did only two commits, 5 years ago with minor modifications. This new change is also not how the upstream project is behaving. (Bitbake rev: 7cea7f7a87da041fc1ad370c5c3d15aabad3a0d4) Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add support for BB_LOADFACTOR_MAXRichard Purdie2024-02-231-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Some ditros don't enable /proc/pressure and it tends to be those which we see bitbake timeout issues on, seemingly as load gets too high and the bitbake processes don't get scheduled in for minutes at a time. Add support for stopping running extra tasks if the system load average goes above a certain threshold by setting BB_LOADFACTOR_MAX. The value used is scaled by CPU number, so a value of 1 would be when the load average equals the number of cpu cores of the system, under one only starts tasks when the load average is below the number of cores. This means you can centrally set a value such as 1.5 which will then scale correctly to different sized machines with differing numbers of CPUs. The pressure regulation is probably more accurate and responsive, however our graphs do show singificant load spikes on some workers and this patch is aimed at trying to avoid those. Pressure regulation is used where available in preference to this load factor regulation when both are set. (Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Re-enable connection pooling with psycopg 3 driverTobias Hagelborn2024-02-231-1/+5
| | | | | | | | | | | | Re-enable connection pooling in case `postgresql+psygopg` driver is used. Async connection pooling is supported in psycopg 3 [psycopg] driver in SQLAlchemy. Allow the connection pool to grow to arbitrary size. (Bitbake rev: 4fe05513b5314c201725e3f8ad54f58d70c56258) Signed-off-by: Tobias Hagelborn <tobiasha@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "bitbake: wget.py: always use the custom user agent"Alexander Kanavin2024-02-201-4/+3
| | | | | | | | | | | | | | | | | | This reverts commit 987ab2a446aab235d6e86e97c05f25fb800d7acc. There's been a report that this breaks downloads from Jfrog Artifactory as self.user_agent is set to 'Mozilla Firefox', and when Artifactory sees that, it sends a response tailored for showing in an interactive browser (which in my opinion it has every right to). If we're using wget, we should say so via wget's default; handling uncooperative servers should be done on per-recipe basis, and ideally with tickets to admins of those servers. (Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Make latest_versionstring extract tags with slashes ↵Peter Kjellerstedt2024-02-192-2/+9
| | | | | | | | | | | | | | correctly Before, everything up to the last slash was removed when extracting the names of the tags. This would lead to that a tag such as "agent/11.0.0" would be incorrectly identified as "11.0.0", which would then be treated as a correct version matching "^(?P<pver>\d+(\.\d+)+)". (Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: A bit of clean-up of latest_versionstring()Peter Kjellerstedt2024-02-191-11/+10
| | | | | | | | | This is mostly preparations for the next commit. (Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Make test_git_latest_versionstring support a max versionPeter Kjellerstedt2024-02-191-11/+14
| | | | | | | | | | | In some cases, the version found by latest_versionstring() may be higher than the real version. Make it possible to specify a maximum version so that this case can be detected. (Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: add comment in try_premirrorsChen Qi2024-02-191-0/+5
| | | | | | | | | | The purpose of ensuring 'incremental fetch' is not easy to see from the codes. So add comments to explain this. (Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: add test case for using premirror in restricted networkChen Qi2024-02-191-0/+19
| | | | | | | | | | | | We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch' failed even with all contents available in PREMIRRORS. Add a test case to ensure no regression in the future. (Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: fix a corner case in try_premirrorChen Qi2024-02-191-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For gitsm recipes, it's possible that some URL is used more than once. e.g., A -> B:rev1 (B is a submodule of A) A -> C (C is a submodule of A) C -> B:rev2 (B is a submodule of C) A anc C are both using B as submodules, but on different revs. Now if we have: B:rev1 -> D B:rev2 -> E Then, the mirror will not be fully used. Say we have all repo mirrors for A, B, C, D, E, then in theory it's not necessary to reach out to any network for downloading. But it's not the case. After downloading B(rev1) and its submodule D from mirrors, the fetch process continues to download C, thus B(rev2) and E. Now it finds that B needs an update because its submodule E needs an update. Of course this is true because E is not downloaded yet. Now the problem comes to whether to use mirror or not. The git.py defines try_premirror to return 'False' when the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and try_mirror returns False, resulting in not using mirror and going to upstream directly. We can see that the mirrors are not fully used. This is usually not problem, as the cost is only some network download. But in case the following two settings are there, we get errors. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.allowed.domain" In such case, the gitsm recipe A will fail to fetch. Note that all contents that A needs are in mirrors and now it's failing to fetch. This is unexpected. Note that the different revs of the same repo in gitsm recipe is not the only way to reveal this problem. For example, there might be a recipe call B that uses B:rev3. Check the protobuf and grpc recipes as an example. For now, we can use the following steps to reproduce this issue. To be clear, the grpc recipe in meta-oe is now 1.60.0. 1. Add in local.conf: DL_DIR = "${TOPDIR}/downloads-premirror" bitbake grpc -c fetch 2. Comment out the DL_DIR setting in local.conf and add the following lines: PREMIRRORS:append = " \ git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ " 3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'. This command succeeds and this shows that the premirror holds everything we need. 4. Add the following lines and then 'bitbake grpc -c fetch'. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.domain" After step 4, the error message is as below: ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used This patch fixes this problem by handling this corner case, that is, if the URL is not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should try premirrors because trying to reach upstream is destined to fail. (Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: add multiple fetches testPavel Zhukov2024-02-191-0/+18
| | | | | | | | | | | Fetch from premirror few times to emulate multiple machines sharing same clonedir or few rebuilds of the package from (pre)mirror Regression test for [Yocto #15369] (Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: Fetch mirror into HEADPavel Zhukov2024-02-191-1/+5
| | | | | | | | | | | | | | | Fix the issue with using of (pre)mirror in case if clonedir exists but outdated. Previous version of the code fetched new mirror content into FETCH_HEAD instead of branch which caused refetch from the upstream. Add new remote add fetch from it instead so the ref can be found by "_contains_ref" Fixes [Yocto #15369] (Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git-make-shallow: support git's safe.bareRepositoryAndré Draszik2024-02-191-16/+18
| | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the git-make-shallow fails miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: 7c63989db4590564516ed150930f4e2fa503e98f) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: support git's safe.bareRepositoryAndré Draszik2024-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the bitbake selftests fail miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: a45e14a7343e36101e45639931322e5649587f57) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git2: support git's safe.bareRepositoryAndré Draszik2024-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the bitbake git fetcher fails miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. While at it, fix one open-coded invocation of git that wasn't using ud.basecmd Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump version to 2.7.3 for hashserv changesRichard Purdie2024-02-192-2/+2
| | | | | | (Bitbake rev: c1e0a0b6ddc9667c9d62319bd9ccd4eb8c64c2a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: hashserv: Postgres adaptations for ignoring duplicate insertsTobias Hagelborn2024-02-191-15/+38
| | | | | | | | | | | | | | | | Hash Equivalence server performs unconditional insert also of duplicate hash entries. This causes excessive error log entries in Postgres. Rather ignore the duplicate inserts. The alternate behavior should be isolated to the postgres engine type. (Bitbake rev: e8d2d178d0fe96f9d6031c97328e8be17d752716) Signed-off-by: Tobias Hagelborn <tobiasha@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add parallel unihash exist APIJoshua Watt2024-02-191-0/+32
| | | | | | | | | Adds API to query if unihashes are known to the server in parallel (Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add parallel query APIJoshua Watt2024-02-191-34/+87
| | | | | | | | | | | | | | | | Implements a new API called get_unihashes() that allows for querying multiple unihashes in parallel. The API is also reworked to make it easier for derived classes to interface with the new API in a consistent manner. Instead of overriding get_unihash() to add custom handling for local hash calculating (e.g. caches) derived classes should now override get_cached_unihash(), and return the local unihash or None if there isn't one. (Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add Client PoolJoshua Watt2024-02-192-0/+163
| | | | | | | | | | Implements a Client Pool derived from the AsyncRPC client pool that allows querying for multiple equivalent hashes in parallel (Bitbake rev: ba4c764d8061c7b88cd4985ca493d6ea6e317106) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add Client Pool objectJoshua Watt2024-02-192-1/+78
| | | | | | | | | | | | Adds an abstract base class that can be used to implement a pool of client connections. The class implements a thread that runs an async event loop, and allows derived classes to schedule work on the loop and wait for the work to be finished. (Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add unihash-exists APIJoshua Watt2024-02-196-33/+151
| | | | | | | | | | | | | | | Adds API to check if the server is aware of the existence of a given unihash. This can be used as an optimization for sstate where a client can query the hash equivalence server to check if a unihash exists before querying the sstate cache. If the hash server isn't aware of the existence of a unihash, then there is very likely not a matching sstate object, so this should be able to significantly cut down on the number of negative hits on the sstate cache. (Bitbake rev: cfe0ac071cfb998e4a1dd263f8860b140843361a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: sqlalchemy: Use _execute() helperJoshua Watt2024-02-191-157/+140
| | | | | | | | | | Use the _execute() helper to execute queries. This helper does the logging of the statement that was being done manually everywhere. (Bitbake rev: 0409a00d62f45afb1b172acbcea17bf17942e846) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add Unihash Garbage CollectionJoshua Watt2024-02-196-116/+684
| | | | | | | | | | | | Adds support for removing unused unihashes from the database. This is done using a "mark and sweep" style of garbage collection where a collection is started by marking which unihashes should be kept in the database, then performing a sweep to remove any unmarked hashes. (Bitbake rev: 433d4a075a1acfbd2a2913061739353a84bb01ed) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: improve the loglevel error message to be more helpfulPaul Gortmaker2024-02-191-1/+1
| | | | | | | | | | | | | | | | | | Coming from a kernel background, I was thinking along the lines of dmesg -n <integer> for loglevel adjustments. So I tried various large and small and even zero number values with no luck before getting frustrated and opening up the python. Let us save others the frustration and give a hint what the args it expects should look like. (Bitbake rev: df184b2a4e80fca847cfe90644110b74a1af613e) Signed-off-by: Paul Gortmaker <paulg@kernel.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskexp_ncurses: ncurses version of taskexp.pyDavid Reyna2024-02-181-0/+1511
| | | | | | | | | | | | | | | | * Create an ncurses version of the GTK app "taskexp.py". * Add these additional features: - Sort tasks in recipes by their dependency order - Print individual and/or recipe-wide dependencies to a file - Add a wild card filter - Show the target recipes on BOLD * Provide a GUI self test * Provide a non-ncurses self test for ptest (Bitbake rev: f49bec66ad51c8cddeceafbbe2445c46e396ee8b) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve setcene performance when encoutering many 'hard' ↵Richard Purdie2024-02-131-1/+11
| | | | | | | | | | | | | | | | | | dependencies "bitbake world -n --setscene-only" shows poor performance as the numbers of tasks increases. Analysys shows this is due to the "deferred" hard dependencies being repeatedly processed which doesn't allow much forward progress in overall task execution. To avoid this, mark when it has been done and don't reprocess until dependencies are updated. We have to be careful as we've seen bugs where these dependencies aren't processed at the right time. They also have to be reproceseed during task migration/rehashing so the code has to "self heal" the data structures. (Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise taskname lookups in next_buildable_taskRichard Purdie2024-02-131-3/+3
| | | | | | | | | | A quick profile of bitbake world showed 147 million calls to taskname_from_tid(). The next_buildable_task function is performance senstive so move the call inside the if block to reduce the number of calls and speed the code up. (Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-26 19:15:13 +0000