summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: documentation: bitbake: add file-checksums to varflags sectionRichard Elberger2023-02-261-0/+17
| | | | | | | | | | | | | Fixes [YOCTO #11605] by: - Adding definition of file-checksums to Variable Flags section. - Describe data to add to list which adds external file dependencies. - Write example on usage to prepend a value to file-checksums list. (Bitbake rev: 6bc65e6402a74c9a65e29333a7e0f1f85dcbcf58) Signed-off-by: Richard Elberger <rich@richelberger.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/npmsw: add more short forms for git operationsMark Asselstine2023-02-262-7/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | >From the npm-install documentation [1] the CLI provides a set of short forms when the install fetches from git. These include "github:" example: npm install github:mygithubuser/myproject "gist:" example: npm install gist:101a11beef "gitlab:" example: npm install gitlab:mygitlabuser/myproject "bitbucket:" example: npm install bitbucket:mybitbucketuser/myproject Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] by Stefan Herbrechtsmeier added support for the "github:" but the others would marked as 'Unsupported dependency'. The other prefixes are added in this commit, along with extending the tests to cover some of these. However, there is one more short form for github which npm-install allows which forgoes the prefix altogether. example: npm install mygithubuser/myproject Unfortunately this format is a bit problematic as it lacks any easily identifiable 'marker' to match against, and it could be either the github short form or install from folder format. Experimentation shows that the folder format requires a leading './' or '/', so we use this to rule out the ambiguity. If this approach to folder and github formats disambiguation is incorrect it won't matter anyways as the folder format is unrecognized by the code as-is and thus with this change or without, things would fail. Since we have to be less strict in the check for git operations we move it to be the last install format which we check, such that the less ambiguous formats can be sorted out first. [1] https://docs.npmjs.com/cli/v9/commands/npm-install [Yocto #14236] (Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix path construction for relative submodule URIMike Crowe2023-02-261-1/+1
| | | | | | | | | | | | | | | | | | | The submodule repository URI contains a path to something not necessarily on the local filesystem. This means that we can't use realpath to normalise it without risking getting bad results if the path happens to match something on the local filesystem. This situation can cause very confusing errors if that matching local path happens to be a symlink to somewhere else. Using normpath rather than realpath means that the path simplification follows simple rules on the string rather than looking at the local filesystem and avoids problems. (Bitbake rev: 47b271e6c8d96960ebe70f80e58f30cc4cbf42e1) Signed-off-by: Mike Crowe <mac@mcrowe.com> Co-authored-by: Dave Craig Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Use internal fetcher function to avoid duplicationRichard Purdie2023-02-261-15/+4
| | | | | | | | | | We currently have two lists of "proxy" or "fetcher" environment exports. Make the one in utils match the one on the fetcher which has a more complete list of variables now. (Bitbake rev: f9165a798a307a6f0fee120d5c3de660d3a44ae8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Drop unused importRichard Purdie2023-02-261-1/+0
| | | | | | | | This import is no longer used anywhere so can be removed. (Bitbake rev: 956128e394581855bf0d03b32a975dc91c2a7e0c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARSCaner Altinbasak2023-02-261-1/+3
| | | | | | | | | These environment variables are needed by gclient and needed to be passed into fetcher. (Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix memory resident cache invalidation issueRichard Purdie2023-02-241-6/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've been seeing weird PRServ failures on the autobuilder. These had one failure always followed by a second. Whilst I can't reproduce the first, if I made that test fail, I could reproduce the second with memory resident bitbake. This was with the tests: prservice.BitbakePrTests.test_import_export_replace_db and then prservice.BitbakePrTests.test_pr_service_deb_arch_dep which was giving a strange looking error: NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa) ERROR: No such task: do_package_write_rpm ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1' where the issue is that selftest.inc written by the test framework and containing PACKAGE_CLASSES = "package_deb" was being ignored. The issue is the cached_statements{} within BBHandler() is not being invalidated at the right time. This patch changes the code to ensure base configuration is not parsed until inotify updates have been processed. (Bitbake rev: cada37c6b9e5862ca2c5a54ad6fd1e1f1939cd9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/siggen: fix debug() callRoss Burton2023-02-241-1/+1
| | | | | | | | | | | | | | | Bitbake f68682 changed the logger's debug() method to be compatible with logging.debug(), but this caller was still using the old API where you can pass an integer as the first argument: WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]') Instead, call bbdebug() which has the priority argument. (Bitbake rev: 18d4f9e8387f7994cf6d46300e25dda1c3a593b2) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.3.1Richard Purdie2023-02-232-2/+2
| | | | | | | | | So that OE-Core can depend on bb.event.check_for_interrupts(), bump our verison number to a development series version. (Bitbake rev: dea1b2f3fc31f28daed5da16e62da8895d6e5716) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/cooker/runqueue: Add ability to interrupt longer running codeRichard Purdie2023-02-233-0/+30
| | | | | | | | | | | | | | | | | | | | Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Replace mkdtemp with TemporaryDirectory and avoid ↵Paulo Neves2023-02-221-4/+1
| | | | | | | | | | | | | | exception masking Due to using mkdtemp instead of TemporaryDirectory we needed to manually cleanup the directory in a try finally block. With tempfile.TemporaryDirectory we can handle the cleanup with a "with" statement and not need to manually clean up oursevels. (Bitbake rev: 36a231d2a45e7c87172196538d18ded81a234774) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Removed unused variables in _contains_lfsPaulo Neves2023-02-221-5/+0
| | | | | | | | | | branchname was set but never used in the context of _contains_lfs method. (Bitbake rev: a64ec64c2faf1ac6ce995cdd4e311d8be6046155) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add real git lfs tests and decoratorPaulo Neves2023-02-221-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. (Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: git-lfs restore _find_git_lfsPaulo Neves2023-02-221-16/+24
| | | | | | | | | | Not restoring the mocked _find_git_lfs leads to other tests failing. (Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Improve idle thread exception handlingRichard Purdie2023-02-201-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | If the inotifier code has an exception, bitbake currently hangs. Catch any exception and exit if seen. Also check the idle thread is alive and exit if it disappears. This should stop bitbake hanging if such a situation arises in future such as this example: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) (Bitbake rev: 358b5b02d5de1ab0f98104c4ec4953e46999b9a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure lock is held with changing notifierRichard Purdie2023-02-201-16/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | We've seen a couple of cases which bitbake hangs due to an inotifer exception such as: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) Ensure we don't destory the inotifier when the idle thread is reading is by holding the lock during setup/teardown. (Bitbake rev: 8fc5c50c2e23017833f93bcd514d708a14fa4266) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop SystemExit usageRichard Purdie2023-02-191-2/+1
| | | | | | | | | Using bb.fatal for a fatal error message is the best practise, switch the code to match other call sites. (Bitbake rev: c27e48fa81c2327a4a355a028884ab457cde3ae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Drop dubious exception handling codeRichard Purdie2023-02-191-5/+0
| | | | | | | | | | | This code appears to be dangerous, it swallows exceptions, turning them into "handled" versions which then show no errors to the user. This is a pretty poor user experience and I can't see why this code should be swallowing such things. Drop the worst bits of code. (Bitbake rev: 13279044f16f2cf2502ebf39d277415f99bb6c18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Improve early exception handlingRichard Purdie2023-02-191-9/+1
| | | | | | | | | | | | | | | | Martin Jansa reported that if you put a syntax error into an imported module such as qa.py in OE, no error is shown. Part of the issue appears to be that the catch_parse_error() decorator only catches certain exceptions and SyntaxError isn't one of them. As far as I can tell we should remove all the special cases and use the more advanced code in all cases, not just expansion errors. I confirmed this now prints a proper error message for a qa.py syntax error. (Bitbake rev: 2365d891847f8e73d1c4661ddfdab8818ff619dc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: action.py: add topdir in bblayers_conf path for remove-layerPedro Baptista2023-02-171-1/+1
| | | | | | | | | | | By including the full path to bblayers.conf the remove-layer command can be executed from any location, not only from the build directory. (Bitbake rev: 25cb4e17e8a4c0310df018b9df184b0155b267eb) Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: action.py: add topdir in bblayers_conf path for add-layerPedro Baptista2023-02-171-1/+2
| | | | | | | | | | | By including the full path to bblayers.conf the add-layer command can be executed from any location, not only from the build directory. (Bitbake rev: 77aa230bf2a053fc0941723a6abbc798ebe53a19) Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: clean up netrc usageRoss Burton2023-02-171-4/+5
| | | | | | | | | | | Assigning a return value which is potentially None to a tuple and catching TypeError is pretty ugly. Rewrite the code to explicitly check the value for clarity. (Bitbake rev: f4ebb27616ac2df27c29a6052b1526a4c48db607) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Evaluate the value of export/unexport/network flagsRichard Purdie2023-02-172-6/+6
| | | | | | | | | | | | | | | | | | | Currently the export/unexport/network flags are acted on only by presence or lack of in the data store. This is deliberate due to performance reasons since a given recipe shell task might export ~80-90 variables and this means expanding flags for every one of them. This does catch users unaware since values which expand to nothing don't work e.g. ${@""} and setting values to "0" wouldn't work either. The downside to this patch is slow down in parsing speed of around 4%. The network flag is easier to change and doesn't affect performance, it was made to operate the same as export/unexport for consitency reasons. (Bitbake rev: 3d96c07f9fd4ba1a84826811d14bb4e98ad58846) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Remove incorrect SystemExit usageRichard Purdie2023-02-171-1/+1
| | | | | | | | | | | Calling SystemExit doesn't work well with server/client usage since the string isn't printed to the right place. Use bb.fatal() instead which prints the right log output and raises and handled exception which then shows correctly on the UI. (Bitbake rev: aaefb43b41a0d9b16a59643136268eb6e5d48cd2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add NODE_EXTRA_CA_CERTS to export listGeorge Kelly2023-02-091-0/+1
| | | | | | | | | | | | | | | | If you are behind a corporate proxy, the npm fetcher uses the proxy IP address already passed in the list of exports. However, it will error if the proxy injects its own root CA certificate. Thus, the NODE_EXTRA_CA_CERTS environment variable must be passed so the user can include their company's root CA as a trusted CA inside node's certificate store. (Bitbake rev: 5daecc25a59ce5bf6c31f5085e6caa7cb543517e) Signed-off-by: George Kelly <george.kelly1097@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix inefficient string concatenationEtienne Cordonnier2023-02-041-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython has an optimization for statements in the form "a = a + b" or "a += b". It seems that this line does not get optimized, because it has a form a = a + b + c: data = data + "./" + f.split("/./")[1] For that reason, it does a copy of data for each iteration, potentially copying megabytes of data for each iteration. Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds instead of 45 seconds on my test setup where SRC_URI points to a big directory. Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython: "do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b" However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup: data = ''.join((data, "./", f.split("/./")[1])) I have changed the other lines to also use += for consistency only, however those were in the form a = a + b and were optimized already. Co-authored-by: JJ Robertson <jrobertson@snap.com> (Bitbake rev: 195750f2ca355e29d51219c58ecb2c1d83692717) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: show how use BB_LOGCONFIG to log warningsMichael Opdenacker2023-02-041-4/+41
| | | | | | | | (Bitbake rev: 22be2dbd8e70322239f70e53ace2a552425e3665) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Suggested-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bblayers/query: Adjust show-layers output layoutPedro Baptista2023-02-041-3/+3
| | | | | | | | | | | Increase the `ljust` value for each column in show-layers output. This improves readability when long layer paths are included (Bitbake rev: a56cd53e0e67a368acace9dab184bd94b9b67b1b) Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bblayers/query: Replace layer directory name with layer name for ↵Pedro Baptista2023-02-041-2/+2
| | | | | | | | | | | | show-layers Replace the layer directory name with the name from BBFILE_COLLECTIONS which is useful when assigning a LAYERDEPENDS value (Bitbake rev: 517cc3f5aee2a7d6362ce0202d4ee709a5378096) Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: include SSL certificate paths in export_proxiesRoss Burton2023-01-261-12/+9
| | | | | | | | | | | | | | | | | | | | | | | | bb.utils.export_proxies() is a poor-man's alternative for the environment setup code in bb/fetch2, but it's used in several places where recipes want to download manually (such as cve-update-db-native). Notably, export_proxies() doesn't pass on the SSL certificate paths from the original environment, so if SSL_CERT_FILE needs to be set (for example, in a buildtools environment) then proxies work but SSL doesn't. In an ideal world export_proxies and the same logic in fetch2 would merge, but until then we can add the SSL_CERT_ variables and duplicate the basic logic: check the datastore first and then the original environment for variables. Also remove the return value as nothing ever checked it. [ YOCTO #15000 ] (Bitbake rev: c19035e8e71c419c5688a86bfc9c946c96f638e8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache ↵Richard Purdie2023-01-265-11/+17
| | | | | | | | | | | | | | | | | | | | | | | | location Currently the codeparser cache is set from CACHE, which is typically in bitbake.conf which means we can't read/write any cache until it is found/read. We may well have python expressions to parse before that happens. The net result is suboptimal functioning of the codeparser cache since it will often be invalidated by data that is never written. This patch changes the codeparser and filechecksum caches to use BB_CACHE as their setting and defaults it to ${TOPDIR}/cache. The patch doesn't change where the "persistent" data such as prserver and hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing cache resists (still currently CACHE). I've left those for a later patch. The patch does ensure data parsed by the core datastore parsing calls is written back since this is now much more useful after this change. (Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: update fixturesTim Orling2023-01-263-21/+75
| | | | | | | | | | | | * Update to latest langdale 4.1.2 and kirkstone 4.0.6 * Re-instate dunfell and update to 3.1.22 - drop comments about bitbake crash (Bitbake rev: f90c45df5cf4640e0714fde13e311db3327f9fd4) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-getvar: Add a quiet command line argumentPaulo Neves2023-01-261-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | bitbake-getvar does not have a way to silence bitbake server's logger and that makes the tool hard to use for text processing. This is especially true when one wants to get a bitbake value to be piped to some other utility and instead we get uncontrolled logging messages or warnings together with bitbake's variable value. Example without quiet: bitbake-getvar --value MACHINE NOTE: Starting bitbake server... qemux86-64 With quiet: bitbake-getvar --value MACHINE --quiet qemux86-64 (Bitbake rev: af354e975d0b4c26d0e91e3c82946b093bc11b45) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fix deprecated threading.Thread.setDaemonTim Orling2023-01-244-4/+4
| | | | | | | | | | | | | | | | | Deprecated in Python 3.10: https://docs.python.org/3/whatsnew/3.10.html#deprecated https://github.com/python/cpython/pull/25174 Fixes warnings like: ...bitbake/lib/bb/ui/uievent.py:68: DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead self.t.setDaemon(True) (Bitbake rev: 323f6ce27a1bfd7159e72f29684674ff495dedee) Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Small optimisation to _findVar()Richard Purdie2023-01-241-7/+9
| | | | | | | | | | | | | Some users of _findVar don't need the override data and even getVarFlag doesn't need it in some common cases (parsing=True). Rearrange the code as the current overridedata approach doesn't need to be in the _findVar code anyway. This removes some search overhead from a critical path. (Bitbake rev: fcb64e1138a20eb19560af3fc5d1fa748cc9cf34) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Add support for new BB_HASH_CODEPARSER_VALS for cache ↵Richard Purdie2023-01-242-12/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | optimisation Currently the codeparser cache ends up being extended for every parse run since there are values in the functions such as the result of os.getpid() from LOGFIFO in OE-Core. Digging into that issue, there are also lots of similar but different functions being parsed where the change might just be a path to WORKDIR, a change in PN or PV or something like DATE/TIME. There is no reason we have to use these changing values when computing the dependenies of the functions. Even with a small tweak like: BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn" the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run to around 1.3MB and remaining static for oe-core and meta-oe. In my local build, admittedly heavily experimented with, the cache had grown to 120MB. The benefits of doing this are: * faster load time for bitbake since the cache is smaller to read from disk and load into memory * being able to skip saving the cache upon shutdown * lower memory footprint for bitbake * faster codeparser data lookups (since there is less data to search) We only use these special values when passing code fragments to the codeparser to parse so the real variable values should otherwise be used in the hash data. The overall effect of this change, combined with others to avoid saving unchanged cache files can be ~2s on a ~16s parse on my local system and results in a more responsive feeling bitbake. It also allows parsing performance to be investigated more consistently. (Bitbake rev: f24bbaaddb36f479a59a958e7fc90ef454c19473) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix parsing race around cache handlingRichard Purdie2023-01-241-8/+8
| | | | | | | | | | | | | When draining the result queue from the parsing processes, cache objects can be created even if they are then immediately destroyed. The reset in the sync code needs to happen after any objects have been created. Change the ordering to fix this. This ordering has caused various cache errors, particularly when interrupting parsing with Ctrl+C. (Bitbake rev: f45a94e6720dacf7f51ac147c115a6f608769093) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Only write files if we have dataRichard Purdie2023-01-241-3/+11
| | | | | | | | | By writing the cache files only if there is data to write, we can save a bit of time. (Bitbake rev: abeff1f80bb1c690b92d535d472dff9df7a56067) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix siggen recipe cache race issueRichard Purdie2023-01-241-2/+1
| | | | | | | | | We need to reset the cache before the start() call which starts the parsing processs, tweak the code to ensure this is the case. (Bitbake rev: 4712d6237df5d2188238294fbdb9f107b068a3a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix lockfile contents check bugRichard Purdie2023-01-241-1/+1
| | | | | | | | We need to check against the first line of the file, fix the typo. (Bitbake rev: 4abc598fb01d426394f4222dfc752e620a8e1b7b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/tests: Add parameter to recipe_uriPavel Zhukov2023-01-201-1/+1
| | | | | | | | | | | | While the parameter is not required it allows testing of possible regression in fetcher code when parameter specified and mirrors are used. (Bitbake rev: 1a21918049091a6d77426dbf8868ffdc14ba1003) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/tests: Add test for MercurialPavel Zhukov2023-01-181-0/+44
| | | | | | | | | | | Basic test for mercurial functionality including PREMIRRORONLY with disabled network (Bitbake rev: c0e349a5fc92e9c90afc53800c7e83495393ff79) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: kill parameters in uri_find_decodedPavel Zhukov2023-01-181-0/+1
| | | | | | | | | | | | | Fixes: File "/bitbake/./lib/bb/fetch2/__init__.py", line 446, in uri_replace if uri_decoded[loc][k] != uri_find_decoded[loc][k]: KeyError: 'module' (Bitbake rev: 80421abd5bf3f1f751219c050fa93970cf1f2088) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Improve lockfile handling at exitRichard Purdie2023-01-141-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | If memory resident bitbake is active and the build directory is renamed upon build completion, several bad things can happen: * the old build directory could be re-created to contain a lockfile leaving an empty directory behind * a lockfile for a new build could be found and attempt to be locked This patch avoids creating an empty directory (not perfectly, but should work in the majority of cases - an empty directory is cosmetic). It also now compares the lock file contents to it's own pid and just exits if it doesn't match, it is clearly then belonging to some new process. This will be combined with bitbake shutdown calls on the autobuilder to ensure "saved" build directories, or build directories being deleted by clobberdir don't do strange things. (Bitbake rev: b986eac18b6a8bf633f5ef15f32f68de4c86173b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Handle sqlite error when cachefile path is too longYoann Congal2023-01-131-1/+20
| | | | | | | | | | | | | | | | | | | Sqlite can't open the cachefile when its path is too long (>= 505 char by my tests). We do have a path length check in sanity.bbclass but this code is called before sanity checks. Treat the error in the exception instead of checking before hand in case sqlite eventually fixes this. Fixes [YOCTO #12374] (Bitbake rev: bf681d173263cd42ffc143655f61abe0573fd83c) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Move heartbeat to idle threadRichard Purdie2023-01-111-22/+22
| | | | | | | | | | Rather than risk the heartbeat event code locking up the server control socket, handle it in the 'idle' thread with the other work. The aim is to remove it as a possible issue with some ongoing hangs. (Bitbake rev: 0f9a0c7853b181817bf01863a26da21412376294) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Move --buildfile help at the end of "Execution" groupYoann Congal2023-01-111-4/+3
| | | | | | | | | | | | | | | Often, '--buildfile/-b' is used to mean "Do not execute anything but this recipe", so it make sense to have it in the "Execution control options" group. Fixes [YOCTO #12018]. (Original suggestion by Ross Burton) (Bitbake rev: 412a6d8701a92efe4dd6db8ec713c0013f7db64d) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix exit handling issuesRichard Purdie2023-01-111-6/+4
| | | | | | | | | | | | | The shutdown() call should definitely not be waiting for idle, since we expect execution and events to continue even after setting either shutdown state. This was causing the UI to appear to hang at Ctrl+C interrupts. Also ensure that if we reset cooker, we stop any active parser processes since these currently appear to be being left behind. (Bitbake rev: 4f73c2eb12ee074f3b7d4717380649f6ca8f3def) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Improve shutdown handlingRichard Purdie2023-01-111-5/+10
| | | | | | | | | | | | There are three levels of shutdown, the initial "wait for current tasks", then "stop current tasks" and "exit now". Change the code so that we don't instantly exit after "stop current tasks" but allow the events to come through from the server first. The new shutdown level allows that to then be broken out of too. (Bitbake rev: af38345b91cfc8a6bb5c38f5753255e7635bfe2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: Don't wait for events indefinitelyRichard Purdie2023-01-111-1/+7
| | | | | | | | | | If for some reason the bitbake server dies, we shouldn't loop indefinitely waiting for events within tinfoil. Add a ping test and exit if things have somehow failed. (Bitbake rev: 32da768ae67cfe0b2d154ce4667a7cfdfa5bf4b0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-14 09:49:15 +0000