summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: Force parser shutdown after catching an exceptionJan Brzezanski2021-03-241-5/+5
| | | | | | | | | | | | | | | | | | | Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. (Bitbake rev: 5d02c98489d3a5836676b9c3fb3bd0157449db2b) Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 915330e1dbae1ee8fd9a0358decf2c294f771961) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add setscene task overlap sanity checkRichard Purdie2021-03-241-0/+4
| | | | | | | | | | | | | We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. (Bitbake rev: 947c2ad65e4c59d01c603f767f11f9aa86b69264) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task execution corruption issueRichard Purdie2021-03-241-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. (Bitbake rev: 0a61f86a8da5356ce0d894ef2ffb96ac6a219db5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: __init__.py: Fix bitbake debug log handlingRichard Purdie2021-03-091-1/+1
| | | | | | | | | | | | For a while I've been puzzled as to why debug logging from runqueue wouldn't appear on the console with -DD. The logic in the bbdebug handling is inverted so fix it and now we see the expected messages from runqueue with -D and -DD. This should then let us debug other issues using those log messages. (Bitbake rev: 7a6517243dbfecfd78f82f6709fbeaab2fb03e2c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/fetch2/__init__.py: drop _PYTHON_SYSCONFIGDATA_NAME unsettingAlexander Kanavin2021-02-111-5/+0
| | | | | | | | | | | | | With introduction of python3targetconfig class in core this is no longer needed. (Bitbake rev: 0a3bf681530bd63fc0036ca81ef868ab53fde56c) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 47b64cfacd7c498ef9ed5486d117f2d69a39f225) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: download LFS content too during do_fetchMatt Hoosier2021-02-022-11/+61
| | | | | | | | | | | | | | | | | Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] (Bitbake rev: c73f8f2f4a6491c6bea54839630af6994c27ad24) Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Ensure hash reflects vardepvalue flags correctlyRichard Purdie2021-01-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | The get_hash() function is used to decide if the base configuration has changed and hence whether a reparse is required. The vardepvalue flag's value was not expanded but it is often used in contexts like: METADATA_REVISION = "${@base_detect_revision(d)}" METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" which in it's unexpanded form means reparsing doesn't happen when it should as the data appears unchanged. Update get_hash to expand the values of vardepvalue so reparsing works as expected. This avoids basehash mismatch errors such as the one recently caused by using METADATA_REVISION in poky.conf's DISTRO_VERSION variable. The issue there could be exposed by a recipe using DISTRO_VERSION with the sequence: bitbake os-release <change the revision of the metadata with a dummy commit> bitbake os-release -C install which was caused because METADATA_REVISION changed but the metadata didn't reparse. (Bitbake rev: cb7277e7bb3a440968fdb918d56fe8fa17bca2e6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Handle cooker daemon startup errorStacy Gaikovaia2020-10-261-2/+4
| | | | | | | | | | | | | | | | | | | | | On startup, bitbake spawns a cooker daemon and waits for it's acknowledgement signal. If the acknowledgement doesn't happen in time,the bitbake object will fail to initialize and exit. The error that occurs in this case isn't handled by the existing try - catch block because SystemExit inherits from a different base Exception class. This commit adds SystemExit to the list of expected bitbake server startup errors. [YOCTO #13993] (Bitbake rev: fec2b85689bba1d26ad6f376bc11cc29bb27cbe5) Signed-off-by: Stacy Gaikovaia <stacy.gaikovaia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update upstream master->main branchname transitionRichard Purdie2020-10-261-1/+1
| | | | | | (Bitbake rev: 8b9471e02528320f6ef8d35840b5618883e85447) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Update documentation links to new URLsReyna, David2020-10-197-13/+13
| | | | | | | | | | | | Update the Toaster documentation links to the new Yocto Project URLs. [YOCTO #14092] (Bitbake rev: 9345d257ced432adc2d16af20ace58cc7c086aab) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump version to 1.48.0 ready for the new releaseRichard Purdie2020-10-191-1/+1
| | | | | | (Bitbake rev: f421de9effc2ba40145373881d20b8e823cf23f8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/toasterui: Fix startup faults from incorrect event sequencingRichard Purdie2020-10-171-2/+4
| | | | | | | | | | | | | | | | | | | | | | Toaster has been failing to start correctly when in interactive mode. The issue is due to setEventMask being called (which triggers parsing) before the environment has been sent from the UI over to the server. This means PATH isn't setup, which causes the sanity checks on HOSTTOOLS to fail in base.bbclass. The fix is to ensure the environment is sent to the server before other commands are run. The pain in debugging this highlights other improvements to the logging are needed. [YOCTO #14079] Reviewed-by: Tim Orling <timothy.t.orling@linux.intel.com> Tested-by: Tim Orling <timothy.t.orling@linux.intel.com> (Bitbake rev: a8efff5c83cd5a25f4b6720e6414a7aa35d04bc7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Enable Gatesgarth branch in place of ZeusDavid Reyna2020-10-172-15/+15
| | | | | | | | | | | | | Toaster directly supports the last two stable branches of Yocto Project. With "Gatesgarth " being released, it is time to replace "Zeus". [YOCTO #14086] (Bitbake rev: de578cd8f8b582c7bea83c75d06b7f667c07578a) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/ui/knotty: fix typo in parseprogressTim Orling2020-10-171-1/+1
| | | | | | | | | | | After parseprogress.finish() it was intended to set parseprogress to None, but a typo means this is not happening. (Bitbake rev: f504d6f6598f62aa20fbf69c30fea95569858edb) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI urlCharlie Davies2020-10-171-6/+7
| | | | | | | | | | | This commit replaces the instances where escaped double quotes are used to support SRC_URI url containing spaces with the more pythonic shlex.quote(). (Bitbake rev: 4f9ba9c794de55bea0343267467bddea99844374) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Show command exceptions in the server log as wellRichard Purdie2020-10-111-0/+1
| | | | | | | | | | There are autobuilder logs where the server commands are failing but we have no debug info in the server log. Improve this to try and understand what is failing. (Bitbake rev: 04d3a79226c9ea448b22f4efbab33876a72c9bdb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: When sending commands we need to process eventsRichard Purdie2020-10-101-1/+10
| | | | | | | | | | | | | The server may be displaying useful information for the user through log messages so we should display anything that has been sent. Its either this or expecting every UI to implement this code around every command call which isn't good API. [YOCTO #14054] (Bitbake rev: 64ae9d7e2fad804dd9e12706c6d76b4b22f9586b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: Ensure exceptions inheriting from BBHandledException are ↵Richard Purdie2020-10-101-1/+5
| | | | | | | | | | | | | | visible Previous changes allowed BBHandledException to be detected but not exceptions which inherit from it. Fix this. The code really needs totally reworking to preserve the exceptions. [YOCTO #14054] (Bitbake rev: ef762d92df6c2554c6248e80212f984d9ec4c651) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid tracebacks if data was never setupRichard Purdie2020-10-011-1/+2
| | | | | | | | Recent changes mean data might not be setup. If its not, avoid tracebacks. (Bitbake rev: 3daff610d9f39d73c80c54d1df46f573666e20db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: use correct umask when writing siginfoRoss Burton2020-09-301-1/+2
| | | | | | | | | | | | | | | We try to write sstate with group-write permissions so that sstate-cache can be shared between multiple users. However the siginfo files are created in various tasks which may set their own umask (such as do_populate_sysroot, 0022). This results in no group write permission on the intermediate directories, which is fatal when sharing a cache. Fix this by wrapping the siginfo mkdir in a umask change to 0002. (Bitbake rev: 75d9ef04a908e366633b255d23ab3275f6860d3a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add umask changing context managerRoss Burton2020-09-301-0/+11
| | | | | | | | | | Add a umask context manager which can be used to temporarily change the umask in a 'with' block. (Bitbake rev: 6c601e68a27e1c60b04c2a61830d1812cc883e09) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: backslash support in file:// URIsLeif Middelschulte2020-09-251-0/+5
| | | | | | | | | | | | Implements backslashes in local filenames. A typical usecase for such a filename is a systemd unit. Example: `dev-disk-by\x2dlabel-FOO.device` (Bitbake rev: 14a35f273b579d5cd5fd92765b89c28f870dd577) Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix handling of `\` in file:// SRC_URILeif Middelschulte2020-09-251-1/+1
| | | | | | | | | | | | | Using backslashes in file:// URIs was broken. Either the resolver would fail or the subsequent `cp` command. Try to avoid this by putting the filenames into quotes. Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=8161 (Bitbake rev: aa857fa2e9cf3b0e43a9049b04ec4b0b3c779b11) Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskexp: update for GTK API changesRoss Burton2020-09-241-1/+6
| | | | | | | | | | | | | | | | | | | The introspected API for GtkTreeModel and friends had some unexpected quirks which have now been fixed, see[1] for details. However, for example Ubuntu 20.04 has the fixed GTK but not an updated pygobject which means taskexp raises an exception on startup. Solve by manually looking at what functions are present and calling the right one. [ YOCTO #14055 ] [1] https://gitlab.gnome.org/GNOME/pygobject/-/commit/9cdbc56fbac4db2de78dc080934b8f0a7efc892a (Bitbake rev: ac7d1114a7e99e6efd6a37b03d170faf678513fb) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: siggen: clean_basepath: improve perfo and readabilityJean-Francois Dagenais2020-09-231-10/+20
| | | | | | | | | | | | | | | This change improves performance by reducing runtime about 33% for typical inputs. (using test_clean_basepath_performance) It is also easier to read, and slightly more resilient to future changes since it doesn't mention 'virtual' anymore. (Bitbake rev: 27b53186fa67d281d29b2f8e15bcff8dc2557b8a) Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com> Co-Developed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/siggen: introduce clean_basepath testcasesJean-Francois Dagenais2020-09-231-0/+91
| | | | | | | | | | | | | | | While discussing with Richard we thought these might help document and safeguard the basic requirements of clean_basepath. A 'bonus' performance testcase is added but commented out since its runtime is long and test machine specific. It is intended for developers to test before and after their changes to the target function as a due diligence verification. (Bitbake rev: ee41549f26952d5f7af19a9b3d8a8b969866e2ef) Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: add unit tests for SRC_URI with spaces in urlCharlie Davies2020-09-231-0/+47
| | | | | | | (Bitbake rev: e7dab75c8d1923abcbbc7c9ac7de215d720ccf26) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch/git: add support for SRC_URI containing spaces in urlCharlie Davies2020-09-231-6/+6
| | | | | | | | | | | | | | | | | | | Microsoft's TFS VCS system allows for spaces in a git repository url. An example of a valid url is: ssh://tfs-my-company.org:22/tfs/My Projects/FooBar This commit adds support for such urls by implementing two changes. Firstly, when bitbake makes a git command line call the url is surrounded by quotes so that the url, regardless of spaces, is treated as one argument. Secondly, additional parsing of various filepath variables, which are based off of the url, are now completed with any spaces in the url replaced with underscores. (Bitbake rev: eb38b6f0935763f7ba19e5618f376fcae1dac41a) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: hashserv: Fix localhost sometimes resolved to a wrong IPAnatol Belski2020-09-231-1/+6
| | | | | | | | | | | | | | | Using localhost for direct builds on host is fine. A case with a misbehavior has been sighted on a Docker build. Even when the host supports IPv6, but Docker is not configured correspondingly - some versions of the asyncio Python module seem to misbehave and try to use IPv6 where it's not supported in the container. This happens at least on some Ubuntu 18.04 based containers, resolving the IP explicitly appears to be the fix. (Bitbake rev: 0e20f91c11afdc17ea776aa02e0cc8b0d59a23d4) Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Block SIGINT in worker processesJoshua Watt2020-09-151-0/+1
| | | | | | | | | | | | | Blocks SIGINT in the worker processes to prevent them from running the parent process signal handler, which causes them to deadlock under certain circumstances. [YOCTO #14034] (Bitbake rev: 9f4207f4b598f549cbd4159841c720276736f23b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Move away from problematic freedesktop.org urlsRichard Purdie2020-09-151-2/+4
| | | | | | | | | | We're either hitting rate limiting with freedesktop.org or the servers have intermittent network connections. Use our own mirror of these repositories instead. (Bitbake rev: a1b7ab5c9d5e64969f5ca0e41c0ac13c723e3761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: Ensure sockets don't leak even when exceptions occurRichard Purdie2020-09-121-12/+16
| | | | | | | | | | | We're seeing leaking open socket connections when errors occur and tinfoil is in use. Improve the exception handling so the sockets are closed even if exceptions occur, allowing more robust behaviour when things go wrong. (Bitbake rev: cefbec9ff47ca973a74ec7300cd736f3e0f0bce0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Fix disconnection handlingRichard Purdie2020-09-122-3/+5
| | | | | | | | | | After the recent init changes, if a client disconnects before issuing a command, the cooker can break in the reset handlers. Add some guards in the code to prevent this. (Bitbake rev: 12605e30e4c4e1ae6a67c97363b892ebf0b9566c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: fix UnboundLocalError when _print_exception raisesChris Laplante2020-09-101-2/+2
| | | | | | | | | | PEP 3110 changed how exceptions work. 'e' is unbound after the 'except' clause. See: https://www.python.org/dev/peps/pep-3110/#semantic-changes (Bitbake rev: b69e97de53eb172ed730993e3b755debaa26f30d) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: get_file_layer(): Improve performanceRobert Yang2020-09-102-5/+19
| | | | | | | | | | | | | | | | | The following code costs a lot of time when there are lot of layers and recipes: for collection in collections: collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or '' My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K (3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a keyword argument to get_file_layer() can fix the problem, it can save about 90% time in my build (6min -> 40s). (Bitbake rev: f08a6601c9bb09622855d62e1cedb92fafd2f71d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: get_file_layer(): Exit the loop when file is matchedRobert Yang2020-09-101-2/+3
| | | | | | | | | | | | | | | | | | | This can make "$ bitbake-layers show-recipes" save about 60% time (14min -> 6min) in my build (more than 3000 recipes) The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with each recipe, and get_file_layer() compare the file with each item in BBFILES which makes it very time consuming when there are a lot of recipes and items in BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't make sense to go on the loop when file is matched. And use fnmatchcase to replace of fnmatch since the comparison should be case-sensitive. (Bitbake rev: 8d64181d29dc262e066a6114dd51e5f2d04f47de) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: Save prioritized BBFILES to BBFILES_PRIORITIZEDRobert Yang2020-09-101-1/+1
| | | | | | | | | | | | The original code saved BBFILES back to BBFILES without any changes which isn't usefule, so remove that line. Now save prioritized BBFILES to BBFILES_PRIORITIZED which can accelerate the query a lot for the one which relies on it such as bb.utils.get_file_layer(). (Bitbake rev: 49bdb5dfa57b41b3ed399961e947c404f9195998) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: process_profilelog: use context managerChris Laplante2020-09-101-15/+14
| | | | | | | (Bitbake rev: 42172900af06baeee559d33b150d5febdf8e960a) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parser worker signal handlers are defaultRichard Purdie2020-09-081-0/+2
| | | | | | | | Otherwise this can interfer with multiprocessing exit handling. (Bitbake rev: b88816c4c84fa4f5ad39c263f5e75b96476e9768) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid parser deadlocksRichard Purdie2020-09-081-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | If you make parsing fail (e.g. add something like: X := "${@d.getVar('MCMACHINES').split()[1]}" to meson.bbclass, then run "while true; do bitbake -g bash; done" it will eventually hang. It appears the cancel_join_thread() call the parsing failure triggers, breaks the results_queue badly enough that it sits in read() indefintely (called from self.result_queue.get(timeout=0.25)). The timeout only applies to lock aquisition, not the read call. I've tried various other approaches such as using cancel_join_thread() in other places but the only way things don't lock up is to avoid cancel_join_thread() entirely for results_queue. I do have a concern that this may adversely affect Ctrl+C handling but equally, its broken now already and this appears to improve things. [YOCTO #14034] (Bitbake rev: 9c61a1cc7be46c23da1f4ef3bee070fb83c4be57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parser is cleaned upRichard Purdie2020-09-061-0/+1
| | | | | | | | | | During cooker shutdown, its possible the parser isn't cleaned up. Fix this (which may partially explain why threads were left hanging around at exit). (Bitbake rev: 928609f30f3a20aaa2f88afc18044a4e10199488) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Note when commands complete in logsRichard Purdie2020-09-051-0/+1
| | | | | | | | | | Its hard to tell from the server logs whether commands complete or not (or how long they take). Add extra info to allow more debugging of server timeouts. (Bitbake rev: 56285ada585ec1481449522282b335bcb5a2671e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Prefix the log data with pid/time informationRichard Purdie2020-09-051-2/+2
| | | | | | | | | | Knowing which process printed which messages and the timestamp of the message is useful for debugging, so add this. Ensure the log parsing isn't affected by using search() instead of match(). (Bitbake rev: 1d043666710df1fa9d9586fd974c0371dd1514b0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Ensure we don't keep looping if some other server ↵Richard Purdie2020-09-051-1/+20
| | | | | | | | | | | | | | | is started Showing "leftover process" messages when a new server has started and is being used by some UI is horrible. Compare the PID data from the lockfile to avoid this (and the ton of confusing log data it generates). Also, move the time.sleep() call to be after the first lock attempt, which reduces noise in the logs significantly. (Bitbake rev: ce1897a31afb5a14997bc3d2f459b90d43eecb7d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Assign a name to the sync thread to aid debuggingRichard Purdie2020-09-051-1/+1
| | | | | | (Bitbake rev: ffdb3d3fa690c35e9a96fc451a5811f5131276f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parser replacement calls parser final_cleanupRichard Purdie2020-09-051-0/+1
| | | | | | | | | This could potentialy account for some of the missing thread cleanup we're seeing. (Bitbake rev: 8f2d690428de8934868b406b79c4699a8ebe902c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Don't show tracebacks if the lockfile is removedRichard Purdie2020-09-051-0/+6
| | | | | | | | | lsof/fuser error if the file doesn't exist. It can be deleted by something else so ignore this if it happens and loop. (Bitbake rev: b100d22ce37b7548b50e59a71802bcc903acd6ea) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Ensure logging is flushedRichard Purdie2020-09-051-21/+23
| | | | | | | | | | | | | The cookerlog output goes to a file and its misleading to look at it and not have it up to date with what the cooker is actually doing. Ensure written data is flushed. Ultimately this should be using python's logging but that is for another day, we need simple fixes right now. (Bitbake rev: d95c3dd2b8ac50423976a7baf0a51e9580871761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Avoid printing binary strings for leftover processesRichard Purdie2020-09-051-1/+1
| | | | | | | | | | The binary string printed into the output is ugly, parse this so the linebreaks come out in the logs and make them much more readable (I was misssing the information initially despite looking for it). (Bitbake rev: c2dd8bb434d5738fedf9019651074b90affff3b2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Fix exception raise statementsRichard Purdie2020-09-051-3/+3
| | | | | | | | Lets use valid python even if it the original happens to work. (Bitbake rev: 343187c57e1459b0e57f90463843782f3a3a8443) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-21 11:14:43 +0000