summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cooker.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: cooker: Improve parsing failure from handled exception usabilityRichard Purdie2022-03-141-1/+3
| | | | | | | | | | | | | | | | When a recipe raises a BBHandledException, it means the error was already shown to the user. Adding an additional one here isn't helpful. What is helpful is to mention that parsing was halted. Tweak the code to do this with improves the messages the user sees and helps understand what happened. (Bitbake rev: b562966435ccd2335cfdccf89a9d7d18dd76a749) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit fdf6ebc8b603fcfd3ed7c64baf486a4adabd25be) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix task-depends.dot for multiconfig targetsRichard Purdie2021-11-211-1/+3
| | | | | | | | | | | | | | | The right hand side of dependencies in the task dependency file generated by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix this. [YOCTO #14621] (Bitbake rev: c1938abf51b57938a21948bb414ad0467e4368d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1d5ca721040c5e39aefa11219f62710de6587701) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle parsing results queue raceRichard Purdie2021-11-211-5/+9
| | | | | | | | | | | | | The previous fix introduced a race where the queue might not be empty but all the parser processes have exited. Handle this correctly to avoid occasional errors. (Bitbake rev: 8eaddb92a5fd14de6b5995aa92a6eed03b90a252) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove debug code, oops :(Richard Purdie2021-11-211-1/+0
| | | | | | | | | (Bitbake rev: ae1bfbf9523e8f6155bb43ee3adba17af3ec9630) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 19291665fa8b6cc331290f2542af3e8e653203f1) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle parse threads disappearing to avoid hangsRichard Purdie2021-11-211-1/+10
| | | | | | | | | | | | | If one of the parse threads disappears during parsing for some reason, bitbake currently hangs. Avoid this (and zombie threads hanging around) by joining() threads which have exited. (Bitbake rev: 920111a330be59e5be2068a8f1a9edcbc6c14402) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit dc86a533d951d13643ce446533370da804782afc) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: check if upstream hash equivalence server is availableJose Quaresma2021-11-121-1/+11
| | | | | | | | | | | | | When the user specify an invalid upstream hash equivalence server in BB_HASHSERVE_UPSTREAM notify the user that we can't connect the server. (Bitbake rev: 7561fdc23f1aff370ead2abc5747c3a1c8b4ae4d) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit be45aeb9a84f30c28711e87e2d2a4a86320a8d94) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Add a dummy event for tinfoil testingRichard Purdie2021-09-261-0/+5
| | | | | | | | | | We need a command genetating an event to test through the tinfoil API. The current test has IO load issues so add a dummy version which won't have the IO constraints. (Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Allow upstream for local hash equivalence serverJoshua Watt2021-09-111-1/+6
| | | | | | | | | | | | | | The hash equivalence server has had the option to support a read-only upstream server for some time now when launched as a standalone program, but there was no way to set the upstream when using a locally started server. Add a new variable called BB_HASHSERVE_UPSTREAM that can be used to specify an upstream server when a local hash equivalence server is used (e.g. BB_HASHSERVE is "auto") (Bitbake rev: 250fa17f1391ff1ee01ab9b51d2a4f9aa35c1d1e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/process: Fix typos in exiting messageMartin Jansa2021-09-011-1/+1
| | | | | | | (Bitbake rev: 1ff1ea3880d293b14ce0fc65e3bc4c938d587a2f) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: handle PRSERV_HOST = "127.0.0.1:0" the same as "localhost:0"Martin Jansa2021-09-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * When using PRSERV_HOST = "localhost:0" inside Docker container (tested with ubuntu 20.04 and 21.04) the self.loop.run_until_complete never completed, so self.address wasn't ever assigned few lines bellow and then self.port = int(self.prserv.address.rsplit(':', 1)[1]) in lib/prserv/serv.py caused a bit ugly exception: bitbake@599696cd20aa:~/nodistro/honister$ bitbake -k pkgconfig-native Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake", line 35, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "/OE/nodistro/honister/bitbake/lib/bb/main.py", line 385, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "/OE/nodistro/honister/bitbake/lib/bb/ui/knotty.py", line 397, in main params.updateToServer(server, os.environ.copy()) File "/OE/nodistro/honister/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer raise Exception("Unable to update the server configuration with local parameters: %s" % error) Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 286, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 491, in updateConfigOpts self.reset() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 1717, in reset self.handlePRServ() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 383, in handlePRServ self.prhost = prserv.serv.auto_start(self.data) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 318, in auto_start singleton.start() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 133, in start self.port = int(self.prserv.address.rsplit(':', 1)[1]) AttributeError: 'NoneType' object has no attribute 'rsplit' * the issue was caused by "localhost" being resolved as IPv6 address ::1 and then asyncio failing to bind it, the same is reproducible with hashserv, but hashserve at least shows nice error message: bitbake$ bitbake-hashserv -l DEBUG -b localhost:0 Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 59, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 53, in main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address * or by bitbake-prserv in prserv.log: bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address * while 127.0.0.1 works fine: bitbake$ bitbake-prserv --start --host=127.0.0.1 --port=42005 bitbake$ cat prserv.log DEBUG: Listening on ('127.0.0.1', 42005) 2021-08-26 22:28:05,828 Listening on ('127.0.0.1', 42005) DEBUG: Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' 2021-08-26 22:28:05,829 Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' NOTE: Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 2021-08-26 22:28:05,831 Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 but 127.0.0.1:0 wasn't handled as "autostart" like localhost:0 is update is_local_special to allow that * /etc/hosts file generated by docker contails localhost for both IPv4 and IPv6: $ grep localhost /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback even when ipv6 is disabled in dockerd as reported in: https://github.com/docker/for-linux/issues/250 * add a check for self.prserv.address to provide better error message: ERROR: Unable to start PR Server, exitting when something bad happens, but in this case you still need to read bitbake-cookerdaemon.log to see the actuall error, in this case: 90 22:30:39.008441 --- Starting bitbake server pid 90 at 2021-08-26 22:30:39.008419 --- 90 22:30:39.023734 Started bitbake server pid 90 90 22:30:39.024286 Entering server connection loop 90 22:30:39.024753 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([]) 90 22:30:39.026314 Processing Client 90 22:30:39.026456 Connecting Client 90 22:30:39.027509 Running command ['setFeatures', [2]] 90 22:30:39.027757 Command Completed 90 22:30:39.028711 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'DISTRO': '', 'PWD': '/OE/nodistro/honister', 'HOME': '/OE', 'MACHINE': 'qemux86', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'PATH': '/OE/nodistro/honister/oe-core/scripts:/OE/nodistro/honister/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'en_US.UTF-8', 'MACHINES': 'qemux86', 'HOSTNAME': '6a439759e3c6', 'TOPDIR': '/OE/nodistro/honister', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHLVL': '1', 'BITBAKE_HOME': '/OE', 'BUILDDIR': '/OE/nodistro/honister/BUILD', 'OLDPWD': '/OE/nodistro/honister/bitbake', '_': '/OE/nodistro/honister/bitbake/bin/bitbake'}, ['/OE/nodistro/honister/bitbake/bin/bitbake', '-k', 'zlib-native']] Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 255, in run self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address 90 22:30:39.530037 Command Completed 90 22:30:39.530913 Processing Client 90 22:30:39.531023 Disconnecting Client 90 22:30:39.531638 No timeout, exiting. 90 22:30:39.632137 Exiting 90 22:30:39.637562 Original lockfile contents: ['90\n'] 90 22:30:39.638107 Exiting as we could obtain the lock (Bitbake rev: c2cdda0e5fc720c60d3b8537fc702cb118981bd2) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: asyncrpc: Catch early SIGTERMJoshua Watt2021-07-291-2/+1
| | | | | | | | | | | | | | | | | If the SIGTERM signal is sent to an asyncrpc server before it has installed the SIGTERM handler in the main loop, it may miss the signal which will can cause the calling process to wait forever on the join(). To resolve this, the calling process should mask of SIGTERM before forking the server process and the server should unmask the signal only after the handler is installed. To simplify the usage of the server, an new helper function called serve_as_process() is added to do this automatically and correctly. Thanks: Scott Murray <scott.murray@konsulko.com> for helping debug (Bitbake rev: ef2865efa98ad20823267364f2159d8d8c931400) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Add REQUIRED_VERSION checksCharlie Davies2021-03-151-2/+8
| | | | | | | | | | | | | Add logic to check that if REQUIRED_VERSION has been set that the cooker class method findBestProviders properly handles the case where the REQUIRED_VERSION has not been found. Fixes [YOCTO #10096] (Bitbake rev: 5df201d746f26154213e6d15d2721cd35b38b59e) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: lib: add Required Version to Bitbake --show-versions commandCharlie Davies2021-03-151-8/+16
| | | | | | | | | | | | | This commit adds a new column to the Bitbake --show-versions command called Required Version. This column will display any packages which have a REQUIRED_VERSION successfully set. Fixes [YOCTO #10096] (Bitbake rev: 90c7d1815e41243323d32b9dbb865757a922578a) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Force parser shutdown after catching an exceptionJan Brzezanski2021-02-211-5/+5
| | | | | | | | | | | | | | | | Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. (Bitbake rev: 915330e1dbae1ee8fd9a0358decf2c294f771961) Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-7/+7
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure reparsing is handled correctlyRichard Purdie2021-02-101-0/+2
| | | | | | | | | | | | | | | From tinfoil, if you edit bblayers.conf and break it, then call parseConfiguration (e.g. by adding a bad layer with bitbake-layers), the system doens't show any parse error yet it should. Add in a call to the updateCache function so that things really are reparsed when requested. Partially fixes [YOCTO #14054] (Bitbake rev: e655f9361b9c3b77906b8e06b5cc76bc5180640e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Don't treat mc recipe (Midnight Commander) as a multiconfig ↵Tomasz Dziendzielski2021-02-061-2/+2
| | | | | | | | | | | | | | | | target When we run `devtool build mc` recipe's task dependencies are expanded to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig and "do_package_sysroot" as multiconfigname. | ERROR: Multiconfig dependency mc:do_populate_sysroot depends on | nonexistent multiconfig configuration named do_populate_sysroot (Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Include all packages a recipe provides in ↵Peter Kjellerstedt2020-12-211-1/+3
| | | | | | | | | | | | | | | | | | | | | | | SkippedPackage.rprovides The provided packages by a skipped recipe are supposed to be listed in SkippedPackage.rprovides, which is used when generating a meaningful error message when a build fails because of a skipped package. Previously this variable only contained the contents of ${RPROVIDES}. However, most recipes don't define RPROVIDES, they define RPROVIDES_<pkg> for each package they provide. Additionally, the recipe provides the packages in PACKAGES without them being included in ${RPROVIDES}. Before this change, having a runtime dependency on a skipped non-recipe package would result in a build error stating that the build failed because the package was skipped, but without providing any reason for why it was skipped. (Bitbake rev: efd026c26a377b826a49b945a8212bf7de8a480a) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid tracebacks if data was never setupRichard Purdie2020-10-011-1/+2
| | | | | | | | Recent changes mean data might not be setup. If its not, avoid tracebacks. (Bitbake rev: 3daff610d9f39d73c80c54d1df46f573666e20db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Block SIGINT in worker processesJoshua Watt2020-09-151-0/+1
| | | | | | | | | | | | | Blocks SIGINT in the worker processes to prevent them from running the parent process signal handler, which causes them to deadlock under certain circumstances. [YOCTO #14034] (Bitbake rev: 9f4207f4b598f549cbd4159841c720276736f23b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Fix disconnection handlingRichard Purdie2020-09-121-2/+3
| | | | | | | | | | After the recent init changes, if a client disconnects before issuing a command, the cooker can break in the reset handlers. Add some guards in the code to prevent this. (Bitbake rev: 12605e30e4c4e1ae6a67c97363b892ebf0b9566c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: Save prioritized BBFILES to BBFILES_PRIORITIZEDRobert Yang2020-09-101-1/+1
| | | | | | | | | | | | The original code saved BBFILES back to BBFILES without any changes which isn't usefule, so remove that line. Now save prioritized BBFILES to BBFILES_PRIORITIZED which can accelerate the query a lot for the one which relies on it such as bb.utils.get_file_layer(). (Bitbake rev: 49bdb5dfa57b41b3ed399961e947c404f9195998) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parser worker signal handlers are defaultRichard Purdie2020-09-081-0/+2
| | | | | | | | Otherwise this can interfer with multiprocessing exit handling. (Bitbake rev: b88816c4c84fa4f5ad39c263f5e75b96476e9768) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid parser deadlocksRichard Purdie2020-09-081-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | If you make parsing fail (e.g. add something like: X := "${@d.getVar('MCMACHINES').split()[1]}" to meson.bbclass, then run "while true; do bitbake -g bash; done" it will eventually hang. It appears the cancel_join_thread() call the parsing failure triggers, breaks the results_queue badly enough that it sits in read() indefintely (called from self.result_queue.get(timeout=0.25)). The timeout only applies to lock aquisition, not the read call. I've tried various other approaches such as using cancel_join_thread() in other places but the only way things don't lock up is to avoid cancel_join_thread() entirely for results_queue. I do have a concern that this may adversely affect Ctrl+C handling but equally, its broken now already and this appears to improve things. [YOCTO #14034] (Bitbake rev: 9c61a1cc7be46c23da1f4ef3bee070fb83c4be57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parser is cleaned upRichard Purdie2020-09-061-0/+1
| | | | | | | | | | During cooker shutdown, its possible the parser isn't cleaned up. Fix this (which may partially explain why threads were left hanging around at exit). (Bitbake rev: 928609f30f3a20aaa2f88afc18044a4e10199488) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Assign a name to the sync thread to aid debuggingRichard Purdie2020-09-051-1/+1
| | | | | | (Bitbake rev: ffdb3d3fa690c35e9a96fc451a5811f5131276f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parser replacement calls parser final_cleanupRichard Purdie2020-09-051-0/+1
| | | | | | | | | This could potentialy account for some of the missing thread cleanup we're seeing. (Bitbake rev: 8f2d690428de8934868b406b79c4699a8ebe902c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/cookerdata: Use BBHandledException, not sys.exit()Richard Purdie2020-09-051-2/+2
| | | | | | | | | | | | | | | Calling sys.exit() in the middle of the code is rather antisocial. We catch this in various places but we shouldn't have to. In all these cases we have already sent events explaining to the user what happened. This means the correct exception is BBHandledException. The recent startup changes have moved the point a lot of this code gets called to inside the UI, with memres it would have always been possible from there anyway. This change makes things much more consistent. (Bitbake rev: 91699f366d24480ff3b19faec78fb9f3181b3e14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure cooker's enviroment is updated on updateConfigRichard Purdie2020-08-251-0/+4
| | | | | | | | | | | | | Only the env variables which were added to the datastore were being updated. We need to update the whole copy, we just only trigger a reparse if any of the variables making it into the datastore change. This avoids a bug where variables such as DISPLAY in a new UI context would break under memory resident bitbake. (Bitbake rev: 4bb71b627767297269e762b414443e15e28bfac4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure BB_ORIGENV is updated by changes to configuration.envRichard Purdie2020-08-251-0/+6
| | | | | | | | Changes to configuration.env were not updating BB_ORIGENV, fix this. (Bitbake rev: c5fbd8452f87e0a2d234eaf27d0450aacdeb8891) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main/server/process: Drop configuration object passingRichard Purdie2020-08-251-2/+2
| | | | | | | | | | | | | | | | | | | The first thing the UIs do is update the server config from the UI. We can just rely upon that and start the server with a standard config, removing the need to pass the confusing configuration object around as well as configParams, which contains a similar copy of some of the data. This makes memory resident bitbake work the same way as the normal mode, removing the opportunity for some class of bugs. The xmlrpcinterface and server_timeout values are passed in at server startup time now and there no longer a second option in the configuration which is effective ignored once the server starts. (Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Defer configuration init to after UI connectionRichard Purdie2020-08-251-9/+8
| | | | | | | | | | | | | Currently we end up parsing the base configuration multiple times as initially, the right settings haven't come from the UI. We can defer this until later in startup using runCommand as a trigger. The advantage to doing this is improved startup times and ultimately we should be able to avoid the double parse of the base configuration. (Bitbake rev: 3caa43b665604475d2c87ba505efb0b9fca9c2e9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/cookerdata: Ensure UI event log is updated from commandlineRichard Purdie2020-08-251-7/+11
| | | | | | | | | Currently the eventlog is not handled correctly for memory resident bitbake. Fix this by allowing adpations to configuration changes. (Bitbake rev: f7d2c9116116659ea42260a3bb96dca100aadae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/cookerdata/main: Improve loglevel handlingRichard Purdie2020-08-251-1/+6
| | | | | | | | | | | Rather than passing debug/verbose/debug_domains around, pass the computed output of these. Ensure that the cooker sets the levels to match the levels currently set in the UI and generally try and make it easier to understand what the code is doing. (Bitbake rev: f0535beecc692a6213be2a22f9eef5956450ecf8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/msg: Cleanup verbose option handlingRichard Purdie2020-08-251-5/+0
| | | | | | | | | | | | The levels of indirection to set these verbose logging options is rather crazy. This attempts to turn things into two specific options with much more specific meanings. For now its all still controlled by the commandline verbose option and should funciton as previously, with the addition that the BB_VERBOSE_LOGS option can now be task specific. (Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Explictly shut down the sync threadRichard Purdie2020-08-241-1/+7
| | | | | | | | | | | | | | | | Hongxu Jia reported a problem where the bb_cache files were not always being written out correctly. This was due to the sync thread being terminated prematurely. Whilst the preceeding changes mean the exit handler for this thread is now correctly called since we switch to using sys.exit() instead of os._exit(), this write can happen after we drop the bitbake lock, leading to potential races. Avoid that headache by adding in explicit thread join() calls before we drop the lock (which atexit or Finalize can't do). (Bitbake rev: afd1900939f7b042297558f4cb01f50f3a299267) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parse_quit thread is closed downRichard Purdie2020-08-241-6/+8
| | | | | | | | | | | Each run through the parser would leak a thread from the queue created to shut the parser down. Close this down correctly and clean up the code flow slightly whilst in the area, making sure this thread does shut down correctly (we don't care if it loses data). (Bitbake rev: 51ba35e9bbd4da8b5f3b3b5b8213cb634a6b0f06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Simplfy idle callback handler functionRichard Purdie2020-08-131-4/+6
| | | | | | | | | | Having the idle callbacks abstracted via the configuration object makes no sense. Its like this for historical reasons from the multiple server backends but we don't need this now so simplfy. (Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle multiconfig name mappings correctlyRichard Purdie2020-07-231-1/+1
| | | | | | | | | | | | | | "bitbake mc:arm:bash mc:arm:busybox" works but "bitbake multiconfig:arm:bash multiconfig:arm:busybox" does not. The reason is the list is modified whilst iterating. Don't do that. [YOCTO #13607] (Bitbake rev: cd041a78d96e656438d93fb1e288080b8a6fe8bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve multiconfig configuration error reportingRichard Purdie2020-07-231-1/+6
| | | | | | | | | This avoids a traceback if an invalid multiconfig is referenced in the bitbake commandline and tweaks the message to make it more understanable. (Bitbake rev: f31d7d0ad57b0ecc2ae06ed4b547c98df2aaa1a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix unmatched files handling leading to misleading warningsRichard Purdie2020-07-211-30/+50
| | | | | | | | | | | | | | | | | | | | | Currently if all recipes in a layer are skipped, there are warnings that the BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this for skipped recipes. The current code is hard to understand, not least as it passes variables which functions modify by reference rather than giving a return value. Update calc_bbfile_priority() to return values rather than modifying them. Refactor the code to try and make it clearer what its doing and fix the skipped recipe issue by passing in the list of parsed files. The function is complicated by the need to not rerun regex matching more than we ever have to which complicates the flow, it would be easier if we just reran operations multiple times. (Bitbake rev: 969cb27b4d978551817612ff4558bec81cfb655c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cache: Cache size optimizationJoshua Watt2020-06-101-7/+1
| | | | | | | | | | | Now that there is a cache object per multiconfig, it is not necessary for each cache object to parse all other multiconfigs. Instead, each cache now only parses the files for it's multiconfig. (Bitbake rev: 3c5c7346adf4ca7ec761c08738f12401ba75b7c8) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cache: Use multiconfig aware cachesJoshua Watt2020-06-101-17/+25
| | | | | | | | | | | | | | | Splits the parsing cache to maintain one cache per multiconfig instead of one global cache. This is necessary now that the files and appends can vary for each multiconfig. A bb.cache.MulticonfigCache dictionary-like proxy object is created instead of a single bb.cache.Cache object. This object will create and properly initialize bb.cache.Cache object for each multiconfig, and each of these caches has a dedicated cache file with a name based on the multiconfig. (Bitbake rev: 5272f2489586479880ae8d046dfcdbe0963ee5bb) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Split file collections per multiconfigJoshua Watt2020-06-101-57/+82
| | | | | | | | | | | | Splits the cooker to track a collection per multiconfig instead of a single collection for all multiconfigs. Practically speaking, this allows each multiconfigs to each have different BBMASKs that apply to it instead of each one using the mask specified in the base configuration. (Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Respect multiconfig parameterJoshua Watt2020-03-071-3/+3
| | | | | | | | | | The cooker had a multiconfig parameter for the findProviders() and findBestProviders() API, but it was being ignored. (Bitbake rev: ea0b68ac2b77676ed1c63f0ee1ae5d300f2b4696) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Reset parse status unpon clientCompleteRichard Purdie2020-03-021-0/+2
| | | | | | | | | | | | | | | | If for example a tinfoil connection edits the datastore, a subsequent connection can be "corrupted" by those changes. By setting the parse status of the caches as False at exit, the behaviour becomes the same as a newly setup server as a new data store is setup. This avoids problems in tests when BB_SERVER_TIMEOUT is set as the server is properly reset between connections. [YOCTO #13812] (Bitbake rev: e66759106e21da2b34a6cdec7aa681ad2204da54) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Reset loghandlerRichard Purdie2020-02-191-0/+3
| | | | | | | | | When parsing, reset the loghandler when finished, else the messages can be misleading. (Bitbake rev: 7af80cd1dd577b05d39a3cc5d5c547a2549e39df) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/siggen: Empty siggen cache during parsingRichard Purdie2020-02-171-1/+2
| | | | | | | | | | | | | | | | | | | When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. (Bitbake rev: 5d98d8e39bba42f458532b1eef3619f2321d8a2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/toaster: replaced deprecated method warn() with warning()Frazer Clews2020-02-061-1/+1
| | | | | | | | | | Removed the deprecated methods as it will only cause problems later on, and since warn() just calls warning(), it shouldn't change anything (Bitbake rev: a194f275235f22411cb2368f06a44f61ceb6a0f3) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: amend code to use proper singleton comparisons where possibleFrazer Clews2020-01-191-4/+4
| | | | | | | | | | | amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. (Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-10-08 21:01:02 +0000