summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/server/process.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: bitbake: Convert to python 3 megacommit This needs breaking up into ↵Richard Purdie2016-05-161-4/+4
| | | | | | | | smaller changes. (Bitbake rev: cf51f19aed208a75d38c14cd585d9b9f115e3ba3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix missing log messages issueRichard Purdie2016-05-141-4/+4
| | | | | | | | | | | | | | | | | Currently if the server dies, its possible that log messages are never displayed which is particularly problematic if one of those messages is the exception and backtrace the server died with. Rather than having the event queue exit as soon as the server disappears, we should pop events from the queue until its empty before exiting. This patch tweaks that code so that even if the server is dead and we're going to exit, we return any events left in the pipe. This makes debugging certain failures much easier. (Bitbake rev: 29f6ade68fb2b506a23a7eb3a00cdcffa291b362) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: xmlrpc: set single use mode differentlyEd Bartosh2016-03-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Currently xmlrpc server implicitly sets itself into single use mode when bitbake server is started with anonymous port (0) or no port is provided in command line. In this mode bitbake shuts down xmlrpc server after build is done. This assumption is incorrect in some cases. For example Toaster uses bitbake in this mode and expects xmlrpc server to stay in memory. Till recent changes single use mode was always unset due to the bug. When the bug was fixed it broke toaster builds as Toaster couldn't communicate with bitbake server in single use mode. Reimplemented logic of setting single use mode. The mode is explicity set when --server-only command line parameter is not provided to bitbake. It doesn't depend on the port number anymore. [YOCTO #9275] [YOCTO #9240] [YOCTO #9252] (Bitbake rev: afc0dd5c532684f6201b1e12bbf4c226ea19062d) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Try connecting 4 times before giving upLucas Dutra Nunes2016-03-091-4/+6
| | | | | | | | | | Instead of trying one time with a timeout of 20 seconds try 4 times with a timeout of 5 seconds, to account for a slow server start. (Bitbake rev: 4a7fe63126dd8177baa5ad21e59e0bebeea8c596) Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Set process names to be meaninfulRichard Purdie2016-01-301-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Handle SIGTERM more gracefullyRichard Purdie2015-09-091-0/+9
| | | | | | | | | | | | | | | | | | | | | Currently if you send a SIGTERM to the bitbake UI process, the system basically hangs if tasks are executing. This is because the server process doesn't actually try any kind of shutdown before exiting. This patch trys executing a stateForceShutdown command first, which is enough to stop any active tasks before the system exits. I also noticed that terminate can execute multiple times, once at SIGTERM from the handler and once from the real exit. Double execution leads to stack traces and potential hangs (writes to dead pipes), so ensure the code only can run once. With these fixes, bitbake much more correctly deals with SIGTERM to the UI process. (Bitbake rev: 1032ddddbe3241da02ebb3608a1c40f9123b9e80) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/server: Add _uiready flag to handle missing error messagesRichard Purdie2015-09-041-1/+1
| | | | | | | | | | | | | | | | | | | | | If you start and suspend a bitbake execution so the bitbake lock is held, then try and run "bitbake -w '' X", you will see bitbake return an error exit code but print no message about what happened at all. The reason is that the -w option creates a "UI" which swallows the messages. The code which handles this exit failure mode thinks a UI has printed the messages and therefore doesn't do so. This adds in an extra parameter to the UI registration code so that we can figure out whether its a primary UI or not and base decisions on whether to display information on that instead. This fixes the error shown above and some bizarre failures on the Yocto Project Autobuilder. [YOCTO #8239] (Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Fix default function parameter assignment to a listPaul Eggleton2015-08-191-1/+3
| | | | | | | | | | | | | | With python you should not assign a list as the default value of a function parameter - because a list is mutable, the result will be that the first time a value is passed it will actually modify the default. Reference: http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments (Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Don't log BBHandledExceptionRichard Purdie2015-06-231-2/+3
| | | | | | | | | | | | If we see a BBHandledException in the idle handler, the understanding is the system handled it, printing a log and traceback is just confusing. Therefore only print these in the cases where its an unknown/unhandled exception. (Bitbake rev: d88ecc2bc44dce8fd92ca3a2c0fd4124a5e464fa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/server: Fix up 100% CPU usage at idleRichard Purdie2015-03-101-0/+3
| | | | | | | | | | | | | | | The recent inotify changes are causing a 100% cpu usage issue in the idle handlers. To avoid this, we update the idle functions to optionally report a float value which is the delay before the function needs to be called again. 1 second is fine for the inotify handler, in reality its more like 0.1s due to the default idle function sleep. This reverts performance regressions of 1.5 minutes on a kernel build and ~5-6 minutes on a image from scratch. (Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix select callRichard Purdie2015-01-231-1/+1
| | | | | | | | | | | | | There was a report that bitbake -e | less would use 100% cpu when it shouldn't really. The issue appears to be a bogus file descriptor in the select call. We shouldn't be blocking if there is event data pending to a *reader* from server context. [YOCTO #7138] (Bitbake rev: 8f166e1a0f3574ae7d1e917a8bb403b87bad15bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Ensure abnormal exits set an error levelRichard Purdie2014-09-021-1/+1
| | | | | | (Bitbake rev: 8f5c1cdae1ee6ce04ae0d04d0b95bd80efbf7534) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Further improve robustness against server shutdownRichard Purdie2014-08-231-3/+14
| | | | | | | | | | | | | | | | | Currently, if an exception occurs in an event handler, the server shuts down but the UI simply hangs. This happens in two places, firstly waiting for events and secondly, sending events to a server which no longer exists. The latter does time out, the former does not. These patches improve both code sections to check if the main server process is alive and if not, trigger things to shut down gracefully. This avoids the timeout in the command sending case too. This resolves various cases where the UI would simply hang indefintely. (Bitbake rev: ac418e1112ff5f9c3157569316902f7a27fba4b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Deal with infinite looping of the serverRichard Purdie2014-08-231-0/+2
| | | | | | | | | | | | | | | Currently if an exception occurs, we just run the idle handler again and again, usually looping indefintely. Chances are the exception that occurred will keep occuring and this is not a good place to be. This was breaking the autobuilders with gigabytes of logs. At least improve things so the cooker shuts down gracefully when this happens. Some trace of the original problem may still be present on the console too! (Bitbake rev: 1f28d8d3311262427938180435b68f0a35c2b330) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Optimise latency when finishing idle functionsRichard Purdie2014-03-101-0/+1
| | | | | | | | | | When idle functions finish, its likely we have some other work to do, so don't sleep in the select call but instead, skip it. This removes small amounts of latency in common commands. (Bitbake rev: 069d6538f83b607cb46c6fe21bf6c596e8b99242) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Drop unnecessary exit delayRichard Purdie2014-03-101-1/+0
| | | | | | | | | | When the server exits, we no longer appear to need this delay. This is likely due to improvements in the various exit codepaths. There is therefore no longer any point in taking the latency hit. (Bitbake rev: 8e75ee29ae07e13f23525c5c6045fbf6cdbe7675) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Use a pipe for quit events instead of Event()Richard Purdie2014-03-101-5/+9
| | | | | | | | | | | | | | Its not possible to notice the change of status of an Event() in the select call we sleep in. It would be possible in python 3.3 but for now use a pipe instead. This removes small latency when bitbake commands finish since the system doesn't sit in the select call. (Debugging these kind of issues is apparent by setting a long sleep for the select call) (Bitbake rev: def28239b0f0d5f1cf13214b263114a5328538b7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Deal more gracefully with SIGTERMRichard Purdie2014-03-101-1/+15
| | | | | | | | | | | | Currently a SIGTERM to the UI process causes the UI simply to lock up. By setting an exit flag, the waitEvent can raise a SIGINT, allowing the UI to break out the event loop and exit. Currently this is results in a traceback but that is more desirable than a hanging process. (Bitbake rev: 0d12041eceeae6bba2034b04913bb13abd67bd15) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Use the setFeatures command on the server instead ↵Richard Purdie2014-03-101-13/+8
| | | | | | | | | | | | | | | of a manger The use of a manager in the process server causes some issues since it remains around for the lifetime of the server even though its only used during initialisation and the system doesn't respond well to SIGTERM events to the extra process (and two threads) the implementation involves. Switching to a dedicated command simplifies the server process structure. (Bitbake rev: 74532a7cf8ccea8b85f1cda5d5bc23d2f3c72a08) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker,xmlrpc,servers: implement CookerFeaturesAlexandru DAMIAN2013-09-221-5/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implementing feature set selection that allows a client to enable specific features in the server at connection time. Only enabling of features is supported, as there is no way to safely remove data loaded into the cooker. Once enabled, a feature will remain enabled for the life of the cooker. Client-server connection now supports specifying the feature set required by the client. This is implemented in the Process server using a managed proxy list, so the server cooker will now load dynamically needed features based on what client connects to it. In the XMLRPC server the feature set is requested by using a parameter for registerUIHandler function. This allows observer-only clients to also specify features for the server. The server code configuration now is completly separated from the client code. All hardcoding of client knowledge is removed from the server. The extra_caches is removed as the client can now specify the caches it needs using the feature. The UI modules now need to specify the desired featureSet. HOB is modified to conform to the featureSet specification. The only feature available is CookerFeatures.HOB_EXTRA_CACHES which forces loading the bb.cache_extra:HobRecipeInfo class. (Bitbake rev: 98e594837aab89ea042cfa9f3740d20a661b14e2) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Rename confusing 'stop' state to 'forceshutdown'Richard Purdie2013-09-141-1/+1
| | | | | | | | | | | | The shutdown state causes the server to finish what its doing, stop was them meant to completely stop it. It doesn't mean the server is stopped though. Renaming the current stop event for forceshutdown gives more meaning to what it actually does. The stopped namespace then becomes available to indicate a completely stopped server. (Bitbake rev: 12e9d33bfae5294e3870dfd1202f63383ad05e92) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process, server/xmlrpc, runqueue: Use select.select() on ↵Richard Purdie2013-09-011-5/+6
| | | | | | | | | | | | | | | | | | | | | | fds, not time.sleep() The existing backend server implementations were inefficient since they were sleeping for the full length of the timeouts rather than being woken when there was data ready for them. It was assumed they would wake and perhaps did when we forked processes directory but that is no longer the case. This updates both the process and xmlrpc backends to wait using select(). This does mean we need to pass the file descriptors to wait on from the internals who know which these file descriptors are but this is a logical improvement. Tests of a pathaolgical load on the process server of ~420 rapid tasks executed on a server with BB_NUMBER_THREAD=48 went from a wall clock measurement of the overall command execution time of 75s to a much more reasonable 24s. (Bitbake rev: 9bee497960889d9baa0a4284d79a384b18a8e826) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add ui event handlers filteringRichard Purdie2013-08-261-4/+8
| | | | | | | | | | | | | | | | | | | Add functionality to allow UIs to update and change the types of events they recieve. To do this we need to add a new command and also need to be able to obtain the current event hander ID. In the case of xmlrpc, this is straightforward, in the case of the process server we need to save the result in a multiprocessing.Value() so we can retrive it. An excplit command was added to the server API to facilitate this. The same function can also be used to mask or unmask specific log messages, allowing the UI to optionally differ from the standard set of message filtering. Based upon work by Cristiana Voicu <cristiana.voicu@intel.com> (Bitbake rev: ba5a6c88785d9889d4172ec79937ac2a5555327e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Improve exit handling and hangsRichard Purdie2013-08-261-19/+17
| | | | | | | | | | | | | | | | | | | | | | | It turns out we have a number of different ways the process server termination can hang. If we call cancel_join_thread() on the event queue, it means that it can be left containing partial data. This means the reading of the event queue in the terminate() function can hang, the timeout and block parameters to Queue.get() don't make any difference. Equally, if we don't call cancel_join_thread(), the join_thread in terminate() will hang giving a different deadlock. The best solution I could find is to loop over the process is_alive() after requesting it stops, trying to join the thread and if that fails, try and flush the event queue again. It wasn't clear what difference a force option should make in this case, we're gracefully trying to empty queues and shut down regardless of whether its a SIGTERM so I've simply removed the force option. (Bitbake rev: c5c8f33ca4b81877a0115887849881001b745bf0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: compat/server/utils: Jettison pre python 2.7.3 workaroundsRichard Purdie2013-06-141-46/+0
| | | | | | | | | Now we've moved to require python 2.7.3, we can jettison the compatibility workarounds/hacks for older python versions. (Bitbake rev: a51c402304f2080a76720f9b31d6dfdbed393bba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake server: create common server infrastructureAlexandru DAMIAN2013-06-071-33/+20
| | | | | | | | | | | | | | | | | | | In an attempt to minimize code duplication, create clear interfaces, and maximize code reuse through OOP, bb.server adds base classes for the BitBakeServer, BitBakeServerConnection and actual server implementations instructed in particular server types. These classes document the minimum interfaces that the derived classes must implement, and provide boilerplate code. Changes to None, Process and XMLRPC servers as to use the common server infrastructure. (Bitbake rev: 6db4a64cef20f8d0aba804db4c4e1eec7b112b46) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/bitbake: Remove launchUI methodRichard Purdie2013-05-301-4/+0
| | | | | | | | | With the removal of the none server type, we can remove the launchUI method and simplify the code slightly. (Bitbake rev: 9bef2f2dd0bcaa59528ebcb3c1ce053b7dff1ec6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Always use separate process for PR ServiceRichard Purdie2013-02-061-1/+1
| | | | | | | | | | | | | | | Using the threading module interacts badly with multiprocessing used elsewhere in bitbake under certain machine loads. This was leading to bitbake hanging on Ctrl+C when the PR Server was being used. This patch converts it to always use the daemonize code which then means the threading code isn't required. [YOCTO #3742] (Bitbake rev: 2d0bbd9398ab839bd2d1e29e50b25d52efb1ce2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process.py: Change timeout error handlingRichard Purdie2012-11-211-2/+2
| | | | | | | | | | | | | | | | | | | In normal usage, we never hit the timeout issue. If we do, it becomes obvious that the current error handling is not good enough. The request may have made it to the server and the answer will get queued. This means the next command may get the return value from the previous command with suitably puzzling results. Without rewriting large sections of code, its not possible to avoid this problem. It is better to increase the timeout to several seconds giving the server a chance to respond and if it does timeout, hard exit since recovery is not possible with the code base today. I'd be happy to see the structure of this code improved but this quick fix at least stops corrupted builds from happening which has to be a good thing. (Bitbake rev: 410c11dd10736873f2dc587fbe9119c38831e693) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: add error to return of runCommandChristopher Larson2012-11-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, command.py can return an error message from runCommand, due to being unable to run the command, yet few of our UIs (just hob) can handle it today. This can result in seeing a TypeError with traceback in certain rare circumstances. To resolve this, we need a clean way to get errors back from runCommand, without having to isinstance() the return value. This implements such a thing by making runCommand also return an error (or None if no error occurred). As runCommand now has a method of returning errors, we can also alter the getCmdLineAction bits such that the returned value is just the action, not an additional message. If a sync command wants to return an error, it raises CommandError(message), and the message will be passed to the caller appropriately. Example Usage: result, error = server.runCommand(...) if error: log.error('Unable to run command: %s' % error) return 1 (Bitbake rev: 717831b8315cb3904d9b590e633000bc897e8fb6) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/server: Remove dead console log codeRichard Purdie2012-11-011-1/+1
| | | | | | | | This code is dead and doesn't do anything so lets remove it. (Bitbake rev: 8d45739f49618757a5d7d79782deda355e3981ec) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/process.py: Ensure queued UI events are queued right before we add ↵Richard Purdie2011-07-071-3/+2
| | | | | | | | our own handler (Bitbake rev: c7a9ef70ba91d47d53074e8d78cbc52f396144a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process.py: Fix issue where early errors weren't making it to the consoleRichard Purdie2011-06-161-1/+1
| | | | | | (Bitbake rev: d97f7d762e3d2f1b0da038d4d99f2531b2490670) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* make exception handling syntax consistentScott Garman2011-06-151-2/+2
| | | | | | | | | | Update exception handling syntax to use the modern style: except ExcType as localvar (Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d) Signed-off-by: Scott Garman <scott.a.garman@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/server/process: Implement getEvent()Richard Purdie2011-06-081-0/+7
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/server/process: Update to new server APIRichard Purdie2011-06-081-32/+75
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add process server from upstream bitbakeRichard Purdie2011-06-081-0/+221
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-11 07:35:13 +0000