summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/server
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: xmlrpc: Show a better error message for connection failuresRichard Purdie2013-10-071-3/+6
| | | | | | | | | Showing a traceback when unable to connect to a bitbake server is rather ugly. This change allows us to show a sensible error message. (Bitbake rev: 26913202f83fbbecdce95da59515af102bcde4a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker,xmlrpc,servers: implement CookerFeaturesAlexandru DAMIAN2013-09-223-13/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implementing feature set selection that allows a client to enable specific features in the server at connection time. Only enabling of features is supported, as there is no way to safely remove data loaded into the cooker. Once enabled, a feature will remain enabled for the life of the cooker. Client-server connection now supports specifying the feature set required by the client. This is implemented in the Process server using a managed proxy list, so the server cooker will now load dynamically needed features based on what client connects to it. In the XMLRPC server the feature set is requested by using a parameter for registerUIHandler function. This allows observer-only clients to also specify features for the server. The server code configuration now is completly separated from the client code. All hardcoding of client knowledge is removed from the server. The extra_caches is removed as the client can now specify the caches it needs using the feature. The UI modules now need to specify the desired featureSet. HOB is modified to conform to the featureSet specification. The only feature available is CookerFeatures.HOB_EXTRA_CACHES which forces loading the bb.cache_extra:HobRecipeInfo class. (Bitbake rev: 98e594837aab89ea042cfa9f3740d20a661b14e2) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Rename confusing 'stop' state to 'forceshutdown'Richard Purdie2013-09-141-1/+1
| | | | | | | | | | | | The shutdown state causes the server to finish what its doing, stop was them meant to completely stop it. It doesn't mean the server is stopped though. Renaming the current stop event for forceshutdown gives more meaning to what it actually does. The stopped namespace then becomes available to indicate a completely stopped server. (Bitbake rev: 12e9d33bfae5294e3870dfd1202f63383ad05e92) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: xmlrpc: delete unused code BitBakeUIEventServerAlexandru DAMIAN2013-09-131-45/+0
| | | | | | | | | | | | | BitBakeUIEventServer is an unused class that pushes UI events over a separate thread. The current version of XMLRPC server works just fine with the classic UI event handlers, so this class is not needed. (Bitbake rev: 8e8e17631d790271b1be747c4b45059ec38ab606) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc: stop server on client exitAlexandru DAMIAN2013-09-011-0/+5
| | | | | | | | | | | | If the server only expects a single client to connect (i.e. no bind parameter set, so there is no way for the clients to get the server port), stop the server after the first client exits. (Bitbake rev: eb6bae56f62082bf147045311154cbae4bca0f4c) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process, server/xmlrpc, runqueue: Use select.select() on ↵Richard Purdie2013-09-012-19/+18
| | | | | | | | | | | | | | | | | | | | | | fds, not time.sleep() The existing backend server implementations were inefficient since they were sleeping for the full length of the timeouts rather than being woken when there was data ready for them. It was assumed they would wake and perhaps did when we forked processes directory but that is no longer the case. This updates both the process and xmlrpc backends to wait using select(). This does mean we need to pass the file descriptors to wait on from the internals who know which these file descriptors are but this is a logical improvement. Tests of a pathaolgical load on the process server of ~420 rapid tasks executed on a server with BB_NUMBER_THREAD=48 went from a wall clock measurement of the overall command execution time of 75s to a much more reasonable 24s. (Bitbake rev: 9bee497960889d9baa0a4284d79a384b18a8e826) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc: Increase timeout to 60sRichard Purdie2013-08-301-1/+1
| | | | | | | | | | | | This is a better value that the earlier infinite timeout yet still allows for servers with high loads. It does mean the bitbake process can hang at exit for the timeout period but that should never happen and only happened for me in some test cases which wouldn't happen in normal use. (Bitbake rev: ab8d926b9bc27c58011e7db9327e031ac76ba34b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add ui event handlers filteringRichard Purdie2013-08-262-5/+13
| | | | | | | | | | | | | | | | | | | Add functionality to allow UIs to update and change the types of events they recieve. To do this we need to add a new command and also need to be able to obtain the current event hander ID. In the case of xmlrpc, this is straightforward, in the case of the process server we need to save the result in a multiprocessing.Value() so we can retrive it. An excplit command was added to the server API to facilitate this. The same function can also be used to mask or unmask specific log messages, allowing the UI to optionally differ from the standard set of message filtering. Based upon work by Cristiana Voicu <cristiana.voicu@intel.com> (Bitbake rev: ba5a6c88785d9889d4172ec79937ac2a5555327e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc/prserv: Add sane timeout to default xmlrpc serverRichard Purdie2013-08-261-4/+18
| | | | | | | | | | | | | | | | | The standard python socket connect has long timouts which make sense for remote connections but not local things like the PR Service. This adds a timeout parameter to the common xmlrpc server creation function and sets it to a more reasonable 5 seconds. Making the PR server instantly exit is a good way to test the effect of this on bitbake. We can remove the bodged timeout in the PRServer terminate function which has the side effect of affecting global scope. (Bitbake rev: 8c01cff94787abbb64fbdf0c16cd63f8f97a7e03) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Improve exit handling and hangsRichard Purdie2013-08-261-19/+17
| | | | | | | | | | | | | | | | | | | | | | | It turns out we have a number of different ways the process server termination can hang. If we call cancel_join_thread() on the event queue, it means that it can be left containing partial data. This means the reading of the event queue in the terminate() function can hang, the timeout and block parameters to Queue.get() don't make any difference. Equally, if we don't call cancel_join_thread(), the join_thread in terminate() will hang giving a different deadlock. The best solution I could find is to loop over the process is_alive() after requesting it stops, trying to join the thread and if that fails, try and flush the event queue again. It wasn't clear what difference a force option should make in this case, we're gracefully trying to empty queues and shut down regardless of whether its a SIGTERM so I've simply removed the force option. (Bitbake rev: c5c8f33ca4b81877a0115887849881001b745bf0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty, xmlrpc: add observer-only modeAlexandru DAMIAN2013-06-171-8/+18
| | | | | | | | | | | | | | | | | | | | | I add an observer only mode for the knotty UI and the XMLRPC server that will allow the UI to register a callback with a server in order to receive events. The observer-UI is able to send read-only commands to the server, and also is able to register as an event handler. Read-only commands are the commands that do not change the state of the server and have been marked as such in the command module. The observer can switch to a full client if it calls addClient at any time, and the server has no other client running. (Bitbake rev: 4de9ee21f1fa4d04937cc7430fb1fc8b7a8f61e2) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: compat/server/utils: Jettison pre python 2.7.3 workaroundsRichard Purdie2013-06-142-139/+11
| | | | | | | | | Now we've moved to require python 2.7.3, we can jettison the compatibility workarounds/hacks for older python versions. (Bitbake rev: a51c402304f2080a76720f9b31d6dfdbed393bba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: xmlrpc: removes the event servers controllerAlexandru DAMIAN2013-06-141-59/+0
| | | | | | | | | | | | | XMLRPC API ran a XMLRPC server at port+2 in order to provide endpoints to register an event server in which to dump the events. This is no longer used, so we remove it. (Bitbake rev: e171a363913a86e56266f4c9d107110c7f5221e3) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: xmlrpc: Allow server to remain memory residentAlexandru DAMIAN2013-06-071-11/+9
| | | | | | | | | | | | | | | | This patch enables the XMLRPC server to remain resident in memory after a task is run, and to accept a new controlling client. To check the server after task completion, do lsof bitbake.lock in the build directory. Kill the server with kill. (Bitbake rev: e823e1f0675ff3794eb39ef0b4df2d7a220f4013) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake server: create common server infrastructureAlexandru DAMIAN2013-06-073-72/+141
| | | | | | | | | | | | | | | | | | | In an attempt to minimize code duplication, create clear interfaces, and maximize code reuse through OOP, bb.server adds base classes for the BitBakeServer, BitBakeServerConnection and actual server implementations instructed in particular server types. These classes document the minimum interfaces that the derived classes must implement, and provide boilerplate code. Changes to None, Process and XMLRPC servers as to use the common server infrastructure. (Bitbake rev: 6db4a64cef20f8d0aba804db4c4e1eec7b112b46) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: xmlrpc remote serverAlexandru DAMIAN2013-05-301-6/+200
| | | | | | | | | | | | | | | | Added code in XMLRPC server that creates a stub local server for a client-only connection and is able to connect to a remote server, and receive events from the remote server. Added the option to start a client with a remote server in bitbake. Original code by Bogdan Marinescu <bogdan.a.marinescu@intel.com> (Bitbake rev: 25b2af76104d5aaf6435de8c158e0407512f97ce) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: xmlrpc transport has identification tokenAlexandru DAMIAN2013-05-301-66/+100
| | | | | | | | | | | | | | | In order to be able to identify different clients over a stateless XMLRPC connection, we add a custom header named Bitbake-token, which identifies each client. We refactor the rest of the code to use the new transport. Based on a patch by Bogdan Marinescu <bogdan.a.marinescu@intel.com> (Bitbake rev: a00c2186bffe848a7cedf31969b904f8f7322ae6) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/bitbake: Remove launchUI methodRichard Purdie2013-05-302-9/+0
| | | | | | | | | With the removal of the none server type, we can remove the launchUI method and simplify the code slightly. (Bitbake rev: 9bef2f2dd0bcaa59528ebcb3c1ce053b7dff1ec6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server: Remove none serverRichard Purdie2013-05-301-203/+0
| | | | | | | | | | | | The process server backend has been serving well as the default for a long time now and the UI model is much better thought out that it used to be. With the move to make bitbake a memory resident process, the none server is now looking rather pointless and complicates the code needlessly. Lets therefore now remove it. (Bitbake rev: 9af03a89605e3db9bce3cea1e0f2d0b6cfaa6fe1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: print is a function in moderm python, covert remaining usersRichard Purdie2013-05-091-1/+1
| | | | | | (Bitbake rev: d0b180d868390a464b6799ad90db6bbe0350e158) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Always use separate process for PR ServiceRichard Purdie2013-02-061-1/+1
| | | | | | | | | | | | | | | Using the threading module interacts badly with multiprocessing used elsewhere in bitbake under certain machine loads. This was leading to bitbake hanging on Ctrl+C when the PR Server was being used. This patch converts it to always use the daemonize code which then means the threading code isn't required. [YOCTO #3742] (Bitbake rev: 2d0bbd9398ab839bd2d1e29e50b25d52efb1ce2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: xmlrpc: Unbreak the transport by adding a missing logfile nameRichard Purdie2012-12-041-1/+1
| | | | | | | | | I appreciate a fixed value is suboptimal but this is better than not working at all! (Bitbake rev: d338a1ad14680af41d218772decd03e98eb2cad5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process.py: Change timeout error handlingRichard Purdie2012-11-211-2/+2
| | | | | | | | | | | | | | | | | | | In normal usage, we never hit the timeout issue. If we do, it becomes obvious that the current error handling is not good enough. The request may have made it to the server and the answer will get queued. This means the next command may get the return value from the previous command with suitably puzzling results. Without rewriting large sections of code, its not possible to avoid this problem. It is better to increase the timeout to several seconds giving the server a chance to respond and if it does timeout, hard exit since recovery is not possible with the code base today. I'd be happy to see the structure of this code improved but this quick fix at least stops corrupted builds from happening which has to be a good thing. (Bitbake rev: 410c11dd10736873f2dc587fbe9119c38831e693) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: add error to return of runCommandChristopher Larson2012-11-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, command.py can return an error message from runCommand, due to being unable to run the command, yet few of our UIs (just hob) can handle it today. This can result in seeing a TypeError with traceback in certain rare circumstances. To resolve this, we need a clean way to get errors back from runCommand, without having to isinstance() the return value. This implements such a thing by making runCommand also return an error (or None if no error occurred). As runCommand now has a method of returning errors, we can also alter the getCmdLineAction bits such that the returned value is just the action, not an additional message. If a sync command wants to return an error, it raises CommandError(message), and the message will be passed to the caller appropriately. Example Usage: result, error = server.runCommand(...) if error: log.error('Unable to run command: %s' % error) return 1 (Bitbake rev: 717831b8315cb3904d9b590e633000bc897e8fb6) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/server: Remove dead console log codeRichard Purdie2012-11-013-5/+5
| | | | | | | | This code is dead and doesn't do anything so lets remove it. (Bitbake rev: 8d45739f49618757a5d7d79782deda355e3981ec) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add client socket info for BitBakeServerConnectionDongxiao Xu2012-02-231-2/+2
| | | | | | | | | | In server/client split model, the client will bind to a specific address and port. We need to pass the values to BitBakeServerConnection(). (Bitbake rev: c8e19c5c389efc06696084c6f9439ba75472c5b7) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: add -B option to bind with interfaceDongxiao Xu2012-02-231-3/+3
| | | | | | | | | | When start bitbake as a server only process, we need to assign certain interface to it. (Bitbake rev: 95b97d2dc6466ea3d99371f5b5bd68f6f3c99074) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* xmlrpc: Change BitbakeServerInfo init functionDongxiao Xu2012-01-061-4/+4
| | | | | | | | | | | Pass host and port to BitbakeServerInfo class instead of the "server" instance. With this change, GUI can connect with server individually by host address and port. (Bitbake rev: 5124351d6c287185723c98e6e4400c5402a5c8a8) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/process.py: Ensure queued UI events are queued right before we add ↵Richard Purdie2011-07-071-3/+2
| | | | | | | | our own handler (Bitbake rev: c7a9ef70ba91d47d53074e8d78cbc52f396144a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process.py: Fix issue where early errors weren't making it to the consoleRichard Purdie2011-06-161-1/+1
| | | | | | (Bitbake rev: d97f7d762e3d2f1b0da038d4d99f2531b2490670) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* make exception handling syntax consistentScott Garman2011-06-151-2/+2
| | | | | | | | | | Update exception handling syntax to use the modern style: except ExcType as localvar (Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d) Signed-off-by: Scott Garman <scott.a.garman@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/server/xmlrpc: Fix typoRichard Purdie2011-06-081-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/server/process: Implement getEvent()Richard Purdie2011-06-081-0/+7
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/server/process: Update to new server APIRichard Purdie2011-06-081-32/+75
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add process server from upstream bitbakeRichard Purdie2011-06-081-0/+221
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake none/xmlrpc servers: Only send pickled events to the xmlrpc serverRichard Purdie2011-06-081-2/+1
| | | | | | | Only the xmlrpc server needs pickled events. Use the function names to signify this requirement. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Cleanup bitbake server init process to be clearer to followRichard Purdie2011-06-082-37/+72
| | | | | | | | Create a standard format server class instance with method calls for each step in the server setup. There should be enough hooks for each of the different server types. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/xmlrpc: only use BBTransport for affected Python versionsJoshua Lock2011-03-141-4/+17
| | | | | | | | | Upstream have fixed the xmlrpclip.Transport() bug from Python #8194 for the Python 2.7.2 release, therefore as we know which versions of the standard library are affected we can only use our copy/paste class when it's needed. Signed-off-by: Joshua Lock <josh@linux.intel.com>
* bitbake/server/none: fix getEvent() to return eventsPaul Eggleton2011-02-241-4/+8
| | | | | | | | | | In the none server, events don't get processed unless the idle_commands function gets called, which previously wasn't happening with getEvent(); thus UIs that use this to get events were not working. Fixes [BUGID #561] Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* bitbake/server/none: remove leftover XMLRPC bits from none serverPaul Eggleton2011-02-241-11/+3
| | | | | | | Remove some comments, imports etc. to do with XMLRPC (inherited from xmlrpc.py which this file was based upon.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* bitbake/runqueue.py: Move SIGCHLD handler to wrap none server sleep callRichard Purdie2011-02-181-0/+7
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Queue up events before the UI is spawnedChris Larson2011-01-042-0/+4
| | | | | | | | | | | | | | - Queue up any events fired to the UI before the UI exists - At exit, check if UIs exist, and if not, flush the queue of LogRecords to the console directly. - When establishing a connection from the UI to the server, flush the queue of events to the queue in the server connection, so the UI will receive them when it begins its event loop. (Bitbake rev: 73488aeb317ed306f2ecf99cc9d3708526a5933c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/xmlrpc: Modify xmlrpc server to work with Python 2.7Joshua Lock2010-12-071-2/+66
| | | | | | | | | | | | | | | | | Python 2.7's library changes some of xmlrpclib's internal implementation such that interacting with a proxy to BitBakes SimpleXMLRPCServer would cause BitBake to crash. The issue was traced to changes in the xmlrpclib.Transport implementation and Python bug #8194 (http://bugs.python.org/issue8194). This patch introduces a workaround by create a subclass of xmlrpclib.Transport, which overrides the offending methods with the Python 2.6.6 implementation copy and pasted from the Python 2.6.6 xmlrpclib, and using this BBTransport implementation for both xmlrpclib.Server objects we create. Signed-off-by: Joshua Lock <josh@linux.intel.com>
* bitbake: Simplfy pre and post cooker hooksRichard Purdie2010-11-282-13/+3
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Rewrite profiling code so its functional for both none and xmlrpc ↵Richard Purdie2010-11-132-6/+29
| | | | | | backends Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/server: Raise a shutdown event if we're seeing exceptionsRichard Purdie2010-08-201-1/+2
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Apply some 2to3 transforms that don't cause issues in 2.6Chris Larson2010-07-022-2/+2
| | | | | | | (Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Apply the 2to3 print function transformChris Larson2010-07-021-2/+2
| | | | | | | (Bitbake rev: ff2e28d0d9723ccd0e9dd635447b6d889cc9f597) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Formatting cleanupsChris Larson2010-07-022-6/+4
| | | | | | | (Bitbake rev: 2caf134b43a44dad30af4fbe33033b3c58deee57) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Don't import xmlrpc server unless neededRichard Purdie2010-01-221-2/+0
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-24 10:58:15 +0000