linux/poky.git/bitbake/lib/bb/server/process.py, branch daisy

bitbake: server/process: Optimise latency when finishing idle functions

2014-03-10T18:10:02+00:00

When idle functions finish, its likely we have some other work to do, so don't sleep in the select call but instead, skip it. This removes small amounts of latency in common commands. (Bitbake rev: 069d6538f83b607cb46c6fe21bf6c596e8b99242) Signed-off-by: Richard Purdie

bitbake: server/process: Drop unnecessary exit delay

2014-03-10T18:10:01+00:00

When the server exits, we no longer appear to need this delay. This is likely due to improvements in the various exit codepaths. There is therefore no longer any point in taking the latency hit. (Bitbake rev: 8e75ee29ae07e13f23525c5c6045fbf6cdbe7675) Signed-off-by: Richard Purdie

bitbake: server/process: Use a pipe for quit events instead of Event()

2014-03-10T18:10:01+00:00

Its not possible to notice the change of status of an Event() in the select call we sleep in. It would be possible in python 3.3 but for now use a pipe instead. This removes small latency when bitbake commands finish since the system doesn't sit in the select call. (Debugging these kind of issues is apparent by setting a long sleep for the select call) (Bitbake rev: def28239b0f0d5f1cf13214b263114a5328538b7) Signed-off-by: Richard Purdie

bitbake: server/process: Deal more gracefully with SIGTERM

2014-03-10T18:09:59+00:00

Currently a SIGTERM to the UI process causes the UI simply to lock up. By setting an exit flag, the waitEvent can raise a SIGINT, allowing the UI to break out the event loop and exit. Currently this is results in a traceback but that is more desirable than a hanging process. (Bitbake rev: 0d12041eceeae6bba2034b04913bb13abd67bd15) Signed-off-by: Richard Purdie

bitbake: server/process: Use the setFeatures command on the server instead of a manger

2014-03-10T18:09:59+00:00

The use of a manager in the process server causes some issues since it remains around for the lifetime of the server even though its only used during initialisation and the system doesn't respond well to SIGTERM events to the extra process (and two threads) the implementation involves. Switching to a dedicated command simplifies the server process structure. (Bitbake rev: 74532a7cf8ccea8b85f1cda5d5bc23d2f3c72a08) Signed-off-by: Richard Purdie

bitbake: bitbake: cooker,xmlrpc,servers: implement CookerFeatures

2013-09-22T11:19:43+00:00

Implementing feature set selection that allows a client to enable specific features in the server at connection time. Only enabling of features is supported, as there is no way to safely remove data loaded into the cooker. Once enabled, a feature will remain enabled for the life of the cooker. Client-server connection now supports specifying the feature set required by the client. This is implemented in the Process server using a managed proxy list, so the server cooker will now load dynamically needed features based on what client connects to it. In the XMLRPC server the feature set is requested by using a parameter for registerUIHandler function. This allows observer-only clients to also specify features for the server. The server code configuration now is completly separated from the client code. All hardcoding of client knowledge is removed from the server. The extra_caches is removed as the client can now specify the caches it needs using the feature. The UI modules now need to specify the desired featureSet. HOB is modified to conform to the featureSet specification. The only feature available is CookerFeatures.HOB_EXTRA_CACHES which forces loading the bb.cache_extra:HobRecipeInfo class. (Bitbake rev: 98e594837aab89ea042cfa9f3740d20a661b14e2) Signed-off-by: Alexandru DAMIAN Signed-off-by: Richard Purdie

bitbake: cooker: Rename confusing 'stop' state to 'forceshutdown'

2013-09-14T07:20:38+00:00

The shutdown state causes the server to finish what its doing, stop was them meant to completely stop it. It doesn't mean the server is stopped though. Renaming the current stop event for forceshutdown gives more meaning to what it actually does. The stopped namespace then becomes available to indicate a completely stopped server. (Bitbake rev: 12e9d33bfae5294e3870dfd1202f63383ad05e92) Signed-off-by: Richard Purdie

bitbake: server/process, server/xmlrpc, runqueue: Use select.select() on fds, not time.sleep()

2013-09-01T14:51:10+00:00

The existing backend server implementations were inefficient since they were sleeping for the full length of the timeouts rather than being woken when there was data ready for them. It was assumed they would wake and perhaps did when we forked processes directory but that is no longer the case. This updates both the process and xmlrpc backends to wait using select(). This does mean we need to pass the file descriptors to wait on from the internals who know which these file descriptors are but this is a logical improvement. Tests of a pathaolgical load on the process server of ~420 rapid tasks executed on a server with BB_NUMBER_THREAD=48 went from a wall clock measurement of the overall command execution time of 75s to a much more reasonable 24s. (Bitbake rev: 9bee497960889d9baa0a4284d79a384b18a8e826) Signed-off-by: Richard Purdie

bitbake: bitbake: Add ui event handlers filtering

2013-08-26T10:29:46+00:00

Add functionality to allow UIs to update and change the types of events they recieve. To do this we need to add a new command and also need to be able to obtain the current event hander ID. In the case of xmlrpc, this is straightforward, in the case of the process server we need to save the result in a multiprocessing.Value() so we can retrive it. An excplit command was added to the server API to facilitate this. The same function can also be used to mask or unmask specific log messages, allowing the UI to optionally differ from the standard set of message filtering. Based upon work by Cristiana Voicu (Bitbake rev: ba5a6c88785d9889d4172ec79937ac2a5555327e) Signed-off-by: Richard Purdie

bitbake: process: Improve exit handling and hangs

2013-08-26T10:29:46+00:00

It turns out we have a number of different ways the process server termination can hang. If we call cancel_join_thread() on the event queue, it means that it can be left containing partial data. This means the reading of the event queue in the terminate() function can hang, the timeout and block parameters to Queue.get() don't make any difference. Equally, if we don't call cancel_join_thread(), the join_thread in terminate() will hang giving a different deadlock. The best solution I could find is to loop over the process is_alive() after requesting it stops, trying to join the thread and if that fails, try and flush the event queue again. It wasn't clear what difference a force option should make in this case, we're gracefully trying to empty queues and shut down regardless of whether its a SIGTERM so I've simply removed the force option. (Bitbake rev: c5c8f33ca4b81877a0115887849881001b745bf0) Signed-off-by: Richard Purdie