linux/poky.git/bitbake/bin/bitbake-worker, branch 2.7_M2

bitbake: runqueue: Pass unique hash to task

2019-01-03T22:47:11+00:00

The unique hash is now passed to the task in the BB_UNIHASH variable [YOCTO #13030] (Bitbake rev: aab80b099f6f259e4b57cba2c26dd385d07c5947) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake-worker: Pass taskhash as runtask parameter

2018-12-07T12:38:58+00:00

Pass the task hash as a parameter to the 'runtask' message instead of passing the entire dictionary of hashes when the worker is setup. This is possible less efficient, but prevents the worker taskhashes from being out of sync with the runqueue in the event that the taskhashes in the runqueue change. [YOCTO #13030] (Bitbake rev: 1e86d8c1bec7ea5d016a5ad2097f999362e29033) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie

bitbake: bitbake: be more explicit when warning about locale choice

2017-11-07T13:32:13+00:00

(Bitbake rev: 286dce008d6e0bd3121393b28ca02de1385519fb) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: cooker: Add BB_LIMITEDDEPS support

2017-01-20T11:52:52+00:00

When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited. We really need a way to know this from the metadata and this new variable provides this in worker context. This means existing code can stop having to guess. (Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402) Signed-off-by: Richard Purdie

bitbake: runqueue: dry-run real tasks when BB_SETSCENE_ENFORCE is set

2016-12-20T15:22:50+00:00

For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used by the installation process for the extensible SDK), we don't actually need the whitelisted real tasks to execute - we just need to have them in the dependency tree so that we get all of the setscene tasks they depend on to run. Therefore we can actually dry-run those real tasks i.e. they won't be run (and thus we won't waste a significant amount of time doing so) and won't be stamped as having run either. We do already have a dry-run mode in BitBake (activated by the -n or --dry-run command line option), but it dry-runs the setscene tasks as well which we don't want here. Note that this has no effect on the checking we are doing with BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are scheduled to run - that's handled separately. Fixes [YOCTO #10369]. (Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b) Signed-off-by: Paul Eggleton Signed-off-by: Richard Purdie

bitbake: runqueue: enable setVariable command to affect task execution

2016-12-14T12:25:07+00:00

Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. (Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4) Signed-off-by: Paul Eggleton Signed-off-by: Richard Purdie

bitbake: bitbake-worker: Further IO performance tweaks

2016-12-08T10:32:21+00:00

Looking further at the CPU loads on systems running large numbers of tasks, the following things helps performance: * Loop on waitpid until there are no processes still waiting * Using select to wait for the cooker pipe to be writable before writing avoiding pointless 100% cpu usage * Only reading from worker pipes that select highlights are readable (Bitbake rev: 9375349e27b08b4d1cfe4825c042d4c82120e00b) Signed-off-by: Richard Purdie

bitbake: bitbake: remove True option to getVarFlag calls

2016-11-30T15:48:09+00:00

getVarFlag() now defaults to expanding by default, thus remove the True option from getVarFlag() calls with a regex search and replace. Search made with the following regex: getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\) (Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd) Signed-off-by: Joshua Lock Signed-off-by: Richard Purdie

bitbake: bitbake: remove True option to getVar calls

2016-11-30T15:48:09+00:00

getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock Signed-off-by: Richard Purdie

bitbake: bitbake-worker: Handle cooker/worker IO deadlocking

2016-11-29T11:22:06+00:00

I noiced builds where tasks seemed to be taking a surprisingly long time. When I looked at the output of top/pstree, these tasks were no longer running despite being listed in knotty. Some were in D/Z state waiting for their exit code to be collected, others were simply not present at all. strace showed communication problems between the worker and cooker, each was trying to write to the other and nearly deadlocking. Eventually, timeouts would allow them to echange 64kb of data but this was only happening every few seconds. Whilst this particularly affected builds on machines with large numbers of cores (and hence highly parallal task execution) and in cases where I had a lot of debug enabled, this situation is clearly bad in general. This patch introduces a thread to the worker which is used to write data back to cooker. This means that the deadlock can't occur and data flows much more freely and effectively. (Bitbake rev: 3cb0d1c78b4c2e4f251a59b86c8da583828ad08b) Signed-off-by: Richard Purdie