summaryrefslogtreecommitdiffstats
path: root/bitbake/bin/bitbake-worker
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: bitbake-worker: Remove unnecessary addDefaultLogFilterJoshua Watt2020-03-131-1/+0
| | | | | | | | | | | Adding the default log filter here is unnecessary because there are no defined logging domains when it is called, which means it does no actual filtering. (Bitbake rev: dcdb8f2c14f09ce34d0a1facc33a441570912c05) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/msg: Use log level instead of debug countJoshua Watt2020-03-131-1/+1
| | | | | | | | | | | | | Passes around the actual logging level as the default log level variable instead of the debug count. This makes it easier to deal with logging levels since the conversion from debug count and verbose flag only has to occur once when logging is initialized and after that actual log levels can be used (Bitbake rev: 41bd155faf7f65cb0727fcce972715769b26ca89) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker child process create group before registering ↵Ivan Efimov2019-11-071-3/+5
| | | | | | | | | | | | | | | | | | SIGTERM handler The bitbake-worker child on the SIGTERM signal handling send the SIGTERM to all processes in it's process group. In cases when the bitbake-worker child got SIGTERM after registering own SIGTERM handler and before the os.setsid() call it can send SIGTERM to unwanted processes. In the worst case during SIGTERM processing the bitbake-worker child can be in the group of the process that started BitBake itself. As a result it can kill processes that not related to BitBake at all. (Bitbake rev: b97b1ef0b1b00848a4a44b34eca123ccf33188f4) Signed-off-by: Ivan Efimov <i.efimov@inango-systems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rework hash equivalenceJoshua Watt2019-09-181-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/hashserv: Allow autostarting of a local hash server using ↵Richard Purdie2019-08-061-0/+1
| | | | | | | | | | | | | | | | | BB_HASHSERVE Its useful, particularly in the local developer model of usage, for bitbake to start and stop a hash equivalence server on local port, rather than relying on one being started by the user before the build. The new BB_HASHSERVE variable supports this. The database handling is moved internally into the hashserv code so that different threads/processes can be used for the server without errors. (Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Enable dynamic task adjustment to hash equivalencyRichard Purdie2019-08-061-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a compelling usecase for tasks being able to notify runqueue that their "unihash" has changed. When this is recieved, the hashes of all subsequent tasks should be recomputed and their new hashes checked against existing setscene validity. Any newly available setscene tasks should then be executed. Making this work effectively needs several pieces. An event is added which the cooker listen for. If a new hash becomes available it can send an event to notify of this. When such an event is seen, hash recomputations are made. A setscene task can't be run until all the tasks it "covers" are stopped. The notion of "holdoff" tasks is therefore added, these are removed from the buildable list with the assumption that some setscene task will run and cover them. The workers need to be notified when taskhashes change to update their own internal siggen data stores. A new worker command is added to do this which will affect all newly spawned worker processes from that worker. An example workflow which tests this code is: Configuration: BB_SIGNATURE_HANDLER = "OEEquivHash" SSTATE_HASHEQUIV_SERVER = "http://localhost:8686" $ bitbake-hashserv & $ bitbake automake-native $ bitbake autoconf-native automake-native -c clean $ bitbake m4-native -c install -f $ bitbake automake-native with the test being whether automake-native is installed from sstate. (Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+3
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Pass unique hash to taskJoshua Watt2019-01-031-3/+4
| | | | | | | | | | | The unique hash is now passed to the task in the BB_UNIHASH variable [YOCTO #13030] (Bitbake rev: aab80b099f6f259e4b57cba2c26dd385d07c5947) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Pass taskhash as runtask parameterJoshua Watt2018-12-071-4/+4
| | | | | | | | | | | | | | | Pass the task hash as a parameter to the 'runtask' message instead of passing the entire dictionary of hashes when the worker is setup. This is possible less efficient, but prevents the worker taskhashes from being out of sync with the runqueue in the event that the taskhashes in the runqueue change. [YOCTO #13030] (Bitbake rev: 1e86d8c1bec7ea5d016a5ad2097f999362e29033) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: be more explicit when warning about locale choiceRoss Burton2017-11-071-2/+1
| | | | | | | (Bitbake rev: 286dce008d6e0bd3121393b28ca02de1385519fb) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add BB_LIMITEDDEPS supportRichard Purdie2017-01-201-0/+2
| | | | | | | | | | | When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited. We really need a way to know this from the metadata and this new variable provides this in worker context. This means existing code can stop having to guess. (Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: dry-run real tasks when BB_SETSCENE_ENFORCE is setPaul Eggleton2016-12-201-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used by the installation process for the extensible SDK), we don't actually need the whitelisted real tasks to execute - we just need to have them in the dependency tree so that we get all of the setscene tasks they depend on to run. Therefore we can actually dry-run those real tasks i.e. they won't be run (and thus we won't waste a significant amount of time doing so) and won't be stamped as having run either. We do already have a dry-run mode in BitBake (activated by the -n or --dry-run command line option), but it dry-runs the setscene tasks as well which we don't want here. Note that this has no effect on the checking we are doing with BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are scheduled to run - that's handled separately. Fixes [YOCTO #10369]. (Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: enable setVariable command to affect task executionPaul Eggleton2016-12-141-2/+10
| | | | | | | | | | | | Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. (Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Further IO performance tweaksRichard Purdie2016-12-081-4/+9
| | | | | | | | | | | | | | Looking further at the CPU loads on systems running large numbers of tasks, the following things helps performance: * Loop on waitpid until there are no processes still waiting * Using select to wait for the cooker pipe to be writable before writing avoiding pointless 100% cpu usage * Only reading from worker pipes that select highlights are readable (Bitbake rev: 9375349e27b08b4d1cfe4825c042d4c82120e00b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVarFlag callsJoshua Lock2016-11-301-1/+1
| | | | | | | | | | | | | | getVarFlag() now defaults to expanding by default, thus remove the True option from getVarFlag() calls with a regex search and replace. Search made with the following regex: getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\) (Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar callsJoshua Lock2016-11-301-2/+2
| | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Handle cooker/worker IO deadlockingRichard Purdie2016-11-291-16/+36
| | | | | | | | | | | | | | | | | | | | | | | | I noiced builds where tasks seemed to be taking a surprisingly long time. When I looked at the output of top/pstree, these tasks were no longer running despite being listed in knotty. Some were in D/Z state waiting for their exit code to be collected, others were simply not present at all. strace showed communication problems between the worker and cooker, each was trying to write to the other and nearly deadlocking. Eventually, timeouts would allow them to echange 64kb of data but this was only happening every few seconds. Whilst this particularly affected builds on machines with large numbers of cores (and hence highly parallal task execution) and in cases where I had a lot of debug enabled, this situation is clearly bad in general. This patch introduces a thread to the worker which is used to write data back to cooker. This means that the deadlock can't occur and data flows much more freely and effectively. (Bitbake rev: 3cb0d1c78b4c2e4f251a59b86c8da583828ad08b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: print full traceback instead of message onlyMarkus Lehtonen2016-11-041-2/+3
| | | | | | | | | | | | | | Print full traceback instead of just the exception message in the child() function inside fork_off_task(). This makes debugging a lot easier as the function catches a generic "Exception" and the exception message alone might not give much information. [YOCTO #10393] (Bitbake rev: 9c7cc981408c9b4bbbff98ae93ff22199f6a8219) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Initial multi-config supportRichard Purdie2016-08-181-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds the notion of supporting multiple configurations within a single build. To enable it, set a line in local.conf like: BBMULTICONFIG = "configA configB configC" This would tell bitbake that before it parses the base configuration, it should load conf/configA.conf and so on for each different configuration. These would contain lines like: MACHINE = "A" or other variables which can be set which can be built in the same build directory (or change TMPDIR not to conflict). One downside I've already discovered is that if we want to inherit this file right at the start of parsing, the only place you can put the configurations is in "cwd", since BBPATH isn't constructed until the layers are parsed and therefore using it as a preconf file isn't possible unless its located there. Execution of these targets takes the form "bitbake multiconfig:configA:core-image-minimal core-image-sato" so similar to our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND. Implementation wise, the implication is that instead of tasks being uniquely referenced with "recipename/fn:task" it now needs to be "configuration:recipename:task". We already started using "virtual" filenames for recipes when we implemented BBCLASSEXTEND and this patch adds a new prefix to these, "multiconfig:<configname>:" and hence avoid changes to a large part of the codebase thanks to this. databuilder has an internal array of data stores and uses the right one depending on the supplied virtual filename. That trick allows us to use the existing parsing code including the multithreading mostly unchanged as well as most of the cache code. For recipecache, we end up with a dict of these accessed by multiconfig (mc). taskdata and runqueue can only cope with one recipecache so for taskdata, we pass in each recipecache and have it compute the result and end up with an array of taskdatas. We can only have one runqueue so there extensive changes there. This initial implementation has some drawbacks: a) There are no inter-multi-configuration dependencies as yet b) There are no sstate optimisations. This means if the build uses the same object twice in say two different TMPDIRs, it will either load from an existing sstate cache at the start or build it twice. We can then in due course look at ways in which it would only build it once and then reuse it. This will likely need significant changes to the way sstate currently works to make that possible. (Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Build datastores from databuilder objectRichard Purdie2016-08-181-10/+13
| | | | | | | | | | | | | | Rather than passing in a datastore to build on top of, use the data builder object in the cache and base the parsed recipe from this. This turns things into proper objects building from one another rather than messy mixes of static and class functions. This sets things up so we can support parsing and building multiple configurations. (Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: don't reassign sys.stdoutEd Bartosh2016-07-081-2/+1
| | | | | | | | | | | | | | | | | | | | | | Worker needs input stream in binary mode as it reads binary content from it. Current code does it by detaching a buffer from sys.stdin and assigning it back to sys.stdin. Detached buffer is io.BufferedReader in binary mode. This operation is implicit as its purpose is not easily understandable from the code. Replacing it with fdopen(sys.stdin.fileno(), 'rb') should make the code more understandable. Assigning the buffer to sys.stdin is not needed as worker doesn't use sys.stdin. Moreover, it leads to difficult to debug issues down the stack. For example, devpyshell doesn't work without reopening sys.stdin in text mode. This is not needed anymore after this fix as sys.stdin is not changed in worker code and remains in text mode. (Bitbake rev: b26bcff4c4d72775f1def7e769015464953b955c) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to python 3Richard Purdie2016-06-021-29/+31
| | | | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Implement support for per-task exportsChristopher Larson2016-05-191-0/+10
| | | | | | | (Bitbake rev: 4506ccf1495c6ed6e8ed678f4baa166bc94d1761) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bin/bitbake-worker: Fix invalid bb.msg.fatal usageRichard Purdie2016-05-131-1/+2
| | | | | | | | | The logging domain specified to bb.msg.fatal was invalid. Replace with a logger.critical() call instead. (Bitbake rev: 1ffd8737e065a3cd634c74cd67e634d785ea93a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update logger.warn() -> logger.warning()Richard Purdie2016-05-111-1/+1
| | | | | | | | | | python deprecated logger.warn() in favour of logger.warning(). This is only used in bitbake code so we may as well just translate everything to avoid warnings under python 3. Its safe for python 2.7. (Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker, bitbake-worker: Fix spelling of "received"Phil Blundell2016-02-011-2/+2
| | | | | | | | | I before E, except after C... (Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df) Signed-off-by: Phil Blundell <pb@pbcl.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Set process names to be meaninfulRichard Purdie2016-01-301-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Guard against multiprocessing corruption of event dataRichard Purdie2015-10-011-0/+9
| | | | | | | | | | | | | | | | | | | | In the forked child, we may use multiprocessing. There is only one event pipe to the worker controlling process and if we're unlucky, multiple processes can write to it at once corrupting the data by intermixing it. We don't see this often but when we do, its quite puzzling. I suspect it only happens in tasks which use multiprocessng (do_rootfs, do_package) and is much more likely to happen when we have long messages, usually many times PAGE_SIZE since PAGE_SIZE writes are atomic. This makes it much more likely within do_roofs, when for example a subprocess lists the contents of a rootfs. To fix this, we give each child a Lock() object and use this to serialise writes to the controlling worker. (Bitbake rev: 3cb55bdf06148943960e438291f9a562857340a3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Ensure pipe closure doesn't crash before killpg()Richard Purdie2015-09-121-1/+5
| | | | | | | | | | If the pipe is closed, we want to ensure that we kill any child processes by triggering the sigterm handler before we exit. This code does that, hopefully avoiding the remaining process left behind issues on the autobuilder. (Bitbake rev: 60f6c2818f38c4d9c2d9aaa42acf3071636f4a3b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Handle SIGKILL of parent gracefullyRichard Purdie2015-09-091-2/+7
| | | | | | | | | | | | | | If we SIGKILL cooker (the parent process), ensure the worker notices and shuts down gracefully. To do this: * trigger the sigterm handler if the parent exits * ensure broken pipe writes don't trigger backtraces which interfer with other exit work * notice if our command pipe is broken due to EOF and sigterm if so (Bitbake rev: c43d6a8d711db8d3bd9a1976b9f8e3efdb4cb4ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Simple code cleanupRichard Purdie2015-09-091-3/+2
| | | | | | | | start/end are unused here and we can improve the code conditional blocks. (Bitbake rev: 68f53dd77fe0bbfa044bd037a9484e0e1c9088b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Fix regression with unbuffered logsJason Wessel2015-05-261-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | I noticed that I was seeing loss of the log files when hitting control-c while debugging a function in bitbake. In fact if you take a recipe and replace its compile function as shown below let it run for a few seconds and hit control-c, you will see first hand that log data is not there. do_compile () { while [ 1 ] ; do echo -n "Output date: " date sleep 1 done } It turns out there was a regression introduced by commit: d0f0e5d9e69 which created the bitbake worker. Since the bitbake worker is started in its own process space, it needs the exact same code added from commit: 88429f018b where the problem was fixed the first time around. (Bitbake rev: 8d1748f75763b4a66516cc46d5457ee6404b1b68) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Use setsid() rather than setpgid()Richard Purdie2015-01-081-2/+5
| | | | | | | | | | | | | | | | | | | | | | The bug has a long discussion of this. Basically, in some environments, the exact details of which aren't understood, a Ctrl+C signal to the UI is being transmitted to all the process children. Looking at the output of "ps ax -O tpgid", its clear the main process is still the terminal owner of these processes. stty -a on a problematic system shows: "-ignbrk brkint" and on a working system shows: "-ignbrk -brkint" The description of brkint would suggest this is the problem, setting up that terminal environment wasn't able to reproduce the problem though. It was confirmed that using setsid() caused the problem to be resolved and is probably the right thing to be doing anyway, so lets do it. [YOCTO #6949] (Bitbake rev: 461aa73fff0ab616032d28c4fd0322eb88838be6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: exit normally when SIGHUPRobert Yang2014-11-201-1/+8
| | | | | | | | | | | | | | Fixed: 1) Run "bitbake recipe" in the terminal 2) Close the terminal while building 3) $ ps aux | grep bitbake-worker There will be many processes, and they will keep the resources (e.g., memory), and won't exit unless kill or kill -9. (Bitbake rev: 40d2ae0723de2bf5fee343faafb4afda40546839) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Fix bitbake -nRichard Purdie2014-09-221-2/+3
| | | | | | | | | | | | Without this you see: File "bitbake/bin/bitbake-worker", line 201, in fork_off_task os._exit(child()) TypeError: an integer is required (Bitbake rev: cd477b5e77ab0373248b8a8fa30e1c7b8ea984fd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue/bitbake-worker: Improve siggen data transfer interfaceRichard Purdie2014-09-111-1/+1
| | | | | | | | | | | | | We need to transfer some of the siggen data from the core/cooker into the worker instances. There was a partial API created for this but its ugly and its not possible to extend it from the siggen class. This patch completes the interface/abstraction for the data and means the class can extend/customise it in any siggen class. (Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Extra profiling data dumpRichard Purdie2014-08-281-4/+32
| | | | | | | | | | | | | | Currently we get no profiling oversight into either the main bitbake worker process, or the overall parsing before task execution. This adds in extra profiling hooks so we can truly capture all parts of bitbake's execution into the profile data. To do this we modify the 'magic' value passed to bitbake-worker to trigger the profiling, before the configuration data is sent over to the worker. (Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Improve sigterm handlerRichard Purdie2014-08-231-1/+6
| | | | | | | | | | | | | | When processes terminate, we really want all of the child processes to terminate too. This was not happening for worker processes which spawned their own multiprocessing pools, leading to build hangs. This change ensures any sigterm gets passed to the whole process group. In local tests, this resolved some hanging process workloads I could generate. It does rely on signals being delivered in a timely fashion and there is a multiprocessing bug we have to work around there. (Bitbake rev: 96f8ea07ace1379380fab2d78eb592fa40c867d4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Drop BBHASH variablesRichard Purdie2014-04-231-4/+0
| | | | | | | | | | | | | Iterating through and calling setVar on this number of variables has significant overhead in the profiling data. By not setting this, we save 3,000 calls to setVar which gives a noticeable improvement to the speed of task execution. The BBHASH variables have since been replaced by accessing that data through the siggen code and going forward, that is the preferred way work with it. (Bitbake rev: 92526eadd09d19938762290e0492076174367583) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Ensure children have default sigterm handlerRichard Purdie2014-03-111-0/+2
| | | | | | | | | | The children of the worker should have the default SIGTERM handler, else they'll try and do cleanup which should only happen in the parent leading to all kinds of bizarre build failures. (Bitbake rev: a53c8d1f846d94082aa459996c4114f10970b8ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Gracefully handle SIGTERMRichard Purdie2014-03-101-0/+8
| | | | | | | | | | Currently if bitbake-worker handles a SIGTERM, it leaves the child processes to complete or hang. It shouldn't do this so hook the SIGTERM event and gracefully shutdown any children. (Bitbake rev: 551406f3f9ee94de09d2da6e16fea054c6dbfdb7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Share BB_TASKDEPDATA with tasksRichard Purdie2013-11-261-3/+4
| | | | | | | | | | | | | | | | | Currently tasks have no knowledge of which other tasks they depend upon. This makes it impossible to do at least two things which would be desirable/interesting: a) Have the ability to create per recipe sysroots b) Allow the aclocal files to be present only for the entries in DEPENDS (directly and indirectly) By exporting task data through this new variable, tasks can inspect their dependencies and then take actions based upon this. (Bitbake rev: 84f1dde717dac22435005b79d03ee0b80a3e8e62) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/bitbake-worker: Fix dry run fakeroot issuesRichard Purdie2013-11-241-1/+2
| | | | | | | | | | | | | | When using the dry run option (-n), bitbake would still try and fire a specific fakeroot worker. This is doomed to failure since it might well not have been built. Add in some checks to prevent the failures. [YOCTO #5367] (Bitbake rev: f34d0606f87ce9dacadeb78bac35879b74f10559) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Ensure ${DATE} and ${TIME} are consistentPeter Kjellerstedt2013-09-061-0/+2
| | | | | | | | | | | | | | | Due to the worker split the ${DATE} and ${TIME} variables could end up with different values for different workers. E.g., a task like do_rootfs that is run within a fakeroot environment had a slightly different view of the time than another task that was not fakerooted which made it impossible to correctly refer to the image generated by do_rootfs from the other task. (Bitbake rev: 756cc69ebf8bfe8455d0c90f288dd51be2499773) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: ensure BUILDNAME is available during executionPaul Eggleton2013-09-041-0/+1
| | | | | | | | | | | | | | | BUILDNAME is set from cooker by default, so since the worker split it will not be set when executing functions. In OpenEmbedded this results in /etc/version (which is populated from BUILDNAME) not having any content. Pass this variable value through to the worker explicitly to fix the issue. Fixes [YOCTO #4818]. (Bitbake rev: 92940b0427d9b2b3f95e27c230ec1e36638a34bc) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: import needed signal moduleValentin Popa2013-07-311-0/+1
| | | | | | | | | | | | bitbake-worker makes use of the signal module but it doesn't import it. This patch fixes the issue. [YOCTO #4750] (Bitbake rev: c2ed639690f135994199eb24d964e37f57259e3a) Signed-off-by: Valentin Popa <valentin.popa@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Adapt autostart to bitbake-workerRichard Purdie2013-06-141-0/+1
| | | | | | | | | | With the change to bitbake-worker we need to ensure the workers know how to contact the PR service, the magic 0 port and singleton is no longer enough. (Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Split runqueue to use bitbake-workerRichard Purdie2013-06-141-0/+358
This is a pretty fundamental change to the way bitbake operates. It splits out the task execution part of runqueue into a completely separately exec'd process called bitbake-worker. This means that the separate process has to build its own datastore and that configuration needs to be passed from the cooker over to the bitbake worker process. Known issues: * Hob is broken with this patch since it writes to the configuration and that configuration isn't preserved in bitbake-worker. * We create a worker for setscene, then a new worker for the main task execution. This is wasteful but shouldn't be hard to fix. * We probably send too much data over to bitbake-worker, need to see if we can streamline it. These are issues which will be followed up in subsequent patches. This patch sets the groundwork for the removal of the double bitbake execution for psuedo which will be in a follow on patch. (Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-13 19:53:04 +0000