summaryrefslogtreecommitdiffstats
path: root/bitbake/bin/bitbake-worker
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: bitbake: Convert to python 3 megacommit This needs breaking up into ↵Richard Purdie2016-05-161-29/+31
| | | | | | | | smaller changes. (Bitbake rev: cf51f19aed208a75d38c14cd585d9b9f115e3ba3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bin/bitbake-worker: Fix invalid bb.msg.fatal usageRichard Purdie2016-05-131-1/+2
| | | | | | | | | The logging domain specified to bb.msg.fatal was invalid. Replace with a logger.critical() call instead. (Bitbake rev: 1ffd8737e065a3cd634c74cd67e634d785ea93a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update logger.warn() -> logger.warning()Richard Purdie2016-05-111-1/+1
| | | | | | | | | | python deprecated logger.warn() in favour of logger.warning(). This is only used in bitbake code so we may as well just translate everything to avoid warnings under python 3. Its safe for python 2.7. (Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker, bitbake-worker: Fix spelling of "received"Phil Blundell2016-02-011-2/+2
| | | | | | | | | I before E, except after C... (Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df) Signed-off-by: Phil Blundell <pb@pbcl.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Set process names to be meaninfulRichard Purdie2016-01-301-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Guard against multiprocessing corruption of event dataRichard Purdie2015-10-011-0/+9
| | | | | | | | | | | | | | | | | | | | In the forked child, we may use multiprocessing. There is only one event pipe to the worker controlling process and if we're unlucky, multiple processes can write to it at once corrupting the data by intermixing it. We don't see this often but when we do, its quite puzzling. I suspect it only happens in tasks which use multiprocessng (do_rootfs, do_package) and is much more likely to happen when we have long messages, usually many times PAGE_SIZE since PAGE_SIZE writes are atomic. This makes it much more likely within do_roofs, when for example a subprocess lists the contents of a rootfs. To fix this, we give each child a Lock() object and use this to serialise writes to the controlling worker. (Bitbake rev: 3cb55bdf06148943960e438291f9a562857340a3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Ensure pipe closure doesn't crash before killpg()Richard Purdie2015-09-121-1/+5
| | | | | | | | | | If the pipe is closed, we want to ensure that we kill any child processes by triggering the sigterm handler before we exit. This code does that, hopefully avoiding the remaining process left behind issues on the autobuilder. (Bitbake rev: 60f6c2818f38c4d9c2d9aaa42acf3071636f4a3b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Handle SIGKILL of parent gracefullyRichard Purdie2015-09-091-2/+7
| | | | | | | | | | | | | | If we SIGKILL cooker (the parent process), ensure the worker notices and shuts down gracefully. To do this: * trigger the sigterm handler if the parent exits * ensure broken pipe writes don't trigger backtraces which interfer with other exit work * notice if our command pipe is broken due to EOF and sigterm if so (Bitbake rev: c43d6a8d711db8d3bd9a1976b9f8e3efdb4cb4ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Simple code cleanupRichard Purdie2015-09-091-3/+2
| | | | | | | | start/end are unused here and we can improve the code conditional blocks. (Bitbake rev: 68f53dd77fe0bbfa044bd037a9484e0e1c9088b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Fix regression with unbuffered logsJason Wessel2015-05-261-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | I noticed that I was seeing loss of the log files when hitting control-c while debugging a function in bitbake. In fact if you take a recipe and replace its compile function as shown below let it run for a few seconds and hit control-c, you will see first hand that log data is not there. do_compile () { while [ 1 ] ; do echo -n "Output date: " date sleep 1 done } It turns out there was a regression introduced by commit: d0f0e5d9e69 which created the bitbake worker. Since the bitbake worker is started in its own process space, it needs the exact same code added from commit: 88429f018b where the problem was fixed the first time around. (Bitbake rev: 8d1748f75763b4a66516cc46d5457ee6404b1b68) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Use setsid() rather than setpgid()Richard Purdie2015-01-081-2/+5
| | | | | | | | | | | | | | | | | | | | | | The bug has a long discussion of this. Basically, in some environments, the exact details of which aren't understood, a Ctrl+C signal to the UI is being transmitted to all the process children. Looking at the output of "ps ax -O tpgid", its clear the main process is still the terminal owner of these processes. stty -a on a problematic system shows: "-ignbrk brkint" and on a working system shows: "-ignbrk -brkint" The description of brkint would suggest this is the problem, setting up that terminal environment wasn't able to reproduce the problem though. It was confirmed that using setsid() caused the problem to be resolved and is probably the right thing to be doing anyway, so lets do it. [YOCTO #6949] (Bitbake rev: 461aa73fff0ab616032d28c4fd0322eb88838be6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: exit normally when SIGHUPRobert Yang2014-11-201-1/+8
| | | | | | | | | | | | | | Fixed: 1) Run "bitbake recipe" in the terminal 2) Close the terminal while building 3) $ ps aux | grep bitbake-worker There will be many processes, and they will keep the resources (e.g., memory), and won't exit unless kill or kill -9. (Bitbake rev: 40d2ae0723de2bf5fee343faafb4afda40546839) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Fix bitbake -nRichard Purdie2014-09-221-2/+3
| | | | | | | | | | | | Without this you see: File "bitbake/bin/bitbake-worker", line 201, in fork_off_task os._exit(child()) TypeError: an integer is required (Bitbake rev: cd477b5e77ab0373248b8a8fa30e1c7b8ea984fd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue/bitbake-worker: Improve siggen data transfer interfaceRichard Purdie2014-09-111-1/+1
| | | | | | | | | | | | | We need to transfer some of the siggen data from the core/cooker into the worker instances. There was a partial API created for this but its ugly and its not possible to extend it from the siggen class. This patch completes the interface/abstraction for the data and means the class can extend/customise it in any siggen class. (Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Extra profiling data dumpRichard Purdie2014-08-281-4/+32
| | | | | | | | | | | | | | Currently we get no profiling oversight into either the main bitbake worker process, or the overall parsing before task execution. This adds in extra profiling hooks so we can truly capture all parts of bitbake's execution into the profile data. To do this we modify the 'magic' value passed to bitbake-worker to trigger the profiling, before the configuration data is sent over to the worker. (Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Improve sigterm handlerRichard Purdie2014-08-231-1/+6
| | | | | | | | | | | | | | When processes terminate, we really want all of the child processes to terminate too. This was not happening for worker processes which spawned their own multiprocessing pools, leading to build hangs. This change ensures any sigterm gets passed to the whole process group. In local tests, this resolved some hanging process workloads I could generate. It does rely on signals being delivered in a timely fashion and there is a multiprocessing bug we have to work around there. (Bitbake rev: 96f8ea07ace1379380fab2d78eb592fa40c867d4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Drop BBHASH variablesRichard Purdie2014-04-231-4/+0
| | | | | | | | | | | | | Iterating through and calling setVar on this number of variables has significant overhead in the profiling data. By not setting this, we save 3,000 calls to setVar which gives a noticeable improvement to the speed of task execution. The BBHASH variables have since been replaced by accessing that data through the siggen code and going forward, that is the preferred way work with it. (Bitbake rev: 92526eadd09d19938762290e0492076174367583) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Ensure children have default sigterm handlerRichard Purdie2014-03-111-0/+2
| | | | | | | | | | The children of the worker should have the default SIGTERM handler, else they'll try and do cleanup which should only happen in the parent leading to all kinds of bizarre build failures. (Bitbake rev: a53c8d1f846d94082aa459996c4114f10970b8ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Gracefully handle SIGTERMRichard Purdie2014-03-101-0/+8
| | | | | | | | | | Currently if bitbake-worker handles a SIGTERM, it leaves the child processes to complete or hang. It shouldn't do this so hook the SIGTERM event and gracefully shutdown any children. (Bitbake rev: 551406f3f9ee94de09d2da6e16fea054c6dbfdb7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Share BB_TASKDEPDATA with tasksRichard Purdie2013-11-261-3/+4
| | | | | | | | | | | | | | | | | Currently tasks have no knowledge of which other tasks they depend upon. This makes it impossible to do at least two things which would be desirable/interesting: a) Have the ability to create per recipe sysroots b) Allow the aclocal files to be present only for the entries in DEPENDS (directly and indirectly) By exporting task data through this new variable, tasks can inspect their dependencies and then take actions based upon this. (Bitbake rev: 84f1dde717dac22435005b79d03ee0b80a3e8e62) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/bitbake-worker: Fix dry run fakeroot issuesRichard Purdie2013-11-241-1/+2
| | | | | | | | | | | | | | When using the dry run option (-n), bitbake would still try and fire a specific fakeroot worker. This is doomed to failure since it might well not have been built. Add in some checks to prevent the failures. [YOCTO #5367] (Bitbake rev: f34d0606f87ce9dacadeb78bac35879b74f10559) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Ensure ${DATE} and ${TIME} are consistentPeter Kjellerstedt2013-09-061-0/+2
| | | | | | | | | | | | | | | Due to the worker split the ${DATE} and ${TIME} variables could end up with different values for different workers. E.g., a task like do_rootfs that is run within a fakeroot environment had a slightly different view of the time than another task that was not fakerooted which made it impossible to correctly refer to the image generated by do_rootfs from the other task. (Bitbake rev: 756cc69ebf8bfe8455d0c90f288dd51be2499773) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: ensure BUILDNAME is available during executionPaul Eggleton2013-09-041-0/+1
| | | | | | | | | | | | | | | BUILDNAME is set from cooker by default, so since the worker split it will not be set when executing functions. In OpenEmbedded this results in /etc/version (which is populated from BUILDNAME) not having any content. Pass this variable value through to the worker explicitly to fix the issue. Fixes [YOCTO #4818]. (Bitbake rev: 92940b0427d9b2b3f95e27c230ec1e36638a34bc) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: import needed signal moduleValentin Popa2013-07-311-0/+1
| | | | | | | | | | | | bitbake-worker makes use of the signal module but it doesn't import it. This patch fixes the issue. [YOCTO #4750] (Bitbake rev: c2ed639690f135994199eb24d964e37f57259e3a) Signed-off-by: Valentin Popa <valentin.popa@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Adapt autostart to bitbake-workerRichard Purdie2013-06-141-0/+1
| | | | | | | | | | With the change to bitbake-worker we need to ensure the workers know how to contact the PR service, the magic 0 port and singleton is no longer enough. (Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Split runqueue to use bitbake-workerRichard Purdie2013-06-141-0/+358
This is a pretty fundamental change to the way bitbake operates. It splits out the task execution part of runqueue into a completely separately exec'd process called bitbake-worker. This means that the separate process has to build its own datastore and that configuration needs to be passed from the cooker over to the bitbake worker process. Known issues: * Hob is broken with this patch since it writes to the configuration and that configuration isn't preserved in bitbake-worker. * We create a worker for setscene, then a new worker for the main task execution. This is wasteful but shouldn't be hard to fix. * We probably send too much data over to bitbake-worker, need to see if we can streamline it. These are issues which will be followed up in subsequent patches. This patch sets the groundwork for the removal of the double bitbake execution for psuedo which will be in a follow on patch. (Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-21 02:17:09 +0000