summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/runqueue.py
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: build/runqueue: Add noextra stamp file parameter to fix multiconfig ↵Richard Purdie2016-09-021-2/+2
| | | | | | | | | | | | | | | | | | builds We can't execute the same task for the same package_arch multiple times as the current setup has conflicting directories. Since these would usually have the same stamp/hash, we want to execute in sequence rather than in parallel, so for the purposes of task execution, don't consider the "extra-info" on the stamp files. We need to add a parameter to the stamp function to achieve this. This avoids multiple update-rc.d populate_sysroot tasks executing in parallel and breaking multiconfig builds. (Bitbake rev: a9041fc96a14e718c0c1d1676e705343b9e872d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Initial multi-config supportRichard Purdie2016-08-181-210/+264
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds the notion of supporting multiple configurations within a single build. To enable it, set a line in local.conf like: BBMULTICONFIG = "configA configB configC" This would tell bitbake that before it parses the base configuration, it should load conf/configA.conf and so on for each different configuration. These would contain lines like: MACHINE = "A" or other variables which can be set which can be built in the same build directory (or change TMPDIR not to conflict). One downside I've already discovered is that if we want to inherit this file right at the start of parsing, the only place you can put the configurations is in "cwd", since BBPATH isn't constructed until the layers are parsed and therefore using it as a preconf file isn't possible unless its located there. Execution of these targets takes the form "bitbake multiconfig:configA:core-image-minimal core-image-sato" so similar to our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND. Implementation wise, the implication is that instead of tasks being uniquely referenced with "recipename/fn:task" it now needs to be "configuration:recipename:task". We already started using "virtual" filenames for recipes when we implemented BBCLASSEXTEND and this patch adds a new prefix to these, "multiconfig:<configname>:" and hence avoid changes to a large part of the codebase thanks to this. databuilder has an internal array of data stores and uses the right one depending on the supplied virtual filename. That trick allows us to use the existing parsing code including the multithreading mostly unchanged as well as most of the cache code. For recipecache, we end up with a dict of these accessed by multiconfig (mc). taskdata and runqueue can only cope with one recipecache so for taskdata, we pass in each recipecache and have it compute the result and end up with an array of taskdatas. We can only have one runqueue so there extensive changes there. This initial implementation has some drawbacks: a) There are no inter-multi-configuration dependencies as yet b) There are no sstate optimisations. This means if the build uses the same object twice in say two different TMPDIRs, it will either load from an existing sstate cache at the start or build it twice. We can then in due course look at ways in which it would only build it once and then reuse it. This will likely need significant changes to the way sstate currently works to make that possible. (Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Abstract worker functionality to an object/arrayRichard Purdie2016-08-181-56/+66
| | | | | | | | | | | | | | | With the introduction of multi-config and the possibility of distributed builds we need arrays of workers rather than the existing two. This refactors the code to have a dict() of workers and a dict of fakeworkers, represented by objects. The code can iterate over these. This is separated out from the multi-config changes since its separable and clearer this way. (Bitbake rev: 8181d96e0a4df0aa47287669681116fa65bcae16) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix two minor issues with the initialising tasks progressPaul Eggleton2016-08-171-2/+3
| | | | | | | | | | | | | | A couple of fixes for the "Initialising tasks" progress bar behaviour: * Properly finish the progress bar when using bitbake -S * Finish the progress bar before calling BB_HASHCHECK_FUNCTION (so that in OE when that shows its own "Checking sstate mirror object availability" progress bar it gets shown on the next line as it should). (Bitbake rev: de6759d8e9990e426e6d6464a2e05381cd4c12d6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: improve exception loggingEd Bartosh2016-07-191-2/+2
| | | | | | | | | | | | | | | Runqueue errors direct the user to view the "failure below", but no additional error message is available. Log the stacktrace so that the user can see what went wrong. Also fix a typo in the log message. (Bitbake rev: e191f401e372ee181bc02250232ad9cb9a0e9477) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: report progress for "Preparing RunQueue" stepPaul Eggleton2016-07-081-3/+67
| | | | | | | | | | | | | | | | | | | | | When "Preparing RunQueue" shows up you can expect to wait up to 30 seconds while it works - which is a bit long to leave the user waiting without any kind of output. Since the work being carried out during this time is divided into stages such that it's practical to determine internally how it's progressing, replace the message with a progress bar. Actually what happens during this time is two major steps rather than just one - the runqueue preparation itself, followed by the initialisation prior to running setscene tasks. I elected to have the progress bar cover both as one (there doesn't appear to be much point in doing otherwise from a user perspective). I did however describe it as "initialising tasks". (Bitbake rev: 591e9741e108487ff437e77cb439ef2dbca42e03) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add ability to enforce that tasks are setscenedPaul Eggleton2016-07-081-1/+80
| | | | | | | | | | | | | | | | | | | | | | | | Add the ability to enter a mode where only a specified whitelist of tasks can be executed outright; everything else must be successfully provided in the form of a setscene task (or covered by a setscene task). Any setscene failure outside of the whitelist will cause the build to fail immediately instead of running the real task, and any real tasks that would execute outside of the whitelist cause an immediate build failure when it comes to executing the runqueue as well. The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting of pn:taskname pairs. A single % character can be substituted for the pn value to match any target explicitly specified on the bitbake command line. Wildcards * and ? can also be used as per standard unix file name matching for both pn and taskname. Part of the implementation of [YOCTO #9367]. (Bitbake rev: 624722c067a7fdd0c0f5d8be611e1f6666ecc4a2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Use tid instead of taskid in find_chains()George McCollister2016-06-161-1/+1
| | | | | | | | | | | In 2c88afb6 find_chains()'s taskid argument was renamed to tid but taskid is still used as key to explored_deps dictionary. Use tid instead of taskid. (Bitbake rev: 29a34ae8f5306d2779bcc761c52f1f9d13a0c0c5) Signed-off-by: George McCollister <george.mccollister@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskdata/runqueue: Rewrite without use of ID indirectionRichard Purdie2016-06-151-500/+458
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I'm not sure what possesed me when I wrote this code originally but its indirection of everyting to use numeric IDs and position dependent lists is horrific. Given the way python internals work, its completely and utterly pointless from performance perspective. It also makes the code hard to understand and debug since any numeric ID has to be translated into something human readable. The hard part is that the IDs are infectous and spread from taskdata into runqueue and even partly into cooker for the dependency graph processing. The only real way to deal with this is to convert everything to use a more sane data structure. This patch: * Uses "<fn>:<taskname>" as the ID for tasks rather than a number * Changes to dict() based structures rather than position dependent lists * Drops the build name, runtime name and filename ID indexes On the most part there shouldn't be user visible changes. Sadly we did leak datastructures to the setscene verify function which has to be rewritten. To handle this, the variable name used to specifiy the version changes from BB_SETSCENE_VERIFY_FUNCTION to BB_SETSCENE_VERIFY_FUNCTION2 allowing multiple versions of bitbake to work with suitably written metadata. Anyone with custom schedulers may also need to change them. I believe the benefits in code readability and easier debugging far outweigh those issues though. It also means we have a saner codebase to add multiconfig support on top of. During development, I did have some of the original code coexisting with the new data stores to allow comparision of the data and check it was working correcty, particuarly for taskdata. I have also compared task-depends.dot files before and after the change. There should be no functionality changes in this patch, its purely a data structure change and that is visible in the patch. (Bitbake rev: 2c88afb60da54e58f555411a7bd7b006b0c29306) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change buildable/running lists to setsRichard Purdie2016-06-151-36/+27
| | | | | | | | | | | | | | Using positions in lists for flags is an odd choice and makes the code hard to maintain. Maintaining a list is slow since list searches are slow (watch bitbake -n slow massively with it) but we can use a set() instead. This patch uses python sets to maintain the lists of tasks in each state and this prepares for changing the task IDs from being integers. (Bitbake rev: 8c1ed57f6ea475b714eca6673b48e8e5f5f0f9c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to python 3Richard Purdie2016-06-021-61/+56
| | | | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: always emit bb.event.DepTreeGeneratedPatrick Ohly2016-05-141-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | The data included in the event is useful for implementing a pre-build check that warns about unexpected components, for example because of an incorrect configuration or changed dependencies. Such a check can be done in a .bbclass that gets inherited globally. But in contrast to a UI, such a class cannot request that the event shall be emitted, and thus the event has to be emitted whether there is a consumer or not. This was done conditionally earlier out of concerns about the performance impact. But now events are handled more efficiently, so that concern no longer seems valid: in some simple testing (admittedly on a fast build workstation), the two lines (generating the data and emitting the event with it) only took about 0.05 seconds (measured with timeit). That was for a build with roughly 500 recipes (from pn-buildlist aka depgraph['pn']), triggered via the command line. That was even with a consumer of the data active and doing some work, so it should be even faster when there is no consumer. (Bitbake rev: 5ddaf5b7ed1001d2dd3f67e7a6d704afa85479d2) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve timestamp comparisonsRichard Purdie2016-05-131-1/+5
| | | | | | | | | | python3 cares more about invalid type comparisons. Add break statements and better tests to make the code paths clearer and avoid type issues in python3. No code functionality change. (Bitbake rev: 2c39ebdd2762d027f007a6a769fdf023cdf3da2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix missing fakeworker under dry runRichard Purdie2016-05-111-1/+1
| | | | | | | | | We shouldn't try and use fakeworker when performing a dry_run. This makes the core match the other fakeworker execution points. (Bitbake rev: 49bea821a2edad5e19c3a566d1a80c23718dede9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update logger.warn() -> logger.warning()Richard Purdie2016-05-111-2/+2
| | | | | | | | | | python deprecated logger.warn() in favour of logger.warning(). This is only used in bitbake code so we may as well just translate everything to avoid warnings under python 3. Its safe for python 2.7. (Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve 'mulitiple .bb files are due to be built' messageRichard Purdie2016-04-121-2/+70
| | | | | | | | | | | | | | | | | | | | | | | | | When multiple recipes which both provide something are being built, bitbake informs us that most likely one of them provides something the other doesn't, which is usually correct, but unfortunately it's rather painful to figure out exactly what that is. This patch dumps two sets of information, one is the provides information for each recipe, filtered so only common components are removed. The other is a list of dependees on the recipe, since sometimes this can easily identify why something is being built. Its not straightforward for bitbake to obtain the information but since the warning/error code path isn't the normal one, we can afford to go through some less than optimal processing to aid debugging. Also provide the same information even if we're showing a warning since its still useful. [YOCTO #8032] (Bitbake rev: 96fc889b8e62ba4463c71158c4b7286c48d68cd8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/runqueue: save task file dependency cache onto diskMarkus Lehtonen2016-02-181-0/+1
| | | | | | | | | | | | Before this patch the usage of cache was quite useless as the file checksums were not actually cached on disk but re-calculated every time. This patch utilises the new writeout_file_checksum_cache() method of the SignatureGenerator class to do the job. (Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Set process names to be meaninfulRichard Purdie2016-01-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "runqueue.py: Ensure one setscene function doesn't mask out ↵Richard Purdie2016-01-191-5/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | another which needs to run" This reverts commit b22592af8145a8c7c4ada2fa7c1dee2e753eca46. That commit isn't entirely clear about why this change is needed but I do have a usecase where this breaks things. If for example you run "bitbake X -c packagedata" and that packagedata is in sstate, you'd expect this to work. If sstate doesn't contain a do_populate_sysroot for a dependency, you would still expect the command above to succeed and you would not expect it to rebuild that dependency. With the current code, this isn't what happens. The code finds the sstate for do_populate_sysroot missing, this makes the task "uncovered" and this in turn makes it unskippable. The example I found with this was avahi-ui, where it would trigger a build of libdaemon to obtain its populate_sysroot. Since this behaviour seems completely incorrect, revert the older patch and we'll address any issues that crop up as a result. (Bitbake rev: 36a9840a5da17cc14561881fdd6a4f2cb0a75e49) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix setscene task dependenciesRichard Purdie2016-01-111-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | Debugging suggests that setscene tasks are being a little greedy about their dependencies, for example, lsof is insisting that gcc-runtime's do_package is installed. If it isn't, its requiring gcc to rebuild. If gcc-runtime do_package_write_xxx and do_packagedata is available, there is no reason do_package should be needed. The reason this is happening appears to be from the batching up of task dependencies code, rather than setscene tasks stopping when passing over a setscene task, they were being carried forward. This patch fixes it so the data is 'zeroed' when passing over a setscene task boundary, which gives the dependency graph that is expected. After this patch, lsof will rebuild quite happily without gcc-runtime:do_package being present, as expected. This should lead to less dependencies being installed for builds from sstate and generally better performance in general. (Bitbake rev: f8bcb0a1e3b008b71c9a7cd21f76d0906f2d8068) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskdata: refactor get_providermapEd Bartosh2016-01-111-1/+1
| | | | | | | | | | | | | | | Added optional parameter 'prefix' to filter out names that don't start with specified prefix. Changed existing call of get_providermap according to changed API. Optimized the code: got rid of extra loop and temporary list variable virts. (Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main/runqueue: Add --setscene-only option to bitbakeRichard Purdie2016-01-111-3/+6
| | | | | | | | | | | | | Its turning out that we really need a way to have bitbake just run the setscene tasks but not any real tasks, particularly for SDK operations. Add an option for this since its pretty straight forward. This allows various nasty workarounds in OE-Core to be removed. (Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add support for <task>- syntaxRichard Purdie2015-12-181-8/+18
| | | | | | | | | | | It can be useful to run all tasks up to but not including a specific task. The main reason this was never added was the lack of a good syntax. This patch uses the syntax <taskname>- to denote this behaviour which is simple, not invasive and fits what we need from good syntax IMO, hence we can add this. (Bitbake rev: 99ccfd411ab3f7baa111f9f3d50fae68816a9a83) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add handling of virtual/xxx provider mappingsRichard Purdie2015-10-011-0/+9
| | | | | | | | | | | | This firstly prints debug messages which show how bitbake decided to resolve the virtual/xxx providers which is useful for debugging. If the siggen has a tasks_resolved() method, it calls this, passing in the mappings, allowing that to do things with the resolved names. (Bitbake rev: d473fc84acddfd69a7207affcd89f65ea2ecf730) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/runqueue: Allow bitbake commands starting with do_Alex Franco2015-09-041-1/+3
| | | | | | | | | | | | | | | The output of "bitbake, -c listtasks pkg" lists tasks with their real names (starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always gets unconditionally prepended to task names. This patch handles this error by checking whether a task starts with "do_" prior to prepending it with it when the task runlist is being constructed (and a few other corner cases). [YOCTO #7818] (Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df) Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: Add provides to taskdepdataMariano Lopez2015-08-101-1/+2
| | | | | | | | | | | | | | | Currently bitbake is the only one that knows the relation between PN and PROVIDES. In some cases it is needed to have this relation in the data store (the bootloader it's a good case). This adds the PROVIDES to the taskdata, so it would be in the data store as one field of BB_TASKDEPDATA. (Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Sanity check BB_NUMBER_THREADSRichard Purdie2015-06-231-0/+3
| | | | | | | | | | | | Bitbake does really weird things with negative or zero numbers of threads which is confusing to the user. Add a sanity check for this. When you have code doing arithmetic on the values and a VM reconfigures to only a single thread, negative numbers are easier than you'd think. (Bitbake rev: 32166ac3c85ff3c04081580ae76bd63590d6ff3e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Handle BBHandledException correctlyRichard Purdie2015-06-231-0/+7
| | | | | | | | | | | | If we see a BBHandledException in runqueue, the understanding is the system handled it, printing a log and traceback is just confusing. Therefore only print these in the cases where its an unknown/unhandled exception. (Bitbake rev: 29d28e22ce431c3d3aabdb88ff4d8cca67a1cfad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve handling of fakeworker failing to startRichard Purdie2015-06-231-1/+6
| | | | | | | | | | | Currently if the fakeworker failes to start the output from bitbake is confusing. Improve the error handling to give a clear indication of what failed. Patch from Chris Larson. (Bitbake rev: ad286d6fed7a580bec36a92c7b7e205322ac407b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Handle cases where siginfo is now a parameter to the ↵Richard Purdie2015-05-151-2/+7
| | | | | | | | | | | | | | hashvadlidate function In some cases we need to check specifically for siginfo files, in some cases we need to check for the actual sstate objects themselves. Therefore make this a parameter to the function. A fallback to the previous function style is maintained for now. (Bitbake rev: 18d3a03e1b07c98b2dce46eb94f30de1a2b4320b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: pass finalized metadata to scenequeue callbacksChristopher Larson2015-04-081-4/+4
| | | | | | | | | | | | This ensures that _append, _prepend, overrides, etc are functional when used on sstate variables (e.g. SSTATE_DIR). [YOCTO #7564] (Bitbake rev: 2e683c25b856b431198573f7f352d841587275e6) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix 100% cpu use after keyboard interruptRichard Purdie2014-12-091-5/+4
| | | | | | | | | | After Ctrl+C is pressed to interrupt bitbake, it loops continually, running at 100% cpu. This patch selects on the correct file descriptors resolving the excess cpu usage. (Bitbake rev: 497404e8484b7ca7c11e459bf0845642156eb677) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Make printed 'runqueue' be consistently capitalizedGary Thomas2014-11-061-1/+1
| | | | | | | | | | | | | This has always bothered me: NOTE: Preparing runqueue NOTE: Executing RunQueue Tasks This patch changes the messages to be consistent. (Bitbake rev: 72ac9f9227fbfb4dc8b933b357d21aa0e4060959) Signed-off-by: Gary Thomas <gary@mlbassoc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue/bitbake-worker: Improve siggen data transfer interfaceRichard Purdie2014-09-111-3/+1
| | | | | | | | | | | | | We need to transfer some of the siggen data from the core/cooker into the worker instances. There was a partial API created for this but its ugly and its not possible to extend it from the siggen class. This patch completes the interface/abstraction for the data and means the class can extend/customise it in any siggen class. (Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Extra profiling data dumpRichard Purdie2014-08-281-2/+5
| | | | | | | | | | | | | | Currently we get no profiling oversight into either the main bitbake worker process, or the overall parsing before task execution. This adds in extra profiling hooks so we can truly capture all parts of bitbake's execution into the profile data. To do this we modify the 'magic' value passed to bitbake-worker to trigger the profiling, before the configuration data is sent over to the worker. (Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix setscene tasks not runningRichard Purdie2014-08-281-0/+4
| | | | | | | | | | | | | | | | | | | | Currently, if you have hard dependencies between setscene tasks (like avahi on base-passwd through useradd.bbclass), other dependencies may not be installed even if these exist in sstate. For example, avahi -> expat -> pigz-native (and avahi -> base-passwd) yet if you cleansstate base-passwd: bitbake gzip-native:do_clean avahi:do_clean expat:do_clean pigz-native:do_clean base-passwd:do_cleansstate bitbake avahi | tee you will currently see pigz-native being rebuilt even though it was in sstate. The fix for this is to continue to iterate dependency chains around hard blocked dependencies as per this patch. After this patch is applied, you will see pigz-native installed from sstate. (Bitbake rev: f787957a224e8c2682a19e5c4a4d9c86bdce52ba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: Fix typoes/grammar in comments.Robert P. J. Day2014-08-191-6/+6
| | | | | | | (Bitbake rev: 000fa81013205dd9bc907ff7a61f06f57637212d) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: Correct several misspellings of "notifing".Robert P. J. Day2014-08-191-7/+7
| | | | | | | (Bitbake rev: 4e9aef14d747c37444a4fc683f9641906906afe9) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add sceneQueueComplete eventRichard Purdie2014-08-031-0/+12
| | | | | | | | | | Its useful to have an event emitted when all of the sceneQueue tasks have completed since the metadata can hook this for processing. Therefore add such an event. (Bitbake rev: 38d4f65bf1cbcdd5a2d60dff0e1d2859c34ed62e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command/runqueue: Fix shutdown logicRichard Purdie2014-07-211-1/+2
| | | | | | | | | | | | | | If you hit Ctrl+C at the right point, the system processes the request but merrily continues building. It turns out finish_runqueue() is called but this doesn't stop the later generation and execution of the runqueue. This patch adjusts some of the conditionals to ensure the build really does stop. (Bitbake rev: 39b08c604ba713100e174c136f81f18eca6ef33d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Do not write out stamp files in dry_run modeRichard Purdie2014-04-231-1/+2
| | | | | | | | | In dry run mode, stamps for noexec tasks are being written out which is incorrect. Avoid this. (Bitbake rev: aa6448a0552ba2947ac262b8b5314a593d1058d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task weighting algorithmRichard Purdie2014-04-231-2/+2
| | | | | | | | | | | | | | | When looking at a list of tasks, do_patch and do_unpack were being given equal priority when one clearly depends on another. The reason for this was the default task weights of 0 being to tasks. This is therefore changed to 1 to allow correct weighting of dependencies which means the scheduler has better information available to it about tasks. Weight endpoints differently (10) for clearer debugging of priorities. (Bitbake rev: 12dc1d17fac3e8ec420fcafb06186d32fd847d89) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix handling of zero priority taskRichard Purdie2014-04-231-1/+1
| | | | | | | | | | | | | The zero priority task should be run first but was being confused with the None value the priority field defaulted to. Check for None explicitly to avoid this error. In the real world this doesn't change much but it confused the debug output from the schedulers. (Bitbake rev: 49c9d8c9400f74c804c2f36462639236e0841ff0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Address issues with incomplete sstate setsRichard Purdie2014-04-011-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The first part of the sstate code checks en-mass whether given checksums are available. The next part of the code then either triggers those setscene tasks either running them or skipping them if they've been covered by others. The problems was that this second part would always skip a task if it was unavailable in the first part, even if it would have otherwise been covered by other tasks. This mean the mere presence of an artefact (or lack of presence) could cause a different build failure. The issue reproduces if you run a build and populate an sstate feed, then run a second build off that feed, then run a third build off the sstate feed of the second build (which is reduced in size). The fix is rather than immediately skipping tasks if the checksum is unavailable, create a list of missing tasks, then, if that task cannot be covered by others we can skip it later. The deferral makes the behaviour the same even when the cache is "incomplete". [YOCTO #6081] (Bitbake rev: 5edb1a3e3f454ba6e65551174d86229db2f99636) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sstate task dependency problemsRichard Purdie2014-04-011-0/+4
| | | | | | | | | | | | | If a setscene task has [depends], its possible they may still get executed out of order. The issue is that the dependencies are set to set() for all tasks involved. This patch adds back in explict dependencies within these chains to avoid the setscene task failures. [YOCTO #6069] (Bitbake rev: 724c889eed3b03d3199810c185086d3973af826c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Pass in commandline options to dump_sigs()Richard Purdie2014-03-271-3/+3
| | | | | | | | | This allows the commandline options to be processed in the dump signature code. (Bitbake rev: ef8537a2e9b48f4fe065a165c102935aee2c9029) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Force -S option to take a parameterRichard Purdie2014-03-271-3/+6
| | | | | | | | | | | | | There is no easy way to make this change. We really need parameters for the -S (dump signatures) handling code. Such a parameter can then be used within the codebase to handle the signatures in different ways. For now, "none" is the recommended default and "printdiff" will execute the new (and more expensive) comparison algorithms. (Bitbake rev: b9873588696507dfb6aade6821f6f75cb9a19e0a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sceneQueueEvent to use the correct hashesRichard Purdie2014-03-251-1/+1
| | | | | | | | | | The runqueue should be using the "realtask" ID to lookup the task hash, not the "task" ID. This patch resolves corruption issues where incorrect task hashes were displayed within toaster. (Bitbake rev: 84be1a27f89d1bf63c21f06d831df0a66a5db860) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Remove use of waitpid on worker processesRichard Purdie2014-03-191-16/+12
| | | | | | | | | | | | | Use of waitpid on the worker processes is a bad idea since it conflicts directly with subprocess internals. Instead use the poll() method and returncode to determine if the process has exitted, if it has, we can shut down the system. This should resolve the hangs once and for all, famous last words. (Bitbake rev: 60969cd62e21e7d4af161bf8504b7643a879c73f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Revert child signal handler for nowRichard Purdie2014-03-191-28/+17
| | | | | | | | | | | | | | | | | | | | | | | | | We're running into processes using 100% cpu. It appears theses are locked in a subprocess.poll() type loop where the process has exited but the code is looping as its not handling the ECHILD error. http://bugs.python.org/issue14396 http://bugs.python.org/issue15756 This is likely due to one or both of the above bugs. The question is what actually grabbed the child exit code as it wasn't this code. Its likely there is therefore some other code racing and taking that code, it may be some kind of race like: http://hg.python.org/cpython/rev/767420808a62/ where the fix effectively catches the childs codes in a different part of the system. We could try and get everyone onto python 2.7.4 where the above bugs are fixed however for now its safer to admit defeat and go back to polling explictly for our worker exit codes. (Bitbake rev: 5b9a099ec2a1dc954b614e12a306595f55b6a99e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-15 15:02:12 +0000