summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/runqueue.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Allow recrdeptask not to exist for all recipesRichard Purdie2017-04-101-0/+3
| | | | | | | | | | | | | | Currently if you specify a recrdeptask, it must exist for all recipes or you get a python traceback. This is a bug and it should be possible to have recipes which don't have the specified task. As well as preventing such a traceback (which shouldn't happen, it should be a user readable error), this allows us to fix issues in OE-Core which would otherwise trigger the traceback. (Bitbake rev: f93a77f2f188e18de9e3d812e86d77c2f3c71889) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: minor typo fixAndre McCurdy2017-03-151-1/+1
| | | | | | | (Bitbake rev: f2c151cd345788a135452f63f622a2bc0b10c2d4) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add option to run all tasks in specific build targetMatthew McClintock2017-03-131-0/+17
| | | | | | | | | | | | | | | | | | | | For example: $ bitbake core-image-minimal --runall unpack ... NOTE: Tasks Summary: Attempted 326 tasks of which 88 didn't need to be rerun and all succeeded. $ bitbake core-image-minimal --runall patch NOTE: Tasks Summary: Attempted 453 tasks of which 332 didn't need to be rerun and all succeeded. This can replace fetchall as well: $ bitbake core-image-minimal --runall fetch NOTE: Tasks Summary: Attempted 135 tasks of which 119 didn't need to be rerun and all succeeded. (Bitbake rev: 7c0fa6ba66cdb956b37d94055307cde857633df9) Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix collapsed setscene dependency treeRichard Purdie2017-02-191-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When we removed the postinst sstate dependency handling code from setscene_depvalid, we noticed things being installed into the old style sysroot for rootfs tasks which should not have been there, causing a performance regression. Analysis revealed that setscene dependencies were "bubbling" over sstate tasks when they should have been stopping there. The 'continue' added by this patch avoids this issue and eusures sstate tasks remain contained to their specific chains. There was another bug in the code this exposed where the acconting for tasks as they were removed from sq_revdeps was not correct. In fixing this, what looks like a workaround in another test can then be simplified. After this change, populate_sysroot tasks are no longer depending on package_write_rpm tasks for example, which would make no sense. A before/after analysis of image dependencies only revealed improved dependencies after this change. Recipe specific sysroots did highlight the issue here since the behaviour of the sysroot dependencies (and processing with depvalid) was not matching what bitbake itself was doing, with bitbake being incorrect. Failures were 'safe' in that too many dependencies would get installed. (Bitbake rev: 5ef2cb50041fa7106c8de170af73d2a54cb0b1f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: revised completion schedulerPatrick Ohly2017-01-201-24/+101
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The idea is that tasks which complete building a recipe (like do_package_qa) are more important than tasks which start building new recipes (do_fetch) or those which increase disk usage (do_compile). Therefore tasks get ordered like this (most important first, do_rm_work before do_build because the enhanced rm_work.bbclass was used): 1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build 2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build 3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build ... 464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build 465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work 466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work 467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work ... 3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install 3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install 3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base 3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base ... 3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base 3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base 3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage 3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck 3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip 3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules 3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir 3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images 3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile 3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile ... The order of the same task between different recipes is the same as with the speed scheduler, i.e. more important recipes come first. (Bitbake rev: 70e297e5c285ce0a02e9efd3117ff62cdc77ec12) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix traceback when using -bRichard Purdie2017-01-201-0/+2
| | | | | | | | | Without this, bitbake -b of image recipes cause tracebacks since the list of providers is empty. (Bitbake rev: c53e43f3c6a675c0934a7a4e358fd66b049ffca3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: use multiple processes to dump signatures.Jianxun Zhang2017-01-171-5/+27
| | | | | | | | | | | | | | | | | | | | | This change significantly shortens the time on reparsing stage of '-S' option. Each file is reparsed and then dumped within a dedicated process. The maximum number of the running processes is not greater than the value of BB_NUMBER_PARSE_THREADS if it is set. The dump_sigs() in class SignatureGeneratorBasic is _replaced_ by a new dump_sigfn() interface, so calls from the outside and subclasses are dispatched to the implementation in the base class of SignatureGeneratorBasic. Fixes [YOCTO #10352] (Bitbake rev: 99d3703edd77a21770b366c6ad65a3c0f5183493) Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Only start fakeroot workers when neededRichard Purdie2016-12-221-8/+7
| | | | | | | | | | | | | Fakeroot workers usually have dependencies that need to be ready before they can be started. Starting them as a block therefore doesn't work as the dependencies may or may not have been built. Therefore start the multiconfig fakeworkers individually upon demand. [YOCTO #10344] (Bitbake rev: ab15486c636e451ed5a62e561980946887901c3b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure setscene tasks with overlapping stamps don't ↵Richard Purdie2016-12-221-1/+5
| | | | | | | | | | | | | | | parallel execute In multiconfig, mutliple tasks can execute which share the same stamp file. These must not execute in parallel, the idea is the first should execute, the subsequent ones should see a valid stamp and get skipped. The normal task execution code has stamps code to handle this, this adds similar code to the setscene execute() function to handle the issue there too. (Bitbake rev: 937acf267fa9e45f538695b2cf8aa83232a96240) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix setscene issues with multiconfigRichard Purdie2016-12-221-20/+20
| | | | | | | | | | | | | | | | setscene was being entirelu skipped for multiconfig variants as the tasks were simply not being spotted. If the default config was also being built it masked the problem. When this was fixed by using taskfn instead of fn in lookups against dataCache, several other instances of this problem were highlighted. This goes through and corrects the setscene code to correclty use taskfn instead of fn in the appropriate places meaning setscene tasks for multiconfig now work correctly. (Bitbake rev: cead85cf219c07b40aeb7c0d948c25f0429136db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure pseudo executes from the correct place (use the ↵Richard Purdie2016-12-221-2/+3
| | | | | | | | | | | | | | right datastore with multiconfig) The location of the fakeroot command and the various environmental values need to be taken from the right multiconfig datastore, not the shared one. This patch ensures the right one is used for cases like a split TMPDIR. [YOCTO #10344] (Bitbake rev: 813a7600bc26b7132ac36d1515571d321f566afb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: dry-run real tasks when BB_SETSCENE_ENFORCE is setPaul Eggleton2016-12-201-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used by the installation process for the extensible SDK), we don't actually need the whitelisted real tasks to execute - we just need to have them in the dependency tree so that we get all of the setscene tasks they depend on to run. Therefore we can actually dry-run those real tasks i.e. they won't be run (and thus we won't waste a significant amount of time doing so) and won't be stamped as having run either. We do already have a dry-run mode in BitBake (activated by the -n or --dry-run command line option), but it dry-runs the setscene tasks as well which we don't want here. Note that this has no effect on the checking we are doing with BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are scheduled to run - that's handled separately. Fixes [YOCTO #10369]. (Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Drop expanded_dataRichard Purdie2016-12-161-4/+4
| | | | | | | | | | | | | | | Some of our metadata assumes that BuildStarted and BuildCompleted events see the same data store. This is the case for buildTarget but not for buildFile and recent changes mean this is now a problem. The update_data() call is now an empty operation and there is no difference between the expanded_data and data so we can simply remove the expanded_data and its references and use data everywhere. This has been inteded for a while but the above issue makes this more pressing to finally clean up. (Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: enable setVariable command to affect task executionPaul Eggleton2016-12-141-0/+1
| | | | | | | | | | | | Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. (Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Send BB_TASKDEPDATA for setscene tasksRichard Purdie2016-12-081-2/+42
| | | | | | | | | | We now have code in OE that needs BB_TASKDEPDATA for setscene tasks. Therefore generate and send this data. In this case its a "pre collapsed" tree but that is fine for the use cases in question. (Bitbake rev: 38b857d086af43af6ea3aa60d3876a2c9b225401) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add the taskhash to BB_TASKDEPDATARichard Purdie2016-12-081-1/+2
| | | | | | | | | | Its useful to know the task hash in code using TASKDEPDATA so add this data to the data structure. The recipe specific sysroots in OE need this data. (Bitbake rev: 758867e8dc74283bb1f031e158ec54cefdd5c2a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: monitor disk space at regular time intervalsPatrick Ohly2016-12-071-5/+19
| | | | | | | | | | | | | | | | | | | | | Hooking the disk monitor into the regular heatbeat event instead of the runqueue solves two problems: - When there is just one long running task which fills up the disk, the previous approach did not notice that until after the completion of the task because _execute_runqueue() only gets called on task state changes. As a result, aborting a build did not work in this case. - When there are many short-lived tasks, disk space was getting checked very frequently. When the storage that is getting checked is on an NFS server, that can lead to noticable traffic to the server. (Bitbake rev: 4547eea26803a9cd355d8b045197bcbdbb36a9ad) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar callsJoshua Lock2016-11-301-16/+16
| | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: Remove redundant whitelist checksRandy Witt2016-10-111-1/+4
| | | | | | | | | | | | | The whitelist checks for BB_SETSCENE_ENFORCE were running for every call to execute(). Since the task list doesn't change for each call into execute, the checks only need to be ran once. [YOCTO #10369] (Bitbake rev: f65e631ab6705dfd9188f19ee423eca33bca7d7d) Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task id string manipulationsRichard Purdie2016-10-091-46/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | Some task id manipulations were suboptimal: * taskfn_fromtid and fn_from_tid were effectively the same function * many calls to split_tid(), then taskfn_fromtid() * taskfn_fromtid() called split_tid() internally This patch adds split_tid_mcfn() to replace split_tid() and returns the "taskfn" variant being used in many places. We update all core calls to the new function and ignore the return values we don't need since the function call overhead of the split_tid wrapper is higher than ignoring a return value. The one remaining standalone use of taskfn_fromtid is replaced with fn_from_tid. I couldn't see any external usage so it was dropped. There is external usage of split_tid so a wrapper remains for it. Combined together these changes should improve some of the runqueue task manipulation performance. (Bitbake rev: 1bf2ef874fbe47f1320007efa0bdeef8d630b8a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.runqueue: fix unexpected process death logicChristopher Larson2016-10-091-10/+5
| | | | | | | | | | | | | | | | | | | | | `if w in self.rq.worker` when w *is* self.rq.worker doesn't make a great deal of sense, and results in this error: File ".../poky/bitbake/lib/bb/runqueue.py", line 2372, in runQueuePipe.read(): name = None > if w in self.rq.worker: name = "Worker" TypeError: unhashable type: 'dict' Most likely this was meant to be 'is' rather than 'in', but rather than checking after the fact, just include the name in the iteration, instead. While we're here, also clean up and fix the broken error message. (Bitbake rev: 267e025cad44c8bd0fb157f1f7a2e08df117ba84) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure worker failure is accounted for in task statisticsRichard Purdie2016-09-301-0/+1
| | | | | | | | | | | | | If the worker fails to launch, ensure the task is shown as failed rather than a confusing "all succeeded" message. Patch from Juro Bystricky [YOCTO #10335] (Bitbake rev: 0e9a2ff96d138641501874a1cd7aa6cc7e94d727) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Handle missing sstate dependencies betterRichard Purdie2016-09-211-5/+4
| | | | | | | | | | | | | | | | | | | | | | | If you "bitbake glibc-locale" then delete the libpcre-native sstate and "bitbake glibc-locale -C package_write_rpm", it will fail with rpmbuild missing the libprce library. The reason is that libpcre-native fails to install from sstate (since it isn't present) but doesn't get built and hence rpm-native tries to run without its dependencies. The simplest fix is not to add "covered" tasks which have failed to install sstate. I can't help feeling there is more to this issue but this does fix the current problem and shouldn't have adverse affects. It is an unusual situation to have missing dependencies in sstate since they're usually all present or not at all. I've taken the opportunity to remove some old cruft from when we had numeric task ids, the code can be simpler now. (Bitbake rev: ba566b46d530b495f12f3a74f76434717b22a020) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/runqueue: Add noextra stamp file parameter to fix multiconfig ↵Richard Purdie2016-09-021-2/+2
| | | | | | | | | | | | | | | | | | builds We can't execute the same task for the same package_arch multiple times as the current setup has conflicting directories. Since these would usually have the same stamp/hash, we want to execute in sequence rather than in parallel, so for the purposes of task execution, don't consider the "extra-info" on the stamp files. We need to add a parameter to the stamp function to achieve this. This avoids multiple update-rc.d populate_sysroot tasks executing in parallel and breaking multiconfig builds. (Bitbake rev: a9041fc96a14e718c0c1d1676e705343b9e872d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Initial multi-config supportRichard Purdie2016-08-181-210/+264
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds the notion of supporting multiple configurations within a single build. To enable it, set a line in local.conf like: BBMULTICONFIG = "configA configB configC" This would tell bitbake that before it parses the base configuration, it should load conf/configA.conf and so on for each different configuration. These would contain lines like: MACHINE = "A" or other variables which can be set which can be built in the same build directory (or change TMPDIR not to conflict). One downside I've already discovered is that if we want to inherit this file right at the start of parsing, the only place you can put the configurations is in "cwd", since BBPATH isn't constructed until the layers are parsed and therefore using it as a preconf file isn't possible unless its located there. Execution of these targets takes the form "bitbake multiconfig:configA:core-image-minimal core-image-sato" so similar to our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND. Implementation wise, the implication is that instead of tasks being uniquely referenced with "recipename/fn:task" it now needs to be "configuration:recipename:task". We already started using "virtual" filenames for recipes when we implemented BBCLASSEXTEND and this patch adds a new prefix to these, "multiconfig:<configname>:" and hence avoid changes to a large part of the codebase thanks to this. databuilder has an internal array of data stores and uses the right one depending on the supplied virtual filename. That trick allows us to use the existing parsing code including the multithreading mostly unchanged as well as most of the cache code. For recipecache, we end up with a dict of these accessed by multiconfig (mc). taskdata and runqueue can only cope with one recipecache so for taskdata, we pass in each recipecache and have it compute the result and end up with an array of taskdatas. We can only have one runqueue so there extensive changes there. This initial implementation has some drawbacks: a) There are no inter-multi-configuration dependencies as yet b) There are no sstate optimisations. This means if the build uses the same object twice in say two different TMPDIRs, it will either load from an existing sstate cache at the start or build it twice. We can then in due course look at ways in which it would only build it once and then reuse it. This will likely need significant changes to the way sstate currently works to make that possible. (Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Abstract worker functionality to an object/arrayRichard Purdie2016-08-181-56/+66
| | | | | | | | | | | | | | | With the introduction of multi-config and the possibility of distributed builds we need arrays of workers rather than the existing two. This refactors the code to have a dict() of workers and a dict of fakeworkers, represented by objects. The code can iterate over these. This is separated out from the multi-config changes since its separable and clearer this way. (Bitbake rev: 8181d96e0a4df0aa47287669681116fa65bcae16) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix two minor issues with the initialising tasks progressPaul Eggleton2016-08-171-2/+3
| | | | | | | | | | | | | | A couple of fixes for the "Initialising tasks" progress bar behaviour: * Properly finish the progress bar when using bitbake -S * Finish the progress bar before calling BB_HASHCHECK_FUNCTION (so that in OE when that shows its own "Checking sstate mirror object availability" progress bar it gets shown on the next line as it should). (Bitbake rev: de6759d8e9990e426e6d6464a2e05381cd4c12d6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: improve exception loggingEd Bartosh2016-07-191-2/+2
| | | | | | | | | | | | | | | Runqueue errors direct the user to view the "failure below", but no additional error message is available. Log the stacktrace so that the user can see what went wrong. Also fix a typo in the log message. (Bitbake rev: e191f401e372ee181bc02250232ad9cb9a0e9477) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: report progress for "Preparing RunQueue" stepPaul Eggleton2016-07-081-3/+67
| | | | | | | | | | | | | | | | | | | | | When "Preparing RunQueue" shows up you can expect to wait up to 30 seconds while it works - which is a bit long to leave the user waiting without any kind of output. Since the work being carried out during this time is divided into stages such that it's practical to determine internally how it's progressing, replace the message with a progress bar. Actually what happens during this time is two major steps rather than just one - the runqueue preparation itself, followed by the initialisation prior to running setscene tasks. I elected to have the progress bar cover both as one (there doesn't appear to be much point in doing otherwise from a user perspective). I did however describe it as "initialising tasks". (Bitbake rev: 591e9741e108487ff437e77cb439ef2dbca42e03) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add ability to enforce that tasks are setscenedPaul Eggleton2016-07-081-1/+80
| | | | | | | | | | | | | | | | | | | | | | | | Add the ability to enter a mode where only a specified whitelist of tasks can be executed outright; everything else must be successfully provided in the form of a setscene task (or covered by a setscene task). Any setscene failure outside of the whitelist will cause the build to fail immediately instead of running the real task, and any real tasks that would execute outside of the whitelist cause an immediate build failure when it comes to executing the runqueue as well. The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting of pn:taskname pairs. A single % character can be substituted for the pn value to match any target explicitly specified on the bitbake command line. Wildcards * and ? can also be used as per standard unix file name matching for both pn and taskname. Part of the implementation of [YOCTO #9367]. (Bitbake rev: 624722c067a7fdd0c0f5d8be611e1f6666ecc4a2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Use tid instead of taskid in find_chains()George McCollister2016-06-161-1/+1
| | | | | | | | | | | In 2c88afb6 find_chains()'s taskid argument was renamed to tid but taskid is still used as key to explored_deps dictionary. Use tid instead of taskid. (Bitbake rev: 29a34ae8f5306d2779bcc761c52f1f9d13a0c0c5) Signed-off-by: George McCollister <george.mccollister@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskdata/runqueue: Rewrite without use of ID indirectionRichard Purdie2016-06-151-500/+458
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I'm not sure what possesed me when I wrote this code originally but its indirection of everyting to use numeric IDs and position dependent lists is horrific. Given the way python internals work, its completely and utterly pointless from performance perspective. It also makes the code hard to understand and debug since any numeric ID has to be translated into something human readable. The hard part is that the IDs are infectous and spread from taskdata into runqueue and even partly into cooker for the dependency graph processing. The only real way to deal with this is to convert everything to use a more sane data structure. This patch: * Uses "<fn>:<taskname>" as the ID for tasks rather than a number * Changes to dict() based structures rather than position dependent lists * Drops the build name, runtime name and filename ID indexes On the most part there shouldn't be user visible changes. Sadly we did leak datastructures to the setscene verify function which has to be rewritten. To handle this, the variable name used to specifiy the version changes from BB_SETSCENE_VERIFY_FUNCTION to BB_SETSCENE_VERIFY_FUNCTION2 allowing multiple versions of bitbake to work with suitably written metadata. Anyone with custom schedulers may also need to change them. I believe the benefits in code readability and easier debugging far outweigh those issues though. It also means we have a saner codebase to add multiconfig support on top of. During development, I did have some of the original code coexisting with the new data stores to allow comparision of the data and check it was working correcty, particuarly for taskdata. I have also compared task-depends.dot files before and after the change. There should be no functionality changes in this patch, its purely a data structure change and that is visible in the patch. (Bitbake rev: 2c88afb60da54e58f555411a7bd7b006b0c29306) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change buildable/running lists to setsRichard Purdie2016-06-151-36/+27
| | | | | | | | | | | | | | Using positions in lists for flags is an odd choice and makes the code hard to maintain. Maintaining a list is slow since list searches are slow (watch bitbake -n slow massively with it) but we can use a set() instead. This patch uses python sets to maintain the lists of tasks in each state and this prepares for changing the task IDs from being integers. (Bitbake rev: 8c1ed57f6ea475b714eca6673b48e8e5f5f0f9c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to python 3Richard Purdie2016-06-021-61/+56
| | | | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: always emit bb.event.DepTreeGeneratedPatrick Ohly2016-05-141-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | The data included in the event is useful for implementing a pre-build check that warns about unexpected components, for example because of an incorrect configuration or changed dependencies. Such a check can be done in a .bbclass that gets inherited globally. But in contrast to a UI, such a class cannot request that the event shall be emitted, and thus the event has to be emitted whether there is a consumer or not. This was done conditionally earlier out of concerns about the performance impact. But now events are handled more efficiently, so that concern no longer seems valid: in some simple testing (admittedly on a fast build workstation), the two lines (generating the data and emitting the event with it) only took about 0.05 seconds (measured with timeit). That was for a build with roughly 500 recipes (from pn-buildlist aka depgraph['pn']), triggered via the command line. That was even with a consumer of the data active and doing some work, so it should be even faster when there is no consumer. (Bitbake rev: 5ddaf5b7ed1001d2dd3f67e7a6d704afa85479d2) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve timestamp comparisonsRichard Purdie2016-05-131-1/+5
| | | | | | | | | | python3 cares more about invalid type comparisons. Add break statements and better tests to make the code paths clearer and avoid type issues in python3. No code functionality change. (Bitbake rev: 2c39ebdd2762d027f007a6a769fdf023cdf3da2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix missing fakeworker under dry runRichard Purdie2016-05-111-1/+1
| | | | | | | | | We shouldn't try and use fakeworker when performing a dry_run. This makes the core match the other fakeworker execution points. (Bitbake rev: 49bea821a2edad5e19c3a566d1a80c23718dede9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update logger.warn() -> logger.warning()Richard Purdie2016-05-111-2/+2
| | | | | | | | | | python deprecated logger.warn() in favour of logger.warning(). This is only used in bitbake code so we may as well just translate everything to avoid warnings under python 3. Its safe for python 2.7. (Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve 'mulitiple .bb files are due to be built' messageRichard Purdie2016-04-121-2/+70
| | | | | | | | | | | | | | | | | | | | | | | | | When multiple recipes which both provide something are being built, bitbake informs us that most likely one of them provides something the other doesn't, which is usually correct, but unfortunately it's rather painful to figure out exactly what that is. This patch dumps two sets of information, one is the provides information for each recipe, filtered so only common components are removed. The other is a list of dependees on the recipe, since sometimes this can easily identify why something is being built. Its not straightforward for bitbake to obtain the information but since the warning/error code path isn't the normal one, we can afford to go through some less than optimal processing to aid debugging. Also provide the same information even if we're showing a warning since its still useful. [YOCTO #8032] (Bitbake rev: 96fc889b8e62ba4463c71158c4b7286c48d68cd8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/runqueue: save task file dependency cache onto diskMarkus Lehtonen2016-02-181-0/+1
| | | | | | | | | | | | Before this patch the usage of cache was quite useless as the file checksums were not actually cached on disk but re-calculated every time. This patch utilises the new writeout_file_checksum_cache() method of the SignatureGenerator class to do the job. (Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Set process names to be meaninfulRichard Purdie2016-01-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "runqueue.py: Ensure one setscene function doesn't mask out ↵Richard Purdie2016-01-191-5/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | another which needs to run" This reverts commit b22592af8145a8c7c4ada2fa7c1dee2e753eca46. That commit isn't entirely clear about why this change is needed but I do have a usecase where this breaks things. If for example you run "bitbake X -c packagedata" and that packagedata is in sstate, you'd expect this to work. If sstate doesn't contain a do_populate_sysroot for a dependency, you would still expect the command above to succeed and you would not expect it to rebuild that dependency. With the current code, this isn't what happens. The code finds the sstate for do_populate_sysroot missing, this makes the task "uncovered" and this in turn makes it unskippable. The example I found with this was avahi-ui, where it would trigger a build of libdaemon to obtain its populate_sysroot. Since this behaviour seems completely incorrect, revert the older patch and we'll address any issues that crop up as a result. (Bitbake rev: 36a9840a5da17cc14561881fdd6a4f2cb0a75e49) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix setscene task dependenciesRichard Purdie2016-01-111-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | Debugging suggests that setscene tasks are being a little greedy about their dependencies, for example, lsof is insisting that gcc-runtime's do_package is installed. If it isn't, its requiring gcc to rebuild. If gcc-runtime do_package_write_xxx and do_packagedata is available, there is no reason do_package should be needed. The reason this is happening appears to be from the batching up of task dependencies code, rather than setscene tasks stopping when passing over a setscene task, they were being carried forward. This patch fixes it so the data is 'zeroed' when passing over a setscene task boundary, which gives the dependency graph that is expected. After this patch, lsof will rebuild quite happily without gcc-runtime:do_package being present, as expected. This should lead to less dependencies being installed for builds from sstate and generally better performance in general. (Bitbake rev: f8bcb0a1e3b008b71c9a7cd21f76d0906f2d8068) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskdata: refactor get_providermapEd Bartosh2016-01-111-1/+1
| | | | | | | | | | | | | | | Added optional parameter 'prefix' to filter out names that don't start with specified prefix. Changed existing call of get_providermap according to changed API. Optimized the code: got rid of extra loop and temporary list variable virts. (Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main/runqueue: Add --setscene-only option to bitbakeRichard Purdie2016-01-111-3/+6
| | | | | | | | | | | | | Its turning out that we really need a way to have bitbake just run the setscene tasks but not any real tasks, particularly for SDK operations. Add an option for this since its pretty straight forward. This allows various nasty workarounds in OE-Core to be removed. (Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add support for <task>- syntaxRichard Purdie2015-12-181-8/+18
| | | | | | | | | | | It can be useful to run all tasks up to but not including a specific task. The main reason this was never added was the lack of a good syntax. This patch uses the syntax <taskname>- to denote this behaviour which is simple, not invasive and fits what we need from good syntax IMO, hence we can add this. (Bitbake rev: 99ccfd411ab3f7baa111f9f3d50fae68816a9a83) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add handling of virtual/xxx provider mappingsRichard Purdie2015-10-011-0/+9
| | | | | | | | | | | | This firstly prints debug messages which show how bitbake decided to resolve the virtual/xxx providers which is useful for debugging. If the siggen has a tasks_resolved() method, it calls this, passing in the mappings, allowing that to do things with the resolved names. (Bitbake rev: d473fc84acddfd69a7207affcd89f65ea2ecf730) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/runqueue: Allow bitbake commands starting with do_Alex Franco2015-09-041-1/+3
| | | | | | | | | | | | | | | The output of "bitbake, -c listtasks pkg" lists tasks with their real names (starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always gets unconditionally prepended to task names. This patch handles this error by checking whether a task starts with "do_" prior to prepending it with it when the task runlist is being constructed (and a few other corner cases). [YOCTO #7818] (Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df) Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: Add provides to taskdepdataMariano Lopez2015-08-101-1/+2
| | | | | | | | | | | | | | | Currently bitbake is the only one that knows the relation between PN and PROVIDES. In some cases it is needed to have this relation in the data store (the bootloader it's a good case). This adds the PROVIDES to the taskdata, so it would be in the data store as one field of BB_TASKDEPDATA. (Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Sanity check BB_NUMBER_THREADSRichard Purdie2015-06-231-0/+3
| | | | | | | | | | | | Bitbake does really weird things with negative or zero numbers of threads which is confusing to the user. Add a sanity check for this. When you have code doing arithmetic on the values and a VM reconfigures to only a single thread, negative numbers are easier than you'd think. (Bitbake rev: 32166ac3c85ff3c04081580ae76bd63590d6ff3e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-11 18:58:52 +0000