summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue.py: revised completion schedulerPatrick Ohly2017-01-201-24/+101
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The idea is that tasks which complete building a recipe (like do_package_qa) are more important than tasks which start building new recipes (do_fetch) or those which increase disk usage (do_compile). Therefore tasks get ordered like this (most important first, do_rm_work before do_build because the enhanced rm_work.bbclass was used): 1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build 2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build 3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build ... 464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build 465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work 466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work 467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work ... 3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install 3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install 3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base 3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base ... 3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base 3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base 3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage 3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck 3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip 3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules 3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir 3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images 3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile 3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile ... The order of the same task between different recipes is the same as with the speed scheduler, i.e. more important recipes come first. (Bitbake rev: 70e297e5c285ce0a02e9efd3117ff62cdc77ec12) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build.py: add preceedtask() APIPatrick Ohly2017-01-201-0/+16
| | | | | | | | | | | | | | | | | | | The API is required by the revised rm_work.bbclass implementation, which needs to know all tasks that do_build depends so that it can properly inject itself between do_build and those tasks. The new API primarily hides the internal implementation of the "after" and "before" dependency tracking. Because tasks defined as precondition via "recrdeptask" may or may not be relevant (they are for rm_work.bclass), the API also includes support for that. There's no default value for including recrdeptasks, so developers have to think about what they need. (Bitbake rev: 9289ab40e77906e983a2f79cd7602ee95be5025a) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid recursive errorsRichard Purdie2017-01-201-0/+9
| | | | | | | | | | If PATH contains WORKDIR which contains PV which contains SRCPV we can end up in circular recursion within the fetcher. This code change allows for the recursion to be broken by giving PV a temporary dummy value in a data store copy. (Bitbake rev: ce1e70b8018340b54dba3a81d7d379182cb77514) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add BB_LIMITEDDEPS supportRichard Purdie2017-01-203-0/+4
| | | | | | | | | | | When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited. We really need a way to know this from the metadata and this new variable provides this in worker context. This means existing code can stop having to guess. (Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/ast: Add RecipeTaskPreProcess event before task finalisationRichard Purdie2017-01-202-0/+12
| | | | | | | | | | | There are various pieces of code which need to run after the tasks are finalised but before bitbake locks in on the task dependencies. This adds such an event so dependency changes in anonymous python can be accounted for and acted upon by these specific event handlers. (Bitbake rev: 4dcd0e53f5ff4bf4f2d6cbdc51ff33a5f5f206af) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix traceback when using -bRichard Purdie2017-01-201-0/+2
| | | | | | | | | Without this, bitbake -b of image recipes cause tracebacks since the list of providers is empty. (Bitbake rev: c53e43f3c6a675c0934a7a4e358fd66b049ffca3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Only add universe targets where the task existsRichard Purdie2017-01-201-0/+9
| | | | | | | | | | | | | Currently "bitbake universe -c fetch" will error if every task in the system doesn't have a fetch task. This seems unreasonable so filter the list based upon whether the task exists. This implementation does assume if a task exits for a given provider name, it can run, even if the preferred provider that ends up getting built might be the namespace without the task. This is unlikely to be a real world issue at this point. (Bitbake rev: 438986f4e39974930ee3205ca4c4d620090e57bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: new multiconfig '*' syntax supportJuro Bystricky2017-01-191-1/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently you cannot build a target for all the configured multiconfigs without specifying a list. The list can be quite long, requiring to type several lines of text. This enhancement is to support globbing so that you can do this, e.g. instead of: $ bitbake multiconfig:A:bash multiconfig:B:bash bash you can do: $ bitbake multiconfig:*:bash There are real world use cases where it is desirable to use multiconfig with two different tasks. For example: SDKs with multiple toolchains but also containing set of additional host tools, or multiconfig builds requiring one image for the main CPU(s) and a different co-image for a companion CPU. For this reason, two variations of the new syntax are supported. For example, the following: $ bitbake multiconfig:*:meta-toolhchain would expand to: $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain meta-toolchain However the following: $ bitbake multiconfig:*:meta-toolhchain hosttools would expand to: $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain hosttools In other words, if the user specified the "default" task explicitly, it replaces the implicit "default" task. [YOCTO#10680] (Bitbake rev: 1559ad1fc3ae8b1e4e75b210f598b90be192571b) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: bin/toaster whitelist TOASTER_DIRbrian avery2017-01-171-0/+1
| | | | | | | | | | | | | | | Some of the tinfoil2 changes resulted in TOASTER_DIR being cleared by the memory resident bitbake server toaster starts up. This prevented toaster from being able to connect to its sqlite database. Adding TOASTER_DIR to the BB_ENV_EXTRAWHITE list stops the cooker from clearing out the TOASTER_DIR variable. [YOCTO #9252] (Bitbake rev: 2420953b3d03551d8254609300ca572717aecdcd) Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Add force flag to git checkout command in update_submodulesOla Redell2017-01-171-1/+1
| | | | | | | | | | | | When the gitsm fetcher is used with a repo that includes a .gitattributes file that makes git modify files on cloning (e.g. line break characters), the subsequent checkout performed in the update_submodules function fails. This is fixed by adding the force flag (-f) to the checkout command. (Bitbake rev: c05e1396625b14e66d795408ea2ae4cd2afc3209) Signed-off-by: Ola Redell <ola.redell@retotech.se> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: use multiple processes to dump signatures.Jianxun Zhang2017-01-172-7/+29
| | | | | | | | | | | | | | | | | | | | | This change significantly shortens the time on reparsing stage of '-S' option. Each file is reparsed and then dumped within a dedicated process. The maximum number of the running processes is not greater than the value of BB_NUMBER_PARSE_THREADS if it is set. The dump_sigs() in class SignatureGeneratorBasic is _replaced_ by a new dump_sigfn() interface, so calls from the outside and subclasses are dispatched to the implementation in the base class of SignatureGeneratorBasic. Fixes [YOCTO #10352] (Bitbake rev: 99d3703edd77a21770b366c6ad65a3c0f5183493) Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Replace slightly elderly bbmake references!Richard Purdie2017-01-091-3/+3
| | | | | | | | | bbmake hasn't existed for a while (a decade+) so we should probably remove the references. (Bitbake rev: 4170cf874850b950f31d2e36c895c110c0b096c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/persist_data/utils: Drop obsolete python2 importsRichard Purdie2017-01-093-28/+7
| | | | | | | | | These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. (Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/cooker: BBCooker stops notifier at shutdownAníbal Limón2017-01-061-0/+2
| | | | | | | | | | | | | At end of BBCooker needs to release fd's associated with pyinotify watchers to avoid: Too many open files (EMFILE) error in different scenarios like several instances of tinfoil. [YOCTO #10873] (Bitbake rev: ae6045b84978940c365c95c33d6996359c3e299d) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Tweak stdout manipulation to be stream safeRichard Purdie2017-01-051-4/+17
| | | | | | | | | | | | | | | | | | | | | We've been seeing oe-selftest failures under puzzling circumstances. It turns out if you run oe-selftest on a machine with xmlrunner installed and have the recent tinfoil2 changes, the launching of PR server crashes leading to selftest hanging if using an autoloaded PR server. The reason is that xmlrunner uses an io.StringIO object as stdout/stderr instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method. We have to deal with both cases and in the python way, we try and then seek forgivness if we see an AttributeError or UnSupportedOperation exception. Unfortunately we have to deal with both cases as we may be performing a traditiional double fork() from the commandline, or a larger python program. [YOCTO #10866] (Bitbake rev: 26243f04e3af652291d13e85c084057104fe155b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Convert multiconfig to use BB_CURRENT_MCRichard Purdie2016-12-221-2/+3
| | | | | | | | | | | | | | | | | People are struggling with multiconfig as the point the conf file is injected into the data store is not what people expect. We can't really use a post config since that is too late and we can't really use a pre config file since that is too early. In OE terms, we need something right around the local.conf point so it behaves in a similar way. A way to handle this is to set the new variable BB_CURRENT_MC to be the currently selected multiconfig, then the metadata itself can choose when to inject the approriate configuration. (Bitbake rev: 518b9015c2be8d3894277a8e54890d6f04d656c0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Only start fakeroot workers when neededRichard Purdie2016-12-221-8/+7
| | | | | | | | | | | | | Fakeroot workers usually have dependencies that need to be ready before they can be started. Starting them as a block therefore doesn't work as the dependencies may or may not have been built. Therefore start the multiconfig fakeworkers individually upon demand. [YOCTO #10344] (Bitbake rev: ab15486c636e451ed5a62e561980946887901c3b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure setscene tasks with overlapping stamps don't ↵Richard Purdie2016-12-221-1/+5
| | | | | | | | | | | | | | | parallel execute In multiconfig, mutliple tasks can execute which share the same stamp file. These must not execute in parallel, the idea is the first should execute, the subsequent ones should see a valid stamp and get skipped. The normal task execution code has stamps code to handle this, this adds similar code to the setscene execute() function to handle the issue there too. (Bitbake rev: 937acf267fa9e45f538695b2cf8aa83232a96240) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix setscene issues with multiconfigRichard Purdie2016-12-221-20/+20
| | | | | | | | | | | | | | | | setscene was being entirelu skipped for multiconfig variants as the tasks were simply not being spotted. If the default config was also being built it masked the problem. When this was fixed by using taskfn instead of fn in lookups against dataCache, several other instances of this problem were highlighted. This goes through and corrects the setscene code to correclty use taskfn instead of fn in the appropriate places meaning setscene tasks for multiconfig now work correctly. (Bitbake rev: cead85cf219c07b40aeb7c0d948c25f0429136db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix clean_basepath to work with multiconfigRichard Purdie2016-12-221-0/+5
| | | | | | | | | | Tasknames can now start with "multiconfig:" which broke the virtual: comparison code and lead to unpredictable checksums with nativesdk recipes. This adds in handling for the new additional prefix which unbreaks nativesdk builds when using multiconfig. (Bitbake rev: 4da296c0c93391ec37f8b6e2245071657759771e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/uihelper: Show better information about multiconfig tasks on UIRichard Purdie2016-12-222-1/+5
| | | | | | | | | | Currently the UI shows X is building, possibly multiple times but doesn't say which of the multilibs that might be. This adds a prefix to the task name so the mulitconfig being built can be identified. (Bitbake rev: d93ddd185dde53f50aea06ada094bb0d9b5ba4f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure pseudo executes from the correct place (use the ↵Richard Purdie2016-12-221-2/+3
| | | | | | | | | | | | | | right datastore with multiconfig) The location of the fakeroot command and the various environmental values need to be taken from the right multiconfig datastore, not the shared one. This patch ensures the right one is used for cases like a split TMPDIR. [YOCTO #10344] (Bitbake rev: 813a7600bc26b7132ac36d1515571d321f566afb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fixup case with no useful netrc dataMark Asselstine2016-12-211-1/+1
| | | | | | | | | | | | | | | | Commit 873e33d0479e977520106b65d149ff1799195bf6 [fetch2/wget: add Basic Auth from netrc to checkstatus()] causes "Fetcher failure for URL: 'https://www.example.com/'. URL https://www.example.com/ doesn't work." on new builds when a user has a .netrc file but there is no default and no matching host. The call to netrc.authenticators() will return None in these cases and the attempted assignment to the 3-tuple will raise a TypeError exception. Add the TypeError to the exceptions caught to get around this issue. (Bitbake rev: c0c0af40ebddaf9dc99353c580a65d4c04295613) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: add Basic Auth from netrc to checkstatus()Matthew McClintock2016-12-201-2/+14
| | | | | | | | | | | | | | fetch2/wget uses urllib to check the status of the mirrors, wget will use netrc to pass login and password information however checkstatus will skip that. This adds netrc login and password to checkstatus so both will work the same. (Bitbake rev: 873e33d0479e977520106b65d149ff1799195bf6) Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: dry-run real tasks when BB_SETSCENE_ENFORCE is setPaul Eggleton2016-12-202-11/+14
| | | | | | | | | | | | | | | | | | | | | | | | For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used by the installation process for the extensible SDK), we don't actually need the whitelisted real tasks to execute - we just need to have them in the dependency tree so that we get all of the setscene tasks they depend on to run. Therefore we can actually dry-run those real tasks i.e. they won't be run (and thus we won't waste a significant amount of time doing so) and won't be stamped as having run either. We do already have a dry-run mode in BitBake (activated by the -n or --dry-run command line option), but it dry-runs the setscene tasks as well which we don't want here. Note that this has no effect on the checking we are doing with BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are scheduled to run - that's handled separately. Fixes [YOCTO #10369]. (Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: use with instead of open/close2.3_M1Ross Burton2016-12-161-11/+11
| | | | | | | | | | This is more pythonic and can handle unclosed file warnings better than the previous code structure. (Bitbake rev: 50633012a64a3b5f0662145e29ff426374fb7683) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Drop expanded_dataRichard Purdie2016-12-163-21/+19
| | | | | | | | | | | | | | | Some of our metadata assumes that BuildStarted and BuildCompleted events see the same data store. This is the case for buildTarget but not for buildFile and recent changes mean this is now a problem. The update_data() call is now an empty operation and there is no difference between the expanded_data and data so we can simply remove the expanded_data and its references and use data everywhere. This has been inteded for a while but the above issue makes this more pressing to finally clean up. (Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar calls (take 2)Joshua Lock2016-12-163-3/+3
| | | | | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (a follow on patch to fix up a few recent introductions) (Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/build: enable access to logger within tasksPaul Eggleton2016-12-142-0/+12
| | | | | | | | | | | | | | | | | | In certain circumstances it can be useful to get access to BitBake's logger within a task; the main example is in OpenEmbedded's image construction code where we want to be able to check the log file for errors and warnings, but we don't want to see any errors or warnings that were emitted through the logger; so we need a way to exclude those. In order to do this, pass the logger object into the task via a BB_TASK_LOGGER variable, and add a logging handler class to bb.utils that can be added to it in order to keep a list of warnings/errors that have been emitted. (Bitbake rev: f1cd6fab604f14d8686b1d783cbfe012d923ee42) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: don't change UI process signal handler on terminatePaul Eggleton2016-12-141-1/+0
| | | | | | | | | | | | | | | | | | | | | On terminating the connection to the server, we were disabling SIGINT - and this is executed on the UI side. I'm not sure whether the intention here was to undo the SIGINT disabling we did in the server, and it was just a mistake that it disabled rather than restored and it's run on the wrong side, or whether we wanted to stop the user from breaking out of the shutdown code - the commit message provides no clues either way. Regardless, we do not want to permanently disable Ctrl+C here - it's legitimate to terminate the connection to the server and then re-establish it within the same process; at least currently, devtool modify by virtue of using tinfoil in two separate parts of the code does this, and the result of this disabling is that during the second tinfoil usage we can potentially be parsing all recipes without the ability to easily interrupt the process. (Bitbake rev: 58c60a951229dcbd8253863fb24228d046c23f6e) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: add means of ignoring basehash mismatchPaul Eggleton2016-12-141-1/+2
| | | | | | | | | | | | If you run the setVariable command to set variables then you end up causing the basehash to not match the previously computed values, which triggers error messages. These mismatches are expected, so add a means of disabling them. (Bitbake rev: 5a80c0e210f26526afbe8f266b7b1a9c03334967) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: enable setVariable command to affect task executionPaul Eggleton2016-12-144-2/+13
| | | | | | | | | | | | Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. (Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: support serialisationPaul Eggleton2016-12-142-0/+36
| | | | | | | | | | | | | The COW object used within VariableHistory can't be serialised itself, but we can convert it to a dict when serializing and then back when deserialising. This finally allows DataSmart objects to be serialized. NOTE: "serialisation" here means pickling, not over XMLRPC or any other transport. (Bitbake rev: bbbb2a53d5decf3b613a92c4ff77c84bfc5d4903) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: allow buildFile warning to be hidden programmaticallyPaul Eggleton2016-12-143-7/+12
| | | | | | | | | | If we want to use this function/command internally, we don't want this warning shown. (Bitbake rev: 5cfbb60833e7b12d698c1c2970c17ccf2a4971bf) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: pass datastore to server when expanding python referencesPaul Eggleton2016-12-144-11/+18
| | | | | | | | | | | | | | | | | | | | If you're expanding a value that refers to the value of a variable in python code, we need to ensure that the datastore that gets used to get the value of that variable is the client-side datastore and not just the part of it that's on the server side. For example, suppose you are in client code doing the following: d.setVar('HELLO', 'there') result = d.expand('${@d.getVar("HELLO", True)}') result should be "there" but if the client part wasn't taken into account, it would be whatever value HELLO had in the server portion of the datastore (if any). (Bitbake rev: cbc22a0a9aadc8606b927dbac0f1407ec2736b35) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: implement server-side recipe parsingPaul Eggleton2016-12-142-17/+81
| | | | | | | | | | | | | It's not really practical for us to parse recipes on the client side, we need to do it on the server because that's where we have the full python environment (including any "pure" python functions defined in classes). Thus, add some functions to tinfoil do this including a few shortcut functions. (Bitbake rev: 8f635815d191c9d848a92d51fdbf5e9fd3da1727) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: remotedata: enable transporting datastore from the client to the serverPaul Eggleton2016-12-141-0/+42
| | | | | | | | | | | | | | | | | | | | | | For the purposes of server-side parsing and expansion allowing for client-side use of the datastore, we need a means of sending a datastore from the client back to the server, where the datastore probably consists of a remote (server-side) original plus some client-side modifications. To do this we need to take care of a couple of things: 1) xmlrpc can't handle nested dicts, so if you enable memres and simply try passing a serialised datastore then things break. Instead of serialising the entire datastore, just take the naive option of transferring the internal dict alone (as a list of tuples) for now. 2) Change the TinfoilDataStoreConnector object into simply the handle (number) when transmitting; it gets substituted with the real datastore when the server receives it. (Bitbake rev: 784d2f1a024efe632fc9049ce5b78692d419d938) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: rewrite as a wrapper around the UIPaul Eggleton2016-12-146-119/+665
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rewrite tinfoil as a wrapper around the UI, instead of the earlier approach of starting up just enough of cooker to do what we want. This has several advantages: * It now works when bitbake is memory-resident instead of failing with "ERROR: Only one copy of bitbake should be run against a build directory". * We can now connect an actual UI, thus you get things like the recipe parsing / cache loading progress bar and parse error handling for free * We can now handle events generated by the server if we wish to do so * We can potentially extend this to do more stuff, e.g. actually running build operations - this needs to be made more practical before we can use it though (since you effectively have to become the UI yourself for this at the moment.) The downside is that tinfoil no longer has direct access to cooker, the global datastore, or the cache. To mitigate this I have extended data_smart to provide remote access capability for the datastore, and created "fake" cooker and cooker.recipecache / cooker.collection adapter objects in order to avoid breaking too many tinfoil-using scripts that might be out there (we've never officially documented tinfoil or BitBake's internal code, but we can still make accommodations where practical). I've at least gone far enough to support all of the utilities that use tinfoil in OE-Core with some changes, but I know there are scripts such as Chris Larson's "bb" out there that do make other calls into BitBake code that I'm not currently providing access to through the adapters. Part of the fix for [YOCTO #5470]. (Bitbake rev: 3bbf8d611c859f74d563778115677a04f5c4ab43) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: provide a means to shut down from the client in memres modePaul Eggleton2016-12-143-0/+16
| | | | | | | | | | In memory resident mode we don't really want to actually shut down since it's only the client going away. (Bitbake rev: 74db369c46043116359101cab70486afd82372c0) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: implement remote datastore functionalityPaul Eggleton2016-12-142-6/+72
| | | | | | | | | | | | | | | | | This allows you to maintain a local reference to a remote datastore. The actual implementation of the remote connection is delegated to a connector object that the caller must define and supply. There is support for getting variable values and expanding python references (i.e. ${@...} remotely, however setting variables remotely is not supported - any variable setting is done locally as if the datastore were a copy (which it kind of is). Loosely based on an earlier prototype implementation by Qing He. (Bitbake rev: a3edc3eefa2d03c4ad5d12187b32fa4dc495082a) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc: send back 503 response with correct encodingPaul Eggleton2016-12-141-1/+1
| | | | | | | | | | | | If you send back a string here you get "TypeError: 'str' does not support the buffer interface" errors in bitbake-cookerdaemon.log and "IncompleteRead(0 bytes read, 22 more expected)" errors on the client side. (Bitbake rev: 0d659a7dfe5fb096f8aa4380320f9e2a464b3cb5) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: fix --observe-only optionPaul Eggleton2016-12-141-3/+7
| | | | | | | | | | | If we're in observe-only mode then we cannot run commands that would affect the server's state, including getSetVariable, so prevent that from being called in observe-only mode. (Bitbake rev: 2c5a8661430edebff67ab4a108995033d182b5d6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: make quiet option a level optionPaul Eggleton2016-12-142-9/+30
| | | | | | | | | | | | | | | | | Allow you to specify -q / --quiet more than once to reduce the messages even further. It will now operate as follows: Level Option Result ----- ------ ---------------------------------------- 0 Print usual output 1 -q Only show progress and warnings or above 2 -qq Only show warnings or above 3+ -qqq Only show errors (Bitbake rev: 6cf2582e17c28ca04f5cfb59858c4a9778c700d4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: fix resetting of reference on variablehistoryPaul Eggleton2016-12-141-1/+1
| | | | | | | | | | | | | There is no "datasmart" member, only dataroot. This dates back to the original implementation of variable history support - it's surprising we haven't noticed the issue until now, but I guess it's rare to change a copy of a datastore in a manner which using the old reference would cause an issue. (Bitbake rev: febd5534b07edfdef15cedb0578730c582c7373f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: views Remove old code that converts template context to JSONMichael Wood2016-12-121-31/+2
| | | | | | | | | | Remove the template context to JSON decorator function as this is deprecated by having a proper REST API. (Bitbake rev: b65a8193368ffa1d15af24a6acde8dce6bd4d383) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: js test Update js tests to use correct url for layer RESTMichael Wood2016-12-121-1/+5
| | | | | | | | | Update js tests so that we use the new Layer REST API. (Bitbake rev: 81764ce3ebf0d2fcb3dc6965f6f07931d39e5524) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Switch front end to use Layer get REST APIMichael Wood2016-12-125-6/+31
| | | | | | | | | | Switch the front end to use the proper REST API for retrieving layer information. (Bitbake rev: 5ea25c49091f4d4b5007af948e063ed25ba5766f) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: typeaheads Layers add url to layer REST API to the layerMichael Wood2016-12-121-0/+3
| | | | | | | | | | Add the url to the Layer typeahead so that this can be used later on by the front end code to look up layer details. (Bitbake rev: d195f24a1b30ae8698bff5e87308347b9596a2e2) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: api Add GET REST API for Layer informationMichael Wood2016-12-121-3/+52
| | | | | | | | | Add a get API for returning information about layers. (Bitbake rev: 3fa5170c98c699d2a7a8380b696fc599efcc5dee) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: importlayer Add git revision typeahead to that input fieldMichael Wood2016-12-124-9/+28
| | | | | | | | | | | Add the front end mechanism to load the typeahead for the git revision field on importing a layer. Also fix one indentation issue and update the js test. (Bitbake rev: 28114be42174095b812a93d4b5a0e01e953d74f8) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-12 05:57:46 +0000