summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cooker.py
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: cooker: Handle datastore tracking for showEnvironment server side onlyRichard Purdie2017-08-311-1/+10
| | | | | | | | | | The current passing of "tracking" backwards and forwards, client to server is ugly and complex and error prone. Instead, set this during showEnvironment commands triggering a reset there if/as required. (Bitbake rev: 9dc7f384db0479569ff93a76a623d5395fecaf47) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/cooker: Drop unused paramRichard Purdie2017-08-311-1/+1
| | | | | | | | Drop pointless unused function parameter. (Bitbake rev: 8104b33656de0b619943bd7a9884eb650ccafbf4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ignore common bitbake files for the parse cache invalidationRichard Purdie2017-08-131-0/+3
| | | | | | | | | | | | | | Writes to the cookerdaemon log and/or the lockfile were meaning the parser cache was always being invalidated and reparsed. This is unnecessary so spot accesses to these two common cases and ignore the files from a reparse perspective. This doesn't remove many sources of reparse but does improve several common cases. (Bitbake rev: 218e4b6418992588312b8ef5949b84ef43263d1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve inotify overflow handlingRichard Purdie2017-08-131-1/+4
| | | | | | | | | | Add a proper function for clearing the mtime cache. Clean up the inotify event overflow case to err on the side of caution and clear any potentially now out of sync caches. (Bitbake rev: ec60459fe2ba16966544eebff43b061abb7ed3ba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure we handle inotify before running the next commandRichard Purdie2017-08-131-7/+10
| | | | | | | | | | | | | | The inotify watch events are handled "at idle" which could in theory mean a command could run before any preceeding inotify events have been processed. This leads to a theoretical race window where those events may have a signficicant effect on the command. Add a mechanism to allow us to ensure all pending events are processed before running commands. (Bitbake rev: bf76cd7e5881adf264b8ba64e27a5b6ca9df4fde) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Track directories searched for bbappend/bb filesRichard Purdie2017-08-131-3/+21
| | | | | | | | | | | | | | | | | Some of the directories searched by BBFILES are not currently being added to the inotify watch list. This can mean that added append files are not noticed leading to misleading metadata results when using BB_SERVER_TIMEOUT != 0. We use glob to expand the BBFILES references and without writing our own implentation, figuring out the directories it searches is hard. We use some horrible hacks here to intecept the listdir calls, I'm open to better ways to handle this but this does ensure we have the right watches set. (Bitbake rev: 4d508d35a224e3a25d2d59c8415ab7985964b14f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/siggen: Reset siggen when reparsingRichard Purdie2017-08-131-0/+1
| | | | | | | | | | | | If we don't do this, we get basehash mismatch errors occurring from the reparse which would then set bitbake's error exit code. This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order to fail with a non-zero BB_SERVER_TIMEOUT. (Bitbake rev: e4c6ca9440f63761560b49bbe12654441f54687e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure all tasks are shown in task dependency treeRichard Purdie2017-08-031-3/+3
| | | | | | | | | | | | "bitbake -g m4-native -c do_unpack" doesn't list any dependencies in task-depends.dot. This is because no header describing the task was added unless a task has dependencies. Tweak the code to fix this. [YOCTO #10893] (Bitbake rev: e2fbe728769f09ef4178e2ef4d69a2f60ffe3ebb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process/cooker: Allow UI process to know if the cooker was started ↵Richard Purdie2017-07-301-1/+5
| | | | | | | | | | | | | | | | | | | successfully Currently if the server fails to start, the user sees no error message and the server will be repeatedly attempted to be started until some longer timeouts expire. There are error messages in the cookerdeamon log but nobody thinks to look there. Add in a pipe which can be used to tell the starting process whether the cooker did actually start or not. If it fails to start, no further attempts can be made and if present, the log file can be shown to the user. [YOCTO #11834] (Bitbake rev: 57000d44beb1aeba37dfc70782b0d6418943acc5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/process: Drop server_main functionRichard Purdie2017-07-241-22/+0
| | | | | | | | | | | | | | Now that there is only one server, this abstraction is no longer needed and causes indrection/confusion. The server shutdown is also broken with the cooker post_server calls happening too late, leading to "lock held" warnings in the logs if PRServ is enabled. Remove the abstraction and put the shutdown calls in the right order with respect to the locking. (Bitbake rev: c0ddde7cf680225127d6285685652b905ed176c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: add internal mode to build_file() functionPaul Eggleton2017-07-211-9/+25
| | | | | | | | | | | | | | | | | In OE's devtool we want to repeatedly run build_file() without showing unnecessary messages and triggering buildhistory for each call. build_file() is just a wrapper around the buildFile command. Change the final "hidewarning" parameter of the buildFile command to "internal" and have this call a new buildFileInternal() function without triggering any of the normal build events, silencing the normal info messages from the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and avoiding calling parseConfiguration() which we've already done at this point. (Bitbake rev: ba53e067a2d448dd63b4ca252557ce98aa8e6321) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata/cooker: Restore original datastore upon client disconnectRichard Purdie2017-07-211-0/+2
| | | | | | (Bitbake rev: 67ae612ab890965357d24a9bd35610cf813b79ba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server: Rework the server API so process and xmlrpc servers coexistRichard Purdie2017-07-211-57/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes the way bitbake server works quite radically. Now, the server is always a process based server with the option of starting an XMLRPC listener on a specific inferface/port. Behind the scenes this is done with a "bitbake.sock" file alongside the bitbake.lock file. If we can obtain the lock, we know we need to start a server. The server always listens on the socket and UIs can then connect to this. UIs connect by sending a set of three file descriptors over the domain socket, one for sending commands, one for receiving command results and the other for receiving events. These changes meant we can throw away all the horrid server abstraction code, the plugable transport option to bitbake and the code becomes much more readable and debuggable. It also likely removes a ton of ways you could hang the UI/cooker in weird ways due to all the race conditions that existed with previous processes. Changes: * The foreground option for bitbake-server was dropped. Just tail the log if you really want this, the codepaths were complicated enough without adding one for this. * BBSERVER="autodetect" was dropped. The server will autostart and autoconnect in process mode. You have to specify an xmlrpc server address since that can't be autodetected. I can't see a use case for autodetect now. * The transport/servetype option to bitbake was dropped. * A BB_SERVER_TIMEOUT variable is added which allows the server to stay resident for a period of time after the last client disconnects before unloading. This is used if the -T/--idle-timeout option is not passed to bitbake. This change is invasive and may well introduce new issues however I believe the codebase is in a much better position for further development and debugging. (Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix whitespacePaul Eggleton2017-07-181-4/+4
| | | | | | | (Bitbake rev: a5f9603988b0a897696e529391b4d95c5e339833) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Use multiple BuildStarted events for multiconfigRichard Purdie2017-07-081-12/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | Currently builds in multiple TMPDIRs with multiconfig can break since the BuildStarted event is used to create directory strutures in several cases (e.g. buildstats.bbclass) and there is only on BuildStarted event generated in a multiconfig build. We have two options, a) to add a new MultiConfigBuildStarted event which is generated once per multiconfig, or b) allow multiple BuildStarted events. Having reviewed the code and current users of BuildStarted, sending one event per multiconfig seems like its the best way forward and the existing code looks able to cope with the duplication of events. I did also check toaster and I think that can handle this issue too (multiconfig builds may have other issues there). I'm therefore proposing we send multiple BuildStarted events for multiconfig and for consistency, send multiple BuildCompleted events too. We need to ensure that BUILDNAME, BUILDSTART and DATE/TIME are set consistently in all the different multiconfig datastores. These events can write to the datastore so copies are not used. buildFile was also cleaned up to ensure it uses the right datastore in various places. (Bitbake rev: 0b00f0382780ab5390a5c3f756a9b4efafe0aec8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: ensure graceful exit after exception during BuildCompleted ↵2.4_M1Paul Eggleton2017-06-141-2/+4
| | | | | | | | | | | | | | | | handler If an event handler for bb.event.BuildCompleted fails, we still need to call finishAsyncCommand() or else BitBake will just exit immediately without showing any error summary, or worse in the case of memory resident mode BitBake will hang and if you Ctrl+C to break out, the command won't be marked as finished which means that no further commands will be able to be executed until the server is manually restarted. (Bitbake rev: 5639faa3eef55cc476a82e810e61ca228cbdf221) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix always loading cache on every UI start with memresPaul Eggleton2017-06-141-5/+12
| | | | | | | | | | | | | | | | | | | | | The main point of memory resident bitbake is to avoid loading data unnecessarily on every bitbake invocation. Unfortunately the code that updated options from the UI was simply treating the fact that either of the "prefile" or "postfile" options were in the list of options passed in as an indication that the configuration was invalid, which was bad because these are always passed in. We only need to mark the configuration as invalid and thus reload it (and thus reload the cache) if the option value has actually changed. At the same time, the recently handled "tracking" option needs to be treated in a similar manner since the configuration needs to be reparsed if that has changed. Also, add a few extra debug messages to aid debugging this code in future. (Bitbake rev: 18dfd144d0da6ff662308ce4f9d135dc11412133) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command / cooker: drop Hob support commands and functionsPaul Eggleton2017-06-121-193/+0
| | | | | | | | | | | | | | | | | | | Drop a number of the commands and support functions that were created solely to support functionality in the now-removed Hob UI. In most cases we now have this functionality elsewhere e.g. to modify config files and recipes we have bb.utils.edit_metadata_file() and friends, and in OE we have oe.recipeutils.patch_recipe_file() and friends which build on top of the former. Additionally, some of it represented pretty egregious incursion of OE metadata-specific references into BitBake code. For now I have left in the find*File functions and commands as they are relatively generic and possibly still useful, but they might be removed in future. (Bitbake rev: a322f13183c66a28d93cc4bc3d839d95fc1d90f4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: add BB_CMDLINE to enable access to UI command line with memresPaul Eggleton2017-06-121-1/+6
| | | | | | | | | | | | | | | | | | | | | | | In OpenEmbedded's buildhistory class we want access to the bitbake command line that launched the build, and up to now we were simply using sys.argv from within the event handler to get that. Unfortunately that doesn't work in memory resident mode, since the event handler is naturally executing within the server and thus will give you the command that launched the bitbake server which is much less interesting. Add a dynamic variable BB_CMDLINE to provide access to this, set from sys.argv within the UI process in updateToServer(). (Note that BB_CMDLINE isn't currently passed through to the worker, so this is only really readable from event handlers plus any explicit getVariable calls - in theory an observe-only UI could read it for example.) Part of the fix for [YOCTO #11634]. (Bitbake rev: 8ad31a3eff5cfcb7b1c462578a582dafcbc426d4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: set config valid flags in initConfigurationData()Paul Eggleton2017-06-121-6/+2
| | | | | | | | | | | | | | If we set these flags here then we can not only tidy up some of the code that calls initConfigurationData() (and reset() which simply calls the former), we also avoid calling initConfigurationData() an extra time unnecessarily during startup (since setting baseconfig_valid = False in updateConfigOpts() also resulted in initConfigurationData() being called from updateCacheSync() at the end of the command.) (Bitbake rev: 74d2ef99a920aa9e70470ba8e600ea973a3c672a) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix file changes not triggering task re-execution with memresPaul Eggleton2017-06-121-1/+13
| | | | | | | | | | | | | | | | | | | | | | | | If you build a recipe, modify one of the files referred to in SRC_URI, then build it again, it should re-run do_fetch and the tasks that depend upon it to incorporate the modified file. With memory resident mode this was not working unless you restarted the server, because the mtime of the file was cached and thus we never checked the actual file. Clear out the mtime caches in buildTargets() to ensure this isn't an issue. I did a minor refactoring to break this out to its own function since we were already clearing out the build mtime cache here. (The test I used for this was . ./oe-init-build-env-memres, bitbake -c fetch mdadm, modify the comments at the top of one of the patches referred to in the recipe, and then bitbake -c fetch mdadm again and it should re-execute the fetch task ("... 0 didn't need to be re-run".) Fixes [YOCTO #10732]. (Bitbake rev: fe4f3487368bc35043a1ba464e1c69d2c9cbefc6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix undefined dep in generatePkgDepTreeDataAndy Voltz2017-06-051-2/+2
| | | | | | | | | | The loop for populating the depends of the tree data should use dep instead of item. (Bitbake rev: 37c298b27e3f72b21513124237cfeffb22c0e2f0) Signed-off-by: Andy Voltz <andy.voltz@timesys.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: use correct multiconfig prefix in dependency graphPatrick Ohly2017-03-231-1/+1
| | | | | | | | | | | | The dependency graph in the bb.event.DepTreeGenerated and the corresponding pn-buildlist and task-depends.dot from "bitbake -g" contained entries like multiconfig:qemuarm.gcc (dot as last separator) instead of the correct multiconfig:qemuarm:gcc (colon as separator). (Bitbake rev: cccd1578d84f041cd0c2dcddb91f317c69af70de) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/cooker: Sort output data shown by -eRichard Purdie2017-03-221-1/+1
| | | | | | | | | | Displaying the environment data in a sorted manner makes it easier to compare data between different setups and means you can know where to find specific entries. (Bitbake rev: baa417e44b009149eb1dfb07d5a488740b6e68ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: detect malformed BBMASK expressions which begin with a ↵Andre McCurdy2017-02-191-0/+5
| | | | | | | | | | | | separator When constructing an older style single regex, it's possible for BBMASK to end up beginning with '|', which matches and masks _everything_. (Bitbake rev: 56ad67017e601c7e0f6085ca84e29c28d8d4519f) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Rewrite dependency dot file generationRichard Purdie2017-02-191-53/+45
| | | | | | | | | | | | | | | | | | | | | | | | The package-depends.dot and pn-depends.dot files are inaccurate, missing out key dependencies such those made via the [depends] flags. As such they can be misleading to the user. They mainly exist for historical reasons, coming from a time before we had task based execution. This commit removes the two dated file formats and replaces them with a recipe-depends.dot which is a flattened version of task-depends.dot. The old format files are removed if present so that the user can't get confused about why data might not match between files. The code is also rewritten to use 'with f: f.write()' syntax as is more commonly used now. Also update the docs to match the change. (Bitbake rev: d3e182bc18ff2894f1efc8aad3d508dd432c996e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: run sanity checks for multiconfigJuro Bystricky2017-02-151-1/+2
| | | | | | | | | | | | Also run sanity check for all additional multiconfig configurations as listed in BBMULTICONFIG, not just the "default" configuration. [YOCTO#10810] (Bitbake rev: b9777374cbe63cdcf80767033f628c26d23e90f9) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: Drop now unneeded update_data callsRichard Purdie2017-02-151-3/+0
| | | | | | | | | | Now that the datastore works dynamically we don't need the update_data calls so we can just remove them. They're not actually done anything at all for a while. (Bitbake rev: 2300beb50333bb620013b058a7309e7f2042101d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add BB_LIMITEDDEPS supportRichard Purdie2017-01-201-0/+1
| | | | | | | | | | | When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited. We really need a way to know this from the metadata and this new variable provides this in worker context. This means existing code can stop having to guess. (Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Only add universe targets where the task existsRichard Purdie2017-01-201-0/+9
| | | | | | | | | | | | | Currently "bitbake universe -c fetch" will error if every task in the system doesn't have a fetch task. This seems unreasonable so filter the list based upon whether the task exists. This implementation does assume if a task exits for a given provider name, it can run, even if the preferred provider that ends up getting built might be the namespace without the task. This is unlikely to be a real world issue at this point. (Bitbake rev: 438986f4e39974930ee3205ca4c4d620090e57bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: new multiconfig '*' syntax supportJuro Bystricky2017-01-191-1/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently you cannot build a target for all the configured multiconfigs without specifying a list. The list can be quite long, requiring to type several lines of text. This enhancement is to support globbing so that you can do this, e.g. instead of: $ bitbake multiconfig:A:bash multiconfig:B:bash bash you can do: $ bitbake multiconfig:*:bash There are real world use cases where it is desirable to use multiconfig with two different tasks. For example: SDKs with multiple toolchains but also containing set of additional host tools, or multiconfig builds requiring one image for the main CPU(s) and a different co-image for a companion CPU. For this reason, two variations of the new syntax are supported. For example, the following: $ bitbake multiconfig:*:meta-toolhchain would expand to: $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain meta-toolchain However the following: $ bitbake multiconfig:*:meta-toolhchain hosttools would expand to: $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain hosttools In other words, if the user specified the "default" task explicitly, it replaces the implicit "default" task. [YOCTO#10680] (Bitbake rev: 1559ad1fc3ae8b1e4e75b210f598b90be192571b) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/cooker: BBCooker stops notifier at shutdownAníbal Limón2017-01-061-0/+2
| | | | | | | | | | | | | At end of BBCooker needs to release fd's associated with pyinotify watchers to avoid: Too many open files (EMFILE) error in different scenarios like several instances of tinfoil. [YOCTO #10873] (Bitbake rev: ae6045b84978940c365c95c33d6996359c3e299d) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Drop expanded_dataRichard Purdie2016-12-161-14/+12
| | | | | | | | | | | | | | | Some of our metadata assumes that BuildStarted and BuildCompleted events see the same data store. This is the case for buildTarget but not for buildFile and recent changes mean this is now a problem. The update_data() call is now an empty operation and there is no difference between the expanded_data and data so we can simply remove the expanded_data and its references and use data everywhere. This has been inteded for a while but the above issue makes this more pressing to finally clean up. (Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: enable setVariable command to affect task executionPaul Eggleton2016-12-141-0/+1
| | | | | | | | | | | | Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. (Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: allow buildFile warning to be hidden programmaticallyPaul Eggleton2016-12-141-4/+5
| | | | | | | | | | If we want to use this function/command internally, we don't want this warning shown. (Bitbake rev: 5cfbb60833e7b12d698c1c2970c17ccf2a4971bf) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: rewrite as a wrapper around the UIPaul Eggleton2016-12-141-3/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rewrite tinfoil as a wrapper around the UI, instead of the earlier approach of starting up just enough of cooker to do what we want. This has several advantages: * It now works when bitbake is memory-resident instead of failing with "ERROR: Only one copy of bitbake should be run against a build directory". * We can now connect an actual UI, thus you get things like the recipe parsing / cache loading progress bar and parse error handling for free * We can now handle events generated by the server if we wish to do so * We can potentially extend this to do more stuff, e.g. actually running build operations - this needs to be made more practical before we can use it though (since you effectively have to become the UI yourself for this at the moment.) The downside is that tinfoil no longer has direct access to cooker, the global datastore, or the cache. To mitigate this I have extended data_smart to provide remote access capability for the datastore, and created "fake" cooker and cooker.recipecache / cooker.collection adapter objects in order to avoid breaking too many tinfoil-using scripts that might be out there (we've never officially documented tinfoil or BitBake's internal code, but we can still make accommodations where practical). I've at least gone far enough to support all of the utilities that use tinfoil in OE-Core with some changes, but I know there are scripts such as Chris Larson's "bb" out there that do make other calls into BitBake code that I'm not currently providing access to through the adapters. Part of the fix for [YOCTO #5470]. (Bitbake rev: 3bbf8d611c859f74d563778115677a04f5c4ab43) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: provide a means to shut down from the client in memres modePaul Eggleton2016-12-141-0/+7
| | | | | | | | | | In memory resident mode we don't really want to actually shut down since it's only the client going away. (Bitbake rev: 74db369c46043116359101cab70486afd82372c0) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle inofity queue overflows more gracefullyRichard Purdie2016-12-081-0/+8
| | | | | | | | | | | If many files change and the inotify queue overflows, rather than print a traceback, invalidate the caches and warn the user. [YOCTO #10676] (Bitbake rev: 058f8517c041b80e8b591ad7d34a68281b2d03fc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix world taskgraph generation issueRichard Purdie2016-12-081-4/+10
| | | | | | | | | | | | | | The processing of the "do_" prefix to tasks is currently inconsistent and has resulted in "bitbake world -g" being broken as task prefixes don't get handled correctly. Make the "do_" task prefix handling consistent through various codepaths. [YOCTO #10651] (Bitbake rev: 3d7186353e804c9410096c408bc337a98c8b33fe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: convert type which needs to be marshalledSujith H2016-12-071-1/+1
| | | | | | | | | | | | | | We assume that the value taken by variable v can be string, integer or any type which can be marshalled by xmlrpc. This change would help us to convert the non marshallable types to string. So that we don't get exception from xmlrpc. [YOCTO #10740] (Bitbake rev: efb0e47479b3526bc047112f7200087c5844bba4) Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar callsJoshua Lock2016-11-301-21/+21
| | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/cooker.py: fix for BBFILE_PATTERN = ""Robert Yang2016-11-041-0/+4
| | | | | | | | | | | | | | | | | | | There would be error when BBFILE_PATTERN = None: BBFILE_PATTERN_foo not defined This is the correct behaviour, but when the layer sets BBFILE_PATTERN = "", it would match all the remaining recipes, and cause "No bb files matched BBFILE_PATTERN" warnings for all the layers which behind it. When a layer sets BBFILE_PATTERN = "" (for example, a layer only provides git repos and source tarballs), now it means has no recipes. This is different from BBFILE_PATTERN_IGNORE_EMPTY, the later one means that it *may* not have any recipes. (Bitbake rev: 91c3b34625fac2a0f093a4b46a46e89f813e7972) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task id string manipulationsRichard Purdie2016-10-091-6/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Some task id manipulations were suboptimal: * taskfn_fromtid and fn_from_tid were effectively the same function * many calls to split_tid(), then taskfn_fromtid() * taskfn_fromtid() called split_tid() internally This patch adds split_tid_mcfn() to replace split_tid() and returns the "taskfn" variant being used in many places. We update all core calls to the new function and ignore the return values we don't need since the function call overhead of the split_tid wrapper is higher than ignoring a return value. The one remaining standalone use of taskfn_fromtid is replaced with fn_from_tid. I couldn't see any external usage so it was dropped. There is external usage of split_tid so a wrapper remains for it. Combined together these changes should improve some of the runqueue task manipulation performance. (Bitbake rev: 1bf2ef874fbe47f1320007efa0bdeef8d630b8a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/providers: Only add target to world build if task existsRichard Purdie2016-09-221-3/+3
| | | | | | | | | | | A "bitbake world -c unpack" currently breaks as not all tasks have an unpack task. This change allows addition of world targets only if the specified task exists which makes certain commands possible when otherwise you just get errors which can't easily be avoided. (Bitbake rev: ca4f5e6d01b5c8cf315f59bc86194d63c0d3d042) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: add LAYERRECOMMENDS processingJoe Slater2016-09-211-3/+34
| | | | | | | | | | | | | | | | | | | Add recommended layers to collection_depends[] so that dynamic priority assignment will work for both depends and recommends. Recommended layers do not cause an error or warning if they are not in the collection list, but debug messages are output for level 3 and above. explode_dep_versions2 returns a dictionary, so we change the variable deplist to depDict. The dictionary values are lists which are either empty or contain only one version specification. (Bitbake rev: 20cdc3d609f8aea992f97c3db336574d3a549973) Signed-off-by: Joe Slater <jslater@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: check if target contains taskEd Bartosh2016-09-161-1/+2
| | | | | | | | | | | | | | Task name was incorrectly added to the targets that already contained :task suffix and fired with BuildInit event. This caused Toaster to create incorrect Target objects and show them in UI. [YOCTO #10221] (Bitbake rev: b7faf1af3bd3110fba347fbe6e432fc4ee66590a) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: record events on cooker exitEd Bartosh2016-09-081-1/+1
| | | | | | | | | | | | | | | | | | | Bitbake collects all events in special event queue when called with -w option. However, it starts to write events to the eventlog only after BuildStarted event is received. In some cases this event is not received at all, e.g. when bitbake is run with --parse-only command line option. It makes sense to write all collected events when CookerExit event received to make sure all events are written into the eventlog even if BuildStarted event is not fired. [YOCTO #10145] (Bitbake rev: 57912de63fa83550c0ae658eb99b76e9cc91a8d1) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: Catch when stdout doesn't have a file descriptorMariano Lopez2016-09-021-9/+12
| | | | | | | | | | | | | | | | | Currently, there is a check to remove the TOSTOP attribute from a tty to avoid hangs. It assumes that sys.stdout will have a file descriptor and this is not always true, some IO classes will throw exceptions when trying to get its file descriptor. This will add a check for such cases and avoid throwing an exception. [YOCTO #10162] (Bitbake rev: cb4f8f6efa28ef2b13bc738a0118b876baa15b3e) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Initial multi-config supportRichard Purdie2016-08-181-125/+169
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds the notion of supporting multiple configurations within a single build. To enable it, set a line in local.conf like: BBMULTICONFIG = "configA configB configC" This would tell bitbake that before it parses the base configuration, it should load conf/configA.conf and so on for each different configuration. These would contain lines like: MACHINE = "A" or other variables which can be set which can be built in the same build directory (or change TMPDIR not to conflict). One downside I've already discovered is that if we want to inherit this file right at the start of parsing, the only place you can put the configurations is in "cwd", since BBPATH isn't constructed until the layers are parsed and therefore using it as a preconf file isn't possible unless its located there. Execution of these targets takes the form "bitbake multiconfig:configA:core-image-minimal core-image-sato" so similar to our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND. Implementation wise, the implication is that instead of tasks being uniquely referenced with "recipename/fn:task" it now needs to be "configuration:recipename:task". We already started using "virtual" filenames for recipes when we implemented BBCLASSEXTEND and this patch adds a new prefix to these, "multiconfig:<configname>:" and hence avoid changes to a large part of the codebase thanks to this. databuilder has an internal array of data stores and uses the right one depending on the supplied virtual filename. That trick allows us to use the existing parsing code including the multithreading mostly unchanged as well as most of the cache code. For recipecache, we end up with a dict of these accessed by multiconfig (mc). taskdata and runqueue can only cope with one recipecache so for taskdata, we pass in each recipecache and have it compute the result and end up with an array of taskdatas. We can only have one runqueue so there extensive changes there. This initial implementation has some drawbacks: a) There are no inter-multi-configuration dependencies as yet b) There are no sstate optimisations. This means if the build uses the same object twice in say two different TMPDIRs, it will either load from an existing sstate cache at the start or build it twice. We can then in due course look at ways in which it would only build it once and then reuse it. This will likely need significant changes to the way sstate currently works to make that possible. (Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Build datastores from databuilder objectRichard Purdie2016-08-181-12/+11
| | | | | | | | | | | | | | Rather than passing in a datastore to build on top of, use the data builder object in the cache and base the parsed recipe from this. This turns things into proper objects building from one another rather than messy mixes of static and class functions. This sets things up so we can support parsing and building multiple configurations. (Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-27 16:07:15 +0000