summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cooker.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: cooker/siggen: Empty siggen cache during parsingRichard Purdie2020-02-181-1/+2
| | | | | | | | | | | | | | | | | | | When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. (Bitbake rev: ef29309d0b512b64d024e383e7baff22c727711c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/toaster: replaced deprecated method warn() with warning()Frazer Clews2020-02-111-1/+1
| | | | | | | | | | | | | Removed the deprecated methods as it will only cause problems later on, and since warn() just calls warning(), it shouldn't change anything (Bitbake rev: c131015f1ac152f1fea4b83a3d451c3e4d05ebec) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a194f275235f22411cb2368f06a44f61ceb6a0f3) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Keep track of watched files using a set instead of a listPeter Kjellerstedt2020-01-281-8/+8
| | | | | | | | | | | | | When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. (Bitbake rev: 16799ada1cda8e021cb04c43e469225790525723) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e96df260e47d160dbd36bfc92c31ef06266f662) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove a left-over comment about expanded_dataPeter Kjellerstedt2019-11-251-4/+0
| | | | | | | | | | | | This should have been removed together with expanded_data in commit e3694e73 (cooker/command: Drop expanded_data). (Bitbake rev: 987996f01d55bc6433aeb7f43c209eb12f6d796b) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Don't daemonize server processJoshua Watt2019-09-271-1/+0
| | | | | | | | | | | | | | | The hash server process is terminated and waited on with join(), so it should not be a daemon. Daemonizing it cause races with the server cleanup, especially in the selftest because the process may not have terminated and cleanup up its socket before the test cleanup runs and tries to do it. [YOCTO #13542] (Bitbake rev: 7c829675581818f92d57056b57fbd3880829b6bd) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rework hash equivalenceJoshua Watt2019-09-181-8/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "bitbake: cooker: Ensure bbappends are found in stable order"Martin Jansa2019-09-161-1/+0
| | | | | | | | | | | | | | This reverts commit 94c0c7f15c7a6244a8576ed948ffc21afb96ba82. This ignores the layer priority, making the issue much worse. E.g. I'm seeing a lot of failures caused by missing users, because base-passwd bbappends applied in unexpected order caused different passwd.master to be found in re-ordered FILESPATH. (Bitbake rev: 2dc862237dba82da37c8ac9289e0a21409b1305c) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Ensure bbappends are found in stable orderWes Lindauer2019-09-071-0/+1
| | | | | | | | | | | | | | | Thanks to wildcards in bbappend filenames, it's possible to have multiple bbappends that apply to the same recipe in the same directory. In order to get sstate hits between different workspaces, we want to apply those bbappend files in a consistent order. Since readdir() returns files in a non-deterministic order between workspaces (based on inode number and/or time of creation), we'll need to sort its result in order to have any consistency. (Bitbake rev: 94c0c7f15c7a6244a8576ed948ffc21afb96ba82) Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: remove generation of recipe-depends.dotChen Qi2019-08-281-21/+4
| | | | | | | | | | | | | | | | | | | | | | The information of recipe-depends.dot is misleading. e.g. $ grep xz recipe-depends.dot | grep bzip2 "bzip2" -> "xz" "xz" -> "bzip2" Users would wonder why they get some circular dependency. The information is derived from removing the task names of task-depends.dot. It's not giving people any additonal information, and it's misleading. So we remove the generation of this file. (Bitbake rev: 4c484cc01e3eee7ab2ab0359fd680b4dbd31dc30) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve hash server startup code to avoid exit tracebacksRichard Purdie2019-08-141-12/+17
| | | | | | | | | | | | | At exit the hashserv code was causing tracebacks as join() wasn't being called from the thread that started the process. Ensure that the hashserver is started from the pre_serve hook which is the final thread the cooker runs in. This avoids the traceback at the expense of some horrific poking into data stores which will ultimately need improving through a proper API. (Bitbake rev: 05888700e5f6cba48a26c8a4c447634a28e3baa6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Cleanup the queue before call process.join()Robert Yang2019-08-081-0/+8
| | | | | | | | | | | | | | | | | | | | | | Fixed: $ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p Press *one* Ctrl-C when the parsing process is at about 50%, then the processes are not exited: Keyboard Interrupt, closing down... Timeout while waiting for a reply from the bitbake server It hangs at process.join(), according to: https://docs.python.org/3.7/library/multiprocessing.html Cleanup the queue before call process.join() can fix the problem. (Bitbake rev: 3eddfadd19b2ce4c061861abf0c340e3825b41ff) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Switch from threads to multiprocessingRichard Purdie2019-08-061-9/+11
| | | | | | | | | | There were hard to debug lockups when trying to use threading to start hashserv as a thread. Switch to multiprocessing which doesn't show the same locking problems. (Bitbake rev: be23d887c8e244f1ef961298fbc9214d0fd0968a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/hashserv: Allow autostarting of a local hash server using ↵Richard Purdie2019-08-061-1/+14
| | | | | | | | | | | | | | | | | BB_HASHSERVE Its useful, particularly in the local developer model of usage, for bitbake to start and stop a hash equivalence server on local port, rather than relying on one being started by the user before the build. The new BB_HASHSERVE variable supports this. The database handling is moved internally into the hashserv code so that different threads/processes can be used for the server without errors. (Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: lib: Cleanup /usr/bin/env pythonRobert Yang2019-06-281-1/+0
| | | | | | | (Bitbake rev: cc712f3257904960247a7532cfc4611f3dccd36c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure mcdeps are processed even if only one multiconfigRichard Purdie2019-06-111-1/+5
| | | | | | | | | If you have no BBMULTICONFIG set but set mcdepends, they're currently ignored. We can handle them correctly with this small tweak. (Bitbake rev: 578f0c02f6a13f4315e7c2ce8b5e876dd2025055) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add compability handling for multiconfig: prefix migrationRichard Purdie2019-06-101-0/+3
| | | | | | | | | | This allows "multiconfig:" targets to continue to work by internally mapping them to the new "mc:" naming, allowing older builds to work as before. (Bitbake rev: c4d90890547af642e99cc541af3415df3559563e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: multiconfig: Switch from 'multiconfig' -> 'mc'Richard Purdie2019-06-101-9/+9
| | | | | | | | | | | | | | After real world use its clear the "multiconfig:" prefix to multiconfig tasks, whilst clear, is also clumbersome. Switch to use the short version instead. mcdepends will continue to work with "multiconfig:" for now as well. The commandline will only accept mc: going forward. [YOCTO #11168] (Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Show base multiconfig environmentJoshua Watt2019-05-301-10/+21
| | | | | | | | | | | | | | | | Adds support to the 'bitbake -e' command so that it can display the base environment for a multiconfig. It was previously possible to get the base environment for the main environment by running "bitbake -e", but there was no support for getting the base environment for a multiconfig without specifying a recipe. A user can now print the base environment for the multiconfig "foo" by running: $ bitbake -e multiconfig:foo (Bitbake rev: 3d657af8a6120193d45d01968605b30075a56198) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Rename __depends in all multiconfigsJoshua Watt2019-05-301-2/+3
| | | | | | | | | | | | | | | | | | | The renaming of the __depends variable to __base_depends and file watches needs to occurs for all multiconfigs, not just the base config. Failing to do this for all multiconfigs will result in a huge increase in the size of the parsing cache (about 5x for a single mulitconfig) because all multiconfig caches will still depend on the base config files. This will also seen a similar jump in the amount of time required to load the parsing cache from memory, both because the cache is larger and because of explosion of additional existence checks that must be done for the base files. [YOCTO #13359] (Bitbake rev: 34137a00f60280e9e806070c6507a0fc6284b834) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-041-2/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-041-13/+0
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+2
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix bbfile_config_priorities when BBFILE_PATTERN is emptyRobert Yang2019-03-241-1/+1
| | | | | | | | | | | | | | | | | | The layer was not in bbfile_config_priorities when BBFILE_PATTERN is empty, this caused "bitbake-layers show-layers" can't show these layers, this was incorrect since these layer did exist. Add these layer to bbfile_config_priorities can fix the problem. Fixed: Add BBFILE_PATTERN_core = "" in oe-core/meta/conf/layer.conf $ bitbake show-layers | grep oe-core There was nothing, now the layer is shown (Bitbake rev: 0ff5cdb0cca9266ca29127639494bcfd95e36831) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Tweak multiconfig dependency resolutionRichard Purdie2019-02-251-29/+24
| | | | | | | | | | | | | | | | | | | There were a couple of problems with the multiconfig dependency resolution: - the "if mc" condition triggering this code wasn't correct, it needs to be "if more than one multiconfig" configured - after adding providers we need to call add_unresolved again and rebuild mcdeps within the "while new" loop By fixing these issues we allow various other combinations of multiconfig builds to work which previously didn't. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: 4359b037de578095db2595f119dfb8e3340e1414) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Split recipes to parse amongst threads ahead of timeRichard Purdie2019-01-081-11/+6
| | | | | | | | | | | | | | | | We have two choices, split the recipes amongst the parsing threads in blocks ahead of time, or have a queue which parsers pull from when idle. The optimum approach depends on how similar the pieces are. For the single recipe reparse case, there is currently a significant wait for the feeder thread to start (around 0.25s in a 2s command). Its possible splitting into blocks in advance may be unluckly for some other workloads but experimentally it seems to work better overall for me at least. (Bitbake rev: ae79868861568d673a70472e85a4bde9e2d84a8f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove Feeder() since its no longer needed with moderm ↵Richard Purdie2019-01-081-37/+5
| | | | | | | | | | | | | multiprocessing There used to be many bugs in multiprocessing and we implemented our own feeder process to avoid them. Now that we have python 3.x, these are fixed and just using the standard Queue mechanism appears to work fine. We can therefore drop the unneeded code and simplify. (Bitbake rev: b2d39fc37fcf3c81a562ec1ef4f8b4c1a493fc57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix indirect multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2019-01-031-20/+31
| | | | | | | | | | | | | | | | | | When an indirect multiconfig dependency exists, such as: A depends on B, B has a multiconfig dependency to C,and our build target is A, the multiconfig dependency to C is not processed on time, hence no providers are added for it, causing an exception in the runqueue because the dependency does exist in it. Call add_unresolved() for all available multiconfigs before processing providers for multiconfig dependencies, detecting mcdepends on time so providers for them can be added correctly. (Bitbake rev: 8a6bc7584ad61b4de98af92a86066602006262f9) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add some timing debug messages to the server startupRichard Purdie2018-12-081-0/+19
| | | | | | | | | We're seeing slow startup in bitbake, add some timeing debug messages so the logs are more useful for debugging when its slow. (Bitbake rev: 8d1fc115b8a176009f1f3a8ce840b422e7e0b45e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2018-11-221-8/+1
| | | | | | | | | | | | | | | | | | | | | | | | | When multiconfig is enabled the cooker adds providers for all the targets to be built on all the multiconfig variables that were set, regardless if there is a dependency to it or not. This causes an issue when a certain target is incompatible with one or more of the multiconfigs, e.g. the target is not in COMPATIBLE_MACHINE for one of the MACHINEs being built, causing the cooker to error out since no providers can be found for that certain target on that multiconfig. This patch modifies the behavior to only look for PROVIDERS for a target on the multiconfig that was selected to be built, PROVIDERS are then looked for in other multiconfigs only when there is a defined dependency to them. [YOCTO #12985] (Bitbake rev: f2106a3a767542359fdde238abcf5fe35ab3a144) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/runqueue: Turn universe warnings into verbnotesRichard Purdie2018-09-101-1/+1
| | | | | | | | | If the user puts universe on the commandline, they don't really want warnings so use the new verbnote level instead. (Bitbake rev: 0c87ade5678e503899e3a6cdda5329f6fc212b63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: Fix incorrect bb files matched warningMark Hatle2018-09-051-1/+4
| | | | | | | | | | | | | | | | | | | | In the case of a sublayer of an existing layer, where the sublayer and main layer share a path, the system may not match the paths properly resulting in: No bb files matched BBFILE_PATTERN_sublayer '^/path/main/sublayer' because it has already matched the main layer. Fix this issue by sorting the collection items based on the pattern, using longest to shortest. Obviously regex wildcards could still be an issue but these are typically not used, so this simply fix should work in the existing cases. (Bitbake rev: 1787cef7221b88f6920ea70fadaffc117d84c7aa) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add support for multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2018-08-011-3/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds the capability for tasks from different multiconfigs to depend on one another. These dependencies can be enabled using the following format: task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on" For the sake of simplicity consider the following example: Assuming we have set up multiconfig builds, one for qemux86 and one for qemuarm, named x86 and arm respectively. Adding the following line to an image recipe (core-image-sato): do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs" Would state that core-image-sato:do_image from x86 will depend on core-image-minimal:do_rootfs from arm so it can be executed. This patch makes modifications to: - cooker: To glue both multiconfigs in one place and make sure the dependencies can be provided. - taskdata: To parse and add a new kind of dependency (mcdepends) to the taskdata object. - runqueue: To differentiate tasks from different multiconfigs, add the specified dependencies to the corresponding tasks, and create a working runqueue that contains tasks from both multiconfigs. - siggen: To avoid looking for tasks from different multiconfigs on objects where they dont belong. The taskdata objects are still not aware of the concept of multiconfig, so each object doesnt know which multiconfig its building, hence why the mcdepends are added to all taskdata objects equally (we really dont expect many of these), but the actual dependencies are added only to the required tasks by the runqueue. (Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix environment double key expansion issueRichard Purdie2018-03-041-3/+2
| | | | | | | | | | | The base configuration needs key expansion and anon python execution, the parsed configurations do not. Fix this consistently, its been broken and causing double key expansion for a while, only relised when we started double anonymous python exeution too. (Bitbake rev: 6138897de5ac6becf3bff56ce7a78f3ec208fcdf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Run registered anonymous python before displaying environmentRichard Purdie2018-03-041-1/+3
| | | | | | | | | | | The output of bitbake -e can differ from what actually is used due to anonymous python making changes to the data store. Execute any anonymous python added in the base configuration to make things more consistent. (Bitbake rev: bcdc2f73e3b4a10b1e479c2891f251d9507a9e30) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: don't stop file notifier when cooker is shutdownRobert Yang2018-02-141-2/+0
| | | | | | | | | | | | | | | | | | | | | It should be live/exited with server rather than cooker, fixed: $ bitbake --server-only -T -1 Set MACHINE = "qemux86" in conf/local.conf $ bitbake quilt Set MACHINE = "qemuppc" in conf/local.conf $ bitbake quilt [snip] ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed. [snip] This is because the server doesn't know local.conf is changed since the notifiers are stopped, so it doesn't reparse, and then we would get the errors, let the notifiers live/exited with server can fix the problem. (Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: fix for BBFILE_PATTERN matches bbappendRobert Yang2018-02-141-3/+7
| | | | | | | | | | | | | | | | | | | | | The old code couldn't handle nestled layers correctly, e.g.: parent_layer/sub_layer/foo.bb Note there are two layers, parent_layer and sub_layer. And in parent_layer/conf/layer.conf: BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/" This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb, so it warns that no files matched sub_layer, this is the expected behavior, but it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it should warn and let the user fix the problem. Check the bbappend in already "matched set" before return it as matched by "unmatched set" can fix the problem. (Bitbake rev: ec90245d28e52ea718d2ce084eb304cdc4355c9c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: skip when BBFILE_PATTERN is emptyRobert Yang2018-02-141-0/+1
| | | | | | | | | There is nothing to do when BBFILE_PATTERN is empty. (Bitbake rev: b7da31641c24c53d47ea45a7119d1bd353011b39) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Sort task graph (dot) file outputRichard Purdie2018-01-291-6/+6
| | | | | | | | | This means two different graph files can easily be compared, currently you'd have to sort them as the output is randomized. (Bitbake rev: 5f7c6ec785f70beb1a4a1bbc0eb83cfa6cd7740d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix typo in bitbake -g messagePaul Eggleton2017-11-211-1/+1
| | | | | | | | | Flatened -> flattened. (Bitbake rev: 80f72ac6c56ba8f3e2f7b5f0cb95bee6c0101323) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix watching directories with Python 3.6+Paul Eggleton2017-10-311-1/+11
| | | | | | | | | | | | | | | | | | | | | | | | In Python 3.6, glob.glob() was reimplemented to use os.scandir() (which itself appeared in Python 3.5), thus our monkey patching of os.listdir() here was no longer effective. The end result was not only that bitbake wouldn't notice added recipes or bbappends with BB_SERVER_TIMEOUT set when being run with Python 3.6 (the shipped Python version on Fedora 26 and some other distribution versions), it also broke devtool modify, devtool upgrade and devtool extract since they rely on the ability to create a bbappend on the fly and have bitbake pick it up. To fix it, do the same monkey patching for os.scandir(), which needs to be conditional upon that actually existing since we have to support Python 3.4 that doesn't have it. Long term we should probably look for a better way to handle this that doesn't involve monkey patching Python library code. Fixes [YOCTO #12185]. (Bitbake rev: d02e90db32e7ee341c2ba3be79b0627d8796bdd6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: ensure monkey-patching in collect_bbfiles() gets undone on ↵Paul Eggleton2017-09-171-19/+19
| | | | | | | | | | | | | | | | error In collect_bbfiles() we're monkey-patching os.listdir in order to find which directories to watch, and then undoing that when we're finished - however if an exception occurred for any reason there was nothing to ensure the latter occurred. This may not have caused any issues, but as this kind of thing really ought to be secured using try...finally just in case, so do that. (Bitbake rev: 013047484a03185c0ce281c53c1db4949cdc4e69) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix watching empty directoriesPaul Eggleton2017-09-171-3/+6
| | | | | | | | | | | | | | | | | | | The code that was supposed to watch directories along BBFILES for creation of new files wasn't working in the case where the directory did not initially contain any matching files - since in updateCache() we are passing the directory path to add_filewatch() and the latter function calls os.path.dirname() on the path on the assumption that it is a file path, and thus the parent of the directory got watched but not the directory itself. (If the directory wasn't empty everything worked fine since add_filewatch() was called elsewhere with the path to one of the files in that directory, and thus the directory got watched). Add a parameter to add_filewatch() to tell it we are passing it directory path(s) rather than file path(s). (Bitbake rev: 47a34dee08fcc25d896a1bdf16fa86267f0b898f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process/cooker: Improve readypipe handlingRichard Purdie2017-09-121-5/+1
| | | | | | | | | | | | | Issues in start are not being correctly detected by the current readypipe code. Change it to use specific "ready" or "fail" messages to correctly determine the correct failure mode and avoid bitbake seeming to hang (it does currently timeout eventually). [YOCTO #12062] (Bitbake rev: 60d4791e3dd05729d2a2adf6f3b203c80d466a73) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: clear extra config settings and remote datastores on client ↵Paul Eggleton2017-09-051-0/+2
| | | | | | | | | | | | | | | | | | disconnect When the UI disconnects, we can throw away any server-side remote datastores we created in response to calls from the UI, and we *must* drop everything in extraconfigdata or it will taint any future operations. Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors when running devtool.DevtoolTests.test_devtool_update_recipe_local_files within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded. (Bitbake rev: 1ca2eec459424892391f060442ef38cf28d6a54a) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: Fix layer priority processingMark Hatle2017-09-051-7/+7
| | | | | | | | | | | | | | | | | If you have a layer with a blank BBFILE_PATTERN the layer was ignored when processing the list of layers with priorities. This list is not only used for processing recipes, but also by additional programs such as bitbake-layers show-layers. Without this change, a layer that provides configuration or classes only does now show up in show-layers, which is used by the yocto-compat-layer.py script. This causes a failures in the compatibility check. (Bitbake rev: a0eaf8c0f228f984bafff09e4e9739f758dc1a9b) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parseConfiguration clears parsecache_validRichard Purdie2017-09-021-0/+2
| | | | | | | | | | | | BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers was failing and highlighted that since parseConfiguation clears data structures, it needs to also clear parsecache_valid as it no longer contains correct data. (Bitbake rev: 7234f33a7eb38ad51a8345f6689bc26e29f29f92) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure buildFile doesn't have lasting side effectsRichard Purdie2017-08-311-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_build_image fails, the reason is that internally, the limited_deps flag is set in the bitbake server and then never cleared. This causes the sysroots to be setup incorrectly (as per the limited dependency case) and builds break. There is also potential for corruption of recipecaches. Add shutdown/cleanup code to ensure these effects don't 'stick'. This bug is particularly nasty as you can destroy TMPDIR with large sysroots in build work directories which are prone to break. Also ensure mtime cache is cleared (to match buildTargets) and that no lasting changes are made to siggen either which ensures: BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_upgrade_git devtool.DevtoolTests.test_devtool_virtual_kernel_modify works. (Bitbake rev: 0a7ee8c8378bba9877c260b1aee782878f1935b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: ensure we can run buildFileInternal() after cache is populatedPaul Eggleton2017-08-311-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | If you run some other operations that result in the cache being populated, and then call buildFileInternal(), then you can end up in a situation where the cache already contains information about the recipe. For example in OE this can now happen when you use devtool upgrade. Normally this doesn't cause any problems, unless you have a non-absolute path in BBLAYERS - in buildFileInternal() we are calling matchfile() which will convert the filename to absolute, but later when taskdata goes to find the providers of the recipe it finds the non-absolute path, sets up the task information using this and then the runqueue can't find any tasks matching the absolute path. To fix this, back out the optimisation I did earlier in bitbake rev ba53e067a2d448dd63b4ca252557ce98aa8e6321 to avoid calling parseConfiguration() again, which is unfortunate but does result in the cached information being that causes the problem being cleared out. This fixes "Task do_unpack does not exist for target ..." running devtool upgrade within intel-iot-refkit. (Bitbake rev: f120355eaec4571ba6d60fc5f7ae9e1f31d846d1) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Allow changes in PRSERV_HOST to be responded toRichard Purdie2017-08-311-8/+11
| | | | | | | | | | | | | | When we reload the configuration, PRSERV_HOST can change. Therefore restart the PR Server depending on the new configuration at reparse. Note that the server has to be started from the right process, it can't be in the UI which shuts down as that shutdown triggers a shutdown of its children and the PR Server shuts down too. This is why we need pre_serve() which ensures its executed in the right context. (Bitbake rev: 971272e84f4efe7ebd0037e164ba54f013a2a34e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Change to consistent prefile/postfile handlingRichard Purdie2017-08-311-13/+10
| | | | | | | | | | | | | | Currently the original prefile and postfile passed when starting bitbake server are 'sticky'. With the new memory resident model this doesn't make sense as the server the system is started with isn't special. This patch changes the code so the prefile/postfile are used if specified on the commandline and not used otherwise. This makes the behaviour much more predictable and expected and as an added bonus simplifies the code. (Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-13 02:20:18 +0000