summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Ensure we clear the stamp cacheRichard Purdie2019-08-141-0/+3
| | | | | | | | | | When the task hashes change we need to ensure the stampcache is cleared out else tasks don't rerun when they should as we're basing decisions on stale cache data. (Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add missing setscene task corner caseRichard Purdie2019-08-141-0/+2
| | | | | | | | | We weren't marking this special case of setscene task as buildable leading to runqueue task failures. (Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add further debug informationRichard Purdie2019-08-141-0/+11
| | | | | | | | | Further testing shows we should test some extra datastructures to help pinpoint logic errors more precisely. This adds some further data structure sanity checks. (Bitbake rev: 83c4370b25c3a14cc946965c5c5f83ea28f488a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop debug statement causing performance issuesRichard Purdie2019-08-141-2/+0
| | | | | | | | | | | | This debug statement could result in a long list of tasks which when repeatedly sent over our IPC, slowed down the builds immensely. Remove it in favour of other more targeted debugging added recently, bringing back some lost performance, particularly on builds with large numbers of tasks. (Bitbake rev: 85fe627fdb6510f0942917964386fad9d8c479c8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix event timing raceRichard Purdie2019-08-142-71/+74
| | | | | | | | | | | | The event from the task notifiing of hash equivalency should only be processed when the task completes. This can otherwise result in a race where a dependent task may run before the original task completes causing various failures. To make this work reliably, the code had to be restructured quite a bit. (Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Recompute holdoff tasks from scratchRichard Purdie2019-08-141-1/+1
| | | | | | | | | | | | The changed_setscene variable here is just odd and not needed. Worse, it could prevent some tasks from being removed from the holdoff tasks list. The list is being rebuilt and should work as intended just from the other data, this is a leftover from previous versions of the code as far as I can tell. (Bitbake rev: 030b9f2b3ce6ed40e79304eb0ffee6c6613f43be) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve scenequeue debuggingRichard Purdie2019-08-141-19/+42
| | | | | | | | | | | | Whilst we had good runqueue failure mode debug, it hadn't adapted to the scenequeue changes. Run the scenequeue sanity tests at the end of a build and output the results regardless of whether all setscene tasks completed or not. This *massively* improves the ability to debug runqueue problems. (Bitbake rev: b9b2177473c0b95a23bd519a201e1d2ba101c6c1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix next_buildable_task performance problemRichard Purdie2019-08-141-5/+7
| | | | | | | | | | | | | | Looking at the profile information, a lot of time is being spent in next_buildable_task. This is probably due to the generator expressions not working well with the empty test. The easiest way to improve things is to switch to using set manipulations. We also don't need to update self.buildable the way the original code did as we don't rely on that anywhere. (Bitbake rev: 3bcf9ad4964b7e42d1a02ce231e9db42a81ead2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Wait for covered tasks to complete before trying setsceneRichard Purdie2019-08-141-0/+4
| | | | | | | | | If tasks are in the covered list of tasks for a given setscene task, it needs to wait for those to complete before we can start. (Bitbake rev: fdee640c26750b852eb68f5c80437377aa300ed8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve hash server startup code to avoid exit tracebacksRichard Purdie2019-08-142-12/+18
| | | | | | | | | | | | | At exit the hashserv code was causing tracebacks as join() wasn't being called from the thread that started the process. Ensure that the hashserver is started from the pre_serve hook which is the final thread the cooker runs in. This avoids the traceback at the expense of some horrific poking into data stores which will ultimately need improving through a proper API. (Bitbake rev: 05888700e5f6cba48a26c8a4c447634a28e3baa6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Add further hash equivalence testsRichard Purdie2019-08-144-12/+166
| | | | | | | | | Add some extra hash equivalence runqueue tests based on recent scenarios that caused problems during testing. (Bitbake rev: 373b085ead992a725b2230ededd992b4c61a1a05) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve setscene task handling logicRichard Purdie2019-08-141-75/+34
| | | | | | | | | | | | | | The previous tasks_covered and tasks_notcovered were basically unstable data structures. We couldn't always tell whether tasks should be covered or not when trying to repair the sturcture if sstate tasks reran. In the end its simpler to throw the lists away and rebuild them based upon current data rather than trying to patch it adhoc. This turns out to be simpler and much more reliable and I've much more confidence in this code. (Bitbake rev: 52ee2ba2c617d928569f5afa404925c8b6f317bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix corruption issueRichard Purdie2019-08-141-1/+1
| | | | | | | | | | | We need to copy this set, not modify the original else all kinds of weird and bad things break, mostly from circular references. We'll not go into how much sleep I lost tracking down the fallout from this. (Bitbake rev: 49927546d2b306830c98f6f9da4a6ad828f6a3a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: layerindexlib: Fix parsing of recursive layer dependenciesMark Hatle2019-08-131-2/+19
| | | | | | | | | | | | | | | | | [YOCTO #13447] When running bitbake-layers layerindex-fetch from 'master', there is a circular dependency between meta-oe and meta-python. This triggered a maximum recursion depth exception. To fix the exception, as we walk down a branch (depth first search), we track the layers we've already seen. If we are about to recurse into a layer we've already seen we report a warning and then stop recursion. (Bitbake rev: d6155d095513be3f500d089c4ed4c4b89949d560) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Fix for the Second Keyboard InterruptRobert Yang2019-08-081-4/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: $ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p Press the first Ctrl-C when the parsing process is at about 50%: Keyboard Interrupt, closing down... Then presss the second Ctrl-C: File "/path/to/bitbake/bitbake/lib/bb/ui/knotty.py", line 619, in main event = eventHandler.waitEvent(0.25) File "/path/to/bitbake/lib/bb/server/process.py", line 591, in waitEvent self.eventQueueNotify.wait(delay) File "/usr/lib/python3.5/threading.py", line 549, in wait signaled = self._cond.wait(timeout) File "/usr/lib/python3.5/threading.py", line 297, in wait gotit = waiter.acquire(True, timeout) KeyboardInterrupt Capture the second KeyboardInterrupt during stateShutdown is running can fix the problem. There may be still tracebacks for the third KeyboardInterrupt, but I'm leaning to not fix it since we aimed for supporting 2 KeyboardInterrupts only. (Bitbake rev: 8c26b451f22193ef1c544e2017cc84515566c1b8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Cleanup the queue before call process.join()Robert Yang2019-08-081-0/+8
| | | | | | | | | | | | | | | | | | | | | | Fixed: $ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p Press *one* Ctrl-C when the parsing process is at about 50%, then the processes are not exited: Keyboard Interrupt, closing down... Timeout while waiting for a reply from the bitbake server It hangs at process.join(), according to: https://docs.python.org/3.7/library/multiprocessing.html Cleanup the queue before call process.join() can fix the problem. (Bitbake rev: 3eddfadd19b2ce4c061861abf0c340e3825b41ff) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump version to 1.43.1 for API changesRichard Purdie2019-08-062-2/+2
| | | | | | (Bitbake rev: f43778c2d19e70d4befd483b06aaf247fc65c799) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Add hashserv+runqueue testRichard Purdie2019-08-064-7/+59
| | | | | | | | Add a test which tests the runqueue adaptations for hash equivalency. (Bitbake rev: 477321d0780df177c1582db119c2bb6795912fc6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Drop bb.build.FuncFailedRichard Purdie2019-08-063-49/+12
| | | | | | | | | | | | | Its hard to see what this exception adds in the current codebase. The logfile attribute is effectively ignored, the exception doesn't serve a defined purpose and mostly seems to be worked around. Remove it entirely. If this does cause output problems, we'll figure out better ways to address those. (Bitbake rev: cfeffb602dd5319f071cd6bcf84139ec77f2d170) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Clean up task reference formatsRichard Purdie2019-08-065-95/+82
| | | | | | | | | | | | | | | | | | Currently siggen uses the format "<filename>.<taskname>" for referencing tasks whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the separator everywhere. This is an API breaking change since the cache is affected, as are siginfo files and any custom signature handlers such as those in OE-Core. Ultimately this will let us clean up and the accessor functions from runqueue, removing all the ".rsplit(".", 1)[0]" type code currently all over the place. Once a standard is used everwhere we can update the code over time to be more optimal. (Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Clean up BB_HASHCHECK_FUNCTION APIRichard Purdie2019-08-062-53/+23
| | | | | | | | | | | | | This function uses an old API which uses offsets into lists as a communication mechanism. Update the API to use "tid" which is used universally in runqueue now. We can also add kwargs support to the funciton definition to drop some of the backwards compaiblility hoops we had to jump though with different function argument combinations. (Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Switch from threads to multiprocessingRichard Purdie2019-08-063-15/+27
| | | | | | | | | | There were hard to debug lockups when trying to use threading to start hashserv as a thread. Switch to multiprocessing which doesn't show the same locking problems. (Bitbake rev: be23d887c8e244f1ef961298fbc9214d0fd0968a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Use separate threads for answering requests and handling themRichard Purdie2019-08-061-1/+40
| | | | | | | | | | Experience with the prserv shows that having two threads, one accepting and queueing connections and one handling the requests leads to much more reliable behaviour than having everything in a single thread. (Bitbake rev: a03d60671a53d9ff70e07cc42fe35f6f8776dac2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Use a memory journalRichard Purdie2019-08-061-1/+1
| | | | | | | | | | | | We've seen PR Server timeouts on the autobuilder, this is likely from the journal being blocked on disk IO generated by the build. Since we're running with synchronous off, we may as well put the journal into memory and avoid any IO related stalls. (Bitbake rev: ee3fc6030e653f3244b065fc89aafd2a7c36ae04) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Turn off sqlite synchronous modeRichard Purdie2019-08-061-0/+2
| | | | | | | | | | | | | | | | | | We're seeing performance problems with hashserv running on a normal build system. The cause seems to be the large amounts of file IO that builds involve blocking writes to the database. Since sqlite blocks on the sync calls, this causes a significant problem. Since if we lose power we have bigger problems, run with synchronous=off to avoid locking and put the jounral into memory to avoid any write issues there too. This took writes from 120s down to negligible in my tests, which means hashserv then responds promptly to requests. (Bitbake rev: 7ae56a4d4fcf66e1da1581c70f75e30bfdf3ed83) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/hashserv: Allow autostarting of a local hash server using ↵Richard Purdie2019-08-066-10/+37
| | | | | | | | | | | | | | | | | BB_HASHSERVE Its useful, particularly in the local developer model of usage, for bitbake to start and stop a hash equivalence server on local port, rather than relying on one being started by the user before the build. The new BB_HASHSERVE variable supports this. The database handling is moved internally into the hashserv code so that different threads/processes can be used for the server without errors. (Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve determinismRichard Purdie2019-08-061-13/+13
| | | | | | | | | | | Whilst this isn't strictly necessary, its helpful if the log output is consistent and its also helpful if bugs either appear or don't appear for a specific configuration. Ensuring the various iterations we make are deterministic (sorted) helps with this. (Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Enable dynamic task adjustment to hash equivalencyRichard Purdie2019-08-062-10/+161
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a compelling usecase for tasks being able to notify runqueue that their "unihash" has changed. When this is recieved, the hashes of all subsequent tasks should be recomputed and their new hashes checked against existing setscene validity. Any newly available setscene tasks should then be executed. Making this work effectively needs several pieces. An event is added which the cooker listen for. If a new hash becomes available it can send an event to notify of this. When such an event is seen, hash recomputations are made. A setscene task can't be run until all the tasks it "covers" are stopped. The notion of "holdoff" tasks is therefore added, these are removed from the buildable list with the assumption that some setscene task will run and cover them. The workers need to be notified when taskhashes change to update their own internal siggen data stores. A new worker command is added to do this which will affect all newly spawned worker processes from that worker. An example workflow which tests this code is: Configuration: BB_SIGNATURE_HANDLER = "OEEquivHash" SSTATE_HASHEQUIV_SERVER = "http://localhost:8686" $ bitbake-hashserv & $ bitbake automake-native $ bitbake autoconf-native automake-native -c clean $ bitbake m4-native -c install -f $ bitbake automake-native with the test being whether automake-native is installed from sstate. (Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Convert to use self.unitaskhashesRichard Purdie2019-08-061-4/+8
| | | | | | | | | Rather than metadata driven sqlite databases for communication, use bitbake's unitaskhashes variable instead. (Bitbake rev: a0d941c787cf3ef030d190903279d311bc05d752) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add new unitaskhashes data variable which is cachedRichard Purdie2019-08-062-2/+19
| | | | | | | | | | | We need to preserve unihash task hashes between runs. Use the new SimpleCache class to create such a class within the signature generator which is loaded at init time and saved when builds complete. The default is unpopulated but metadata sig handlers can populate this cache. (Bitbake rev: 1f326f2c29c2664a5daaeeb0c1fd332630efbdba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve scenequeue processing logicRichard Purdie2019-08-061-9/+3
| | | | | | | | | | | | Rather than a special copy of the data structure which we change, compute the logic using set operations from other data we have. This means we can add tasks back into the scenequeue without having to worry about reversing operations on this variable with all the potential bugs that might involve. (Bitbake rev: b707d0cbc25fa336a1e95ff588f1ea37eee063eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Add SimpleCache classRichard Purdie2019-08-061-0/+53
| | | | | | | | | | This adds a simple version of the MultiProcessCache which can be used to save and load cache data, useful for a new usecase we have in sigdata/runqueue. (Bitbake rev: 19a6e35600ae6d2d1bcecca6e68ab8c37674774e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Import unihash code from OE-CoreRichard Purdie2019-08-061-0/+163
| | | | | | | | | | | | This code is closely tied with the hash server in bitbake and also means we can't relibably test the hashserv runqueue functionality without OE metadata. Moving this to bitbake as a MixIn class makes most sense and encourages code collaboration and reuse as well as enabling easier and more accurate testing of the APIs. (Bitbake rev: 7bb79099a6c1b463d6ae9226c4cab5e76a965675) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: SQL OptimizationsJoshua Watt2019-08-061-10/+27
| | | | | | | | | | | | | | | | | | | | | | | | Implements a number of optimizations to the SQL used in the hash equivalence server: 1) Two indexes are created for the two methods (method, taskhash and method outhash) by which rows are found in order to speed up the lookup 2) An extra SELECT to lookup the just inserted row was removed. This SELECT is unnecessary since all of the information about the newly inserted row is already available. 3) A uniqueness constraint was added to the table. This should allow the server to be multithreaded in the future since duplicate inserts can be detected (and ignored). This change requires bumping the database version to '2', since a uniqueness constraint can't be added to an existing table. 4) Some comments are added to clarify the trick SELECT statement used when inserting new equivalent hashes (Bitbake rev: 7aec8632e67b4f0ab7b72692c40a42f6926608c3) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: avoid 'maximum recursion depth' RuntimeErrors when ↵Chris Laplante via bitbake-devel2019-07-271-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | handling 403 codes The code says that some servers respond with 403 codes when they really mean 405 codes. But we still need to account for legitimate 403 codes. Before this change, I noticed that sstate mirror checking was taking a very long time when I purposely entered incorrect credentials into my .netrc file for our sstate mirror. Instrumenting the code, I discovered tracebacks like the following for every mirror access attempt: File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus fetcher.checkstatus() File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus ret = try_mirrors(self, self.d, ud, mirrors, True) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors ret = try_mirror_url(fetch, origud, uds[index], ld, check) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url found = ud.method.checkstatus(fetch, ud, ld) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus opener.open(r) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(*args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405 unverifiable=True)) ... (repeats until recursion depth is reached) Solution is to make sure we only attempt the GET request once when handling 403/405 error codes. (Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: Use npm pack to download node modules instead of wgetMads Andreasen2019-07-271-22/+23
| | | | | | | | | | Using npm pack to download the main node module and its dependencies allow for the use of private npm modules and access to them via .npmrc (Bitbake rev: e5eda3871893e4eadeb311aeb997e183675598f4) Signed-off-by: Mads Andreasen <mads@andreasen.cc> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Sync list of fs_types with oe-coreDaniel Ammann2019-07-271-5/+5
| | | | | | | (Bitbake rev: 1dddfe3512b6390958abb91b21f074568ae4e8db) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix handling of tainted sig filesRichard Purdie2019-07-191-2/+2
| | | | | | | | | | The addition of some debugging code meant that comparisions between sig files with a taint and without a taint weren't working. Tweak the logic to avoid tracebacks if one side doesn't have a taint. (Bitbake rev: f5ea06fc2b6713c9f8e85ecf7cb981ae9a84d896) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Create a symlink for current cachefileRobert Yang2019-07-191-0/+9
| | | | | | | | | | | | So that people or other tools can easily know which one is being used, just like what we did for run.do_task and log.do_task, otherwise, we have no way to know it. I usually use "ls -t", but it isn't reliable since the one which is being used may not the latest one. (Bitbake rev: cf286dff653eed542bf347ca46234c224944d5b0) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix non setscene tasks targets being lostRichard Purdie2019-07-182-1/+18
| | | | | | | | | | | | If you specify both setscene and non-setscene tasks on the commandline, the non-setscene tasks could be missed, e.g. "bitbake X:do_patch X:do_populate_sysroot" and do_patch would fail to run. Fix the problem in runqueue and add a testcase. (Bitbake rev: 75292fdec5d9c0b5b3c554c4b7474a63656f7e12) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/clearcase: Fix class import errorsCHerzig@Gauselmann.de2019-07-161-1/+3
| | | | | | | (Bitbake rev: 9a5152fa4613a1164cbf2a0248460e75207b2624) Signed-off-by: Christian Herzig <cherzig@gauselmann.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Allow common sstate tasks to become validRichard Purdie2019-07-161-0/+6
| | | | | | | | | | | As the logic in bitbake improves, the logic in the tests needs to as well. Afer we built a task for the first time, allow its setscene hash verification status to change, mirroring what would happen in a multiconfig build. (Bitbake rev: 27ec2e69ab3e32972caf8b072b2945736696d83d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise multiconfig with overlapping setsceneRichard Purdie2019-07-166-11/+98
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently if a multiconfig build contains different configurations which have overlapping sstate artefacts, it will build them multiple times. This is clearly suboptimal and not what users want/expect. This adds code to detect this and stall all but one of the setscne tasks so that once its built, it can be found by the other tasks. We take care to iterate the multiconfigs in order so try and avoid dependency loops. We also match on PN+taskname+taskhash since this is what we know sstate in OE-Core would use. There are some tasks even within a multiconfig which match hashes (mostly do_populate_lic tasks) but those have a much higher chance of circular dependency so aren't work attempting to optimise. If a deadlock does occur the build will be slower but there is code to unbreak such a deadlock so it hopefully doens't break anything. Comments are injected into the test tasks so they have different task hashes and a new test for this optimisation is added. (Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Abstract hash verification functionRichard Purdie2019-07-161-40/+42
| | | | | | | | | Pull the common pieces of the hash verification code into a single function and reduce code duplication. (Bitbake rev: d0c39e05cef841c6f29cc6c919df6cbf271a9bda) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Whitespace fixRichard Purdie2019-07-161-19/+19
| | | | | | | | Fix some unwanted extra indentation. (Bitbake rev: 460a5c2e3e1d72f2da16fbc96832fadc82e72c52) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Simplify some convoluted logicRichard Purdie2019-07-161-6/+1
| | | | | | | | This was left from when task IDs complicated the code, simplify. (Bitbake rev: ae36b5c693bb9f13c88199e78e3c31616852eafb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: uihelper: No longer listen to scenequeue task startedRichard Purdie2019-07-151-1/+1
| | | | | | | | | | With the merge of the scenequeue with real tasks, this now confuses the statistics. The real tasks are the definitive progress so monitor only those. (Bitbake rev: 20956b508a082224139c8f56b68299edff6e0443) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: Add initial scenario based test for runqueueRichard Purdie2019-07-1510-0/+442
| | | | | | | | | | We need some tests for runqueue, its been something which has been hard to test for a long time. Add some dummy metadata to allow this, mirroring the OE structure in spirit. (Bitbake rev: 37564d7440c5d7aa05ec537f3b79026b1c83bb68) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Complete the merge of scenequeue and normal task executionRichard Purdie2019-07-151-117/+109
| | | | | | | | | | | | This combines the scqenequeue and normal task execution into one function and simplifies the state engine accordingly. This is the final set of cleanup to fully merge things without adding the extra noise to the previous commits. (Bitbake rev: 56f3396d8c7cfbebd175877c9d773e4e35f8dea1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Alter setscenewhitelist handlingRichard Purdie2019-07-151-49/+30
| | | | | | | | | | | | | | | | Since there is now parallel execution of setscene and normal tasks, the way setscenewhitelist handling worked can't function the way it did. Paul and I never liked its error output anyway. This code tries a different approach, checking the task at execution time but printing the uncovered task list. This code may need improvement after real world usage but can work with the new task flows. (Bitbake rev: a08d8ba5f5194a09391b1904ee31c04c5f0b1e28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-18 05:43:05 +0000