summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: bitbake-user-manual: Improve the example for Removal (Override ↵Martin Jansa2019-09-071-4/+8
| | | | | | | | | | | | Style Syntax) * to better show how it works with spaces and multiple values (Bitbake rev: 89dd570ebd7046f5bce4a8b7f3b2b50b1cf65589) Signed-off-by: Herb Kuta <herb.kuta@lge.com> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Ensure bbappends are found in stable orderWes Lindauer2019-09-071-0/+1
| | | | | | | | | | | | | | | Thanks to wildcards in bbappend filenames, it's possible to have multiple bbappends that apply to the same recipe in the same directory. In order to get sstate hits between different workspaces, we want to apply those bbappend files in a consistent order. Since readdir() returns files in a non-deterministic order between workspaces (based on inode number and/or time of creation), we'll need to sort its result in order to have any consistency. (Bitbake rev: 94c0c7f15c7a6244a8576ed948ffc21afb96ba82) Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: show warning when renaming the archive with bad checksum failedMartin Jansa2019-09-031-1/+2
| | | | | | | | | | | | | | | | | | | | | * noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: 9a1bf4ba9ec00c2a222d820f8f83d1f056b021d6) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix movefile() exception handling with python3Martin Jansa2019-09-031-1/+1
| | | | | | | | | | | | | | | | | | | | * with python3 this fails with: File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile 0795: try: 0796: os.rename(src, destpath) 0797: renamefailed = 0 0798: except Exception as e: *** 0799: if e[0] != errno.EXDEV: 0800: # Some random error. 0801: print("movefile: Failed to move", src, "to", dest, e) 0802: return None 0803: # Invalid cross-device-link 'bind' mounted or actually Cross-Device Exception: TypeError: 'OSError' object is not subscriptable (Bitbake rev: d6e43c443ddbbe467c4380c48d2bc28ae18504a1) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Resolve fetch error in bitbake-selftestArmin Kuster2019-09-011-2/+2
| | | | | | | | | | | | | | | | | | | | | FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) AssertionError: '' is not true : Could not find upstream version for db [YOCTO #13496] The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget. Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the recipe. Also change the version being checked. (Bitbake rev: 4cf5bb761c561ddea86f2875be35d05abc8486e1) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: remove generation of recipe-depends.dotChen Qi2019-08-281-21/+4
| | | | | | | | | | | | | | | | | | | | | | The information of recipe-depends.dot is misleading. e.g. $ grep xz recipe-depends.dot | grep bzip2 "bzip2" -> "xz" "xz" -> "bzip2" Users would wonder why they get some circular dependency. The information is derived from removing the task names of task-depends.dot. It's not giving people any additonal information, and it's misleading. So we remove the generation of this file. (Bitbake rev: 4c484cc01e3eee7ab2ab0359fd680b4dbd31dc30) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: respect force flag in runall and runonlyJoshua Watt2019-08-281-14/+18
| | | | | | | | | | | | | | | | | Specifying the force flag will now cause runall and runonly to invalidate the tasks before running them. This allows a --runall or --runonly to force the tasks to run, even if they would have otherwise been skipped, e.g.: bitbake -f --runall fetch Will run all do_fetch tasks even if they wouldn't be necessary (for example, skipped by setscene) (Bitbake rev: 71e52d3822016027106f2a2e74b8dfdf20f5dc1e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise build_taskdepdata slightlyRichard Purdie2019-08-211-6/+6
| | | | | | | | | | Rather than repeatedly calling mc_from_tid() do this in the parent, removing around a million function calls. Takes time spent in this function from 40s to 36s. (Bitbake rev: 28b3f0d8867804799420689c314ac4a8f01efb8c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Further optimise holdoff tasksRichard Purdie2019-08-211-24/+25
| | | | | | | | | | There are other data structures which can be reprocessed at the same time as holdoff_tasks, further improving build efficiency in various places. (Bitbake rev: 02090b3456b7a2de12e72dfeaabfd3b631609924) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise holdoff task handlingRichard Purdie2019-08-211-2/+10
| | | | | | | | | | | We don't need to process the holdoff task list until we're executing tasks which saves some data manipulation, at the cost of some data structures not being correct at all times. This saves significant amounts of time in various profile charts of larger builds. (Bitbake rev: 270f076111b12eab358417b0c4cf9c70d7cc787a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Small but critical fixRichard Purdie2019-08-211-0/+1
| | | | | | | | | | | | | We've observed do_package and do_package_setscene running in parallel. The reason is that holdoff_tasks wasn't getting updated. Looking at the code, it would seem the reason is that the task was in pending_migrations and hence changed wasn't set and holdoff_tasks wasn't updated. Fix this. It only affects builds with rehashing enabled. (Bitbake rev: e26e61e84575669bd223f6ab316798097ed95ec8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Delay the setup of the siggen slightly to allow ↵Richard Purdie2019-08-211-1/+1
| | | | | | | | | | | | metadata defined siggens If we define a metadata siggen it can fail due to the early init here. Move slightly later to avoid those failures which allows fixes in OE to the check-layer script related to the hash equiv siggen. (Bitbake rev: fdf5c341f3393173876a753c46c9bd067eb2b353) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure target_tids is filteredRichard Purdie2019-08-161-0/+1
| | | | | | | | | | | bitbake <target> --runonly=fetch failed as the target_tids list included entries which were no longer targeted task ids. Fix this. (Bitbake rev: 94e848ae6544e628a19cb97115279b0b1678967c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Ensure we don't accumulate sockets in TIME_WAIT stateRichard Purdie2019-08-161-0/+6
| | | | | | | | | | This can cause a huge backlog of closing sockets on the server and in our case we don't really want/need the protection TCP is trying to give us so work around it. (Bitbake rev: 7bc79fdf60519231da7c0c7b5b6143ce090ed830) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: server/process: Handle BBHandledException to avoid ↵Robert Yang2019-08-161-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | unexpected exceptions The parseBaseConfiguration() raises bb.BBHandledException(), but BitBakeServer() didn't handle it, so we always got unexpected exceptions when there were errors. For example: === Case 1: * Add "print "hello"' in base.bbclass' def oe_import() function def oe_import(d): print "hello" [snip] $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Last 60 lines of server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 21 print "hello" ^ SyntaxError: Missing parentheses in call to 'print' <The first exception> During handling of the above exception, another exception occurred: <Tracebacks> <The second exception> During handling of the above exception, another exception occurred: <Tracebacks> <The third exception> During handling of the above exception, another exception occurred: <Tracebacks> [snip] Now it looks like: $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 21: The code lines resulting in this error were: 0001:def oe_import(d): *** 0002: print "hello" 0003: import sys 0004: 0005: bbpath = d.getVar("BBPATH").split(":") 0006: sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath] SyntaxError: Missing parentheses in call to 'print' (base.bbclass, line 21) === Case 2: * Add 'HOSTTOOLS += "hello"' to conf/local.conf: $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): <Tracebacks> [snip] During handling of the above exception, another exception occurred: [snip] <Tracebacks> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello The error message is printed by bb.fatal() which raises bb.BBHandledException(), but BitBakeServer() doesn't handle it, so we got it. Now it looks like: ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello No unexpected exceptions anymore. [YOCTO #13267] (Bitbake rev: 6e6865e6371dbd31a136eae64cc5b1fa5f5bee33) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure data is handled correctlyRichard Purdie2019-08-151-0/+2
| | | | | | | | | This doesn't appear to have ill effects right now but there is a correctness issue which this so fix it. (Bitbake rev: a5e084a266f63c2fd370122327615e49beaeb94e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix data corruption problemRichard Purdie2019-08-151-0/+3
| | | | | | | | | This was overwriting data in the parent which caused all kinds of odd/weird failures. (Bitbake rev: 4c5aeb424247a9d0c907524ffacd9c61fcdc0852) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Fix testsRichard Purdie2019-08-152-5/+5
| | | | | | | | | There were paths being accidentally included in some of the hashserv tests. Remove that and update the hashes so the tests work independently of paths. (Bitbake rev: 6ddb9f09cb60c2354fa6a67cce412c4dc1e7dc2d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure we clear the stamp cacheRichard Purdie2019-08-141-0/+3
| | | | | | | | | | When the task hashes change we need to ensure the stampcache is cleared out else tasks don't rerun when they should as we're basing decisions on stale cache data. (Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add missing setscene task corner caseRichard Purdie2019-08-141-0/+2
| | | | | | | | | We weren't marking this special case of setscene task as buildable leading to runqueue task failures. (Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add further debug informationRichard Purdie2019-08-141-0/+11
| | | | | | | | | Further testing shows we should test some extra datastructures to help pinpoint logic errors more precisely. This adds some further data structure sanity checks. (Bitbake rev: 83c4370b25c3a14cc946965c5c5f83ea28f488a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop debug statement causing performance issuesRichard Purdie2019-08-141-2/+0
| | | | | | | | | | | | This debug statement could result in a long list of tasks which when repeatedly sent over our IPC, slowed down the builds immensely. Remove it in favour of other more targeted debugging added recently, bringing back some lost performance, particularly on builds with large numbers of tasks. (Bitbake rev: 85fe627fdb6510f0942917964386fad9d8c479c8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix event timing raceRichard Purdie2019-08-142-71/+74
| | | | | | | | | | | | The event from the task notifiing of hash equivalency should only be processed when the task completes. This can otherwise result in a race where a dependent task may run before the original task completes causing various failures. To make this work reliably, the code had to be restructured quite a bit. (Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Recompute holdoff tasks from scratchRichard Purdie2019-08-141-1/+1
| | | | | | | | | | | | The changed_setscene variable here is just odd and not needed. Worse, it could prevent some tasks from being removed from the holdoff tasks list. The list is being rebuilt and should work as intended just from the other data, this is a leftover from previous versions of the code as far as I can tell. (Bitbake rev: 030b9f2b3ce6ed40e79304eb0ffee6c6613f43be) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve scenequeue debuggingRichard Purdie2019-08-141-19/+42
| | | | | | | | | | | | Whilst we had good runqueue failure mode debug, it hadn't adapted to the scenequeue changes. Run the scenequeue sanity tests at the end of a build and output the results regardless of whether all setscene tasks completed or not. This *massively* improves the ability to debug runqueue problems. (Bitbake rev: b9b2177473c0b95a23bd519a201e1d2ba101c6c1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix next_buildable_task performance problemRichard Purdie2019-08-141-5/+7
| | | | | | | | | | | | | | Looking at the profile information, a lot of time is being spent in next_buildable_task. This is probably due to the generator expressions not working well with the empty test. The easiest way to improve things is to switch to using set manipulations. We also don't need to update self.buildable the way the original code did as we don't rely on that anywhere. (Bitbake rev: 3bcf9ad4964b7e42d1a02ce231e9db42a81ead2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Wait for covered tasks to complete before trying setsceneRichard Purdie2019-08-141-0/+4
| | | | | | | | | If tasks are in the covered list of tasks for a given setscene task, it needs to wait for those to complete before we can start. (Bitbake rev: fdee640c26750b852eb68f5c80437377aa300ed8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve hash server startup code to avoid exit tracebacksRichard Purdie2019-08-142-12/+18
| | | | | | | | | | | | | At exit the hashserv code was causing tracebacks as join() wasn't being called from the thread that started the process. Ensure that the hashserver is started from the pre_serve hook which is the final thread the cooker runs in. This avoids the traceback at the expense of some horrific poking into data stores which will ultimately need improving through a proper API. (Bitbake rev: 05888700e5f6cba48a26c8a4c447634a28e3baa6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Add further hash equivalence testsRichard Purdie2019-08-144-12/+166
| | | | | | | | | Add some extra hash equivalence runqueue tests based on recent scenarios that caused problems during testing. (Bitbake rev: 373b085ead992a725b2230ededd992b4c61a1a05) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve setscene task handling logicRichard Purdie2019-08-141-75/+34
| | | | | | | | | | | | | | The previous tasks_covered and tasks_notcovered were basically unstable data structures. We couldn't always tell whether tasks should be covered or not when trying to repair the sturcture if sstate tasks reran. In the end its simpler to throw the lists away and rebuild them based upon current data rather than trying to patch it adhoc. This turns out to be simpler and much more reliable and I've much more confidence in this code. (Bitbake rev: 52ee2ba2c617d928569f5afa404925c8b6f317bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix corruption issueRichard Purdie2019-08-141-1/+1
| | | | | | | | | | | We need to copy this set, not modify the original else all kinds of weird and bad things break, mostly from circular references. We'll not go into how much sleep I lost tracking down the fallout from this. (Bitbake rev: 49927546d2b306830c98f6f9da4a6ad828f6a3a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: layerindexlib: Fix parsing of recursive layer dependenciesMark Hatle2019-08-131-2/+19
| | | | | | | | | | | | | | | | | [YOCTO #13447] When running bitbake-layers layerindex-fetch from 'master', there is a circular dependency between meta-oe and meta-python. This triggered a maximum recursion depth exception. To fix the exception, as we walk down a branch (depth first search), we track the layers we've already seen. If we are about to recurse into a layer we've already seen we report a warning and then stop recursion. (Bitbake rev: d6155d095513be3f500d089c4ed4c4b89949d560) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Fix for the Second Keyboard InterruptRobert Yang2019-08-081-4/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: $ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p Press the first Ctrl-C when the parsing process is at about 50%: Keyboard Interrupt, closing down... Then presss the second Ctrl-C: File "/path/to/bitbake/bitbake/lib/bb/ui/knotty.py", line 619, in main event = eventHandler.waitEvent(0.25) File "/path/to/bitbake/lib/bb/server/process.py", line 591, in waitEvent self.eventQueueNotify.wait(delay) File "/usr/lib/python3.5/threading.py", line 549, in wait signaled = self._cond.wait(timeout) File "/usr/lib/python3.5/threading.py", line 297, in wait gotit = waiter.acquire(True, timeout) KeyboardInterrupt Capture the second KeyboardInterrupt during stateShutdown is running can fix the problem. There may be still tracebacks for the third KeyboardInterrupt, but I'm leaning to not fix it since we aimed for supporting 2 KeyboardInterrupts only. (Bitbake rev: 8c26b451f22193ef1c544e2017cc84515566c1b8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Cleanup the queue before call process.join()Robert Yang2019-08-081-0/+8
| | | | | | | | | | | | | | | | | | | | | | Fixed: $ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p Press *one* Ctrl-C when the parsing process is at about 50%, then the processes are not exited: Keyboard Interrupt, closing down... Timeout while waiting for a reply from the bitbake server It hangs at process.join(), according to: https://docs.python.org/3.7/library/multiprocessing.html Cleanup the queue before call process.join() can fix the problem. (Bitbake rev: 3eddfadd19b2ce4c061861abf0c340e3825b41ff) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump version to 1.43.1 for API changesRichard Purdie2019-08-062-2/+2
| | | | | | (Bitbake rev: f43778c2d19e70d4befd483b06aaf247fc65c799) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Add hashserv+runqueue testRichard Purdie2019-08-064-7/+59
| | | | | | | | Add a test which tests the runqueue adaptations for hash equivalency. (Bitbake rev: 477321d0780df177c1582db119c2bb6795912fc6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Drop bb.build.FuncFailedRichard Purdie2019-08-063-49/+12
| | | | | | | | | | | | | Its hard to see what this exception adds in the current codebase. The logfile attribute is effectively ignored, the exception doesn't serve a defined purpose and mostly seems to be worked around. Remove it entirely. If this does cause output problems, we'll figure out better ways to address those. (Bitbake rev: cfeffb602dd5319f071cd6bcf84139ec77f2d170) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Clean up task reference formatsRichard Purdie2019-08-065-95/+82
| | | | | | | | | | | | | | | | | | Currently siggen uses the format "<filename>.<taskname>" for referencing tasks whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the separator everywhere. This is an API breaking change since the cache is affected, as are siginfo files and any custom signature handlers such as those in OE-Core. Ultimately this will let us clean up and the accessor functions from runqueue, removing all the ".rsplit(".", 1)[0]" type code currently all over the place. Once a standard is used everwhere we can update the code over time to be more optimal. (Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Clean up BB_HASHCHECK_FUNCTION APIRichard Purdie2019-08-062-53/+23
| | | | | | | | | | | | | This function uses an old API which uses offsets into lists as a communication mechanism. Update the API to use "tid" which is used universally in runqueue now. We can also add kwargs support to the funciton definition to drop some of the backwards compaiblility hoops we had to jump though with different function argument combinations. (Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Switch from threads to multiprocessingRichard Purdie2019-08-063-15/+27
| | | | | | | | | | There were hard to debug lockups when trying to use threading to start hashserv as a thread. Switch to multiprocessing which doesn't show the same locking problems. (Bitbake rev: be23d887c8e244f1ef961298fbc9214d0fd0968a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Use separate threads for answering requests and handling themRichard Purdie2019-08-061-1/+40
| | | | | | | | | | Experience with the prserv shows that having two threads, one accepting and queueing connections and one handling the requests leads to much more reliable behaviour than having everything in a single thread. (Bitbake rev: a03d60671a53d9ff70e07cc42fe35f6f8776dac2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Use a memory journalRichard Purdie2019-08-061-1/+1
| | | | | | | | | | | | We've seen PR Server timeouts on the autobuilder, this is likely from the journal being blocked on disk IO generated by the build. Since we're running with synchronous off, we may as well put the journal into memory and avoid any IO related stalls. (Bitbake rev: ee3fc6030e653f3244b065fc89aafd2a7c36ae04) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Turn off sqlite synchronous modeRichard Purdie2019-08-061-0/+2
| | | | | | | | | | | | | | | | | | We're seeing performance problems with hashserv running on a normal build system. The cause seems to be the large amounts of file IO that builds involve blocking writes to the database. Since sqlite blocks on the sync calls, this causes a significant problem. Since if we lose power we have bigger problems, run with synchronous=off to avoid locking and put the jounral into memory to avoid any write issues there too. This took writes from 120s down to negligible in my tests, which means hashserv then responds promptly to requests. (Bitbake rev: 7ae56a4d4fcf66e1da1581c70f75e30bfdf3ed83) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/hashserv: Allow autostarting of a local hash server using ↵Richard Purdie2019-08-066-10/+37
| | | | | | | | | | | | | | | | | BB_HASHSERVE Its useful, particularly in the local developer model of usage, for bitbake to start and stop a hash equivalence server on local port, rather than relying on one being started by the user before the build. The new BB_HASHSERVE variable supports this. The database handling is moved internally into the hashserv code so that different threads/processes can be used for the server without errors. (Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve determinismRichard Purdie2019-08-061-13/+13
| | | | | | | | | | | Whilst this isn't strictly necessary, its helpful if the log output is consistent and its also helpful if bugs either appear or don't appear for a specific configuration. Ensuring the various iterations we make are deterministic (sorted) helps with this. (Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Enable dynamic task adjustment to hash equivalencyRichard Purdie2019-08-062-10/+161
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a compelling usecase for tasks being able to notify runqueue that their "unihash" has changed. When this is recieved, the hashes of all subsequent tasks should be recomputed and their new hashes checked against existing setscene validity. Any newly available setscene tasks should then be executed. Making this work effectively needs several pieces. An event is added which the cooker listen for. If a new hash becomes available it can send an event to notify of this. When such an event is seen, hash recomputations are made. A setscene task can't be run until all the tasks it "covers" are stopped. The notion of "holdoff" tasks is therefore added, these are removed from the buildable list with the assumption that some setscene task will run and cover them. The workers need to be notified when taskhashes change to update their own internal siggen data stores. A new worker command is added to do this which will affect all newly spawned worker processes from that worker. An example workflow which tests this code is: Configuration: BB_SIGNATURE_HANDLER = "OEEquivHash" SSTATE_HASHEQUIV_SERVER = "http://localhost:8686" $ bitbake-hashserv & $ bitbake automake-native $ bitbake autoconf-native automake-native -c clean $ bitbake m4-native -c install -f $ bitbake automake-native with the test being whether automake-native is installed from sstate. (Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Convert to use self.unitaskhashesRichard Purdie2019-08-061-4/+8
| | | | | | | | | Rather than metadata driven sqlite databases for communication, use bitbake's unitaskhashes variable instead. (Bitbake rev: a0d941c787cf3ef030d190903279d311bc05d752) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add new unitaskhashes data variable which is cachedRichard Purdie2019-08-062-2/+19
| | | | | | | | | | | We need to preserve unihash task hashes between runs. Use the new SimpleCache class to create such a class within the signature generator which is loaded at init time and saved when builds complete. The default is unpopulated but metadata sig handlers can populate this cache. (Bitbake rev: 1f326f2c29c2664a5daaeeb0c1fd332630efbdba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve scenequeue processing logicRichard Purdie2019-08-061-9/+3
| | | | | | | | | | | | Rather than a special copy of the data structure which we change, compute the logic using set operations from other data we have. This means we can add tasks back into the scenequeue without having to worry about reversing operations on this variable with all the potential bugs that might involve. (Bitbake rev: b707d0cbc25fa336a1e95ff588f1ea37eee063eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Add SimpleCache classRichard Purdie2019-08-061-0/+53
| | | | | | | | | | This adds a simple version of the MultiProcessCache which can be used to save and load cache data, useful for a new usecase we have in sigdata/runqueue. (Bitbake rev: 19a6e35600ae6d2d1bcecca6e68ab8c37674774e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-10 08:27:10 +0000