summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/runqueue.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Optimize loop in scenequeue data constructionBenjamin Szőke2025-05-271-2/+1
| | | | | | | | | | | | In scenequeue data generation loop, progress bar update in each cycle cause a significant loss of speed. Remove progress bar update in for loop. It was become quite fast (faster 30 times, about) without progress bar update. (Bitbake rev: f775e53f1196e3e5d0ded277c061c5c268b3b28d) Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: runqueue: Verify mcdepends are validMark Hatle2025-03-031-0/+2
| | | | | | | | | | | | | | | In order to avoid a potentially confusing backtrace, check that the mcdepend is valid when we add it. Add a test case to ensure invalid configurations are caught and trigger an error. [RP: Reworked test case to simplify and improve code] (Bitbake rev: ff523497270f37b484b44a4445c2194791bcb6ff) Signed-off-by: Mark Hatle <mark.hatle@amd.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: Remove double importsMichael Estner2025-02-271-1/+0
| | | | | | | | | * Remove double imports mentioned by pylint (Bitbake rev: 741db6719efca5aa9ef2c15e60cdd624e4aa1a8d) Signed-off-by: Michael Estner <michaelestner@web.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix scenetask processing performance issueRichard Purdie2024-11-291-1/+5
| | | | | | | | | | | Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. (Bitbake rev: 923c19b9713e398d8e66e6d4422dfd4c18a03486) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise setscene loop processingRichard Purdie2024-11-291-2/+8
| | | | | | | | | | Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. (Bitbake rev: 00f4d932e3af0eeb333339cbe942010fc76dee0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix performance of multiconfigs with large overlapRichard Purdie2024-11-291-2/+4
| | | | | | | | | | | | | | | | | | | There have been complaints about the performance of large multiconfig builds for a while. The key missing data point was that the builds needed to have large overlaps in sstate objects. This can be simulated by building the same things with just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of deferred tasks. The issue is that the expensive checks in the setscene loop were hit every time through runqueue's execute function before the check on deferred tasks. This leads to task execution starvation as that only happens once per iteration. Move the skip check earlier in the function which speeds things up enormously and should improve performance of such builds for users. (Bitbake rev: 9c6c506757f2b3e28c8b20513b45da6b4659c95f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker/cooker: Increase default pipe sizeRichard Purdie2024-11-211-1/+1
| | | | | | | | | | | | | | | | The default pipe size is 64kb on builds, which can be inefficient for larger log files from workers. Increase the pipe size to 512kb since build systems have decent amounts of memory and this is a more efficient way of batching the data. Tweak the default read sizes to match the pipe size for efficiency. Since the contstant is only present in python 3.10 onwards, add some compatibility code. (Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid dumpsigs idle loop blockingRichard Purdie2024-11-071-29/+44
| | | | | | | | | | | | | | | | | | | | | We're seeing some failures on hosts where slow "idle" loop iterations are causing bitbake server timeouts. These seem to happen particularly in the dump_signatures() function within runqueue. That isn't entirely surprising since it creates a pool of threads to execute work an at best can take around 10s to execture and return control backto the main loop. On a slow system, it is understandable this can take longer, particularly as these functions are creating large chunks of IO. Since the work is being done in threads, we can launch them, return to idle and check on the results periodically as they complete. This should hopefully address some of the remaining timeout issues we see on the autobuilder in oe-selftest sstate tests. (Bitbake rev: e66f1b643b4b77404ba31f2704cda5af9bf00a57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid save_unitaskhashesRichard Purdie2024-06-071-3/+0
| | | | | | | | | | The save comes with an IO overhead which can slow down the rehash loop in bitbake a lot. We only needed to do this when recipes were doing unihash cache copying. Now they aren't doing that, drop this IO pain point. (Bitbake rev: dfc15ef99302dea22a051c9eb8398ffd5cf1fc20) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb: Use namedtuple for Task dataJoshua Watt2024-05-281-18/+27
| | | | | | | | | | | | Task dependency data is becoming unwieldy with the number of indices it contains. Convert it to use a named tuple instead, which allows members to be indexed by a named property or an index (which allows it to retain backward compatibility). (Bitbake rev: 26446cca4d22734c3f1b328a205c169dadb7e494) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve rehash get_unihash parallelismRichard Purdie2024-05-281-2/+10
| | | | | | | | | Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. (Bitbake rev: c1949d5350342eaaf6ab988d7bfba99496d55523) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Process unihashes in parallel at initRichard Purdie2024-05-281-8/+10
| | | | | | | | | | | | | | Improve the runqueue init code to call unihash queries in parallel since this is faster and more efficient, particularly on slower links with longer round trip times. The call to the function from cooker is unneeded since that function calls prepare() and hence this functionality will already have run, so drop that obsolete call. (Bitbake rev: 721c97a115a7a4bf21955be79391bd6e0099f40e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Allow rehash loop to exit in case of interruptsRichard Purdie2024-05-281-1/+3
| | | | | | | | | | The initial hash serve loop exits in the case where interrupts are present but probably checks a bit too often. Tweak that and also allow the slow rehash loop to break on interrupt, improving bitbake Ctrl+C response. (Bitbake rev: 4534365591fd17bcc2b684900863b67bc69519ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add timing warnings around slow loopsRichard Purdie2024-05-281-0/+22
| | | | | | | | | | | | | With hashserve enabled, there are two slow paths/loops, one at initial runqueue generation and also during the rehash process when new outhashes are found. Add timing information at the hashserve log level for when these loops take longer than 30s or 60s overall. This will leave evidence in the logs when things are running particularly slowly. (Bitbake rev: 6c357ede08e0b2a93bdaad2c1d631994faf2b784) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue: Report which dependencies affect the taskhashjoshua Watt2024-05-081-3/+5
| | | | | | | | | | | | | Report which task dependencies in BB_TASKDEPDATA are included in the taskhash. This allows tasks to identify which tasks dependencies may change without the task re-running. Knowing this information is important for tasks that want to transfer information from dependencies (such as SPDX) (Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: allow '=' in environment variable valuesdavid d zuhn2024-03-221-1/+1
| | | | | | | | | | | | Limit the split to key & value (2 items) instead of the n items one can get if there are '=' characters in the value. Fixes [YOCTO #15447] (Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb) Signed-off-by: david d zuhn <david.zuhn@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add support for BB_LOADFACTOR_MAXRichard Purdie2024-02-231-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Some ditros don't enable /proc/pressure and it tends to be those which we see bitbake timeout issues on, seemingly as load gets too high and the bitbake processes don't get scheduled in for minutes at a time. Add support for stopping running extra tasks if the system load average goes above a certain threshold by setting BB_LOADFACTOR_MAX. The value used is scaled by CPU number, so a value of 1 would be when the load average equals the number of cpu cores of the system, under one only starts tasks when the load average is below the number of cores. This means you can centrally set a value such as 1.5 which will then scale correctly to different sized machines with differing numbers of CPUs. The pressure regulation is probably more accurate and responsive, however our graphs do show singificant load spikes on some workers and this patch is aimed at trying to avoid those. Pressure regulation is used where available in preference to this load factor regulation when both are set. (Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve setcene performance when encoutering many 'hard' ↵Richard Purdie2024-02-131-1/+11
| | | | | | | | | | | | | | | | | | dependencies "bitbake world -n --setscene-only" shows poor performance as the numbers of tasks increases. Analysys shows this is due to the "deferred" hard dependencies being repeatedly processed which doesn't allow much forward progress in overall task execution. To avoid this, mark when it has been done and don't reprocess until dependencies are updated. We have to be careful as we've seen bugs where these dependencies aren't processed at the right time. They also have to be reproceseed during task migration/rehashing so the code has to "self heal" the data structures. (Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise taskname lookups in next_buildable_taskRichard Purdie2024-02-131-3/+3
| | | | | | | | | | A quick profile of bitbake world showed 147 million calls to taskname_from_tid(). The next_buildable_task function is performance senstive so move the call inside the if block to reduce the number of calls and speed the code up. (Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve performance for executing tasksRichard Purdie2024-02-131-10/+21
| | | | | | | | | | | | | Now that runqueue performance profiling works again we can see a lot of time is lost in build_taskdepdata. Whilst we can't compute that in advance, we can compute the individual entries. Therefore put a cache in place to compute those and save overhead in starting up tasks. (Bitbake rev: c4519b542702ba25023e53d77b275a6fa571ec50) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall all bugRichard Purdie2024-01-121-18/+24
| | | | | | | | | | | | | Where chains of RDEPENDS are multiple levels deep, the runall code was not accounting for this and recursing deeply enough to gather all dependencies. Fix this by iterating over the result until no more dependencies are found. Tested-by: Jonas Gorski <jonas.gorski@bisdn.de> Reported-by: Jonas Gorski <jonas.gorski@bisdn.de> (Bitbake rev: 966f25dfc23a6d17b2b6d3e0100e9ae264f99025) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: rework 'bitbake -S printdiff' logicAlexander Kanavin2024-01-101-15/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously printdiff code would iterate over tasks that were reported as invalid or absent, trying to follow dependency chains that would reach the most basic invalid items in the tree. While this works in tightly controlled local builds, it can lead to bizarre reports against industrial-sized sstate caches, as the code would not consider whether the overall target can be fulfilled from valid sstate objects, and instead report missing sstate signature files that perhaps were never even created due to hash equivalency providing shortcuts in builds. This commit reworks the logic in two ways: - start the iteration over final targets rather than missing objects and try to recursively arrive at the root of the invalid object dependency. A previous version of this patch relied relies on finding the most 'recent' signature in stamps or sstate in a different function later, and recursively comparing that to the current signature, which is unreliable on real world caches. - if a given object can be fulfilled from sstate, recurse only into its setscene dependencies; bitbake wouldn't care if dependencies for the actual task are absent, and neither should printdiff I wrote a recursive function for following dependencies, as doing recursive algorithms non-recursively can result in write-only code, as was the case here. [YOCTO #15289] (Bitbake rev: aadeca63da5d96160ce4d6d71da556e2e033f9b7) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: prioritize local stamps over sstate signatures in ↵Alexander Kanavin2024-01-051-0/+3
| | | | | | | | | | | | | | | | | | | printdiff Even with the reworked printdiff code, sstate which is heavily used in parallel can throw races at the tests: if a new matching, but otherwise unrelated sstate signature appears between writing out local stamps and listing matching sstate files, then that signature will be deemed 'the latest' and the actual local stamp will be discarded. This change ensures the scenario does not happen. It also makes use of the reworked find_siginfo(), particularly the 'sstate' entry in returned results. (Bitbake rev: c8574b796dabb69699c70540dd95a44d8f7388ab) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure version of siggen is verifiedRichard Purdie2024-01-051-0/+1
| | | | | | | | | | Since we need to change the form of the siggen function, we need to add versioning and some verison checks. This means if a newer bitbake is used with older metadata we can detect it. (Bitbake rev: 721556568413508213d22c29985e305a45a8d68a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs/runqueue: adapt to reworked find_siginfo()Alexander Kanavin2024-01-051-5/+5
| | | | | | | | | | In particular having 'time' explicitly used as a sorting key should make it more clear how the entries are being sorted. (Bitbake rev: 5439aca056c84ab4410aaf24bdb68e896191d8e1) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: add debugging for find_siginfo() callsAlexander Kanavin2024-01-051-0/+4
| | | | | | | (Bitbake rev: 52f5503e8cf048331134233fb681db6dc736ae38) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Remove tie between rqexe and starts_workerRichard Purdie2023-12-171-15/+8
| | | | | | | | | | | We've been moving to try and separate several pieces of runqueue. Allow start_worker to operate separately to rqexe since they don't need to be tied. This allows rqexe to be available to print_diff for future improvements. (Bitbake rev: 834e452243ff2eea6e8e2e7f4935b5233ffb4b00) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve inter setscene task dependency handlingRichard Purdie2023-12-061-13/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The way the code currently handles dependencies between setscene tasks is fairly poor, basically by deleting chunks of dependencies and adding reversed dependency relationships. This was once the best way to handle things but now a lot of the surrounding code has changed and this approach is suboptimal and can be improved. This change firstly adds debug logging for "hard" setscene task dependencies since previously the codepaths were missing from logs making them very hard to read. The changes to the setscene dependency graph are removed entirely this these altered graphs were a significant source of problems. Instead, if a hard dependency is run into, we mark the hard dependency as buildable and defer the task until the hard dependencies are met. The code now also skips the check_dependencies() code for hard dependencies since previously that code was having to list all possible hard dependencies. We don't need to do that as we can safely assume hard dependencies are required. With these changes to runqueue's behaviour, we stand some chance of being able to fix other bugs in OE-Core related to useradd for example. (Bitbake rev: 367789b53c1c22ec26e0f4836cdf2bdd9c7d84fa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Refactor StaleSetSceneTasks event out of ↵Richard Purdie2023-11-161-18/+18
| | | | | | | | | | | | | | build_scenequeue_data The event at the end of the function doesn't really belong there, move it out so the function becomes cleaner. It also avoids having to pass a number of parameters around, another sign this is the correct thing to do. This continues refactoring started a while ago to improve and modularise the code. (Bitbake rev: f712b927a84e8b8deaa8c907e8f9bcdec681f2aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Move 'cantskip' into sqdataRichard Purdie2023-11-161-7/+7
| | | | | | | | | Conceptually, this belongs in sqdata so this is the next step in refactoring this code, continuing what was started a while ago. (Bitbake rev: 2cf267cb657cd1864db43955cc010abba7a64093) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix errors when using -S printdiffRichard Purdie2023-11-091-2/+2
| | | | | | | | | | | | | | | | | When running printdiff, messages like: "Can't find a task we're supposed to have written out?" might be shown. This happens when hashequivalence is enabled and is due to the use of "hash" when "unihash" should be used (which defaults back to hash). Changing this fixed various errors oe-selftest highlighted. Also print the task ID when erroring about a missing task as it aids debugging. (Bitbake rev: e7783bbe0559abb91f5038cf83575c0f25138af2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall option for setscene tasksRichard Purdie2023-11-091-0/+1
| | | | | | | | | | | If --runall is used with setscene tasks, it will still skip them if it can. By marking the tasks as targets, this avoids that skipping and means --runall deploy_source_date_epoch works as expected for example. (Bitbake rev: 8c6cd1e8dd224bc1cc92736cbb7e113fd1fb5aae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: add header with length of messageEtienne Cordonnier2023-11-061-12/+22
| | | | | | | | | | | | | | | | | | | | | | | | | The IPC mechanism between runqueue.py and bitbake-worker is currently not scalable: The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker has no information about the size of the message. Therefore, the bitbake-worker is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")" for each chunk received. This does not scale, because queue.find has a linear complexity relative to the size of the queue, and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI. The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup 35 seconds before this fix, and 1.5 seconds after this fix). This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be received, and does not need to constantly search the whole queue for </tag>. (Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: set has 'add', not 'append' methodChris Laplante2023-10-231-1/+1
| | | | | | | | | | Discovered via pylint (Bitbake rev: 1a353cda696b7f59386ad2d78a57005b90a37da4) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: clarify that 'closest' signature means 'most recent' ↵Alexander Kanavin2023-10-231-1/+1
| | | | | | | | | | | | | | | | | (and not closest in its content) This is printed by 'bitbake -S printdiff' and more accurately reflects what the code does. Most of the time the most recent item should be what the user wants to see when debugging sstate misses, but there could also be printdiff-all (print differences with all matching sstate objects), or printdiff-N (N most recent, and not just the latest). (Bitbake rev: 315f48f965a54da5cec92908d91aa61c2d450add) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/lib: spawn server/worker using the current Python interpreterRoss Burton2023-09-261-2/+4
| | | | | | | | | | | | | | | | | The user may have invoked ./bin/bitbake using a different Python interpreter than whatever python3 is on $PATH (for example, explicitly using a different version). However, as the server and workers are spawned directly they'll use the hashbang and thus a different Python. We also ensure that argv[0] is set to sys.executable instead of 'bitbake-server' or 'bitbake-worker', so that sys.executable is set to the right value inside the child. Without this the server won't be able to start any workers. (Bitbake rev: b44d5d2a53d3082c8ce94e09c0cf833e33e25aec) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker/runqueue: Avoid unnecessary bytes object copiesEtienne Cordonnier2023-09-221-2/+2
| | | | | | | | | | | | | | | | | | declaring queue=b"" creates an object of types bytes(). bytes() is an immutable object, and therefore doing "self.queue = self.queue + r" creates a new object containing "self.queue" concatenated with "r". On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker, so those copies significantly slow down the initialization of the bitbake-worker. Rather use bytearray() which a mutable type, and use extend() to avoid copies. In my test setup, byterray.extend() is 10.000 times faster than copying the queue, for a queue size of 180MB. (Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: show number of currently running bitbake threads when ↵Martin Jansa2023-09-051-1/+1
| | | | | | | | | | | | | | | | | | pressure changes * it might be a bit confusing as it shows number of threads before making the decision to start more tasks and also it can show only a few tasks running, but not because of pressure when there just aren't many tasks left or wait for their dependencies to be finished first * example output: NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads (Bitbake rev: b0b114f31f20c5fcde31e6c308937ad4102dfe0a) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: show more pressure dataMartin Jansa2023-08-241-8/+10
| | | | | | | | | | | | | | * with latest bitbake I'm seeing very low number of bitbake tasks executed in parallel, probably due to pressure regulation show the values this is based on in the note * also simplify a bit by counting the pressure and exceeds signs only once (Bitbake rev: 21c17968f801e406ef7f328656587fadd9ef7f5d) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: fix PSI check logicChen Qi2023-08-111-4/+5
| | | | | | | | | | | | | | | | | | The current calculation is not correct because if tdiff is less than 1.0, it's not taken into consideration when calculating the current pressure. Also, make it clear that the 1.0s is the psi accumulation cycle, which might be changed in the future. We have this cycle because it could largely avoid the 0 result issue, that is, if the interval between checks are too small, the result might be 0. With this accumulation logic, which has been there but let's make it clear, this 0 result problem could be mitigated. (Bitbake rev: 95fa8fb5fb4d5a72e79b11d69792613bfd494e72) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: convert deferral messages from bb.note to bb.debugDenys Dmytriyenko2023-07-191-3/+3
| | | | | | | | | | | | | | | Using multiconfig to target baremetal pieces of the system and building corresponding toolchains for them results in hundreds and hundreds of "Deferring %s after %s" and "Deferred task %s now buildable". To clean up the output and to reduce risk of missing important warnings, convert these notice messages to debug messages. (Bitbake rev: 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318) Signed-off-by: Denys Dmytriyenko <denis@denix.org> Signed-off-by: Denys Dmytriyenko <denys@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add pressure change loggingRichard Purdie2023-07-191-0/+4
| | | | | | | | | | It is currently hard to tell when bitbake is throttling task execution due to system pressure changes. Add notes to the console output to make this clearer, only generating output when the values change. (Bitbake rev: a6056599922fb2fe3f54c5c86ac7ea604f469adc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix deferred task/multiconfig race issueRichard Purdie2023-06-281-4/+12
| | | | | | | | | | | | | | | If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix handling of virtual files in layername calculationRichard Purdie2023-06-201-2/+4
| | | | | | | | | | I'd tested target configurations but in real world use, it became clear the layername functionality in worker context was failing for virtual class extensions. Fix this. (Bitbake rev: 888778ebfa85677ff36065dfcd0f8a7684edfc80) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Pass hashfn in taskdep dataJoshua Watt2023-06-011-2/+4
| | | | | | | | | | | Include the hashfn (the value of BB_HASHFILENAME) in the task dependency data. This allows tasks to get a specific unique hash for dependent tasks when one is available. (Bitbake rev: 4dbecf6059e495246267b09d0f43086d51e6df2c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a ↵Richard Purdie2023-05-251-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | recipe There are times when it would be useful for code to know which layer (or collection in old bitbake terms) it is contained within. Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe so that it is possible to determine this. To do it, we need to pass data from the cooker into the recipe endpoints, since only the top level cooker information knows about the layer structure which makes the patch a bit painful. The idea is that this would make layer overrides possible: OVERRIDES .= ":layer-${FILE_LAYERNAME}" which then opens possibilities like: WARN_QA:append:layer-core = " patch-fuzz" as an example where OE-Core could enable specific QA tests only for that specific layer. (Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix PSI check calculationChen Qi2023-04-131-4/+9
| | | | | | | | | | | | | | | The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. (Bitbake rev: b4763c2c93e7494e0a27f5970c19c1aac66c228b) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/cooker/runqueue: Add ability to interrupt longer running codeRichard Purdie2023-02-231-0/+15
| | | | | | | | | | | | | | | | | | | | Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop SystemExit usageRichard Purdie2023-02-191-2/+1
| | | | | | | | | Using bb.fatal for a fatal error message is the best practise, switch the code to match other call sites. (Bitbake rev: c27e48fa81c2327a4a355a028884ab457cde3ae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: builtins fix for 'd' deletionRichard Purdie2022-12-291-1/+1
| | | | | | | | | | | | | | | | | | | I've been seeing event handlers where 'd' seems to disappear half way through event handler execution. This is problematic when multiple threads are active since this code assumes single threading. The easiest fix is to change the handler function calls to contain d as a parameter as we do elsewhere for other functions. This will break any non-text handlers but I was only able to spot one of those in runqueue. It will also break handlers than call functions that assume 'd' is in the global namespace but those failures should be obvious and we can fix those to pass d around. This solution avoids manipulating builtins which was always a horrible thing to do anyway and solves the issue without needing locking, thankfully. (Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-13 12:20:32 +0000