summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/runqueue.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Refactor StaleSetSceneTasks event out of ↵Richard Purdie2023-11-161-18/+18
| | | | | | | | | | | | | | build_scenequeue_data The event at the end of the function doesn't really belong there, move it out so the function becomes cleaner. It also avoids having to pass a number of parameters around, another sign this is the correct thing to do. This continues refactoring started a while ago to improve and modularise the code. (Bitbake rev: f712b927a84e8b8deaa8c907e8f9bcdec681f2aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Move 'cantskip' into sqdataRichard Purdie2023-11-161-7/+7
| | | | | | | | | Conceptually, this belongs in sqdata so this is the next step in refactoring this code, continuing what was started a while ago. (Bitbake rev: 2cf267cb657cd1864db43955cc010abba7a64093) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix errors when using -S printdiffRichard Purdie2023-11-091-2/+2
| | | | | | | | | | | | | | | | | When running printdiff, messages like: "Can't find a task we're supposed to have written out?" might be shown. This happens when hashequivalence is enabled and is due to the use of "hash" when "unihash" should be used (which defaults back to hash). Changing this fixed various errors oe-selftest highlighted. Also print the task ID when erroring about a missing task as it aids debugging. (Bitbake rev: e7783bbe0559abb91f5038cf83575c0f25138af2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall option for setscene tasksRichard Purdie2023-11-091-0/+1
| | | | | | | | | | | If --runall is used with setscene tasks, it will still skip them if it can. By marking the tasks as targets, this avoids that skipping and means --runall deploy_source_date_epoch works as expected for example. (Bitbake rev: 8c6cd1e8dd224bc1cc92736cbb7e113fd1fb5aae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: add header with length of messageEtienne Cordonnier2023-11-061-12/+22
| | | | | | | | | | | | | | | | | | | | | | | | | The IPC mechanism between runqueue.py and bitbake-worker is currently not scalable: The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker has no information about the size of the message. Therefore, the bitbake-worker is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")" for each chunk received. This does not scale, because queue.find has a linear complexity relative to the size of the queue, and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI. The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup 35 seconds before this fix, and 1.5 seconds after this fix). This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be received, and does not need to constantly search the whole queue for </tag>. (Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: set has 'add', not 'append' methodChris Laplante2023-10-231-1/+1
| | | | | | | | | | Discovered via pylint (Bitbake rev: 1a353cda696b7f59386ad2d78a57005b90a37da4) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: clarify that 'closest' signature means 'most recent' ↵Alexander Kanavin2023-10-231-1/+1
| | | | | | | | | | | | | | | | | (and not closest in its content) This is printed by 'bitbake -S printdiff' and more accurately reflects what the code does. Most of the time the most recent item should be what the user wants to see when debugging sstate misses, but there could also be printdiff-all (print differences with all matching sstate objects), or printdiff-N (N most recent, and not just the latest). (Bitbake rev: 315f48f965a54da5cec92908d91aa61c2d450add) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/lib: spawn server/worker using the current Python interpreterRoss Burton2023-09-261-2/+4
| | | | | | | | | | | | | | | | | The user may have invoked ./bin/bitbake using a different Python interpreter than whatever python3 is on $PATH (for example, explicitly using a different version). However, as the server and workers are spawned directly they'll use the hashbang and thus a different Python. We also ensure that argv[0] is set to sys.executable instead of 'bitbake-server' or 'bitbake-worker', so that sys.executable is set to the right value inside the child. Without this the server won't be able to start any workers. (Bitbake rev: b44d5d2a53d3082c8ce94e09c0cf833e33e25aec) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker/runqueue: Avoid unnecessary bytes object copiesEtienne Cordonnier2023-09-221-2/+2
| | | | | | | | | | | | | | | | | | declaring queue=b"" creates an object of types bytes(). bytes() is an immutable object, and therefore doing "self.queue = self.queue + r" creates a new object containing "self.queue" concatenated with "r". On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker, so those copies significantly slow down the initialization of the bitbake-worker. Rather use bytearray() which a mutable type, and use extend() to avoid copies. In my test setup, byterray.extend() is 10.000 times faster than copying the queue, for a queue size of 180MB. (Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: show number of currently running bitbake threads when ↵Martin Jansa2023-09-051-1/+1
| | | | | | | | | | | | | | | | | | pressure changes * it might be a bit confusing as it shows number of threads before making the decision to start more tasks and also it can show only a few tasks running, but not because of pressure when there just aren't many tasks left or wait for their dependencies to be finished first * example output: NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads (Bitbake rev: b0b114f31f20c5fcde31e6c308937ad4102dfe0a) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: show more pressure dataMartin Jansa2023-08-241-8/+10
| | | | | | | | | | | | | | * with latest bitbake I'm seeing very low number of bitbake tasks executed in parallel, probably due to pressure regulation show the values this is based on in the note * also simplify a bit by counting the pressure and exceeds signs only once (Bitbake rev: 21c17968f801e406ef7f328656587fadd9ef7f5d) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: fix PSI check logicChen Qi2023-08-111-4/+5
| | | | | | | | | | | | | | | | | | The current calculation is not correct because if tdiff is less than 1.0, it's not taken into consideration when calculating the current pressure. Also, make it clear that the 1.0s is the psi accumulation cycle, which might be changed in the future. We have this cycle because it could largely avoid the 0 result issue, that is, if the interval between checks are too small, the result might be 0. With this accumulation logic, which has been there but let's make it clear, this 0 result problem could be mitigated. (Bitbake rev: 95fa8fb5fb4d5a72e79b11d69792613bfd494e72) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: convert deferral messages from bb.note to bb.debugDenys Dmytriyenko2023-07-191-3/+3
| | | | | | | | | | | | | | | Using multiconfig to target baremetal pieces of the system and building corresponding toolchains for them results in hundreds and hundreds of "Deferring %s after %s" and "Deferred task %s now buildable". To clean up the output and to reduce risk of missing important warnings, convert these notice messages to debug messages. (Bitbake rev: 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318) Signed-off-by: Denys Dmytriyenko <denis@denix.org> Signed-off-by: Denys Dmytriyenko <denys@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add pressure change loggingRichard Purdie2023-07-191-0/+4
| | | | | | | | | | It is currently hard to tell when bitbake is throttling task execution due to system pressure changes. Add notes to the console output to make this clearer, only generating output when the values change. (Bitbake rev: a6056599922fb2fe3f54c5c86ac7ea604f469adc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix deferred task/multiconfig race issueRichard Purdie2023-06-281-4/+12
| | | | | | | | | | | | | | | If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix handling of virtual files in layername calculationRichard Purdie2023-06-201-2/+4
| | | | | | | | | | I'd tested target configurations but in real world use, it became clear the layername functionality in worker context was failing for virtual class extensions. Fix this. (Bitbake rev: 888778ebfa85677ff36065dfcd0f8a7684edfc80) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Pass hashfn in taskdep dataJoshua Watt2023-06-011-2/+4
| | | | | | | | | | | Include the hashfn (the value of BB_HASHFILENAME) in the task dependency data. This allows tasks to get a specific unique hash for dependent tasks when one is available. (Bitbake rev: 4dbecf6059e495246267b09d0f43086d51e6df2c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a ↵Richard Purdie2023-05-251-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | recipe There are times when it would be useful for code to know which layer (or collection in old bitbake terms) it is contained within. Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe so that it is possible to determine this. To do it, we need to pass data from the cooker into the recipe endpoints, since only the top level cooker information knows about the layer structure which makes the patch a bit painful. The idea is that this would make layer overrides possible: OVERRIDES .= ":layer-${FILE_LAYERNAME}" which then opens possibilities like: WARN_QA:append:layer-core = " patch-fuzz" as an example where OE-Core could enable specific QA tests only for that specific layer. (Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix PSI check calculationChen Qi2023-04-131-4/+9
| | | | | | | | | | | | | | | The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. (Bitbake rev: b4763c2c93e7494e0a27f5970c19c1aac66c228b) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/cooker/runqueue: Add ability to interrupt longer running codeRichard Purdie2023-02-231-0/+15
| | | | | | | | | | | | | | | | | | | | Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop SystemExit usageRichard Purdie2023-02-191-2/+1
| | | | | | | | | Using bb.fatal for a fatal error message is the best practise, switch the code to match other call sites. (Bitbake rev: c27e48fa81c2327a4a355a028884ab457cde3ae7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: builtins fix for 'd' deletionRichard Purdie2022-12-291-1/+1
| | | | | | | | | | | | | | | | | | | I've been seeing event handlers where 'd' seems to disappear half way through event handler execution. This is problematic when multiple threads are active since this code assumes single threading. The easiest fix is to change the handler function calls to contain d as a parameter as we do elsewhere for other functions. This will break any non-text handlers but I was only able to spot one of those in runqueue. It will also break handlers than call functions that assume 'd' is in the global namespace but those failures should be obvious and we can fix those to pass d around. This solution avoids manipulating builtins which was always a horrible thing to do anyway and solves the issue without needing locking, thankfully. (Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen: Rework stamps functionsRichard Purdie2022-12-171-12/+13
| | | | | | | | | | | | | | | The current method of passing either a task's datastore, or dataCaches and a filename into the stamp functions is rather horrible. Due to the different contexts, fixing this is hard but we do control the bitbake side of the API usage so we can migrate those to use other functions and then only support a datastore in the public bb.build API which is only called from task context in OE-Core. (Bitbake rev: c79ecec580e4c2a141ae483ec0f6448f70593dcf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: siggen/runqueue: Switch to using RECIPE_SIGGEN_INFO ↵Richard Purdie2022-12-171-13/+13
| | | | | | | | | | | | | | | | | | | feature for signature dumping Now that we have cache support for the taskdep/gendep/lookupcache data, we can switch to use that cooker feature and skip the secondary reparse to write the sig files. This does make the initial parse longer but means the secondary one isn't needed. At present parsing with the larger cache isn't optimal but we have plans in place which will make this faster than the current reparse code being removed here. (Bitbake rev: 5951b5b56449855bc2a30146af65eb287a35fcef) (Bitbake rev: 1252e5bce51ae912ecff9dcc354a371786ff2c72) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Directly store datacaches referenceRichard Purdie2022-12-171-0/+2
| | | | | | | | | | | It is becomming clear the siggen needs access to our cache data but we can't always obtain it in the contexts we need to. Add it directly, meaning over time we should be able to simplify the APIs and stop convoluting new ones! (Bitbake rev: 6b213590ed0e77683cf7fbce6bbe9605ddecf3d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen/runqueue: Drop do_setscene referencesRichard Purdie2022-12-111-1/+1
| | | | | | | | | | do_setscene was from a different era before our modern setscene per task code. It hasn't been used for years so remove some old obsolete references to it. (Bitbake rev: ef72282298f7c4db74383c23bb0251dd06d3c6d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Drop non-multiconfig aware siggen supportRichard Purdie2022-12-111-5/+3
| | | | | | | | | All siggens in common use should now support multiconfig, drop the compatibility code. (Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve error message for missing multiconfigRichard Purdie2022-11-291-0/+2
| | | | | | | | | | | | If you place a multiconfig which isn't enabled into an mcdepends you currently get a traceback from runqueue. We can do better, add some code to tell the user what happened in a more readable way without the traceback. [YOCTO #14970] (Bitbake rev: a4693b70764bb394ee2cf8dd12a5f6fce866008b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix race issues around hash equivalence and sstate reuseRichard Purdie2022-11-221-16/+20
| | | | | | | | | | | | | | | | | | | | | | | | | We identified a use case where a native recipe (autoconf-native) was rebuilt with no change in output yet the sstate for do_package tasks wasn't being used. The issue is that do_package tasks have a hard dependency on pseudo-native:do_populate_sysroot. That task was one of the many tasks being rehashed when autoconf-native's hash was changed. If update_tasks processed a recipe before it had processed pseudo-native, that recipe would be marked as not possible from sstate and would run the full tasks. The fix is to split the processing into two passes, first to handle the existing covered/notcovered updates, then in the second pass, check whether there are "harddep" issues. This defers the do_package tasks until after pseudo-native is installed from sstate as expected and everything works well again. (Bitbake rev: e479d1e418a7d34f0a4663b4a0e22bb11503c8ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add further debug for sstate reuse issuesRichard Purdie2022-11-221-0/+4
| | | | | | | | | | | | Even after enabling all our debugging, we had a reproducible test case where sstate wasn't being reused and it was unclear from the logs why. This patch adds debugging on the possible codepaths that were breaking and allowed the issue to be debugged and fixed. (Bitbake rev: 9233ad685b9b5e9eeb775fc71761712aaf0e876c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: worker/runqueue: Reduce initial data transfer in workerdataRichard Purdie2022-11-201-16/+36
| | | | | | | | | | | | | | | | | When setting up the worker we were transfering large amounts of data which aren't needed until task execution time. Defer the fakeroot and taskdeps data until they're needed for a specific task. This will duplicate some information when executing different tasks for a given recipe but as is is spread over the build run, it shouldn't be an issue overall. Also take the opportunity to clean up the silly length argument lists that were being passed around at the expense of extra dictionary keys. (Bitbake rev: 3a82acdcf40bdccd933c4dcef3d7e480f0d7ad3a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/cookerdata: Move recipe parsing functions from cache to ↵Richard Purdie2022-11-201-2/+1
| | | | | | | | | | | | | | | databuilder When 'NoCache' was written, databuilder/cookerdata didn't exist. It does now and the recipe parsing functionality contained in NoCache clearly belongs there, it isn't a cache function. Move those functions, renaming to match the style in databuilder but otherwise not changing functionality for now. Fix up the callers to match (which make it clear this is the right move). (Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: fix a typoOliver Lang2022-10-251-1/+1
| | | | | | | (Bitbake rev: 82f40261a06d39f0e7748942f480da5b44282fa3) Signed-off-by: Oliver Lang <quantenkeks@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change pressure file warning to a noteRichard Purdie2022-08-241-1/+1
| | | | | | | | | The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: 9bdedc8074990e613c9567e2cd8072f8d885f07f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add memory pressure regulationAryaman Gupta2022-08-121-5/+22
| | | | | | | | | | | | | | | | | | | Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop deadlock breaking force failRichard Purdie2022-08-081-3/+0
| | | | | | | | | | | | | | I'm 99% certain this failing of a scenequeue task corrupts runqueue and causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted and did weird things so drop this code. We've seen builds where the deadlock triggers and it then tries to run tasks where the SQ task already ran with very confusing failures. It is likely it is this code causing it. (Bitbake rev: 8efced47fcb47851a370fd6786df6fb377f99963) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve deadlock warning messagesRichard Purdie2022-08-081-4/+6
| | | | | | | | | | Tweak the deadlock breaking messages to be explict about which task is blocked on which other task. The messages currently imply it is "freeing" the blocking task which is confusing. (Bitbake rev: cf7f60b83adaded180f6717cb4681edc1d65b66d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure deferred tasks are sorted by multiconfigRichard Purdie2022-08-081-5/+8
| | | | | | | | | | | | | We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. (Bitbake rev: 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add cpu/io pressure regulationAryaman Gupta2022-07-281-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <randy.macleod@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix unihash cache mismatch issuesRichard Purdie2022-06-021-0/+3
| | | | | | | | | | | | | | | Very occasionally we see errors in eSDK testing on the autobuilder where the task hashes in the eSDK don't match what was just built. I was able to inspect one of these build directories and noticed that the bb_unihashes.dat file in the eSDK was zero sized. Whilst inspecting the code to understand the cause, I noticed that updated hashes are not saved out in subsequent updates of the values in the rehash process. Add a missing sync call to ensure this happens. (Bitbake rev: 7912dabbcf444a3c3d971cca4a944a8b931e301b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sig file location when using multiconfigRichard Purdie2022-04-301-1/+1
| | | | | | | | | | | | We're using the wrong data store when trying to locate siginfo files, fix this. Thanks to Gregory Lumen <gregorylumen@microsoft.com> for spotting. [YOCTO #14774] (Bitbake rev: 0ed800e19a3197f8e622c8d3b630aae384e60aba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop pointless variable assignmentRichard Purdie2022-04-211-1/+0
| | | | | | | | | This is set at the start of the loop anyway so it does nothing. Drop the pointless code. (Bitbake rev: e6a3173c9cdf349ccbd4cf612868f92cce8717c8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Replace "abort" usage in task handlingScott Murray2022-02-211-6/+6
| | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language replace the use of "abort" with "halt" in code related to handling task failure. (Bitbake rev: 831fb7f2329a3cd95b71e9c85d7d7f0d717f947f) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename allowed multiple provider variableScott Murray2022-02-211-2/+2
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: MULTI_PROVIDER_WHITELIST -> BB_MULTI_PROVIDER_ALLOWED (Bitbake rev: a09546b725fda13c0279638c7c904110da7bf6cd) (Bitbake rev: d035435c1a4951a45481867cf932faa4a6f8f936) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename setscene enforce filtering variableScott Murray2022-02-211-17/+17
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_SETSCENE_ENFORCE_WHITELIST -> BB_SETSCENE_ENFORCE_IGNORE_TASKS (Bitbake rev: 2e243ac06581c4de8c6e697dfba460ca017d067c) (Bitbake rev: f8f7b80a0df4646247e58238a52a7d85a37116d4) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop BB_STAMP_POLICY/BB_STAMP_WHITELISTRichard Purdie2022-01-121-22/+1
| | | | | | | | | | | | The different stamp policies were poor versions of the siggen code and task hashes, predating it and being used by packaged staging. They had many limitations, hence their replacement. I'm not aware of any users of that code any more so I believe it and the assoicated stamp whitelist variable can simply be removed. (Bitbake rev: 98407efc8c670abd71d3fa88ec3776ee9b5c38f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall option handlingRichard Purdie2021-11-081-0/+1
| | | | | | | | | | The previous fix for runall option handling had a small bug in it, it didn't clear the originally processed task list which meant it was running too many tasks. Fix this so the list is reset and rebuild correctly. (Bitbake rev: 87c9e120897ed04dfc64d4752fc602f9bfcb8645) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall option task deletion ordering issueRichard Purdie2021-11-051-19/+16
| | | | | | | | | | | | | | | | The runbuild option handling in runqueue was flawed as items deleted from the main task list may be dependencies and hence cause index errors. Rather than modify runtaskentries straight away, compute a new shorted list and use that as an input to the second phase. This avoids the need to add tasks back to the list meaning delcount can be simplifed to a simple counter. The second use case in runonly doen't re-add items so doesn't have this issue. (Bitbake rev: 3428e3c54eb5cc03ff96f9cee6dc839afee7a419) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Fix string concatination potential performance issuesRichard Purdie2021-11-031-18/+19
| | | | | | | | | | | | | | | | Python scales badly when concatinating strings in loops. Most of these references aren't problematic but at least one (in data.py) is probably a performance issue as the issue is compounded as strings become large. The way to handle this in python is to create lists which don't reconstruct all the objects when appending to them. We may as well fix all the references since it stops them being copy/pasted into something problematic in the future. This patch was based on issues highligthted by a report from AWS Codeguru. (Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Clean up use of len()Richard Purdie2021-11-031-33/+33
| | | | | | (Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-16 09:44:03 +0000