summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: BBHandler/cooker: Implement recipe and global classesRichard Purdie2022-08-127-22/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | We have some confusion for users since some classes are meant to work in the configuration space (or "globally") and some are meant to be selected by recipes individually. The cleanest way I could find to clarify this is to create "classes-global" and "classes-recipe" directories which contain the approproate classes and have bitbake switch scope between them at the appropriate point during parsing. The existing "classes" directory is always searched as a fallback. Once a class is moved to a specific directory, it will no longer be found in the incorrect context. A good example from OE is that INHERIT += "testimage" will no longer work but IMAGE_CLASSES += "testimage" will, which makes the global scope cleaner by only including it where it is useful and intended to be used (images). (Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add copyright headers where missingRichard Purdie2022-08-1228-0/+56
| | | | | | | | | Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Make inherit calls more directlyRichard Purdie2022-08-121-1/+4
| | | | | | | | | | | Rather than recursing into the conf handler code, simply call into the parse code directly when inheriting files as we've already resolved the paths and don't need anything the other codepath brings. This makes the codepath clearer at the expense of some slight duplication. (Bitbake rev: 0f4f3af6d93a0018df58b8a1d8d423c78ba6526d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Allow earlier exit for classes not foundRichard Purdie2022-08-121-0/+3
| | | | | | | | | Rather than relying on later code to error if the class isn't found, exit earlier and more clearly from a code perspective. (Bitbake rev: f7c55c8147329670fd5bc55b1ae3f47f25b89bab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: prefix the tasks with a timestamp in the log task_orderJose Quaresma2022-08-121-1/+3
| | | | | | | | | | This is useful when debugging as it helps understand possible race conditions between tasks of diferent recipes. (Bitbake rev: 950a2ea4c91d6e13d7587104367fa85cc7efe01c) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add memory pressure regulationAryaman Gupta2022-08-121-5/+22
| | | | | | | | | | | | | | | | | | | Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix insufficent entropy in sigtask file namesJoshua Watt2022-08-092-1/+22
| | | | | | | | | | | | | | | | | | | | | | | | | Signature generation uses mkstemp() to get a file descriptor to a unique file and then write the signature into it. However, the unique file name generation in glibc is based on the system timestamp, which means that with highly parallel builds it is more likely than one might expect expected that a conflict will occur between two different builder nodes. When operating over NFS (such as a shared sstate cache), this can cause race conditions and rare failures (particularly with NFS servers that may not correctly implement O_EXCL). The signature generation code is particularly susceptible to races since a single "sigtask." prefix used for all signatures from all tasks, which makes collision even more likely. To work around this, add an internal implementation of mkstemp() that adds additional truly random entropy to the file name to eliminate conflicts. (Bitbake rev: 97955f3c1c738aa4b4478a6ec10a08094ffc689d) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop deadlock breaking force failRichard Purdie2022-08-081-3/+0
| | | | | | | | | | | | | | I'm 99% certain this failing of a scenequeue task corrupts runqueue and causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted and did weird things so drop this code. We've seen builds where the deadlock triggers and it then tries to run tasks where the SQ task already ran with very confusing failures. It is likely it is this code causing it. (Bitbake rev: 8efced47fcb47851a370fd6786df6fb377f99963) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve deadlock warning messagesRichard Purdie2022-08-081-4/+6
| | | | | | | | | | Tweak the deadlock breaking messages to be explict about which task is blocked on which other task. The messages currently imply it is "freeing" the blocking task which is confusing. (Bitbake rev: cf7f60b83adaded180f6717cb4681edc1d65b66d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure deferred tasks are sorted by multiconfigRichard Purdie2022-08-081-5/+8
| | | | | | | | | | | | | We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. (Bitbake rev: 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: npm fetcher: improve description of SRC_URI formatMichael Opdenacker2022-08-041-2/+2
| | | | | | | | | | | | | Using the term "Parameter" which is consistent with the description of SRC_URI parameters in the following text. (Bitbake rev: 87e42f1202162152c779ccc8bbd06f88f0bdab96) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reported-by: Quentin Schulz <foss@0leil.net> Reviewed-by: Quentin Schulz <foss+yocto@0leil.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add cpu/io pressure regulationAryaman Gupta2022-07-281-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <randy.macleod@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: movefile: use the logger for printingJose Quaresma2022-07-281-6/+6
| | | | | | | | (Bitbake rev: 274b2d7a2fa0b43b0b542cb5471ff832e692ea93) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: remove: check the path again the expand python globJose Quaresma2022-07-281-2/+2
| | | | | | | | | | | | | When we call the remove with recurse=True we first check if the remove operation is safe in _check_unsafe_delete_path. But the check is been done on the path instaed of the expanded python glog. (Bitbake rev: 7236488b898309ec5f1880936ddae22a28ccf5d3) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add TCP Keep AlivesJoshua Watt2022-07-201-0/+7
| | | | | | | | | | | | | | | Adds TCP Keep Alive support to the async RPC server. This should help prevent file descriptor exhaustion on the server when client connections are interrupted and the socket never closes (e.g. no FIN is sent from the client). A keep alive is sent after 30 seconds of inactivity, then every 15 seconds after that up to a maximum of 2 minutes. (Bitbake rev: 68f4ce662cad28fed739900addbdee949ad3c1e8) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: bitbake-user-manual: hashserv can be accessed on a ↵Jose Quaresma2022-07-191-1/+1
| | | | | | | | | | dedicated domain (Bitbake rev: b171aa45fb8518dcfbba315b303a4fe9bf2180c6) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Drop DL_DIR fallback for local file fetcherRichard Purdie2022-07-153-18/+8
| | | | | | | | | | | | | | | | | | | | | | A long time ago, we made DL_DIR a final fallback for the local fetcher. Since then we added checksum support and task hashes and the world has changed. There were warnings added some time ago if this fallback triggers and it is now time to drop it entirely. The original use case was for sstate however the sstate code now sets FILESPATH correctly so DL_DIR is no longer needed. There have been a few small bugs exposed by this change, missing mkdir calls and some minor test issues that needed tweaks. In general this simplifies and improves the fetcher code flow though. This completes a cleanup that ensures local files are correctly covered at parse time which ensures rebuilds and reparses happen at the right times. (Bitbake rev: 3e1444e536c71d3885ef6b9d428807163c309640) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure directory exists before creating symlinkRichard Purdie2022-07-151-0/+2
| | | | | | | | | | | | If the mirrors code is trying to create a symlink and the parent directory doesn't exist, as might be the case for sstate mirrors where the fetch is into a subdir, it can silently fail. Ensure the directory exists in this case to avoid issues. (Bitbake rev: eff16e474ee7dc49ae433420a4c8d15d3314a618) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: bb.fatal when trying to checksum non-existing filesPaulo Neves2022-07-152-1/+8
| | | | | | | | | | | | If the local fetcher was not able to find the file anywhere but it was included in the SRC_URI for checksumming just make it a fatal error. Ensure a list of searched locations is included too to match the runtime error that would have resulted. (Bitbake rev: 5e3b2ad90d9cd0f248b1cb740637caa24442d101) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: show SRCREV and git repo in error message about fixed ↵Martin Jansa2022-07-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | SRCREV * in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to be one character shorter and because fetcher uses: if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]): to decide which SRCREV values are fixed SRCREVs this one was considered as tag or branch name, because it was only 39 chars long The original error message wasn't very helpful as it doesn't show which repo or which SRCREV was considered missing: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) with SRCPV included in PV as error recomments it's a bit better: bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16 The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T with this change the first error will read: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) (Bitbake rev: 04dc17bef9b762cef9eecdf91c9f37738d8ae44d) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler/BBHandler: Improve comment error messages and add testsRichard Purdie2022-07-043-4/+32
| | | | | | | | | | | | | | | | | | | Currently if you trigger one of the comment errors, the newline characters are stripped and the line numbers are incorrect. In one case it prints the empty line which is also unhelpful. Rework the code around these errors so the line numbers are correct and the lines in question are more clearly displayed complete with newlines so the user can more clearly see the error. I also added a couple of simplistic test cases to ensure that errors are raised by the two known comment format errors. [YOCTO #11904] (Bitbake rev: 712da71b24445c814d79a206ce26188def8fce0a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: gitsm: fix incorrect handling of git submodule relative urlsGennaro Iorio2022-07-011-1/+1
| | | | | | | | | | | As specified by git submodule manual relative urls can start either with '..' or './', second case was incorrectly managed leading to an interpretation of urls starting with './' as absoulte urls. (Bitbake rev: 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b) Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix logging issues where only the first message was ↵Richard Purdie2022-06-251-2/+5
| | | | | | | | | | | | | | | displayed I realised only the first logging message was being displayed in a given parsing process. The reason turned out to be the UI handler failing with a "pop from empty list". The default handler was then lost and no further messages were processed. Fix this by catching the exception correctly in the connection writer code. (Bitbake rev: d3e64f64525187f1409531a0bd99df576e627f7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: document npm and npmsw fetchersMichael Opdenacker2022-06-221-2/+74
| | | | | | | | | | This addresses bug [YOCTO #10098] (Bitbake rev: cca7999586317435d79bf53df4359cdd8bfadff4) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual: add explicit target for crates fetcherMichael Opdenacker2022-06-221-0/+2
| | | | | | | | | | Like in other sections describing fetchers (Bitbake rev: c9bab35f6aecbf85ee1a19a7b70e15a80b42471f) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Correct description of the ??= operatorJacob Kroon2022-06-211-16/+39
| | | | | | | | | | | | Stating that the assignment is done at the end of parsing is misleading. The weak default value is the value which a variable will expand to if no value has been assigned to it using any of the assignment operators. (Bitbake rev: 8189f58d0449d16f162b6e8d98c4e5edc6bff875) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler: Remove lingering closeOla x Nilsson2022-06-211-2/+0
| | | | | | | | | | | | The f.close() statement should have been removed in 459ad524756a3f9b50feeedf31e33502dceae8d5. (Bitbake rev: 9fc1bab6b7e3c0fca3ddec4bc8c7763d2aff8bab) Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add test for broken mirror tarballPavel Zhukov2022-06-211-0/+25
| | | | | | | | | | | | | | | With PREMIRRORS set and BB_NO_NETWORK = "1" bitbake should not try to fetch into non-initialized git directory if tarball is broken (or not in gzip format) [Yocto 14822] (Bitbake rev: c9aaca3dd2dfdf4a291d6e1f6263037e0f54b4b6) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event.py: ignore exceptions from stdout and sterr operations in atexitMikko Rapeli2022-06-111-2/+8
| | | | | | | | | | | | | | | | | When atexit functions run, stdout and stderr operations may fail, e.g. when output is piped to less but has been exited by the user. This removes error print from output of "bitbake -e sqlite3 | less" if user presses "q" before bitbake has finished processing: [Errno 32] Broken pipeError in atexit._run_exitfuncs: Traceback (most recent call last): File "/home/builder/src/poky/bitbake/lib/bb/event.py", line 135, in print_ui_queue sys.stdout.flush() (Bitbake rev: 35167536c163eb6b7653cbcaad9f65b834d3e2f8) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add tests for premirror using real projectPavel Zhukov2022-06-111-0/+39
| | | | | | | | | | | Existing test uses fake local repo for better performance. Adding test which uses real (but still small) project hosted on yoctoproject.org. (Bitbake rev: 247f3536a691cdaa3f96768d1b42653b1da9ae84) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Drop unnecessary duplicated functionPavel Zhukov2022-06-111-9/+0
| | | | | | | | | Function "git" is defined in super class FetcherTest already. (Bitbake rev: bfed4ad2d50bcd635bb2ef99977e03e879fc79a2) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Drop sre_constants usageRichard Purdie2022-06-111-3/+2
| | | | | | | | | | | | | | | | | As reported by Martin Jansa <Martin.Jansa@gmail.com>: bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated import sre_constants it's deprecated since 3.11 with: https://github.com/python/cpython/issues/91308 The correct replacement for our usage is re.error so use that instead. (Bitbake rev: 3c0cd401472ffee06d5a93bdba566cb033851fcf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Avoid tracebacks at exitRichard Purdie2022-06-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | In theory this should have been worked around but is still occurring. Add it to the list of things to ignore when bitbake is shutting down. Traceback (most recent call last): File "/usr/lib64/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 698, in startCallbackHandler event = self.reader.get() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 722, in get res = self.reader.recv_bytes() File "/usr/lib64/python3.9/multiprocessing/connection.py", line 221, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/lib64/python3.9/multiprocessing/connection.py", line 426, in _recv_bytes return self._recv(size) File "/usr/lib64/python3.9/multiprocessing/connection.py", line 384, in _recv chunk = read(handle, remaining) TypeError: an integer is required (got type NoneType)' (Bitbake rev: 7a28ac4fe478bee1e52e84412da9626495f9c6c7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Remove daemonic thread usageRichard Purdie2022-06-081-5/+9
| | | | | | | | | | | | | We're seeing UI deadlocks occasionally and this is possibly due to the use of a daemonic thread in the UI event queue processing. This thread could terminate holding a threading Lock() which would cause issues for the process when exitting. Change the shutdown process to handle this more cleanly. (Bitbake rev: f5ad8349a5dbff9824a89f5708cfd011d61888c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Avoid risk of exception deadlocksRichard Purdie2022-06-081-14/+9
| | | | | | | | | | | | | | The open coded lock acquire/release in the UI event handler doesn't cover the case an exception occurs and if one did, it could deadlock the code. Switch to use 'with' statements which would handle this possibility. We have seen deadlocks in the UI at exit this so this removes a possible cause. (Bitbake rev: bd12792f28efd2f03510653ec947ebf961315272) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Move files into place atomicallyRichard Purdie2022-06-081-4/+7
| | | | | | (Bitbake rev: cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add tests to cover BB_FETCH_PREMIRRORONLY functionalityPavel Zhukov2022-06-081-0/+63
| | | | | | | | | | | | Basic test to cover functionality of BB_FETCH_PREMIRRORONLY using local git repository. Local repository has been chosen to allow easy manipulation with the repo to simulate behaviour reported in [Yocto 13233] (Bitbake rev: 773e0815ba0c2183afcb169cda525b7625e60e42) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Honour BB_FETCH_PREMIRRORONLY optionPavel Zhukov2022-06-081-4/+9
| | | | | | | | | | | | | | | | | | This should fix [Yocto 13353] and related to [Yocto 13233] as well. Previously if git repo mirror has been updated in between of two builds fetcher of the second build didn't try updated mirror but switched to git clone from upstream instead. This is problem for offline builds. Fix this to raise MirrorException if BB_FETCH_PREMIRRORONLY has been specified by the mirror doesn't contain SRC_REV. [YOCTO #13353] [YOCTO #13233] (Bitbake rev: b47ecab3e3aad5c5c376ec023aa82a51aa0f3b86) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.0.1Richard Purdie2022-06-022-2/+2
| | | | | | | | This allows OE to depend on the unihash copy functionality. (Bitbake rev: 928aa3dc715d1d3be4b7a8d1247ea2ab5ee2d38e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/siggen: Add unihash cache copy functionRichard Purdie2022-06-022-0/+15
| | | | | | | | | | We see rare failures in eSDK generation with zero sized unihash cache files. This is almost certainly due to races in the cache file being updated. Add a copy function where the cache file can be copied with the lock held to avoid this. (Bitbake rev: 9e72a3915e36cb843037040cb68a82077436dbef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix unihash cache mismatch issuesRichard Purdie2022-06-021-0/+3
| | | | | | | | | | | | | | | Very occasionally we see errors in eSDK testing on the autobuilder where the task hashes in the eSDK don't match what was just built. I was able to inspect one of these build directories and noticed that the bb_unihashes.dat file in the eSDK was zero sized. Whilst inspecting the code to understand the cause, I noticed that updated hashes are not saved out in subsequent updates of the values in the rehash process. Add a missing sync call to ensure this happens. (Bitbake rev: 7912dabbcf444a3c3d971cca4a944a8b931e301b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: osc: fix DeprecationWarningMartin Jansa2022-05-221-1/+1
| | | | | | | | | | | * fixes: bitbake/lib/bb/fetch2/osc.py:93: DeprecationWarning: invalid escape sequence '\d'   match = re.match('<directory ?.* rev="(\d+)".*>', output) (Bitbake rev: 6a346df51b96a6c0e1ee516df36eb0b6c292b063) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/osc: Add support to query latest revisionGunjan Gupta2022-05-211-5/+29
| | | | | | | | | | Add support to query latest revision. This makes it possble to use osc fetcher without specifying the rev parameter. (Bitbake rev: aa4cee1bb7415c498e4dc6af4dbb3d0c841faf2e) Signed-off-by: Gunjan Gupta <viraniac@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Add clean_stamp API function to allow removal of task stampsRichard Purdie2022-05-161-5/+9
| | | | | | | | | | | | | | | We currently have no API to be able to remove all the potential stamps of a task. It is unusual to need to do this, particularly as you could race against other things happening in the system but we do have a use case for this in cleaning up sysroots in OE-Core. The alternative is to mess with CLEANMASK in OE-Core but that is just going to add potential for errors. We need the first part of the make_stamp() function so separate that out so it can be called seperately. (Bitbake rev: 4d671504a25863018ac51c21c005cef0a4d8f05c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Do not depend on vardepvalueexclude flagTomasz Dziendzielski2022-05-161-0/+2
| | | | | | | | | | | | | | If SRC_URI contains python function that extends vardepvalueexclude its value is being tracked by sstate-cache, which can lead to rebuilds if value is set dynamically (for example gerrit replicas). Return empty string if vardepvalueexclude is checked to fix this behaviour. (Bitbake rev: f5f9a7b89a7d8321f03184e61ad6d5ed8d0f840e) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/osc: Small fixes for osc fetcherGunjan Gupta2022-05-101-6/+10
| | | | | | | | | | | | | | | | | | | The current fetcher seemed to have some issues that made it difficult when trying to use the same. This patch fixes the following * Make consistent use of the path that needs to be used as oscdir * The path mentioned in os.access in download function was not same as ud.moddir which would result into invoking of fetch command instead of update command even if directory already existed * Before creating oscrc, make sure oscdir exists and create it if it does not exist * Updated the configuration to use apiurl and added a new parameter to control whether http or https needs to be used to connect to apiurl (Bitbake rev: 3ec78686f3c0ea2304097b86a965f9be4b0cb879) Signed-off-by: Gunjan Gupta <viraniac@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: break on first dependent task differenceSchmidt, Adriaan2022-05-101-0/+1
| | | | | | | | | | | | | | compare_sigfiles() recursively calculates differences on all dependent tasks with changed hashes. This is done in arbitrary/alphabetical order, and only the last of those results is returned, while everything else is discarded. This changes the behavior to instead return the first difference and not calculate any more, which significantly speeds up diffs of tasks with many dependencies. (Bitbake rev: ea6a676c9aa2864c2eff40eea41ba09ce903a651) Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git : Use cat as pagerRichard Purdie2022-05-081-1/+1
| | | | | | | | | We don't have less in HOSTTOOLS in OE and this can confuse git. Force the pager to cat to be consistent and minimal everywhere. (Bitbake rev: d3d406e8552fdd865dc58b419a84411736475ad2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: correctly handle file names containing colonsRoland Hieber2022-05-081-1/+1
| | | | | | | | | | | | | | | File names containing colons cause split() to return a list with more than two elements, which will lead to a stack trace ending in: ValueError: too many values to unpack (expected 2) Split only once at the last colon, thereby making sure that only two elements are returned. (Bitbake rev: a70a7376a8708bde07959deb5d5842d7f84ee5f8) Signed-off-by: Roland Hieber <rhi@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Change emphasis in error message to be clearer to usersRichard Purdie2022-05-051-1/+1
| | | | | | | | | Users are misreading this message. Reorder the wording to make the empahsis clearer on the shell environment. (Bitbake rev: dd2475e36af470ea47b0841d181444e36aa6e947) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-26 07:46:29 +0000