summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: utils: Fix lockfile path length issuesRichard Purdie2022-11-221-3/+6
| | | | | | | | | | | | | | | | If the path to bitbake.lock is in a deep directory, bitbake will hang. The reason was that the max file length limiting code (to 255 chars) was including the directory name and it should only act on the filename within the directory. Fix it to just use the base filename. [YOCTO #14766] (Bitbake rev: e3db9c2e9eded3c5cb6040714a6054b44f6b3880) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 89d70e7b71eecfe06592202f326e566c579ba01d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Handle lockfile filenames that are too long for filesystemsRichard Purdie2022-11-221-1/+5
| | | | | | | | | | | | | | | | The fetcher mirror code can go crazy creating lock filenames which exceed the filesystem limits. When this happens, the code will loop/hang. Handle the filename too long exception correctly but also truncate lockfile lengths to under 256 since the worst case situation is lockfile overlap and lack of parallelism. (Bitbake rev: 30d42ef030d03e11322b6b05ea7bbb64ab3d6f21) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 63baf3440b16e41ac6601de21ced94a94bdf1509) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie2022-10-271-2/+2
| | | | | | | | | CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 8ea8e443005ad92f4ad264d9abd9e90e33fb5c17) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Pass lock argument in fileslockedJoshua Watt2022-09-021-2/+4
| | | | | | | | | | | | Pass additional arguments in the fileslocked() context manager to the underlying lockfile() function. This allows the context manager to be used for any types of locks (non-blocking, shared, etc.) that the lockfile() function supports. (Bitbake rev: 048d682b031644fb9f0d41a489bacb873aa27bd7) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change pressure file warning to a noteRichard Purdie2022-08-241-1/+1
| | | | | | | | | The user does need to be told about this but it isn't really a warning, just something they may need to be aware of. Drop the level accordingly. (Bitbake rev: 3b719e8e115b7fde869f62ddc180e045c1b51cdf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: runqueue: add memory pressure regulationAryaman Gupta2022-08-231-5/+22
| | | | | | | | | | | | | | | | | | | | | Prevent new tasks from being scheduled if the memory pressure is above a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY" variable in the conf/local.conf file. This is an extension to the following commit and hence regulates pressure in the same way: 48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. This is why memory pressure is rarely seen but might be useful as a last resort to prevent OOM errors. (Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c) (Bitbake rev: 82b683f8c7a559f4fcab68f6a0fa7dc3dc20fa05) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: runqueue: add cpu/io pressure regulationAryaman Gupta2022-08-231-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. >From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) (Bitbake rev: 66741d216e9d4343e82a94f00cd39751632a5b96) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <randy.macleod@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Update user-agentChristophe Priouzeau2022-08-221-3/+8
| | | | | | | | | | | | | | | With the usage of enterprise proxy, the user-agent defined are too old and refused by proxy configuration. Updating to something more modern is desirable. (Bitbake rev: 17be38290d1e971cd89785e6bf44caef0a6416f8) Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com> Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Move files into place atomicallyRichard Purdie2022-07-151-4/+7
| | | | | | | | | (Bitbake rev: 7fc4cffebf5dcc1d050416c0b7f7d58c765c1d69) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Fix usehead for non-default namesJoey Degges2022-07-152-1/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: a247f56df680382d62910bb9a174e0fdd29e4ca8) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: reduce keep-alive timeout from 5000s (83 minutes) to 10 minutesRoss Burton2022-06-261-3/+4
| | | | | | | | | | | | | | | | | The keep alive timeout is excessively long at 83 minutes (5000 seconds), reduce this to 10 minutes: this should be long enough that it rarely triggers in normal builds, but when it does it has useful information. (Bitbake rev: a496a8952d8542ce814b13f460811d8849d25a3c) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit dcf52157d3635925491783be656c6b76d1efe1a4) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: display active tasks when printing keepAlive() messageRoss Burton2022-06-261-1/+3
| | | | | | | | | | | | | | | | | | | | | | | In interactive bitbake sessions it is obvious what tasks are running when one of them hangs or otherwise takes a long time. However, in non-interactive sessions (such as automated builds) bitbake just prints a message saying that it is "still alive" with no clues as to what tasks are active still. By simply listing the active tasks when printing the keep alive message, we don't need to parse the bitbake log to identify which of the tasks is still active and has presumably hung. (Bitbake rev: 36fe8bae9fec61547ee0b13bcb721033afd3ac0e) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 30f6c3f175617beea8e8bb75dcf255611e3fc2fd) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bin/bitbake-getvar: Add a new command to query a variable value ↵Richard Purdie2022-06-261-0/+48
| | | | | | | | | | | | | | | | | | | | | | (with history) We've talked about having this for long enough. Add a command which queries a single variable value with history. This saves "bitbake -e | grep" and avoids the various pitfalls that has. It also provides a neat example of using tinfoil to make such a query. Parameters to limit the output to just the value, to limit to a variable flag and to not expand the output are provided. [YOCTO #10748] (Bitbake rev: 47ed06d441152f8b6d374cacfac2c668c354423e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4c1881b620e885f55d7772f8626b8a76c2828333) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil/data_smart: Allow variable history emit() to function remotelyRichard Purdie2022-06-262-0/+16
| | | | | | | | | | | | | We can't access the emit() function of varhistory currently as the datastore parameter isn't handled correctly, nor is the output stream. Add a custom wrapper for this function which handles the two details correctly. (Bitbake rev: 144a1cfe8b60c677bb6ec66c242e064c7ba3ed88) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ba0fa084ccd2b1ade96425d158fd31e49e42f286) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Disable gc around critical sectionRichard Purdie2022-04-161-0/+3
| | | | | | | | | | | | | | | | The python gc can trigger whilst we're holding the event stream lock and when cleaning up objects, they can trigger warnings. This translates into a new event which would then need the lock and we can deadlock. Disable gc whilst we hold that lock to avoid this unfortunate and problematic situation. (Bitbake rev: 0784db7dd0fef6f0621ad8d74372f44e87fef950) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 96a6303949cefd469bcf5ed250ff512271354357) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add check for empty SRC_URI hash stringScott Weaver2022-04-051-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | No error was being reported when the hash string was set to empty. For example: SRC_URI[md5sum] = "" On a related note (not a bug): Because whitespace in the string will result in a checksum mismatch, the error message was updated to make it a little clearer why the error was thrown. For example: SRC_URI[md5sum] = " " or SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1" Now creates a message like this: File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5 checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected [YOCTO #14232] (Bitbake rev: d8c2175f6a7b5fdf111d6a073b2c3dbd3c0b061d) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a13510d0028e234ea2f4744b0d0c38558395c70f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: Allow run_command not to wait on eventsRichard Purdie2022-03-281-2/+2
| | | | | | | | | | | | | There are some commands where we want to see the events returned so allow the caller to request this. This also allows us to fix an infamous bug in the tinfoil testsuite in OE-Core. (Bitbake rev: 41bf1fa85a540232dcf92fe473c3b3c4cd7259dd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0e8421c41d97d5d50a553d70c8f775d521f1a199) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Note when commands complete in logsRichard Purdie2022-03-281-0/+1
| | | | | | | | | | | | | Its hard to tell from the server logs whether commands complete or not (or how long they take). Add extra info to allow more debugging of server timeouts. (Bitbake rev: d388f6d159b9d7e1ed3f199f2d1aca0c473cda6d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 56285ada585ec1481449522282b335bcb5a2671e) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Handle upstream master -> main branch changeRichard Purdie2022-02-231-1/+1
| | | | | | (Bitbake rev: d22cc1e587c650fd5f90cda32f5720f8a3105aac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: specify loop for asyncio in python < 3.6Jate Sujjavanich2022-01-251-6/+17
| | | | | | | | | | | | | | | | | [YOCTO #14697] Detect python version 3.5 restoring loop argument where it is still required. In 3.6 auto loop detection is available. Bitbake 1.46 is used in dunfell which lists a minimum python version of 3.5. Omitting this argument leads to a regression and hang during "Initialising tasks" at 44%. (Bitbake rev: be6ecc160ac4a8d9715257b9b955363cecc081ea) Signed-off-by: Jate Sujjavanich <jatedev@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Drop gnu urls from wget connectivity testRichard Purdie2022-01-171-3/+0
| | | | | | | | | | | | These urls are no longer adding much to the test coverage but the intermittent network issues connecting to them are painful. Drop the urls. (Bitbake rev: e6d75e0beb95aa0cdf82bbc0a6b767c7f6cfcfc0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit bdf5739c5d831dc97a7d81568f94a0953c71017f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Update to use exec_module() instead of load_module()Richard Purdie2022-01-171-2/+5
| | | | | | | | | | | | This is deprecated in python 3.12 and Fedora 35 is throwing warnings so move to the new functions. (Bitbake rev: e84c9aa77b61e48f1507edcba7bde65295f4d3ad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 68a18fbcb5959e334cf307d7fa8dc63832edb942) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Add a dummy event for tinfoil testingRichard Purdie2022-01-072-0/+15
| | | | | | | | | | | | | We need a command genetating an event to test through the tinfoil API. The current test has IO load issues so add a dummy version which won't have the IO constraints. (Bitbake rev: c5969eedd035648e3258bade386cc67ce3bb0e03) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a144178584394ea0700ffc2f2bfac94e8690effc) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bblayers/action: When adding layers, catch BBHandledExceptionRichard Purdie2021-12-051-2/+2
| | | | | | | | | | | | | | When adding a layer, parse error can occur, raising BBHandledException. Catch this and error, aborting the layer add to meet user expectations. [YOCTO #14054] (Bitbake rev: f18b65d0b9a6b983d53bde491e1bf2ca56949444) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ceddb5b3d229b83c172656053cd29aeb521fcce0) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure reparsing is handled correctlyRichard Purdie2021-12-051-0/+2
| | | | | | | | | | | | | | | | | | >From tinfoil, if you edit bblayers.conf and break it, then call parseConfiguration (e.g. by adding a bad layer with bitbake-layers), the system doens't show any parse error yet it should. Add in a call to the updateCache function so that things really are reparsed when requested. Partially fixes [YOCTO #14054] (Bitbake rev: a6ad6eaf4cddbe45077d1e6cb667f71602b8a49d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e655f9361b9c3b77906b8e06b5cc76bc5180640e) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process/knotty: Improve early exception handlingRichard Purdie2021-12-022-5/+23
| | | | | | | | | | | | | | | | | | | | | | | The new server startup code means exceptions can happen when we aren't setup to show them to the user correctly, leading to ugly tracebacks. Add in some special case handling of BBHandledException to at least ensure that common case doesn't traceback and the user sees meaningful output. In the future, the logging setup can likely be improved, as can the way runCommand handles exceptions, they all should likely become real exceptions again on the UI side. [YOCTO #14022] [YOCTO #14033] (Bitbake rev: 7fdd43c5cbde38daa013076de2fdedcf3c3d3107) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6059d0e77f60ddb679049bd34478f41b1ab7995d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: When sending commands we need to process eventsRichard Purdie2021-12-021-1/+10
| | | | | | | | | | | | | | | | The server may be displaying useful information for the user through log messages so we should display anything that has been sent. Its either this or expecting every UI to implement this code around every command call which isn't good API. [YOCTO #14054] (Bitbake rev: f20da5247dea524e837c5b6fdeccc79cbafedf90) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 64ae9d7e2fad804dd9e12706c6d76b4b22f9586b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: Ensure exceptions inheriting from BBHandledException are ↵Richard Purdie2021-12-021-1/+5
| | | | | | | | | | | | | | | | | visible Previous changes allowed BBHandledException to be detected but not exceptions which inherit from it. Fix this. The code really needs totally reworking to preserve the exceptions. [YOCTO #14054] (Bitbake rev: 80348b68a34b7ec45a0496a4af7f2ae0c26488f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ef762d92df6c2554c6248e80212f984d9ec4c651) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Add timeout for checkstatus calls (30s)Richard Purdie2021-11-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 30s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] (Bitbake rev: c0348de8121c3a842bf44906f7e2f79e93f7275b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update pcre.org address after github changesRichard Purdie2021-11-091-1/+1
| | | | | | | | | | | | | vcs.pcre.org was a redirect to github which we use for subversion testing. With the protocol changes at github and the removal of the redirect, use a direct address for github. (Bitbake rev: fa471399d41efdf61e95e0be541b45f0621756f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6230ca71eb7eb2a6db162e28a01727d00af5299b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update github urlsRichard Purdie2021-11-081-13/+13
| | | | | | | | | (Bitbake rev: f1a3e9d22b5f4fb01c4a0e4ba03afb1afbba47f1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 07fca7e3ab696ba985b3ef86ab9031d688bf2df2) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch2: Fix quoting warningRichard Purdie2021-11-081-6/+6
| | | | | | | | | | | | | | | Fix: lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence for several lines of the fetch tests. (Bitbake rev: bd8883d756328ca4c8f6bf97f77e17133a6bfb45) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9d84fd557a3fcbae2cdd70b24e69325ad737a01e) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Handle github dropping git:// supportRichard Purdie2021-11-031-0/+4
| | | | | | | | | github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. (Bitbake rev: c222eddcebe892ae209aea7776cfc1147ac1df6e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use os.rename instead of mvRichard Purdie2021-10-231-4/+4
| | | | | | | | | | | | os.rename will overwrite the destination file if present so we can use this instead of the process call overhead. (Bitbake rev: c5b8a2fce98c362ea77d74a8bc472d01b739a98a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b3cccaa6a896c41d8c9be5eebc327f726542d16b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Avoid races over mirror tarball creationRichard Purdie2021-10-231-2/+19
| | | | | | | | | | | | | | | | | There is a potential race over the mirror tarballs where a partial git repo could be extracted causing fetcher failures if the tarball is being rewritten whilst another build accesses it. Create the mirror tarball atomically to avoid this. [YOCTO #14441] (Bitbake rev: e3da0ecbd282da060b52a4bcf3ed36497295fde0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3250bc950c56bd7dd2114df26e5a8e13b04ceac8) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: let asyncio discover the running loopJustin Bronder2021-10-231-2/+2
| | | | | | | | | | | | | | | | | | | >From 3.10 documentation [1]: Deprecated since version 3.8, removed in version 3.10: The loop parameter. This function has been implicitly getting the current running loop since 3.7 This is fixed in master as a side-effect of cf9bc0310b0092bf52b61057405aeb51c86ba137 which is more intrusive but likewise drops the loop parameter. 1. https://docs.python.org/3/library/asyncio-stream.html#asyncio.open_connection (Bitbake rev: 74a1e71b1e677a482fdedc685a71a1798ad63920) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: correct deprecation warning in process.pyAlexander Kanavin2021-10-231-1/+1
| | | | | | | | | | | (Bitbake rev: 1c422329c3b88a91a6faeacbba3b6104d66ba7a1) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit aff52fe21a0b27f6302555c1e52a864550eb46ce) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: adjust parser error check for python 3.10 compatibilityAlexander Kanavin2021-10-231-1/+1
| | | | | | | | | | | | | | The change was introduced in https://github.com/python/cpython/commit/a698d52c3975c80b45b139b2f08402ec514dce75 (Bitbake rev: e0d6af2f6885c93f94c7fbcaeb7b0336c05f02b0) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8d3c6cbbe6ee734495713ae3b99c609527842506) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: do not import imp in layerindexlibAlexander Kanavin2021-10-231-1/+0
| | | | | | | | | | | | | The module is deprecated and unused. (Bitbake rev: 2dca6a22cb7b911c905a10158f1a3c3258de5c11) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 76c9030d6c91cd776a1aa732fb92b7cd4539b503) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fix regexp deprecation warningsAlexander Kanavin2021-10-231-3/+3
| | | | | | | | | | | | | | See here for details: https://docs.python.org/3/library/re.html (Bitbake rev: d5ac4af74ba62a27232ff16931e8f3b22fc43112) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 660e6ad4abb77c6f3c1d48bd64777dd76c05d7e2) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: correct the collections vs collections.abc deprecationAlexander Kanavin2021-10-232-3/+4
| | | | | | | | | | | | | This becomes a hard error in python 3.10. (Bitbake rev: 4b66d498eab22eca29c9e3a7fdfa5c6e54055e78) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ae219e1f7460077f4492b31ac91cef4cf9b17277) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: compat.py: remove file since it no longer actually implements anythingChris Laplante2021-10-234-30/+21
| | | | | | | | | | | | | | Now that compat.py just imports Python standard library stuff, get rid of the layer of indirection. (Bitbake rev: 2a40a776efd7c28fa93a8556accccb79e3a7d0f0) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e2be6defbb9fcf25f9df04c3b452d0dba48dfd03) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: test/fetch: Update urls to match upstream branch name changesRichard Purdie2021-10-191-3/+3
| | | | | | | | | | (Bitbake rev: 93094705c1c66688e0730a1efb17805778c9fa9f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 473e2a5486bd972ad0f808db089abcb8945d3a48) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/msg: Cleanup verbose option handlingRichard Purdie2021-10-026-18/+15
| | | | | | | | | | | | | | | The levels of indirection to set these verbose logging options is rather crazy. This attempts to turn things into two specific options with much more specific meanings. For now its all still controlled by the commandline verbose option and should funciton as previously, with the addition that the BB_VERBOSE_LOGS option can now be task specific. (Bitbake rev: b4117231bf070703b9375af4411bcd160e07fdae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 423c046f2173aaff3072dc3d0882d01b8a0b0212) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Show a readable error for invalid multiconfig nameRichard Purdie2021-10-021-0/+2
| | | | | | | | | | | | | | | | | If a multiconfig starts with a digit, users would see pages of errors as we use the multiconfig as a python function name prefix and python functions cannot start with a digit. We could avoid doing that but it is easier just to ask users to name multiconfigs not starting with digits. This tweak ensures the user sees an easier to understand error. (Bitbake rev: 34301f8a38078c2329e460051a1193c0314bcfd2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f9cddaeef35b2ea0dadf717101ed896f6b857abd) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Improve error handlingRichard Purdie2021-10-021-4/+6
| | | | | | | | | | | | | | | If bitbake-worker fails, return an error code showing that. Also make the thread cleanup code explict in a finally clause as it would otherwise hang. [YOCTO #14393] (Bitbake rev: 97541440e982848ef8bdbced22decdc24eda855b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7e0af70fb53fb13f824ca954b8cc1dffee730233) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Show error for no BBLAYERS in bblayers.confRichard Purdie2021-10-021-0/+3
| | | | | | | | | | | | | | If there is no BBLAYERS set in bblayers.conf show a more helpful error and exit. [YOCTO #14340] (Bitbake rev: 7f7034a6a0893debd8a5288a5765146a8b2ab0a0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 97183e10faf9862b5d9489d6e2c27ac77c3b697d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Improve missing core layer error messageRichard Purdie2021-10-021-0/+2
| | | | | | | | | | | | | | If the core layer is missing from bblayers.conf, the message the user sees is hard to understand. Improve it. [YOCTO #14340] (Bitbake rev: 0b08c0ed89951a2fc1f052d1bcae8b8bc5552c8e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5815a7258ebb8a989e0c6f5798853559d9413f02) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Improve error display for handled exceptionsRichard Purdie2021-10-021-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We don't need tracebacks for BBHandledException. Reduces confusing output like: ERROR: /meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio ERROR: ExpansionError during parsing /meta/recipes-core/images/core-image-tiny-initramfs.bb Traceback (most recent call last): File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_qa', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_cpio', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1038, in follow_chain(task='do_image_complete', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in chain: > bb.fatal("Circular task dependencies as %s depends itself via the chain %s?!" % (task, " -> ".join(chain))) chain.append(task) File "/bitbake/lib/bb/__init__.py", line 165, in fatal: mainlogger.critical(''.join(args), extra=kwargs) > raise BBHandledException() to the real error: ERROR: /media/build1/poky/meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio (Bitbake rev: a189283ba9dba8fdf11436d5e74cd2b95c2500b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 551d4c0576a0a0c3406000029df9238b312f2263) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Catch and error upon circular task referencesRichard Purdie2021-10-021-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | If there are circular task references, error on them rather than show a recursion error. A simple reproducer is: """ do_packageswu () { : } addtask do_packageswu after do_image_complete before do_image_qa """ into image_types.bbclass. There is code in runqueue to detect these but we never get that far with the current codebase. [YOCTO #13140] (Bitbake rev: f613d8d601be75e624e46cfe2351d1a067a9c341) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 339d4d6be515a71311b81fb9e99742af0d8a5130) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-12 19:13:33 +0000