linux/poky.git/bitbake/lib/bb, branch hardknott-3.3.3

bitbake: server: Fix early parsing errors preventing zombie bitbake

2021-09-09T09:19:42+00:00

If the client process never sends cooker data, the server timeout will be 0.0, not None. This will prevent the server from exiting, as it is waiting for a new client. In particular, the client will disconnect with a bad "INHERIT" line, such as: INHERIT += "this-class-does-not-exist" Instead of checking explicitly for None, check for a false value, which means either 0.0 or None. (Bitbake rev: c7b506d11df78cfc4610db6578745eaa6220b13a) Signed-off-by: Joshua Watt Signed-off-by: Richard Purdie (cherry picked from commit 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: data_smart: Allow colon in variable expansion regex

2021-07-30T13:38:24+00:00

Now that ":" is a valid character in variable key names, it needs to be allowed by the variable expansion code too, to match. (Bitbake rev: 9b2d96b27f550da0fa68ba9ea96be98eb3a832a6) Signed-off-by: Richard Purdie

bitbake: data_smart/parse: Allow ':' characters in variable/function names

2021-07-30T13:38:24+00:00

It is becomming increasingly clear we need to find a way to show what is/is not an override in our syntax. We need to do this in a way which is clear to users, readable and in a way we can transition to. The most effective way I've found to this is to use the ":" charater to directly replace "_" where an override is being specified. This includes "append", "prepend" and "remove" which are effectively special override directives. This patch simply adds the character to the parser so bitbake accepts the value but maps it back to "_" internally so there is no behaviour change. This change is simple enough it could potentially be backported to older version of bitbake meaning layers using the new syntax/markup could work with older releases. Even if other no other changes are accepted at this time and we don't backport, it does set us on a path where at some point in future we could require a more explict syntax. I've tested this patch by converting oe-core/meta-yocto to the new syntax for overrides (9000+ changes) and then seeing that builds continue to work with this patch. (Bitbake rev: f8d1bc200460fc1cb5cbf7a7d1986fd86424b22d) Signed-off-by: Richard Purdie

bitbake: fetch2: add check for empty SRC_URI hash string

2021-07-06T22:28:30+00:00

No error was being reported when the hash string was set to empty. For example: SRC_URI[md5sum] = "" On a related note (not a bug): Because whitespace in the string will result in a checksum mismatch, the error message was updated to make it a little clearer why the error was thrown. For example: SRC_URI[md5sum] = " " or SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1" Now creates a message like this: File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5 checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected [YOCTO #14232] (Bitbake rev: 731fb52eb03338c0bdb2a2256c22c64c22bcbace) Signed-off-by: Scott Weaver Signed-off-by: Richard Purdie (cherry picked from commit a13510d0028e234ea2f4744b0d0c38558395c70f) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: fetch/svn: Fix parsing revision of SVN repos with redirects

2021-07-06T22:28:30+00:00

svn was printing a message when encountering HTTP redirects. This confused the revision parser. (Bitbake rev: 40b4e2dcd24d3ed01d35d705928133aaa1c4a93b) Signed-off-by: Harald Brinkmann Signed-off-by: Richard Purdie (cherry picked from commit a944a335f8f4c4fe5df55f3d7d8e757bd2835146) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: server/process: Handle error in heartbeat funciton in OOM case

2021-05-20T20:40:07+00:00

We've seen cases where an OOM error causes bitbake server to hang: 9171 02:21:09.127810 Command Completed Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer server.run() File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run ret = self.main() File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main ready = self.idle_commands(.1, fds) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands bb.event.fire(heartbeat, self.cooker.data) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire fire_class_handlers(event, d) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers execute_handler(name, handler, event, d) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler ret = handler(event) File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval") File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8') File "/usr/lib/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 423, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory We need to wrap the calls in the same high level wrapper as idle function calls and trigger an exit upon an unhandled exception. (Bitbake rev: 0c83e98b0b4129e43e05129074fff60b22a9f2eb) Signed-off-by: Richard Purdie

bitbake: fetch2/wget: when checking latest versions, consider all numerical directories

2021-05-20T20:40:07+00:00

Previously the regex was maching x.y, but wasn't matching x, which is a problem e.g. here: https://download.gnome.org/sources/epiphany/ (the new gnome version scheme adds 40-series at the end). (Bitbake rev: 8eddd1808387025a22d8ad1b009c2894d19bf79b) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

bitbake: runqueue: Handle deferred task rehashing in multiconfig builds

2021-05-06T10:22:44+00:00

If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. (Bitbake rev: b67476d4758915db7a5d9f58bc903ae7501a1774) Signed-off-by: Richard Purdie (cherry picked from commit 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue

2021-05-06T10:22:44+00:00

We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. (Bitbake rev: e70cba8d5861d79ed0da9e760e618af8b759c8a9) Signed-off-by: Richard Purdie (cherry picked from commit edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie

bitbake: bitbake: tests/fetch: remove write protected files too

2021-04-27T14:22:58+00:00

For some reason several git-annex files in Debian 10 buster are read-only and removing them with "rm -rf" fails. Fixes test failures like: $ bitbake-selftest ... rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied EE..................................ssss.sssssssssssssss.sssss....................................................................................................... ====================================================================== ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex fetcher, ud = self.fetch_shallow(uri) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow bb.utils.remove(ud.clonedir, recurse=True) File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path)) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1. Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod the whole temporary directory which should cover any directory name differences between different git-annex versions. Fixes tests failing after chmod call: Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/ (Bitbake rev: 14c5f0735947307b9d69c57f7334fefaea7311b3) Signed-off-by: Mikko Rapeli Signed-off-by: Richard Purdie (cherry picked from commit 7729ef2983c72867e99fad82d671069ba5cb32b2) Signed-off-by: Anuj Mittal Signed-off-by: Richard Purdie