linux/poky.git/bitbake/lib/bb/server, branch 2.8_M3

bitbake: bitbake: server/process: Handle BBHandledException to avoid unexpected exceptions

2019-08-16T09:05:32+00:00

The parseBaseConfiguration() raises bb.BBHandledException(), but BitBakeServer() didn't handle it, so we always got unexpected exceptions when there were errors. For example: === Case 1: * Add "print "hello"' in base.bbclass' def oe_import() function def oe_import(d): print "hello" [snip] $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Last 60 lines of server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 21 print "hello" ^ SyntaxError: Missing parentheses in call to 'print' During handling of the above exception, another exception occurred: During handling of the above exception, another exception occurred: During handling of the above exception, another exception occurred: [snip] Now it looks like: $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 21: The code lines resulting in this error were: 0001:def oe_import(d): *** 0002: print "hello" 0003: import sys 0004: 0005: bbpath = d.getVar("BBPATH").split(":") 0006: sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath] SyntaxError: Missing parentheses in call to 'print' (base.bbclass, line 21) === Case 2: * Add 'HOSTTOOLS += "hello"' to conf/local.conf: $ bitbake -p ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello The error message is printed by bb.fatal() which raises bb.BBHandledException(), but BitBakeServer() doesn't handle it, so we got it. Now it looks like: ERROR: Unable to start bitbake server (None) ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log): ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello No unexpected exceptions anymore. [YOCTO #13267] (Bitbake rev: 6e6865e6371dbd31a136eae64cc5b1fa5f5bee33) Signed-off-by: Robert Yang Signed-off-by: Richard Purdie

bitbake: bitbake: Drop duplicate license boilerplace text

2019-05-04T09:44:10+00:00

With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie

bitbake: bitbake: Add initial pass of SPDX license headers to source code

2019-05-04T09:44:04+00:00

This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie

bitbake: server/process: Add missing exception raise

2019-02-06T08:24:50+00:00

The intent of the code was to catch one kind of error, it was actually swallowing all exceptions and looping indefinitely. Fix it to work as intended. This explains some mystery hangs we've been seeing. (Bitbake rev: d73dbc3580faa1225d95ae4cefac4879ca3c1b2f) Signed-off-by: Richard Purdie

bitbake: process: Rewrite multiple connection handling

2018-12-26T11:10:33+00:00

If the bitbake server recieved multiple connections, it currently closes ones it can't handle (while its dealing with another). This is rather antisocial behaviour which causes clients to quickly run through their retries and abort. Instead, queue any other connections until the current one is closed. This way the client can decide when it wants to stop waiting for the server. If the client is gone by the time we handle it, we handle that gracefully. This also fixes a number of bugs in the connection handling where connections which did drop early were badly handled causing tracebacks in the logs. Also, handle queue incomming connections in a loop to ensure that the main client handling doesn't starve that piece of the system. This code was stress tested by running 50 connection attempts in parallel at once, ensuring the code correctly handled them. (Bitbake rev: 02845a561b38658ac3edf5cc9c34625ed860d34f) Signed-off-by: Richard Purdie

bitbake: process: Handle EWOULDBLOCK in socket connect

2018-12-26T11:10:33+00:00

Now that we set a timeout for the socket, it can return EWOULDBLOCK if a signal or other event happens to wake up even if we don't timeout. If this happens, retry the connection, else we simply see it quickly loop through the retries and abort the connection in a very short interval. (Bitbake rev: d5b0a9a302ac0eafa4f797ac15ea77db87e82b3c) Signed-off-by: Richard Purdie

bitbake: process.py: Set socket timeout to 10 seconds

2018-12-26T11:10:33+00:00

The current value of 2 seconds has shown to be short in wider testing. (Bitbake rev: 8a1f2fcf35d61d83bbafa8fa3ae215fd5f51728b) Signed-off-by: Richard Purdie

bitbake: server/process: Ensure socket has a timeout set

2018-12-15T11:48:33+00:00

We're seeing hangs in oe-selftest where server startup and shutdown are racing. The assumption was a connect would timeout however no timeout is set which can leave processes hanging. Set a short timeout for the connection to avoid this. (Bitbake rev: f02114cb70e8f6f1d32e19c02b758fe0aadecd19) Signed-off-by: Richard Purdie

bitbake: server/process: Increase server startup time delay

2018-12-08T17:17:37+00:00

On loaded production systems we've seen bitbake server take over 40s to start up. Increase the timeout to 90s which tries to avoid failures in selftests. The delays come from setting up the inotify watches (31s) so can't really be avoided. After 5s delay we now warn the user we're waiting for 90s so the interactive exeperience shouldn't be much changed and its very unlikely the user would see that anyway. (Bitbake rev: 7a4e105093c3080c1087cd06abc2883852a23e28) Signed-off-by: Richard Purdie

bitbake: process: Flush key output to logs

2018-12-05T13:11:27+00:00

Small tweak to ensure these items are printed into the log since there is other logging code which looks for the header and this makes it clearer the server did start but is slow somewhere in startup. (Bitbake rev: 7c0b84cac892744225fa0462f918ea9a79b356cb) Signed-off-by: Richard Purdie