linux/poky.git/bitbake/lib/prserv, branch nanbield-4.3.2

bitbake: bitbake: Add copyright headers where missing

2022-08-12T10:49:29+00:00

Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie

bitbake: prserv: make localhost work

2021-09-02T11:05:07+00:00

After [1] introduced, the PR server doesn't work in docker when use below setting like before. PRSERV_HOST = "localhost:0" And it's because the localhost is resolved to an ipv6 address ::1 as the below bitbake-prserv shows. bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address So add the extra logic to make the localhost resolved as expected to make the PR service work especially in docker. [1] 6a2b23e2 prserv: Replace XML RPC with modern asyncrpc implementation (Bitbake rev: 0a11696e0898c3c5108e6d7c5ad28da50e00ea66) Signed-off-by: Mingli Yu Signed-off-by: Richard Purdie

bitbake: prserv: handle PRSERV_HOST = "127.0.0.1:0" the same as "localhost:0"

2021-09-01T17:56:25+00:00

* When using PRSERV_HOST = "localhost:0" inside Docker container (tested with ubuntu 20.04 and 21.04) the self.loop.run_until_complete never completed, so self.address wasn't ever assigned few lines bellow and then self.port = int(self.prserv.address.rsplit(':', 1)[1]) in lib/prserv/serv.py caused a bit ugly exception: bitbake@599696cd20aa:~/nodistro/honister$ bitbake -k pkgconfig-native Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake", line 35, in sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "/OE/nodistro/honister/bitbake/lib/bb/main.py", line 385, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "/OE/nodistro/honister/bitbake/lib/bb/ui/knotty.py", line 397, in main params.updateToServer(server, os.environ.copy()) File "/OE/nodistro/honister/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer raise Exception("Unable to update the server configuration with local parameters: %s" % error) Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 286, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 491, in updateConfigOpts self.reset() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 1717, in reset self.handlePRServ() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 383, in handlePRServ self.prhost = prserv.serv.auto_start(self.data) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 318, in auto_start singleton.start() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 133, in start self.port = int(self.prserv.address.rsplit(':', 1)[1]) AttributeError: 'NoneType' object has no attribute 'rsplit' * the issue was caused by "localhost" being resolved as IPv6 address ::1 and then asyncio failing to bind it, the same is reproducible with hashserv, but hashserve at least shows nice error message: bitbake$ bitbake-hashserv -l DEBUG -b localhost:0 Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 59, in ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 53, in main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address * or by bitbake-prserv in prserv.log: bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address * while 127.0.0.1 works fine: bitbake$ bitbake-prserv --start --host=127.0.0.1 --port=42005 bitbake$ cat prserv.log DEBUG: Listening on ('127.0.0.1', 42005) 2021-08-26 22:28:05,828 Listening on ('127.0.0.1', 42005) DEBUG: Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' 2021-08-26 22:28:05,829 Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' NOTE: Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 2021-08-26 22:28:05,831 Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 but 127.0.0.1:0 wasn't handled as "autostart" like localhost:0 is update is_local_special to allow that * /etc/hosts file generated by docker contails localhost for both IPv4 and IPv6: $ grep localhost /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback even when ipv6 is disabled in dockerd as reported in: https://github.com/docker/for-linux/issues/250 * add a check for self.prserv.address to provide better error message: ERROR: Unable to start PR Server, exitting when something bad happens, but in this case you still need to read bitbake-cookerdaemon.log to see the actuall error, in this case: 90 22:30:39.008441 --- Starting bitbake server pid 90 at 2021-08-26 22:30:39.008419 --- 90 22:30:39.023734 Started bitbake server pid 90 90 22:30:39.024286 Entering server connection loop 90 22:30:39.024753 Accepting [] ([]) 90 22:30:39.026314 Processing Client 90 22:30:39.026456 Connecting Client 90 22:30:39.027509 Running command ['setFeatures', [2]] 90 22:30:39.027757 Command Completed 90 22:30:39.028711 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'DISTRO': '', 'PWD': '/OE/nodistro/honister', 'HOME': '/OE', 'MACHINE': 'qemux86', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'PATH': '/OE/nodistro/honister/oe-core/scripts:/OE/nodistro/honister/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'en_US.UTF-8', 'MACHINES': 'qemux86', 'HOSTNAME': '6a439759e3c6', 'TOPDIR': '/OE/nodistro/honister', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHLVL': '1', 'BITBAKE_HOME': '/OE', 'BUILDDIR': '/OE/nodistro/honister/BUILD', 'OLDPWD': '/OE/nodistro/honister/bitbake', '_': '/OE/nodistro/honister/bitbake/bin/bitbake'}, ['/OE/nodistro/honister/bitbake/bin/bitbake', '-k', 'zlib-native']] Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 255, in run self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address 90 22:30:39.530037 Command Completed 90 22:30:39.530913 Processing Client 90 22:30:39.531023 Disconnecting Client 90 22:30:39.531638 No timeout, exiting. 90 22:30:39.632137 Exiting 90 22:30:39.637562 Original lockfile contents: ['90\n'] 90 22:30:39.638107 Exiting as we could obtain the lock (Bitbake rev: c2cdda0e5fc720c60d3b8537fc702cb118981bd2) Signed-off-by: Martin Jansa Signed-off-by: Richard Purdie

bitbake: prserv: Add read-only mode

2021-08-23T07:30:55+00:00

[YOCTO #13659] (Bitbake rev: 44287430b9804fcbf2440f85a2424792140e4dc9) Signed-off-by: Paul Barker [updated for asyncrpc changes] Signed-off-by: Scott Murray Signed-off-by: Richard Purdie

bitbake: prserv: Replace XML RPC with modern asyncrpc implementation

2021-08-23T07:30:54+00:00

Update the prserv client and server classes to use the modern json and asyncio based RPC system implemented by the asyncrpc module. (Bitbake rev: 6a2b23e27bb61185b8afb382e20ce79f996d9183) Signed-off-by: Paul Barker [updated for asyncrpc changes, client split to separate file] Signed-off-by: Scott Murray Signed-off-by: Richard Purdie

bitbake: prserv: Handle requests in main thread

2021-05-06T10:08:07+00:00

The prserver process is cleanly separated from the main bitbake process so requests can be handled in the main thread. This removes the need for a request queue and a separate request handling thread. (Bitbake rev: 6b09415bed6b5e7c12aaf39b677d9ef72844e233) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: prserv: Extract daemonization from PRServer class

2021-05-06T10:08:07+00:00

The code to start the prservice process as a daemon is extracted out of the PRServer class and simplified. This makes the PRServer class easier to modernise as it no longer needs to worry about process management. (Bitbake rev: 39c7c158c52157b18f5ccbbd673e3298e6402f52) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: prserv: Use multiprocessing to auto start prserver

2021-05-06T10:08:07+00:00

We can use the modern multiprocessing support in Python instead of manually using fork to start the prserver process. To do this we need to set up the signal handlers for the prserver process in the work_forever function (which is now used as the main function for this process). The old code to start the prserver process using fork is not removed in this commit as it is tightly intertwined with the daemonization code which will be refactored in a following commit. (Bitbake rev: b3da56240c0f92efab1c0b293738c35c0f1ee6ab) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: prserv: Add connect function

2021-04-27T14:12:57+00:00

This function abstracts the setup of a PR service client connection so that openembedded-core doesn't need to be updated any time the details are changed. (Bitbake rev: d892287b31f81b075983ba500be265f75b53df64) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie

bitbake: prserv: Drop unused dump_db method

2021-04-27T14:12:57+00:00

(Bitbake rev: ecb7bf34eac02ff58dbc27b3768ceaf4adb1c9cd) Signed-off-by: Paul Barker Signed-off-by: Richard Purdie