| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After [1] introduced, the PR server doesn't work in docker when use
below setting like before.
PRSERV_HOST = "localhost:0"
And it's because the localhost is resolved to an ipv6 address ::1 as
the below bitbake-prserv shows.
bitbake$ bitbake-prserv --start --host=localhost --port=42005
bitbake$ cat prserv.log
Traceback (most recent call last):
File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module>
ret = main()
File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main
ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only)
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon
run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile))
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon
func()
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main
server.serve_forever()
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever
self.start()
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp
self.server = self.loop.run_until_complete(server_coro)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server
return await loop.create_server(factory, host, port, **kwds)
File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address
So add the extra logic to make the localhost resolved as expected to
make the PR service work especially in docker.
[1] 6a2b23e2 prserv: Replace XML RPC with modern asyncrpc implementation
(Bitbake rev: 0a11696e0898c3c5108e6d7c5ad28da50e00ea66)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
| |
(Bitbake rev: 1ff1ea3880d293b14ce0fc65e3bc4c938d587a2f)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* When using PRSERV_HOST = "localhost:0" inside
Docker container (tested with ubuntu 20.04 and 21.04) the
self.loop.run_until_complete never completed, so self.address
wasn't ever assigned few lines bellow and then
self.port = int(self.prserv.address.rsplit(':', 1)[1])
in lib/prserv/serv.py caused a bit ugly exception:
bitbake@599696cd20aa:~/nodistro/honister$ bitbake -k pkgconfig-native
Traceback (most recent call last):
File "/OE/nodistro/honister/bitbake/bin/bitbake", line 35, in <module>
sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
File "/OE/nodistro/honister/bitbake/lib/bb/main.py", line 385, in bitbake_main
return ui_module.main(server_connection.connection, server_connection.events,
File "/OE/nodistro/honister/bitbake/lib/bb/ui/knotty.py", line 397, in main
params.updateToServer(server, os.environ.copy())
File "/OE/nodistro/honister/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer
raise Exception("Unable to update the server configuration with local parameters: %s" % error)
Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last):
File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 90, in runCommand
result = command_method(self, commandline)
File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 286, in updateConfig
command.cooker.updateConfigOpts(options, environment, cmdline)
File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 491, in updateConfigOpts
self.reset()
File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 1717, in reset
self.handlePRServ()
File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 383, in handlePRServ
self.prhost = prserv.serv.auto_start(self.data)
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 318, in auto_start
singleton.start()
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 133, in start
self.port = int(self.prserv.address.rsplit(':', 1)[1])
AttributeError: 'NoneType' object has no attribute 'rsplit'
* the issue was caused by "localhost" being resolved as IPv6 address ::1
and then asyncio failing to bind it, the same is reproducible with hashserv, but
hashserve at least shows nice error message:
bitbake$ bitbake-hashserv -l DEBUG -b localhost:0
Traceback (most recent call last):
File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 59, in <module>
ret = main()
File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 53, in main
server.serve_forever()
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever
self.start()
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp
self.server = self.loop.run_until_complete(server_coro)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server
return await loop.create_server(factory, host, port, **kwds)
File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address
* or by bitbake-prserv in prserv.log:
bitbake$ bitbake-prserv --start --host=localhost --port=42005
bitbake$ cat prserv.log
Traceback (most recent call last):
File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module>
ret = main()
File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main
ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only)
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon
run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile))
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon
func()
File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main
server.serve_forever()
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever
self.start()
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp
self.server = self.loop.run_until_complete(server_coro)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server
return await loop.create_server(factory, host, port, **kwds)
File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address
* while 127.0.0.1 works fine:
bitbake$ bitbake-prserv --start --host=127.0.0.1 --port=42005
bitbake$ cat prserv.log
DEBUG: Listening on ('127.0.0.1', 42005)
2021-08-26 22:28:05,828 Listening on ('127.0.0.1', 42005)
DEBUG: Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3'
2021-08-26 22:28:05,829 Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3'
NOTE: Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39
2021-08-26 22:28:05,831 Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39
but 127.0.0.1:0 wasn't handled as "autostart" like localhost:0 is
update is_local_special to allow that
* /etc/hosts file generated by docker contails localhost for both IPv4 and IPv6:
$ grep localhost /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
even when ipv6 is disabled in dockerd as reported in:
https://github.com/docker/for-linux/issues/250
* add a check for self.prserv.address to provide better error message:
ERROR: Unable to start PR Server, exitting
when something bad happens, but in this case you still need to read
bitbake-cookerdaemon.log to see the actuall error, in this case:
90 22:30:39.008441 --- Starting bitbake server pid 90 at 2021-08-26 22:30:39.008419 ---
90 22:30:39.023734 Started bitbake server pid 90
90 22:30:39.024286 Entering server connection loop
90 22:30:39.024753 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
90 22:30:39.026314 Processing Client
90 22:30:39.026456 Connecting Client
90 22:30:39.027509 Running command ['setFeatures', [2]]
90 22:30:39.027757 Command Completed
90 22:30:39.028711 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'DISTRO': '', 'PWD': '/OE/nodistro/honister', 'HOME': '/OE', 'MACHINE': 'qemux86', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'PATH': '/OE/nodistro/honister/oe-core/scripts:/OE/nodistro/honister/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'en_US.UTF-8', 'MACHINES': 'qemux86', 'HOSTNAME': '6a439759e3c6', 'TOPDIR': '/OE/nodistro/honister', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHLVL': '1', 'BITBAKE_HOME': '/OE', 'BUILDDIR': '/OE/nodistro/honister/BUILD', 'OLDPWD': '/OE/nodistro/honister/bitbake', '_': '/OE/nodistro/honister/bitbake/bin/bitbake'}, ['/OE/nodistro/honister/bitbake/bin/bitbake', '-k', 'zlib-native']]
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 255, in run
self.start()
File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp
self.server = self.loop.run_until_complete(server_coro)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server
return await loop.create_server(factory, host, port, **kwds)
File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address
90 22:30:39.530037 Command Completed
90 22:30:39.530913 Processing Client
90 22:30:39.531023 Disconnecting Client
90 22:30:39.531638 No timeout, exiting.
90 22:30:39.632137 Exiting
90 22:30:39.637562 Original lockfile contents: ['90\n']
90 22:30:39.638107 Exiting as we could obtain the lock
(Bitbake rev: c2cdda0e5fc720c60d3b8537fc702cb118981bd2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Make versionVariableMatch() support pn-foo overrides using the new
override syntax.
(Bitbake rev: 653df4bc413d595d5611d5fa678e7f2e30bb7431)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
| |
Thanks Peter for spotting.
(Bitbake rev: 8c7a54ea7fdc721278380895d01868b96b330b90)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
| |
Document that bitbake-selftest exists and that we appreciate test cases.
(Bitbake rev: 52896ca1fabd22cce01b75cc6fe3412b1ec09b5b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
[YOCTO #13659]
(Bitbake rev: 44287430b9804fcbf2440f85a2424792140e4dc9)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[updated for asyncrpc changes]
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update the prserv client and server classes to use the modern json and
asyncio based RPC system implemented by the asyncrpc module.
(Bitbake rev: 6a2b23e27bb61185b8afb382e20ce79f996d9183)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[updated for asyncrpc changes, client split to separate file]
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
asyncio in older Python 3.x (seen with 3.7) can seemingly hang if
new_event_loop is called repeatedly in the same process. The
reuse of processes in the Bitbake thread pool during parsing seems
to be able to trigger this with the PR server export selftest.
It appears that calling set_event_loop with the new loop avoids the
issue, so that is now done in the asyncrpc Client initializer (with
an explanatory comment). This should be revisited when the day
arrives that Python 3.9 becomes the minimum required for BitBake.
Additionally, it was discovered that using get_event_loop in the
asyncrpc server initialization can trigger hangs in the hashserv
unittests when the second test is run. To avoid this, switch to
calling new_event_loop + set_event_loop in the initialization code
there as well.
(Bitbake rev: bb9a36563505652294b20b4c88199b24fa208342)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reworks the async I/O API so that the async loop is only created in the
child process. This requires deferring the creation of the server until
the child process and a queue to transfer the bound address back to the
parent process
(Bitbake rev: 8555869cde39f9e9a9ced5a3e5788209640f6d50)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
[small loop -> self.loop fix in serv.py]
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The urllib.request.ProxyHandler constructor only reads the $http_proxy
+ $https_proxy environment variables.
$no_proxy is evaluated later when the url is opened.
It is therefore not sufficient to just construct the proxy handler in
the
| with bb.utils.environment(**newenv):
context, but the 'opener.open(r)' call must also be made there.
(Bitbake rev: 076baf4fbd328d247508fd399866a397eb34f67e)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently we inform the user that some package/layer is skipped but we
don't print the reason albeit bitbake knows the reason. So currently it
looks like:
gtk+:
meta-oe 2.24.32 (skipped)
With this change the output prints the skip reason which is very helpful
for debugging:
gtk+:
meta-oe 2.24.32 (skipped: one of 'x11 directfb' needs to be in DISTRO_FEATURES)
(Bitbake rev: d43e72db4f7c8b47d91d99ed54ce30e9ee898de1)
Signed-off-by: Marco Felsch <m.felsch@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
"remove" was accidentally omitted when defining which override operators
should be highlighted
(Bitbake rev: a0248338452f9ec26b588ef83679aca6263e7e76)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
| |
OE-Core did this a while ago, it is simpler if bitbake matches.
(Bitbake rev: a3050aee21b6a23b55232d52f89980a3bbd3a290)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The days of broken certificates are behind us now, so instead of always
passing --no-check-certificate to wget, don't pass it by default and
instead only pass it BB_CHECK_SSL_CERTS = "0".
[ YOCTO #14108 ]
(Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of just exporting the proxy variables when calling into urllib,
use bb.utils.environment() to export all of the known variables that are
needed for proper connectivity.
Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can
find the certificates in buildtools environments
(Bitbake rev: 116637b0e9aabae7f680b102dbf3577b8a58f049)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is a list of environment variable names that need to be exported
when executing external commands, such as 'http_proxy'. To avoid
duplication, make this a top-level variable.
Also add SSL_CERT_FILE, which is used by OpenSSL to locate the
certificate bundle. This is needed in buildtools environments where the
default path isn't valid.
(Bitbake rev: 5c2cf57fd11d91f749a3b562f6f0a863f15013ed)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
bb.utils.environment() is a context manager to alter os.environ inside
a specific block, restoring it after the block is closed.
(Bitbake rev: 9974848f67581ff7d76cef52a94f505af99b4932)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
If run on an empty build directory, taskexp wasn't working as it didn't
send the current environment to the server. This means HOSTTOOLS in oe-core
couldn't be built and gave an error. Add the missing updateToServer call in.
[YOCTO #14408]
(Bitbake rev: 06a0bbe746f879ae539223e7fdb6f07d55d13719)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
When an exception occurs at startup, show it to the user.
[YOCTO #14408]
(Bitbake rev: cc1df1af67cfd3e223b39e2b7ea5f86b8cf78aee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
If an exception occurs in early setup, bitbake could just hang. Return
the exception rather than doing that.
[YOCTO #14408]
(Bitbake rev: c8a4107132ce51f84ae84bf1ceb1c3fd90f156d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently the code can just show nothing as the exception if there was a double
fault, which in this code path is quite likely. This leads to an error log
which effectively says "it failed" with no information about how.
Improve things so we get a nice verbose traceback left in the logs/output
which is preferable to no logs.
(Bitbake rev: e5782b71647d1eb6de53bde7bc4f6019a5589f21)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.
(Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Fix some references missed during the overrides syntax migration.
Thanks to Quentin Schulz <foss@0leil.net> for the patch.
(Bitbake rev: 2fd03ec7b136c694f2ced43b3abb69f719c99ec2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous patches have exposed new issues with this code path,
the issues being around what should happen when the hash of a task
changes and the task is or is not on the deferred task list.
Rather than rebuilding the deferred task list during each rehash
event, build it once at the start of a build. This avoids the problem
of tasks being added back after they have run and also avoids problems
of always ensuring the same task is deferred. It also allows the
'outrightfail' codepath to be handled separately as the conditions
are subtly differnt.
One significant win for the new approch is the build is not continually
printing out lists of deferred tasks, that list remains fairly static
from the start of the build. Logic is added in to ensure a rehashed
task with a hash matching other deferred tasks is deferred along with
them as a small optimization.
An interesting test case for this code was reported by Mark Hatle
with four multiconfigs, each the same apart from TMPDIR and running a
build of:
bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal
which is interesting in that the build of buildtools partially overlaps
core-image-minimal and the build has a rehash event for qemuwrapper-cross
even without any external hash equivalence server or preexisting data.
(Bitbake rev: bb424e0a6d274d398f434f7df63951da9ce305b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Updates the Vim syntax highlighting to account for the new override
syntax and also highlight "append" and "prepend" overrides
(Bitbake rev: 01a6322315a6ff6ab55a349f9fcd1e2d93448bfd)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Setting something like:
BAR:append:unusedoverride
should cause BAR to be None, not "" which was what the datastore was
returning. This caused problems when mixing variables like:
RDEPENDS:${PN}:inactiveoverride
RDEPENDS:${BPN}
since key expansion would report key overlap when there was none. This
is a bug in the datastore. Fix it and add a test too.
[YOCTO #14088]
(Bitbake rev: 699e36c270d863258502d315ed00a1b940bfbf96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"metadata" is used both as singular and as plural.
This fixes a case in which the verb has a singular conjugation,
which conflicts with the absence of article indicating a plural case.
(Bitbake rev: fff7ade48d6cb9381284b93742bb2255976d6b41)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Quentin Schulz <foss@0leil.net>
Reviewed-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
| |
(Bitbake rev: 214d11867ea5dd9f1d7e50e128d45d7cc4eaf7ea)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
The previous syntax of the example was clearly incorrect
The example and description were also ambiguous, one could think that
it was "bitbake recipe" instead of "bitbake <recipename>"
(Bitbake rev: 9d6664bbb68fac9bb4fbcbe0b35beb0a022a5a1f)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: ca88466f6d244042b12b66ccd69e27ca2f057d17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
The examples and tests use non-standard override names, convert these to
the new syntax by hand.
(Bitbake rev: a6c40eca1146c0160da7e4e0bd7ac52fef2029e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
This runs the overrides conversion script in OE-Core over the bitbake code
base including the docs. A handful of things were excluded in toaster
and for the Changelog file.
(Bitbake rev: 47f8d3b24fd52381bf3b41e2f55a53e57841344c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change updates the datastore to use the new override syntax using
colons instead of underscores exclusively. It is expected layers would
have to be converted to work with bitbake after this change.
Supporting mixed syntax isn't possible, it is only feasible to have
one internal representation of overrides.
Whilst we can't warn for every possible override that may be set in the
old format, show errors for _append/_prepend/_remove since those should
never be present.
(Bitbake rev: 7dcf317cc141dc980634f8c18bfa84f83e57206a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Now that ":" is a valid character in variable key names, it needs to be
allowed by the variable expansion code too, to match.
(Bitbake rev: 019251649a38754d5877759d13b664e28dea77de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When parsing an anonymous python function, bitbake generates a name for
the function based on the full path to the file in which it was found.
As not all characters which are valid in file paths are valid in Python
function names we have a translation table. However, this translation
table was missing an entry for '~'.
(Bitbake rev: b201c0b284e25c20685d9d206797c4cc650eeca0)
Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the SIGTERM signal is sent to an asyncrpc server before it has
installed the SIGTERM handler in the main loop, it may miss the signal
which will can cause the calling process to wait forever on the join().
To resolve this, the calling process should mask of SIGTERM before
forking the server process and the server should unmask the signal only
after the handler is installed. To simplify the usage of the server, an
new helper function called serve_as_process() is added to do this
automatically and correctly.
Thanks: Scott Murray <scott.murray@konsulko.com> for helping debug
(Bitbake rev: ef2865efa98ad20823267364f2159d8d8c931400)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
| |
(Bitbake rev: b31f1853d7fcb8b8f8885ca513a0021a5d0301e6)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
If there is an issue with the format of the reply given by the server then we
should print this reply line in the error message. Printing the message which
the client sent doesn't illuminate anything here.
(Bitbake rev: bd8f8d7b055da15cd7bdd0b383061852a0f54cb7)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If a variable has a glossary entry and some rST files write about those
variables, it's better to point to the glossary entry instead of just
highlighting it by surrounding it with two tick quotes.
The script that is used to do the replacement of ``FOO`` by :term:`FOO`
is the following Python code:
import re
from pathlib import Path
from runpy import run_module
import contextlib
import io
import sys
re_term = re.compile(r'variables.html#term-([a-zA-Z_0-9]*)')
terms = []
new_terms = set()
with contextlib.redirect_stdout(io.StringIO()) as f:
run_module('sphinx.ext.intersphinx', run_name='__main__')
objects = f.getvalue()
match = re_term.search(objects)
while match:
if match.group(1):
terms.append(match.group(1))
match = re_term.search(objects, match.end())
for rst in Path('.').rglob('*.rst'):
with open(rst, 'r') as f:
content = "".join(f.readlines())
for term in terms:
content = re.sub(r'``({})``(?!.*\s+[~=-]{{{:d},}})'.format(term, len(term)), r':term:`\1`', content)
with open(rst, 'w') as f:
f.write(content)
This script takes one argument as input: an objects.inv which can be
gotten from doc/_build/html/objetcs.inv after running `make html`.
Note that this excludes from replacement terms that appear in section
titles as it requires refs to be changed too. This can be automated too
if need be but right now it looks a bit confusing to have an anchor link
(for sections) also have a term/reference link in it. I am not sure this
is desired today.
(Bitbake rev: aba88f40c47133ed9bc999e0298aca3bc8490912)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
be alphabetically sorted
Even though, care should be taken to have the terms in the glossary
sources alphabetically ordered, it is possible some terms might be in
the wrong place.
This makes sure that whatever the order of terms in the glossary
sources, the generated medium is correctly sorted.
(Bitbake rev: 98809ebc6ad51f0a94cedccfaff9c11d3744dc0d)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
glossary sources
This reorders a few entry so that they are alphabetically sorted.
(Bitbake rev: 311350ed5cb164d975c2119e60255a409e27dffb)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
-W is for turning warnings into errors, --keep-going to make sure the
whole documentation is built and all warnings will be printed before
failing.
Since there's currently no warning, it's time to introduce it!
(Bitbake rev: 6b0b3d707f662ca7b1d0de99fe032e4f35867d10)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the client process never sends cooker data, the server timeout will
be 0.0, not None. This will prevent the server from exiting, as it is
waiting for a new client. In particular, the client will disconnect with
a bad "INHERIT" line, such as:
INHERIT += "this-class-does-not-exist"
Instead of checking explicitly for None, check for a false value, which
means either 0.0 or None.
(Bitbake rev: 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a library that implements file-like objects (similar to
gzip.GzipFile) that can stream to arbitrary compression programs. This
is utilized to implement a LZ4 and zstd compression API.
(Bitbake rev: 61c3acd058ea018696bd284b3922d0b458838d05)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is becomming increasingly clear we need to find a way to show what
is/is not an override in our syntax. We need to do this in a way which
is clear to users, readable and in a way we can transition to.
The most effective way I've found to this is to use the ":" charater
to directly replace "_" where an override is being specified. This
includes "append", "prepend" and "remove" which are effectively special
override directives.
This patch simply adds the character to the parser so bitbake accepts
the value but maps it back to "_" internally so there is no behaviour
change.
This change is simple enough it could potentially be backported to older
version of bitbake meaning layers using the new syntax/markup could
work with older releases. Even if other no other changes are accepted
at this time and we don't backport, it does set us on a path where at
some point in future we could
require a more explict syntax.
I've tested this patch by converting oe-core/meta-yocto to the new
syntax for overrides (9000+ changes) and then seeing that builds
continue to work with this patch.
(Bitbake rev: 0dbbb4547cb2570d2ce607e9a53459df3c0ac284)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously access to AWS S3 was expected to be preconfigured and
credentials to be stored in ~/.aws/credentials. With this change
one can use Bitbake s3 fetcher without AWS credentials stored
permanently as above, just with them exported as the following
environment variables:
- AWS_ACCESS_KEY_ID,
- AWS_SECRET_ACCESS_KEY.
- AWS_DEFAULT_REGION.
(Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48)
Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commits https://git.openembedded.org/bitbake/commit/?id=78cd63285713fde59506eb2e71a7b7ee59a594ff
and https://git.openembedded.org/bitbake/commit/?id=5cbf6d95fc1009e78e7d0745a49e0bf418b37abb
added few calls to logger.warn(), which is deprecated and instead
should use logger.warning():
https://lists.openembedded.org/g/bitbake-devel/topic/82742194#12377
(Bitbake rev: a28ba2d31cd3aa557d4977e9376c5d01cd863e9a)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It looks like git gc can interrupts the package creation when
BB_GENERATE_MIRRORS_TARBALL is in use.
Log excerpts:
tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output:
tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it
tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
(Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877)
Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
This is the BitBake glossary.
Remove an erroneous statement probably coming from the introduction
to the YP/OE glossary.
(Bitbake rev: 398a1686176c695d103591089a36e25173f9fd6e)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|