linux/poky.git/bitbake, branch scarthgap-5.0.6Mirror of git.yoctoproject.org/pokyhttps://git.enea.com/cgit/linux/poky.git/atom?h=scarthgap-5.0.62024-12-09T14:54:14+00:00bitbake: runqueue: Fix scenetask processing performance issue2024-12-09T14:54:14+00:00Richard Purdierichard.purdie@linuxfoundation.org2024-11-29T18:51:36+00:00urn:sha1:ae41df7b37098fe6a427aa26a3bc711c06e20767
Analysis shows that "bitbake core-image-ptest-all" spends a lot of
time in scenequeue_updatecounters and much of it is rebuilding a set
which doens't change. Reorder the code to avoid that performance
glitch.
(Bitbake rev: f40a3a477d5241b697bf2fb030dd804c1ff5839f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 923c19b9713e398d8e66e6d4422dfd4c18a03486)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: runqueue: Optimise setscene loop processing2024-12-09T14:54:14+00:00Richard Purdierichard.purdie@linuxfoundation.org2024-11-29T16:09:46+00:00urn:sha1:ac3e66f1a666a1177b3677de56356cae10e5fe1a
Rather than looping through things we looped through on the previous execution,
start looping where we left off for setscene processing. This gives speed
improvements depending on the kind of build being executed.
(Bitbake rev: 28569e9796d4b34d7b77b4f79074ab7854850386)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 00f4d932e3af0eeb333339cbe942010fc76dee0f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: runqueue: Fix performance of multiconfigs with large overlap2024-12-09T14:54:14+00:00Richard Purdierichard.purdie@linuxfoundation.org2024-11-29T12:51:23+00:00urn:sha1:4f26b1b6a56f02f0abca53853902600d0b6f5d1e
There have been complaints about the performance of large multiconfig builds
for a while. The key missing data point was that the builds needed to have large
overlaps in sstate objects. This can be simulated by building the same things with
just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
deferred tasks.
The issue is that the expensive checks in the setscene loop were hit every time
through runqueue's execute function before the check on deferred tasks. This leads
to task execution starvation as that only happens once per iteration.
Move the skip check earlier in the function which speeds things up enormously
and should improve performance of such builds for users.
(Bitbake rev: db083cfe9e33c9fd7ffeead7b8c6023a5d581976)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9c6c506757f2b3e28c8b20513b45da6b4659c95f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: fetch2: use persist_data context managers2024-12-06T13:50:24+00:00Chris Laplantechris.laplante@agilent.com2024-11-25T17:12:04+00:00urn:sha1:1876097252d066a523ed8a3031322129f8f38498
Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s.
See https://docs.python.org/3/whatsnew/3.13.html#sqlite3
The previous commit fixed persist_data's context manager to close the
connection, but we were never actually using `with` in the first place.
This change is not necessary on 'master' because persist_data was
removed.
(Bitbake rev: 6c2641f7a9e92c1b82e306f59ddd3c1249c52cbf)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: persist_data: close connection in SQLTable __exit__2024-12-06T13:50:24+00:00Chris Laplantechris.laplante@agilent.com2024-11-25T17:12:03+00:00urn:sha1:52d9241ee5e02879aa3f751b4e3c92e8a8a4e78c
A Connection's __exit__ does not close the connection, but rather deals
with transactions. See https://docs.python.org/3/library/sqlite3.html#how-to-use-the-connection-context-manager
This change is not necessary on 'master' because persist_data was
removed.
(Bitbake rev: c9ea354c2df3d94b9ebe7810e83df9efb6007013)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: fetch/wget: Increase timeout to 100s from 30s2024-12-06T13:50:24+00:00Richard Purdierichard.purdie@linuxfoundation.org2024-11-20T21:26:03+00:00urn:sha1:dbb243bf0bc8ef394640dcfcbbef3397b2617c8f
Testing shows the worst case CDN response time can be up to 100s. The wget fetcher
is used for accessing sstate from the CDN so increase our timeouts there to match
our worst case repsonse times.
(Bitbake rev: c7f282cd27edfd78830b61db586ed669808893a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: Remove custom exception backtrace formatting2024-12-06T13:50:24+00:00Joshua WattJPEWhacker@gmail.com2024-10-08T12:36:25+00:00urn:sha1:d77302b2fe94db6c132310771b7b97c7b534edec
Removes the code in bitbake to show custom backtrace formatting for
exceptions. In particular, the bitbake exception code prints function
arguments, which while helpful is a security problem when passwords and
other secrets can be passed as function arguments.
As it turns out, the handling of the custom serialized exception stack
frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass
formatted exceptions"), which changed the events to pass a preformatted
stacktrack list of strings, but the passing of the serialized data was
never removed.
Change all the code to use the python traceback API to format exceptions
instead of the custom code; conveniently traceback.format_exception()
also returns a list of stack trace strings, so it can be used as a drop
in replacement for bb.exception.format_exception()
(Bitbake rev: c25e7ed128b9fd5b53d28d678238e2f3af52ef8b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: fetch2/git: Use quote from shlex, not pipes2024-11-28T13:36:00+00:00Richard Purdierichard.purdie@linuxfoundation.org2024-11-28T08:13:41+00:00urn:sha1:fd5231a5445f0a61d3a7678d1980a94dc1c80544
The pipes module is removed in python 3.13. It was already using the
quote function from shlex so use that directly instead.
The module already imports shlex too so it is an easy substitution.
(Bitbake rev: 70bd343fb273ad174e56d08c8b80c5594501e030)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: tests/fetch: Use our own mirror of mobile-broadband-provider to decouple from gnome gitlab2024-11-02T13:12:00+00:00Richard Purdierichard.purdie@linuxfoundation.org2024-10-29T20:14:08+00:00urn:sha1:3198afac8770abf3f9c4582aecb6bac2ab291718
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: 377eba2361850adfb8ce7e761ef9c76be287f88c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 91e268b11ed683bd197026f9b36001f6d54ee05c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake: tests/fetch: Use our own mirror of sysprof to decouple from gnome gitlab2024-11-02T13:12:00+00:00Richard Purdierichard.purdie@linuxfoundation.org2024-10-28T09:28:00+00:00urn:sha1:9149036240dbd508dc09939d03465876aa0dc7ed
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: 12e2b11e4038138ad1a02b9d4d36d0904fbc820d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 008808755ed6cfeb6c41273e69ce718f0833c26c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>