linux/poky.git/bitbake/lib/bb/siggen.py, branch kirkstone-4.0.22

bitbake: bitbake-diffsigs: break on first dependent task difference

2023-02-17T15:05:08+00:00

compare_sigfiles() recursively calculates differences on all dependent tasks with changed hashes. This is done in arbitrary/alphabetical order, and only the last of those results is returned, while everything else is discarded. This changes the behavior to instead return the first difference and not calculate any more, which significantly speeds up diffs of tasks with many dependencies. (Bitbake rev: 89f13cd4a927a73de98998c27082c63b07671525) Signed-off-by: Adriaan Schmidt Signed-off-by: Richard Purdie (cherry picked from commit ea6a676c9aa2864c2eff40eea41ba09ce903a651) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: siggen: Fix inefficient string concatenation

2023-02-17T15:05:08+00:00

As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython has an optimization for statements in the form "a = a + b" or "a += b". It seems that this line does not get optimized, because it has a form a = a + b + c: data = data + "./" + f.split("/./")[1] For that reason, it does a copy of data for each iteration, potentially copying megabytes of data for each iteration. Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds instead of 45 seconds on my test setup where SRC_URI points to a big directory. Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython: "do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b" However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup: data = ''.join((data, "./", f.split("/./")[1])) I have changed the other lines to also use += for consistency only, however those were in the form a = a + b and were optimized already. Co-authored-by: JJ Robertson (Bitbake rev: 590ae6fde9da75db3a368e5c0d47920696c33ebf) Signed-off-by: Etienne Cordonnier Signed-off-by: Richard Purdie (cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bitbake: Add copyright headers where missing

2022-10-11T20:57:28+00:00

Where copyright headers were not present, add them to make things clear. (Bitbake rev: e591325b2bd901c381003deb96a7b32a7148e93e) Signed-off-by: Richard Purdie (cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: siggen: Fix insufficent entropy in sigtask file names

2022-10-11T20:57:28+00:00

Signature generation uses mkstemp() to get a file descriptor to a unique file and then write the signature into it. However, the unique file name generation in glibc is based on the system timestamp, which means that with highly parallel builds it is more likely than one might expect expected that a conflict will occur between two different builder nodes. When operating over NFS (such as a shared sstate cache), this can cause race conditions and rare failures (particularly with NFS servers that may not correctly implement O_EXCL). The signature generation code is particularly susceptible to races since a single "sigtask." prefix used for all signatures from all tasks, which makes collision even more likely. To work around this, add an internal implementation of mkstemp() that adds additional truly random entropy to the file name to eliminate conflicts. (Bitbake rev: 63bb5591e833de0e7b552963ad9bc4b39e56fda9) Signed-off-by: Joshua Watt Signed-off-by: Luca Ceresoli (cherry picked from commit 97955f3c1c738aa4b4478a6ec10a08094ffc689d) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: siggen: Drop pointless break statement

2022-05-12T12:41:42+00:00

(Bitbake rev: f9b563dc02535b594975ac0f5dab8e12d11a6ce0) Signed-off-by: Richard Purdie (cherry picked from commit 42809f6acb79e39042e81d54c28efb92b7481e44) Signed-off-by: Steve Sakoman Signed-off-by: Richard Purdie

bitbake: bitbake-diffsigs: make finding of changed signatures more robust

2022-04-14T08:48:33+00:00

In `runtaskhashes`, the keys contain the absolute paths to the recipe. When working with shared sstate caches (where these absolute paths can be different) we see that compare_sigfiles does not identifiy a changed hash of a dependent task as "changed", but instead as "removed"&"added", preventing the function from recursing and continuing the comparison. By calling `clean_basepaths` before comparing the `runtaskhashes` dicts, we avoid this. (Bitbake rev: 7358378b90b68111779e6ae72948e5e7a3de00a9) Signed-off-by: Adriaan Schmidt Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

bitbake: siggen: Add missing reset handler to hashserv signature generator

2022-03-26T09:27:43+00:00

When we have a client connection, we should close that connection when reset() is called on the siggen. Add the missing function. (Bitbake rev: 770b4ea81b6126b0830e51649c40f7a46c64132a) Signed-off-by: Richard Purdie

bitbake: cooker/siggen: Support exit calls and use for hashserv client

2022-03-26T09:27:43+00:00

We have shutdown functions within the async client code but the siggen doesn't currently call them. We notice on python 3.10 (such as on fedora35) that at exit, there is a stray asyncio process left behind. Usually this doesn't cause problems but it could potentially be a cause of a hang. For general cleanliness and completness, add in hooks to call the exit handler. (Bitbake rev: 9ee3fb95330003878fbd64b3ce8897aad96fcd0f) Signed-off-by: Richard Purdie

bitbake: siggen: Improve ambiguous use of 'dependent'

2022-03-23T12:14:46+00:00

I'd never spotted this until it was pointed out but the task isn't dependent, it is a dependency. Fix this confusing reference. (Bitbake rev: 93395559c9dda734a3dec9ffd9bb2156a71a2c17) Signed-off-by: Richard Purdie

bitbake: bitbake: Replace remaining "abort" usage

2022-02-21T23:37:26+00:00

In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language replace the remaining usage of "abort" in documentation, error messages, and comments with halt/fail/exit as appropriate. A couple of external Javascript API calls in Toaster remain, as they cannot currently be changed. (Bitbake rev: bc27762bf3ffb4a20b58eace5302438c4a526626) Signed-off-by: Scott Murray Signed-off-by: Richard Purdie