| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Report which task dependencies in BB_TASKDEPDATA are included in the
taskhash. This allows tasks to identify which tasks dependencies may
change without the task re-running. Knowing this information is
important for tasks that want to transfer information from dependencies
(such as SPDX)
(Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now that the bitbake hash server supports SSL connections, we need to
capture a few environment variables which can affect the ability to
connect via SSL. Note that the variables are only put in place to affect
the environment while actually invoking the server
[RP: Tweak to use BB_ORIGENV as well]
[RP: Tweak to handle os.environ restore correctly]
(Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Adds support for hashserver credentials to be specified in the
SignatureGenerator
(Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Adds API to query if unihashes are known to the server in parallel
(Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Implements a new API called get_unihashes() that allows for querying
multiple unihashes in parallel.
The API is also reworked to make it easier for derived classes to
interface with the new API in a consistent manner. Instead of overriding
get_unihash() to add custom handling for local hash calculating (e.g.
caches) derived classes should now override get_cached_unihash(), and
return the local unihash or None if there isn't one.
(Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
error
* when comparing 2 tmp/stamps/*do_configure.sigdata* I got
TypeError("filename must be a str or bytes object, or a file")
but both files I was comparing were
Zstandard compressed data (v0.8+), Dictionary ID: None
according to "file"
with TypeError catched to show which file it failed to open I got better
error which shows it was trying to read "do_prepare_recipe_sysroot.sigdata"
file now and after a while you might notice that it's not just the expected
file, but a dict with 'path', 'sstate', 'time'.
Fix that in bitbake-diffsigs but keep the TypeError and add OSError
in case it will eventually walk on file which isn't zstd compressed
pipecompress throws CompressionError.
ERROR: Failed to open {'path': '5.15.do_prepare_recipe_sysroot.sigdata.99b12a401341a0df7c3553cb00c87a7674295496bd5c25ed71764ee0d0fb8eb8', 'sstate': False, 'time': 1707136354.991718}: filename must be a str or bytes object, or a file
Traceback (most recent call last):
File "bitbake/bin/bitbake-diffsigs", line 192, in <module>
output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 1039, in compare_sigfiles
recout = recursecb(dep, a[dep], b[dep])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/bin/bitbake-diffsigs", line 102, in recursecb
out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 857, in compare_sigfiles
raise err
File "bitbake/lib/bb/siggen.py", line 853, in compare_sigfiles
with bb.compress.zstd.open(a, "rt", encoding="utf-8", num_threads=1) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/zstd.py", line 12, in open
return bb.compress._pipecompress.open_wrap(ZstdFile, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/_pipecompress.py", line 59, in open_wrap
raise TypeError("filename must be a str or bytes object, or a file")
TypeError: filename must be a str or bytes object, or a file
* if I replace zstd file with just plaintext it fails with:
$ echo foo > foo
$ echo foo > bar
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
with this change it shows the name of the file which it failed to uncompress:
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Failed to open sigdata file 'foo': Process died with 1
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
(Bitbake rev: f3f843a4fd6e0a9e8f6edef5dd3cf1fce29c50ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Since we need to change the form of the siggen function, we need to add versioning
and some verison checks. This means if a newer bitbake is used with older metadata
we can detect it.
(Bitbake rev: 721556568413508213d22c29985e305a45a8d68a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
file checksums are part of the data checksummed
to generate the task hash. The list of file checksums
was not ordered.
In this commit we make sure the task hash checksum takes
a list of checksum data that is ordered by unique file name
thus guaranteeing reproducibility.
(Bitbake rev: 5293a1b36eeb89f57577cb709ec7f293909039a1)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
The debug in the comments was out of date. It is still useful so update the
code sample to the new code needed.
(Bitbake rev: fa2724069ea7028939d816cb5ccd0e7c1bed09a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: 9a98851ef86adea3b05c4eb7c44e7ea3fbbb4420)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The runtaskdep data in siginfo files was written out with full paths to
the bb files, matching bitbake's internal "unique key" ID for recipes/tasks.
When originally implemented this made sense.
Over time, the main use for the data in siginfo files has become to match
against other siginfo files to debug changes of hash calcuations. The
recipename data is not useful for this as the siginfo filenames use PN
instead which can often be derived from the recipe filename but not always.
It is time to throw away the 'tid' data format and switch over the use a
hybrid PN form which includes the multiconfig. That can be easily stripped
off in the find_siginfo code in oe-core.
The other purpose of having a sortable dependency ID is retained and the
multiconfig needs to be included to allow the taskhashes to be processed
and calculated correctly. PN is meant to be unique between recipes, only
one would ever be built so using PN in this location is fine.
The one risk of this change is there isn't any compatibility to the old
format. I'm not convinced we should spend time complicating the code
with it. This change will change the taskhashes everywhere so the only
mixing of old and new siginfo files will be either through hash equivalence
or through users using the tool against old and new info files manually
which will give some weird output but it should be clear they're in
different formats as there would be large paths from the old files not
present in the new ones.
We have options to add backwards compatibility if some issue is found
to need that.
(Bitbake rev: 637933e2e5a59228a8d17aae4160551cab5f2f61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bitbake f68682 changed the logger's debug() method to be compatible with
logging.debug(), but this caller was still using the old API where you
can pass an integer as the first argument:
WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]')
Instead, call bbdebug() which has the priority argument.
(Bitbake rev: 18d4f9e8387f7994cf6d46300e25dda1c3a593b2)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython
has an optimization for statements in the form "a = a + b" or "a += b".
It seems that this line does not get optimized, because it has a form a = a + b + c:
data = data + "./" + f.split("/./")[1]
For that reason, it does a copy of data for each iteration, potentially copying megabytes
of data for each iteration.
Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds
instead of 45 seconds on my test setup where SRC_URI points to a big directory.
Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython:
"do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b"
However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup:
data = ''.join((data, "./", f.split("/./")[1]))
I have changed the other lines to also use += for consistency only, however those were in the form a = a + b
and were optimized already.
Co-authored-by: JJ Robertson <jrobertson@snap.com>
(Bitbake rev: 195750f2ca355e29d51219c58ecb2c1d83692717)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Tweak the code to remove duplication and only set if the attribute isn't
already there to avoid overwriting.
(Bitbake rev: 513e6c4e9233e0d0bc31e1169077fdbf9aaf4ec3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"fn" can mean different things in bitbake, we added support for class
extensions and then mutlticonfigs by extending it. In siggen, it generally
means that mc is prefixed to it and that it is a virtual filename.
Replace "fn" with "mcfn" in the code to make this clearer as if I'm getting
confused, everyone else likely is as well. "mcfn" is sometimes referred
to as taskfn as well but mcfn is probably the easiest to understand as the
taskname isn't included.
(Bitbake rev: e1c1139ab90f8da1b5036db11d943daefbe87859)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The current method of passing either a task's datastore, or
dataCaches and a filename into the stamp functions is rather
horrible.
Due to the different contexts, fixing this is hard but we do control
the bitbake side of the API usage so we can migrate those to use other
functions and then only support a datastore in the public bb.build API
which is only called from task context in OE-Core.
(Bitbake rev: c79ecec580e4c2a141ae483ec0f6448f70593dcf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
One of the challenges in maintaining the code is that it sometimes uses
a datacaches structure and sometimes a datastore. Rather than continue
the current dual API madness, have the worker contexts create a dummy
datacaches structure with the entries we need. Whilst this does need to
be kept in sync with the real structure, that doesn't change and this
allows the code to be simplified.
With this new approach, we can unify the stamps dependency code again.
(Bitbake rev: c6d325fc9b53e9d588ab273ee3c2a99a70fba42c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
feature for signature dumping
Now that we have cache support for the taskdep/gendep/lookupcache data,
we can switch to use that cooker feature and skip the secondary reparse to
write the sig files. This does make the initial parse longer but means the
secondary one isn't needed.
At present parsing with the larger cache isn't optimal but we have plans
in place which will make this faster than the current reparse code being
removed here.
(Bitbake rev: 5951b5b56449855bc2a30146af65eb287a35fcef)
(Bitbake rev: 1252e5bce51ae912ecff9dcc354a371786ff2c72)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
It is becomming clear the siggen needs access to our cache data but we
can't always obtain it in the contexts we need to. Add it directly,
meaning over time we should be able to simplify the APIs and stop
convoluting new ones!
(Bitbake rev: 6b213590ed0e77683cf7fbce6bbe9605ddecf3d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
do_setscene was from a different era before our modern setscene
per task code. It hasn't been used for years so remove some old
obsolete references to it.
(Bitbake rev: ef72282298f7c4db74383c23bb0251dd06d3c6d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
All siggens in common use should now support multiconfig, drop the
compatibility code.
(Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
bitbake cache
Being able to track siggen hash construction data can be useful for cache
debugging. For now, add an extra cache class which contains this information.
It can be enabled in the same way as the hob data cache through a feature flag
to cooker. This allows us to experiment with the data without carrying larger
patches around and ultimately may allow use to have a hash mismatch debugging
mode that is more easily enabled.
(Bitbake rev: 0736a8a03da8b774fafbd28f746bef4705378049)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The basehash data is really internal bitbake data and passing an object
directly is more efficient than creating and then extracting variables.
This will match the format of other data we may optionally wish to
store in the cache so more to the more efficient method. Nothing I
can see is using this data today (and nothing should be).
(Bitbake rev: e621093a1bf37cd75ede3fb77ab6845556870fc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Python handles frozensets a little more optimally than normal sets. Once we
finish parsing, we don't edit this data so we can convert to them.
To do that, we need to stop changing them so process ignore_deps earlier
then we can freeze the data and keep it frozen.
This has the side effect that we need to be careful to sort the data
in some of the variables when calculating the hashes.
Overall this does seem to show a decent parsing time speed improvement
of 20-25% in a local test but this would be highly setup dependent.
Also ensure the sigdata can handle exported frozenset and make it import
back to them instead of sets.
(Bitbake rev: 19475627c363a52da49ec144422c87448ff2a6c5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Where copyright headers were not present, add them to make things
clear.
(Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Signature generation uses mkstemp() to get a file descriptor to a unique
file and then write the signature into it. However, the unique file name
generation in glibc is based on the system timestamp, which means that
with highly parallel builds it is more likely than one might expect
expected that a conflict will occur between two different builder nodes.
When operating over NFS (such as a shared sstate cache), this can cause
race conditions and rare failures (particularly with NFS servers that
may not correctly implement O_EXCL).
The signature generation code is particularly susceptible to races since
a single "sigtask." prefix used for all signatures from all tasks, which
makes collision even more likely.
To work around this, add an internal implementation of mkstemp() that
adds additional truly random entropy to the file name to eliminate
conflicts.
(Bitbake rev: 97955f3c1c738aa4b4478a6ec10a08094ffc689d)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
We see rare failures in eSDK generation with zero sized unihash cache files. This is
almost certainly due to races in the cache file being updated. Add a copy function
where the cache file can be copied with the lock held to avoid this.
(Bitbake rev: 9e72a3915e36cb843037040cb68a82077436dbef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
compare_sigfiles() recursively calculates differences on all dependent
tasks with changed hashes. This is done in arbitrary/alphabetical order, and
only the last of those results is returned, while everything else is discarded.
This changes the behavior to instead return the first difference and not calculate
any more, which significantly speeds up diffs of tasks with many dependencies.
(Bitbake rev: ea6a676c9aa2864c2eff40eea41ba09ce903a651)
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: 42809f6acb79e39042e81d54c28efb92b7481e44)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In `runtaskhashes`, the keys contain the absolute paths to the recipe. When
working with shared sstate caches (where these absolute paths can be different)
we see that compare_sigfiles does not identifiy a changed hash of a dependent
task as "changed", but instead as "removed"&"added", preventing the function
from recursing and continuing the comparison.
By calling `clean_basepaths` before comparing the `runtaskhashes` dicts, we
avoid this.
(Bitbake rev: 7358378b90b68111779e6ae72948e5e7a3de00a9)
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
When we have a client connection, we should close that connection when reset()
is called on the siggen. Add the missing function.
(Bitbake rev: 770b4ea81b6126b0830e51649c40f7a46c64132a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We have shutdown functions within the async client code but the siggen
doesn't currently call them. We notice on python 3.10 (such as on fedora35)
that at exit, there is a stray asyncio process left behind. Usually this
doesn't cause problems but it could potentially be a cause of a hang.
For general cleanliness and completness, add in hooks to call the exit handler.
(Bitbake rev: 9ee3fb95330003878fbd64b3ce8897aad96fcd0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
I'd never spotted this until it was pointed out but the task isn't dependent,
it is a dependency. Fix this confusing reference.
(Bitbake rev: 93395559c9dda734a3dec9ffd9bb2156a71a2c17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining usage of "abort" in documentation, error
messages, and comments with halt/fail/exit as appropriate.
A couple of external Javascript API calls in Toaster remain, as
they cannot currently be changed.
(Bitbake rev: bc27762bf3ffb4a20b58eace5302438c4a526626)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS
BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS
the derived code variables basewhitelist and taskwhitelist have been
renamed to basehash_ignore_vars and taskhash_ignore_tasks,
respectively.
[RP: Added compatibility mapping code and tweaked naming]
(Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Currently the console output for signature dumps e.g. with bitbake-dumpsig
isn't deterministic. Add some sorting to improve that.
(Bitbake rev: a663440b1623f97c9c169df5566e429fbc932a53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently if you have something like:
SRC_URI = "file://foobar;subdir=${S}"
and a file like:
foobar/1/somefile
and then move it to:
foobar/2/somefile
the task checksums don't reflect/notice this. The file-checksum fields
encode two pieces of data, the file path and whether or not the file
exists. Changing the code which uses these fields is problematic.
We can however add a "/./" path element which means "include the bit
after the marker in the checksum" which the path walking code can use
to mark which bits of the path are visible to the fetcher.
I'm not convinced this is great design but it does appear to work.
(Bitbake rev: b4975d2ecf615ac4c240808fbc5a3f879a93846b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The switch to using json has messed up the type handling as the code
does assume that set()s are present. Add a decoder to reconstruct
the set() objects. Also fix the change of tuples to lists for the
file checksums and fix an existing type bug where dicts insteads of
lists was used.
Drop some old siginfo format handling code which is now long since
obsolete.
(Bitbake rev: 2d704842c0928f8dbe78fd081042aa7280af96be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since OE is about to change to zstd compression of sstate, it would make it
timely to convert the siginfo files from pickle which isn't reproducible
to json which is both reproducible and also human readable. At the same time
add zstd compression. This makes the siginfo files smaller, reprodubicle
and easier to debug.
Backwards compatibility mixing the two formats hasn't been supported since
in reality if sstate changes at the same time, files will be in one format
or the new one but comparing mixed formats won't make much sense.
Since json doesn't support sets, we translate them into lists in the files
themselves. We only use sets in bitbake since it makes things easier in
the internal code, sorted lists are fine for the file format.
[YOCTO #13973]
(Bitbake rev: 22c18494c9072788e6e26eb73de70378ae5c5bf5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
The diff output isn't deterministic at the moment as the sets can have differing
ordering. Sort the output so it is consistent.
(Bitbake rev: 117830c1d7ef3e53052fa326e1ca62c5c3946c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change updates the datastore to use the new override syntax using
colons instead of underscores exclusively. It is expected layers would
have to be converted to work with bitbake after this change.
Supporting mixed syntax isn't possible, it is only feasible to have
one internal representation of overrides.
Whilst we can't warn for every possible override that may be set in the
old format, show errors for _append/_prepend/_remove since those should
never be present.
(Bitbake rev: 7dcf317cc141dc980634f8c18bfa84f83e57206a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
os.rename can fail for example an incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than trying to fix every call site, add a wrapper in bb.utils
for renames. We can then handle cross device failures and
fall back to shutil.move. The reason os.rename is still used is
because shutil.move is too slow for speed sensitive sections of code.
[YOCTO #14301]
(Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b)
Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
The Python built-in ConnectionError type can be used instead of a custom
HashConnectionError type. This will make code refactoring simpler.
(Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
target
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.
| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot
(Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
When a single signature is locked, dependent task checksum calculations
fail. This in turn is because get_unihash cannot be cached correctly
by this function. Remove that has turned out to be a poor optimisation
to avoid that bug.
(Bitbake rev: 7ae2b73d131491ab5bf09fd6055e1fdb67a560c6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We try to write sstate with group-write permissions so that sstate-cache
can be shared between multiple users. However the siginfo files are
created in various tasks which may set their own umask (such as
do_populate_sysroot, 0022). This results in no group write permission
on the intermediate directories, which is fatal when sharing a cache.
Fix this by wrapping the siginfo mkdir in a umask change to 0002.
(Bitbake rev: 75d9ef04a908e366633b255d23ab3275f6860d3a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change improves performance by reducing runtime about 33% for typical
inputs. (using test_clean_basepath_performance)
It is also easier to read, and slightly more resilient to future changes since it doesn't
mention 'virtual' anymore.
(Bitbake rev: 27b53186fa67d281d29b2f8e15bcff8dc2557b8a)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Co-Developed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
present
Before this fix, this example basepath (a):
virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile
would get incorrectly "cleaned" into:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb
When searching backwards in `a` trying to isolate the 'virtual:xyz' to add
it to the end of the string, we need to consider `a` still has the recipe
path and taskname. So stoping the rsplit after only 1 split is not enough.
We want to reach the second ':' from the end.
This way, we obtain:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native
reviewed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
(Bitbake rev: d193d93422a0ad62aa35b5d4ca5da8d422f72180)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
The code that handled exceptions from the hash equivalence client was
raising an exception itself because hashserv.client wasn't imported
(Bitbake rev: a76290dfc6f34ff9f6efdb13a6db74b6b4759daf)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|