summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/siggen.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: siggen: Drop pointless break statementRichard Purdie2022-05-121-1/+0
| | | | | | | | | (Bitbake rev: f9b563dc02535b594975ac0f5dab8e12d11a6ce0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 42809f6acb79e39042e81d54c28efb92b7481e44) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: make finding of changed signatures more robustSchmidt, Adriaan2022-04-141-5/+5
| | | | | | | | | | | | | | | | | In `runtaskhashes`, the keys contain the absolute paths to the recipe. When working with shared sstate caches (where these absolute paths can be different) we see that compare_sigfiles does not identifiy a changed hash of a dependent task as "changed", but instead as "removed"&"added", preventing the function from recursing and continuing the comparison. By calling `clean_basepaths` before comparing the `runtaskhashes` dicts, we avoid this. (Bitbake rev: 7358378b90b68111779e6ae72948e5e7a3de00a9) Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add missing reset handler to hashserv signature generatorRichard Purdie2022-03-261-0/+6
| | | | | | | | | When we have a client connection, we should close that connection when reset() is called on the siggen. Add the missing function. (Bitbake rev: 770b4ea81b6126b0830e51649c40f7a46c64132a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/siggen: Support exit calls and use for hashserv clientRichard Purdie2022-03-261-0/+9
| | | | | | | | | | | | | We have shutdown functions within the async client code but the siggen doesn't currently call them. We notice on python 3.10 (such as on fedora35) that at exit, there is a stray asyncio process left behind. Usually this doesn't cause problems but it could potentially be a cause of a hang. For general cleanliness and completness, add in hooks to call the exit handler. (Bitbake rev: 9ee3fb95330003878fbd64b3ce8897aad96fcd0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Improve ambiguous use of 'dependent'Richard Purdie2022-03-231-1/+1
| | | | | | | | | I'd never spotted this until it was pointed out but the task isn't dependent, it is a dependency. Fix this confusing reference. (Bitbake rev: 93395559c9dda734a3dec9ffd9bb2156a71a2c17) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Replace remaining "abort" usageScott Murray2022-02-211-1/+1
| | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language replace the remaining usage of "abort" in documentation, error messages, and comments with halt/fail/exit as appropriate. A couple of external Javascript API calls in Toaster remain, as they cannot currently be changed. (Bitbake rev: bc27762bf3ffb4a20b58eace5302438c4a526626) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename basehas and taskhash filtering variablesScott Murray2022-02-211-24/+40
| | | | | | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS the derived code variables basewhitelist and taskwhitelist have been renamed to basehash_ignore_vars and taskhash_ignore_tasks, respectively. [RP: Added compatibility mapping code and tweaked naming] (Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure dumpsig output is deterministicRichard Purdie2021-12-091-8/+8
| | | | | | | | | Currently the console output for signature dumps e.g. with bitbake-dumpsig isn't deterministic. Add some sorting to improve that. (Bitbake rev: a663440b1623f97c9c169df5566e429fbc932a53) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/checksum/siggen: Fix taskhashes not tracking file directoriesRichard Purdie2021-11-081-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently if you have something like: SRC_URI = "file://foobar;subdir=${S}" and a file like: foobar/1/somefile and then move it to: foobar/2/somefile the task checksums don't reflect/notice this. The file-checksum fields encode two pieces of data, the file path and whether or not the file exists. Changing the code which uses these fields is problematic. We can however add a "/./" path element which means "include the bit after the marker in the checksum" which the path walking code can use to mark which bits of the path are visible to the fetcher. I'm not convinced this is great design but it does appear to work. (Bitbake rev: b4975d2ecf615ac4c240808fbc5a3f879a93846b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix type conversion issuesRichard Purdie2021-10-171-11/+16
| | | | | | | | | | | | | | | The switch to using json has messed up the type handling as the code does assume that set()s are present. Add a decoder to reconstruct the set() objects. Also fix the change of tuples to lists for the file checksums and fix an existing type bug where dicts insteads of lists was used. Drop some old siginfo format handling code which is now long since obsolete. (Bitbake rev: 2d704842c0928f8dbe78fd081042aa7280af96be) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Change file format of siginfo files to use zstd compressed jsonRichard Purdie2021-10-141-12/+17
| | | | | | | | | | | | | | | | | | | | | | Since OE is about to change to zstd compression of sstate, it would make it timely to convert the siginfo files from pickle which isn't reproducible to json which is both reproducible and also human readable. At the same time add zstd compression. This makes the siginfo files smaller, reprodubicle and easier to debug. Backwards compatibility mixing the two formats hasn't been supported since in reality if sstate changes at the same time, files will be in one format or the new one but comparing mixed formats won't make much sense. Since json doesn't support sets, we translate them into lists in the files themselves. We only use sets in bitbake since it makes things easier in the internal code, sorted lists are fine for the file format. [YOCTO #13973] (Bitbake rev: 22c18494c9072788e6e26eb73de70378ae5c5bf5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix sorting in diff outputRichard Purdie2021-09-261-7/+7
| | | | | | | | | The diff output isn't deterministic at the moment as the sets can have differing ordering. Sort the output so it is consistent. (Bitbake rev: 117830c1d7ef3e53052fa326e1ca62c5c3946c45) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Switch to using new override syntaxRichard Purdie2021-08-021-2/+2
| | | | | | | | | | | | | | | | | This change updates the datastore to use the new override syntax using colons instead of underscores exclusively. It is expected layers would have to be converted to work with bitbake after this change. Supporting mixed syntax isn't possible, it is only feasible to have one internal representation of overrides. Whilst we can't warn for every possible override that may be set in the old format, show errors for _append/_prepend/_remove since those should never be present. (Bitbake rev: 7dcf317cc141dc980634f8c18bfa84f83e57206a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add bb.utils.rename() helper function and use for renamingDevendra Tewari2021-05-061-1/+1
| | | | | | | | | | | | | | | | | | | | os.rename can fail for example an incremental build in Docker fails with: OSError: [Errno 18] Invalid cross-device link when source and destination are on different overlay filesystems. Rather than trying to fix every call site, add a wrapper in bb.utils for renames. We can then handle cross device failures and fall back to shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code. [YOCTO #14301] (Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b) Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Use generic ConnectionErrorPaul Barker2021-04-271-3/+3
| | | | | | | | | | The Python built-in ConnectionError type can be used instead of a custom HashConnectionError type. This will make code refactoring simpler. (Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1) Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-3/+3
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Don't treat mc recipe (Midnight Commander) as a multiconfig ↵Tomasz Dziendzielski2021-02-061-1/+1
| | | | | | | | | | | | | | | | target When we run `devtool build mc` recipe's task dependencies are expanded to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig and "do_package_sysroot" as multiconfigname. | ERROR: Multiconfig dependency mc:do_populate_sysroot depends on | nonexistent multiconfig configuration named do_populate_sysroot (Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Remove broken optimisationRichard Purdie2020-11-031-7/+1
| | | | | | | | | | | When a single signature is locked, dependent task checksum calculations fail. This in turn is because get_unihash cannot be cached correctly by this function. Remove that has turned out to be a poor optimisation to avoid that bug. (Bitbake rev: 7ae2b73d131491ab5bf09fd6055e1fdb67a560c6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: use correct umask when writing siginfoRoss Burton2020-09-301-1/+2
| | | | | | | | | | | | | | | We try to write sstate with group-write permissions so that sstate-cache can be shared between multiple users. However the siginfo files are created in various tasks which may set their own umask (such as do_populate_sysroot, 0022). This results in no group write permission on the intermediate directories, which is fatal when sharing a cache. Fix this by wrapping the siginfo mkdir in a umask change to 0002. (Bitbake rev: 75d9ef04a908e366633b255d23ab3275f6860d3a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: siggen: clean_basepath: improve perfo and readabilityJean-Francois Dagenais2020-09-231-10/+20
| | | | | | | | | | | | | | | This change improves performance by reducing runtime about 33% for typical inputs. (using test_clean_basepath_performance) It is also easier to read, and slightly more resilient to future changes since it doesn't mention 'virtual' anymore. (Bitbake rev: 27b53186fa67d281d29b2f8e15bcff8dc2557b8a) Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com> Co-Developed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: clean_basepath: remove recipe full path when virtual:xyz ↵Jean-Francois Dagenais2020-08-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | present Before this fix, this example basepath (a): virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile would get incorrectly "cleaned" into: helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb When searching backwards in `a` trying to isolate the 'virtual:xyz' to add it to the end of the string, we need to consider `a` still has the recipe path and taskname. So stoping the rsplit after only 1 split is not enough. We want to reach the second ':' from the end. This way, we obtain: helloworld/helloworld_1.2.3.bb:do_compile:virtual:native reviewed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> (Bitbake rev: d193d93422a0ad62aa35b5d4ca5da8d422f72180) Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix error when hash equivalence has an exceptionJoshua Watt2020-06-281-0/+1
| | | | | | | | | | The code that handled exceptions from the hash equivalence client was raising an exception itself because hashserv.client wasn't imported (Bitbake rev: a76290dfc6f34ff9f6efdb13a6db74b6b4759daf) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: siggen: Pass all data caches to hash functionsJoshua Watt2020-06-121-16/+62
| | | | | | | | | | | | | | | | | | | Passing all the data caches to the task hashing functions allows them to correctly account for mcdepends in task signatures. This allows tasks to be correctly re-run when a mcdepends changes. By default, the legacy behavior is maintained for derived signature generators by passing a special proxy object that can either be used to access all multiconfigs or the legacy behavior. If a derived signature generator is updated, it can set the supports_multiconfig_datacaces property to instruct bitbake it deals with multiconfigs properly. [YOCTO #13724] (Bitbake rev: 8ff9203de4fce9c104c2987d86980c9f34036b97) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/siggen: Don't expand ignored variablesRichard Purdie2020-04-061-1/+1
| | | | | | | | | | | | | | | | | | If a variable is in the signature whitelist, we'd currently expand it, then later ignore the data. This is problemtic for code which has effects when expanded, recently source date epoch in OE-Core for example. We don't actually need to do this, if we pass the whitelist into the earlier function it can avoid the expansion. This also also give a small performance boost since we avoid running code in some cases. [YOCTO #13581] (Bitbake rev: f483ee4a869fb1dafbe4bdf2da228cdaa40b38bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Lower hash equivalence loggingJoshua Watt2020-03-131-4/+4
| | | | | | | | | | | | | | Lowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json) [YOCTO #13813] (Bitbake rev: 2ddb649ea31afe052f08e3969e36abf6fb515bc2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Log hash equivalence with a different loggerJoshua Watt2020-03-131-8/+9
| | | | | | | | | | Switches the hash equivalence logging to use a different logger so that it can be easily filtered out with python's structured logging. (Bitbake rev: 20bb29ef973e9c5483eb50a74550ea207637367b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/siggen: Empty siggen cache during parsingRichard Purdie2020-02-171-0/+11
| | | | | | | | | | | | | | | | | | | When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. (Bitbake rev: 5d98d8e39bba42f458532b1eef3619f2321d8a2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Avoid cache mismatch issues with locked sigsRichard Purdie2020-02-081-0/+4
| | | | | | | | | If locked sigs are in use this function makes little sense, need to avoid generating mismatch warnings. (Bitbake rev: 27ad9c1d468fba858a4adeb56b605227b415ae0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Cache unihash values to avoid cache lookupRichard Purdie2020-02-081-5/+17
| | | | | | | | | | Add unihash cache of values to speed up cache lookup. This avoids the overhead of the disk based check functions. (Bitbake rev: 5c9cc45b60904a1c355db9bf9c4495f1b25aca37) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Optimise get_unihash disk based cache handlingRichard Purdie2020-02-081-18/+32
| | | | | | | | | | | | | | | | Currently the cache can grow huge since any previously used hash is retained in the cache. This change moves to use one hash per task which improves the speed of the functions considerably. Currently performance is an issue, as are very large cache files and cache load time. By moving to a single hash per task, the shorted filename as a key is no longer usable as the same recipe has multiple variants for the same filename so this has to change. (Bitbake rev: ed764e7fcf04b6d0ba6b4cac7415b1ee8f492865) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Test extra cross/native hashserv methodRichard Purdie2020-01-021-6/+21
| | | | | | | | | | | | | | | | Hack the hashserv to allow extra data to be injected into the hashserv method. This allows OE-Core to handle cases where there are multiple sstate objects for the same taskhash, e.g. native/cross objects based upon BUILD_ARCH or the host distro (when uninative isn't used). This has been tested and proven to be very effective. We will likely rework the code to improve how this is handled but for now this improves automated builds until we can get to that refactoring and more invasive changes. (Bitbake rev: 0a09b0fa03d1afc08037964dc63a18ef7cff9c78) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on ↵Aníbal Limón2019-12-301-2/+3
| | | | | | | | | | | | | | | directories The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory. Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn" (Bitbake rev: 923aff060d8aba8456979c35b16d300ba7c13ff9) Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Split get_tashhash for performanceRichard Purdie2019-12-161-9/+24
| | | | | | | | | | | | | | | There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash. Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does. This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing. (Bitbake rev: 6a32af2808d748819f4af55c443578c8a63062b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix performance issue in get_unihashRichard Purdie2019-12-151-3/+3
| | | | | | | | | | There is a significant performance issue in get_unihash(). The issue turns out to be the lookups of setscene tasks. We can fix this by using a set() instead of the current list. (Bitbake rev: 1e561672d039ebfb8cd0e0654a44dcf48513317c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "siggen: Fix hashequiv performance issues"Richard Purdie2019-12-121-13/+14
| | | | | | | | This reverts commit c4b8440f730c33eaf9f818b856ae81b2f1017fec. The logic in this change is flawed and needs to be re-thought. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix hashequiv performance issuesRichard Purdie2019-12-121-14/+13
| | | | | | | | | | | | | | | | | | | We're seeing huge slowdowns on large builds on the autobuilder. A qemux86 world build was running really slowly, a primary feature was lots of rehashing going on due to an sstate change which caused a rebuild when all output should be identical. This was traced to the hundreds of thousands of calls to get_unihash() from get_taskash(). If we simplify the unitaskhashes data structure, we can bypass the function call and access the data directly. In local profile charts, this significanly sped up process_possible_migrations(), both on profiles and visually. Whilst this change doesn't aid readability, it does solve an otherwise huge performance issue. (Bitbake rev: c4b8440f730c33eaf9f818b856ae81b2f1017fec) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure new unihash propagates through the systemRichard Purdie2019-12-091-0/+1
| | | | | | | | | | Its possible the new unihash may not exist in sstate. Currently the code would create an sstate object with the old hash however this updates it to create the object with the new unihash. (Bitbake rev: abcaa1398031fa5338a43859c661e6d4a9ce863d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Avoid taskhash mismatch errors for nostamp tasks when ↵Richard Purdie2019-12-061-4/+8
| | | | | | | | | | | | | | | | | | | dependencies rehash An example: NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded Is caused by a rehash in a dependency happening somewhere earlier in the build and the taint being reset. Change the code so that nostamp taints are preserved to avoid the issue. (Bitbake rev: 61624a3fc38e8546e01356d5ce7a09f21e7094ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Allow handling of equivalent hashesRichard Purdie2019-12-041-0/+26
| | | | | | | | | | | | | Based on the hashserv's new ability to accept hash mappings, update runqueue to use this through a helper function in siggen. This addresses problems with meta-extsdk-toolchain and its dependency on gdb-cross which caused errors when building eSDK. See the previous commit for more details. (Bitbake rev: 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix hashequiv bug where new hash wasn't referenced correctlyRichard Purdie2019-11-251-0/+1
| | | | | | | | | | | If a hash is reported to the hash server, the stamp written out by the current task didn't account for any new hash the server may have provided. Fix this so the correct stamp is written. This means "bitbake X; bitbake X" no longer rebuilds lots of things when hashequiv is active. (Bitbake rev: 9dc5178f56ab8ae37e1a646c09117c503e48d072) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Remove full path from unitaskhashes keysRichard Purdie2019-09-301-2/+2
| | | | | | | | | | The full paths make the cache useless in the sdk. They also bloat the cache size. They're for human debugging benefit only so compromise and reduce this to the filename. (Bitbake rev: 3b275c4083eae1d3781f0862919af9de83932b0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Avoid writing misleading sigdata filesRichard Purdie2019-09-301-1/+2
| | | | | | | | | | Use the unihash in the output filename of sigdata files else the contents of stamp directories is misleading. Write the unihash into the singature to make it clear what happened. (Bitbake rev: feb01ee54d3706fe93768f332054c7532f7209e4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue: Fix signature mismatch issuesRichard Purdie2019-09-301-1/+5
| | | | | | | | | | | | We need to set the setscene tasklist before we call into the taskhash/unihash code else the behaviour is inconsistent. Avoid reporting hashes for non setscene tasks since we'd never query that. (Bitbake rev: 419a7840b8627278db694029c25df00214d01d96) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure setscenetasks list is available to worker contextRichard Purdie2019-09-301-2/+3
| | | | | | | | | | | The setscenetasks list needs to be available in the worker contexts else the signature behaviour there mismatches what the server does. Add the data to get/set_taskdata to ensure this happens. (Bitbake rev: 632980ef90fe126b7ba3d138f4d574ae05914779) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix attribute error when hashserver failsJoshua Watt2019-09-271-2/+2
| | | | | | | | | | | | The HashConnectionError class was moved to the client module and needs to be updated. [YOCTO #13537] (Bitbake rev: 9fb862685e5e5a2aa534bc25cab1e4158d708b40) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rework hash equivalenceJoshua Watt2019-09-181-48/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Optimise hash equiv queriesRichard Purdie2019-09-161-0/+10
| | | | | | | | | We only have hash equivalence for setscene tasks so only query the server for those, reducing the number of connections needed. (Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Add hashserv+runqueue testRichard Purdie2019-08-061-4/+19
| | | | | | | | Add a test which tests the runqueue adaptations for hash equivalency. (Bitbake rev: 477321d0780df177c1582db119c2bb6795912fc6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Clean up task reference formatsRichard Purdie2019-08-061-81/+71
| | | | | | | | | | | | | | | | | | Currently siggen uses the format "<filename>.<taskname>" for referencing tasks whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the separator everywhere. This is an API breaking change since the cache is affected, as are siginfo files and any custom signature handlers such as those in OE-Core. Ultimately this will let us clean up and the accessor functions from runqueue, removing all the ".rsplit(".", 1)[0]" type code currently all over the place. Once a standard is used everwhere we can update the code over time to be more optimal. (Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Convert to use self.unitaskhashesRichard Purdie2019-08-061-4/+8
| | | | | | | | | Rather than metadata driven sqlite databases for communication, use bitbake's unitaskhashes variable instead. (Bitbake rev: a0d941c787cf3ef030d190903279d311bc05d752) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-04-22 11:35:45 +0000