summaryrefslogtreecommitdiffstats
path: root/scripts/gen-lockedsig-cache
Commit message (Collapse)AuthorAgeFilesLines
* scripts/gen-lockedsig-cache: Only look at new format sstate objectsRichard Purdie2020-01-061-4/+1
| | | | | | | | | | We can have sstate directories which are mixtures of old and new layout entries. Only use the new format ones since these are the only ones bitbake itself will look at. (From OE-Core rev: 610b314ddc757e6ac8ba4d47921aee1f2f35df97) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add extra directory levelRichard Purdie2020-01-061-3/+10
| | | | | | | | | | | | | | | | | | We're having speed issues on the autobuilder due to the numbers of files in sstate directories. We previously split these by the first two characters of the hash. This change extends this to split by the next two characters as well, creating more layers of directories. This should signifiantly speed up eSDK builds on the autobuilder as the current sstate layout simply isn't scaling there but addresses a general complaint. gen-lockedsig-cache needed to be updated for the new split level sstate. Also update tests for new layout. (From OE-Core rev: d05bde16bdad761ed8f4c0a48de60c649aa33e85) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/gen-lockedsig-cache: Don't list paths which don't existRichard Purdie2019-10-091-0/+2
| | | | | | | | | This avoids failures seen on the autobuilder when generating eSDKs and release sstate copies. (From OE-Core rev: 8ffe6ca984e034976322beea9b16c92d46708d15) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gen-lockedsig-cache: Replace glob lookup with hash to filename lookupKonrad Scherer2019-10-021-5/+42
| | | | | | | | | | | | | | | | | | Using the glob function to map signatures to sstate files is very slow when the sstate is large and accessed over nfs. The lookup now only loads the necessary prefixes and doesn't use glob as all. Unfortunately I don't have access to the systems where the performance isse was noticed and on my test system the glob is fast enough that the performance numbers aren't useful. I could verify that file list returned by the new code is the same. [YOCTO #13539] (From OE-Core rev: ad36335b8592e0387dd36066920cd5ffefd375f8) Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib+scripts: Convert to SPDX license headersRichard Purdie2019-05-091-0/+3
| | | | | | | | | | | | | | | | | | | | | | | This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gen-lockedsig-cache: catch os.link errorbrian avery2017-03-221-1/+5
| | | | | | | | | | | | We do a hard link to speed up sdk creation but if your sstate-cache is across a file system boundary, this tries and fails. This patch catches that error and does a copy instead. (From OE-Core rev: fb9fdd7a74917cdcab039aa3a9a9944b18246fea) Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gen-lockedsig-cache: ensure symlinks are dereferencedPaul Eggleton2016-08-171-3/+4
| | | | | | | | | | | | | If you set up a local mirror in SSTATE_MIRRORS then you can end up with symlinks in SSTATE_DIR rather than real files. We don't want these symlinks in the sstate-cache prodcued by gen-lockedsig-cache, so dereference any symlinks before copying. (From OE-Core rev: d65a6ee9e7a9c63b9a16bdb5025af8a7c6433c4f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/populate_sdk_ext: filter sstate within the extensible SDKPaul Eggleton2016-07-261-2/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use the new oe-check-sstate to filter the sstate artifacts shipped with the extensible SDK by effectively running bitbake within the produced eSDK and and getting it to tell us which tasks it will restore from sstate. This has several benefits: 1) We drop the *-initial artifacts from the minimal + toolchain eSDK. This still leaves us with a reasonably large SDK for this configuration, however it does pave the way for future reductions since we are actually filtering by what will be expected to be there on install rather than hoping that whatever cuts we make will match. 2) We verify bitbake's basic operation within the eSDK, i.e. that we haven't messed up the configuration 3) We verify that the sstate artifacts we expect to be present are present (at least in the sstate cache for the build producing the eSDK). Outside deletion of sstate artifacts has been a problem up to now, and this should at least catch that earlier i.e. during the build rather than when someone tries to install the eSDK. This does add a couple of minutes to the do_populate_sdk_ext time, but it seems like the most appropriate way to handle this. Should mostly address [YOCTO #9083] and [YOCTO #9626]. (From OE-Core rev: 4b7b48fcb9b39fccf8222650c2608325df2a4507) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts: python3: change python to python3 in shebangEd Bartosh2016-06-031-1/+1
| | | | | | | (From OE-Core rev: 4b544ff388497cac82b0585f237900595523e1cb) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gen-lockedsig-cache: fix bad destination path joiningPaul Eggleton2016-01-261-1/+1
| | | | | | | | | | | | | | | When copying the sstate-cache into the extensible SDK, if the source path had a trailing / and the destination path did not, there would be a missing / between the path and the subdirectory name, and you'd end up with subdirectories like "sstate-cacheCentOS-6.7". There are functions in os.path for this sort of thing so let's just use them and avoid the problem. (From OE-Core rev: 5eb8f15c48b5f39a10eb2b63b026cf1ebfd05533) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gen-lockedsig-cache: copy correct native sstate into ext SDKPaul Eggleton2016-01-241-3/+3
| | | | | | | | | | | | | | | | | | When constructing the sstate-cache directory for the extensible SDK, we were copying in any matching native sstate packages, and as the signature doesn't actually change when the distro changes (since NATIVELSBSTRING is just a path separator for the artifacts and is not part of the signature) we ended up copying duplicated packages when the distro changed e.g. upon host distro upgrade. Only search in the NATIVELSBSTRING-named subdirectory for native packages and the issue goes away. Fixes [YOCTO #8885]. (From OE-Core rev: 6c6baf6aa1823b8b20123f505e45c2768a193ad5) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/gen-lockedsig-cache: improve outputPaul Eggleton2015-12-011-3/+10
| | | | | | | | | | | * Print some status when running * When incorrect number of arguments specified, print usage text (From OE-Core rev: ac38d245878b618ddf56f9a68834d344500e45a6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/gen-lockedsig-cache: fix race with temp file creationPaul Eggleton2015-10-191-0/+4
| | | | | | | | | | | | | | | | | | As part of populating the sstate-cache with an artifact (.tgz file) we create a temp file and then atomically move it to the final name. Due to the glob used in this script such temp files were being matched, and between the time they were matched and the time the script started copying files, the temp file may have vanished. This fixes random "No such file or directory" failures building the extensible SDK on build setups where the sstate-cache directory is shared amongst multiple build machines. (From OE-Core rev: 588de5198c641ea1cfc3e01a6d129296bd2f706b) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gen-lockedsig-cache: Allow cross-filesystem copies.Randy Witt2015-02-241-2/+8
| | | | | | | | | | | | Since this previously always tried to use hardlinks you couldn't have the source and destination be on different devices. This change allows for that and also prevents failure in situations where the files already existed. (From OE-Core rev: cf675896340ebed7c4830b93d791ddb08999031f) Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstatesig/sstate: Add support for locked down sstate cache usageRichard Purdie2014-09-171-0/+40
I've been giving things some thought, specifically why sstate doesn't get used more and why we have people requesting external toolchains. I'm guessing the issue is that people don't like how often sstate can change and the lack of an easy way to lock it down. Locking it down is actually quite easy so patch implements some basics of how you can do this (for example to a specific toolchain). With an addition like this to local.conf (or wherever): SIGGEN_LOCKEDSIGS = "\ gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \ eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \ eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \ gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \ " the code at the end of the email will force the hashes to those values for the recipes mentioned. The system would then find and use those specific objects from the sstate cache instead of trying to build anything. Obviously this is a little simplistic, you might need to put an override against this to only apply those revisions for a specific architecture for example. You'd also probably want to put code in the sstate hash validation code to ensure it really did install these from sstate since if it didn't you'd want to abort the build. This patch also implements support to add to bitbake -S which dumps the locked sstate checksums for each task into a ready prepared include file locked-sigs.inc (currently placed into cwd). There is a function, bb.parse.siggen.dump_lockedsigs() which can be called to trigger the same functionality from task space. A warning is added to sstate.bbclass through a call back into the siggen class to warn if objects are not used from the locked cache. The SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a warning or a fatal error. A script is provided to generate sstate directory from a locked-sigs file. (From OE-Core rev: 7e14784f2493a19c6bfe3ec3f05a5cf9797a2f22) (From OE-Core rev: 884d4fa3e77cf32836f14a113c11489076f4a84d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>