summaryrefslogtreecommitdiffstats
path: root/meta/classes/sstate.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* classes: Update classes to match new bitbake class scope functionalityRichard Purdie2022-08-121-1364/+0
| | | | | | | | | Move classes to classes-global or classes-recipe as appropriate to take advantage of new bitbake functionality to check class scope/usage. (From OE-Core rev: f5c128008365e141082c129417eb72d2751e8045) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: Add SPDX license identifiersRichard Purdie2022-08-121-0/+2
| | | | | | | | | | | | As stated in our top level license files, the license is MIT unless otherwise stated. Add SPDX identifers accordingly. Replace older license statementa with the standardised syntax. Also drop "All Rights Reserved" expression as it isn't used now, doesn't mean anything and is confusing. (From OE-Core rev: 081a391fe09a21265881e39a2a496e4e10b4f80b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: Add copyright statements to files without oneRichard Purdie2022-08-121-0/+4
| | | | | | | | | Where there isn't a copyright statement, add one to make it explicit. Also drop editor config lines where they were present. (From OE-Core rev: 880c1ea3edc8edef974e65b2d424fc36809ea034) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Use the python3 ThreadPoolExecutor instead of the OE ThreadedPoolJose Quaresma2022-07-151-13/+19
| | | | | | | | | | | For the FetchConnectionCache use a queue where each thread can get an unsed connection_cache that is properly initialized before we fireup the ThreadPoolExecutor. (From OE-Core rev: eb6a6820928472ef194b963b606454e731f9486f) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* abi_version/sstate: Bump hashequiv and sstate versions due to git changesRichard Purdie2022-05-071-1/+1
| | | | | | | | | | | | | The autobuilder sstate was corrupted via incorrect equiavlences caused by the security fix to git and the poor interaction that had with SCM version checks under fakeroot/pseudo. Bump the versions to enable a clean slate to work off. (From OE-Core rev: 3b6672730372e130d4d72b683fc3150911964745) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Allow optimisation of do_create_spdx task dependenciesRichard Purdie2022-03-161-1/+1
| | | | | | | | | do_create_spdx tasks don't need their dependencies so we can optimistion this as we do for some other tasks. (From OE-Core rev: 6723a045c3a46537bb76111f8306b5960e532522) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/scripts: Improve internal variable namingSaul Wold2022-03-101-2/+2
| | | | | | | | | Update internal variable names to improve the terms used. (From OE-Core rev: f408068e5d7998ae165f3002e51bc54b380b8099) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: inside the threadedpool don't write to the shared localdataJose Quaresma2022-03-071-1/+1
| | | | | | | | | | | When inside the threadedpool we make a copy of the localdata to avoid some race condition, so we need to use this new localdata2 and stop write the shared localdata. (From OE-Core rev: 1fa763b2022822a76fde541724e83e1977833d03) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: do not add TARGET_ARCH to pkgarch for cross recipes.Alexander Kanavin2022-03-021-2/+1
| | | | | | | | | | | | This is redundant (target arch is already in PN), and breaks compiling a cross-canadian toolchain, as that needs populating the sysroot with two different native-hosted toolchains built from cross recipes. Inserting TARGET_ARCH allows only one or the other. (From OE-Core rev: 33fc1792cd782feb8dbb4285e3006bb588f7978f) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/scripts: Automated conversion of OE renamed variablesRichard Purdie2022-02-211-13/+13
| | | | | | (From OE-Core rev: aa52af4518604b5bf13f3c5e885113bf868d6c81) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Setup fetcher environment in advanceRichard Purdie2022-02-201-9/+12
| | | | | | | | | | The threading code here can race as the fetcher changes the environment which is shared between the threads. By setting it up in advance, it isn't changed and therefore no longer races. (From OE-Core rev: cabc3cc2eac5916e63340c18d1074411b377ced4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix up additional debugging when fetch fails occurAlexander Kanavin2022-02-101-2/+4
| | | | | | | | | This should print the actual stack trace, for real :) (From OE-Core rev: 9fbc7d6a2b2388e70a76cb97285e2f60bdc79624) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: additional debugging when fetch fails occurAlexander Kanavin2022-02-081-2/+2
| | | | | | | | | | | | | | | | Autobuilder has been showing things like: Checking sstate mirror object availability...ERROR: SState: cannot test file://universal/d9/bc/sstate:xz-native:x86_64-linux:5.2.5:r0:x86_64:8:d9bced04b194d5fc8d778eb8a0d674fa7375a42c8c50a9237e6d7672e9e7a00c_deploy_source_date_epoch.tar.zst: TimeoutError('timed out') ERROR: SState: cannot test file://37/a0/sstate:libgcc-initial:core2-64-poky-linux:11.2.0:r0:core2-64:8:37a0a5aec105a0822df098f15ff2b67d0e7220204742b5d2b1f7958dda6fa5ce_deploy_source_date_epoch.tar.zst: TimeoutError('timed out') ERROR: SState: cannot test file://universal/11/a4/sstate:libpciaccess-native:x86_64-linux:0.16:r0:x86_64:8:11a4d6c3a2e147ef7dd5f31c0ff2a91271dad49b561d8aa24849115081cf1842_deploy_source_date_epoch.tar.zst: TimeoutError('timed out') done. which is not helpful. To find out what really happened and where, the original traceback is needed too. (From OE-Core rev: 80a9052221fb2a12e7c652f2d1764101202fdb90) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* abi_version/sstate: Bump for hash equivalence fixRichard Purdie2022-02-051-1/+1
| | | | | | | | | With the hash equivalence fix, we need to bump the sstate and hash equivalence version numbers to ensure older task hashes aren't matched into the new namespace. (From OE-Core rev: 3f229267e0588c747265e849ee19724033cc6a80) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* staging: Add extra hash handling codeRichard Purdie2022-02-051-1/+1
| | | | | | | | | | | | | | | | | Target build output, stored in do_populate_sysroot or do_package can depend not only upon direct dependencies but also indirect ones. A good example is linux-libc-headers. The toolchain depends on this but most target recipes do not. There are some headers which are not used by the toolchain build and do not change the toolchain task output, hence the task hashes can change without changing the sysroot output of that recipe yet they can influence others. A specific example is rtc.h which can change rtcwake.c in util-linux but is not used in the glibc or gcc build. To account for this, we need to account for the populate_sysroot hashes in the task output hashes. (From OE-Core rev: 1cf62882bbac543960e4815d117ffce0e53bda07) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: A third fix for for touching files inside pseudoPeter Kjellerstedt2022-01-261-5/+9
| | | | | | | | | | | | | | | | | | | | | | | This continues where commit676757f "sstate: fix touching files inside pseudo" and commit 29fc8599 "sstate: another fix for touching files inside pseudo" left off. The previous changes switched from trying to check if the sstate file is writable before touching it, to always touching the sstate file and ignoring any errors. However, if the sstate file is actually a symbolic link that links to nothing, this would actually result in an empty sstate file being created. And this in turn leads to that future setscene tasks will fail when they try to unpack the empty file. Change the code so that if an sstate file linking to nothing already exists, it is overwritten with the new sstate file. Also change it so that the temporary file that is used is always removed, even if ln fails to link the sstate file to it. (From OE-Core rev: 552197a0c4c9f75a9177c00b197ea91296ed9fc4) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve failure to obtain archive message/handlingRichard Purdie2022-01-191-1/+3
| | | | | | | | | | | | | | The bb.fatal() case where sstate failed to find/use an archive in setcene tasks is suboptimal. Bitbakes handling of setscene tasks will be to warn but the fatal will turn this into an error, despite the real task being rerun. In these failure cases other messages would usually have been printed so turn this into a warning and raise a handled exception status so that bitbake knows to fail the task but not print more messages. (From OE-Core rev: 7875baeca7a3ed216b2442fb8771e51efbfa5a4f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Show better exception information for failuresRichard Purdie2022-01-191-2/+2
| | | | | | | | | | | | Errors like SState: cannot test file://universal/5f/10/sstate:m4-native:x86_64-linux:1.4.19:r0:x86_64:7:5f108b175274798b17950b5ee686bf61445e90fb204d7f795e5b879b5603e88b_deploy_source_date_epoch.tar.zst: 'GIT_SSL_CAINFO' aren't useful, try and improve on this. (From OE-Core rev: 1701d40536db098adb26508799bae13733f1a6fd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Preserve permissions when extracting tar archiveJacob Kroon2022-01-151-1/+1
| | | | | | | | | | | | This is done by default when tar is run by the superuser, but for native recipes the corresponding task is not run as root under pseudo, so pass the flag explicitly. Suggested-by: Richard Purdie <richard.purdie@linuxfoundation.org> (From OE-Core rev: 32f01e57e1e915e8121609fb48b0b0254e625957) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: Only allow network in existing network accessing codeRichard Purdie2022-01-121-0/+2
| | | | | | | | | | | | | | | | | | | Use the newly added network task flag against tasks where network access is expected. This is do_fetch, do_checkuri, do_testimage, do_testsdk and do_testsdkext. We can't disable networking in sstate tasks due to sstate downloads and also so we can report hash equivalence to the server so network access is enabled in sstate tasks. Access within build-appliance do_image is also allowed due to the use of pip, this is a poor example made rather obvious now and needs to be reworked. Network access anywhere else in any other task isn't allowed. (From OE-Core rev: 7ce1e88a3ad85bbb925bb9f7167dc0a5fd1c27f4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: explicitly name the checkhashes thread poolRoss Burton2021-11-251-1/+2
| | | | | | | | | Give this thread pool a name for clarity. (From OE-Core rev: 57ea50fcfe81f47b93b9302d1aab2e81dcdd3105) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add a function to get the sstate file name in sstate_checkhashesJose Quaresma2021-11-161-9/+7
| | | | | | | (From OE-Core rev: dc64479264bdf3d60ece02a98b65be35650f8f86) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: only search on the mirrors for the missing filesJose Quaresma2021-11-161-6/+2
| | | | | | | | | | | | | | | | | | | | | | The existence of missed files on the local sstate cache is the condition to search for it on the sstate mirror. The missed collection are populated when we search the files on the local sstate cache. - When we have sstate mirrors we don't need to iterate all files again because we already know what's missing. - When we found a file on the sstate mirror is because of they are missing on the local sstate mirror. So avoid checking if the hash exists on the missed collection. - When we can't find the file on the sstate mirror we don't need to add the hash again to the missed collection as it already exists. (From OE-Core rev: 46e1f9a900e2be1abeda0dc540fc14040ecd0d7c) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: cleanup the sstate mirror progress checkJose Quaresma2021-11-161-4/+5
| | | | | | | | | | We only have the progress bar when there are more than 100 objects. So we can check for this and store the result to show the progress bar. (From OE-Core rev: 18e7a77aa1626e11820804ad64b86a32f97da2ca) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: another fix for touching files inside pseudoJose Quaresma2021-11-051-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | This patch is a fixup for 676757f "sstate: fix touching files inside pseudo" running the 'id' command inside the sstate_unpack_package function shows that this funcion run inside the pseudo: uid=0(root) gid=0(root) groups=0(root) The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ] will always return true and the touch can fail when the real user don't have permission or in readonly filesystem. As the documentation refers: - the file test operator "-w" check if the file has write permission (for the user running the test). - the file test operator "-O" check if you are owner of file We can avoid this test running the touch and mask any return errors that we have. (From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Account for reserved characters when shortening sstate filenamesManuel Leonhardt2021-11-031-1/+1
| | | | | | | | | | | | | | | | | | | | Previously, when shortening sstate filenames, the reserved characters for .siginfo were not considered, when siginfo=False, resulting in differently shortend filenames for the sstate and siginfo files. With this change, the filenames of the truncated sstate and siginfo files have the same basename, just as is already the case for untruncated filenames. Making sure that the .siginfo files always have the filename of the corresponding sstate file plus its .siginfo suffix, also when being truncated, makes it easier to manage the sstate cache and an sstate mirror outside of Bitbake/Yocto. (From OE-Core rev: c2e0e43b7123cf5149833e0072c8edaea3629112) Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de> Cc: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix touching files inside pseudoJose Quaresma2021-10-231-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | running the 'id' command inside the sstate_create_package function shows that this funcion run inside the pseudo: uid=0(root) gid=0(root) groups=0(root) The check for touch files [ ! -w ${SSTATE_PKG} ] will always return true and the touch can fail when the real user don't have permission or in readonly filesystem. As the documentation refers, the file test operator "-w" check if the file has write permission (for the user running the test). We can avoid this test running the touch and mask any return errors that we have. (From OE-Core rev: f6e7445c94443544e92fda97a017ce93393c5f84) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: count the files on mirrors using the pre local filesJose Quaresma2021-10-231-7/+4
| | | | | | | | | | | | | | | | We don't need extra python collections to count the found files on the sstate cache and sstate mirrors. The main found collections provides all the files that were found. Then we only need to store the nunber of files that found on the local sstate cache and the files that found on the sstate cache mirror is derived from that. (From OE-Core rev: 0fe71f31593e87b9188b388860d2ba94bf8266bc) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: don't silently handle all exceptions in sstate_checkhashesRoss Burton2021-10-181-3/+4
| | | | | | | | | | | | If checkstatus returns an exception we should silently handle FetchError, as this means the fetch failed for 'normal' reasons such as file not found. However, other exceptions may be raised, and these should be made visible. (From OE-Core rev: 50d99faf88a1d82cbd939b9bd6e33ebed2b1ffd8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Allow validation of sstate singatures against list of keysDaniel McGregor2021-10-141-1/+4
| | | | | | | | | | Allow a user to validate sstate objects against a list of keys, instead of just any known key in the user's keychain. (From OE-Core rev: 52ba0c5e6e2e3d5d01dc3f01404f0ab1bb29b3b5) Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Switch to ZStandard compressor supportHenry Kleynhans2021-10-141-10/+19
| | | | | | | | | | | | | | | | | | | | | | | | This patch switches the compressor from Gzip to ZStandard for ssate cache files. Zstandard compression provides a significant improvement in decompression speed as well as improvement in compression speed and disk usage over the 'tgz' format in use. Furthermore, its configurable compression level offers a trade-off between time spent compressing sstate cache files and disk space used by those files. The reduced disk usage also contributes to saving network traffic for those sharing their sstate cache with others. Zstandard should therefore be a good choice when: * disk space is at a premium * network speed / resources are limited * the CI server can sstate packages can be created at high compression * less CPU on the build server should be used for sstate decompression (From OE-Core rev: 0710e98f40ec7394567ede1f88f7c7bae306b925) Signed-off-by: Henry Kleynhans <hkleynhans@fb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* abi_version/sstate: Bump to fix rpm corruption issuesRichard Purdie2021-10-081-1/+1
| | | | | | (From OE-Core rev: 14feca0dc781f8e9003f70f317b63b242dc579b6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstatesig: Only apply group/other permissions to pseudo filesRichard Purdie2021-10-081-1/+1
| | | | | | | | | | | | | | | | | We hardlink some files into the build, such as licence files in do_populate_lic tasks. Depending on the umask that the source tree was checked out with, the group permissions would vary. This results in inconsistent task outhashes. Avoid this by ignoring the group/other bits unless we're under pseudo context. Bump the ABI numbers to ensure we don't see cache corruption from earlier builds. (From OE-Core rev: 2bd9c806de7e5197168360e3bd1d223a04a92291) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSIONRichard Purdie2021-10-041-1/+1
| | | | | | | | | | | | | | | At this point the hash equivalence and sstate is 'junk' on the autobuilder unforuntately due to the volume of fixes and also the volume of slighly not quite right patches tested during the development of the fixes. In order to try and help any remaining sanity I might have, bump the version numbers to start with a clean slate so we're working from a known good baseline rather than risk chasing phantom issues. For those upgrading, there wouldn't be much reuse anyway after the changes. (From OE-Core rev: be32692c627a14509de5eb3834e7321c3c5faf25) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstatesig: Add processing for full build paths in sysroot filesRichard Purdie2021-10-041-0/+7
| | | | | | | | | | | | | | | | | | | | | Some files in the populate_sysroot tasks have hardcoded paths in them, particularly if they are postinst-useradd- files or crossscripts. Add some filtering logic to remove these paths. This means that the hashequiv "outhash" matches correcting in more cases allowing for better build artefact reuse. To make this work a new variable is added SSTATE_HASHEQUIV_FILEMAP which maps file globbing to replacement patterns (paths or regex) on a per sstate task basis. It is hoped this shouldn't be needed in many cases. We are in the process to developing QA tests which will better detect issues in this area to allow optimal sstate reuse. (From OE-Core rev: d9852ffbbe728dac33dc081538a08af98f52fd4a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* ssate: Cleanup directtasks handlingRichard Purdie2021-09-231-14/+4
| | | | | | | | | | | | | There are several tasks which should only be triggered by direct dependencies. Some are listed in the dep loop, some are not. They should all be in both cases. Fix this by making a list and using the list in both cases to fix various inconsistencies. (From OE-Core rev: 29d285bc07bd7732c7ea39c0fd03c0b3c0d93e52) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Avoid deploy_source_date_epoch sstate when unneededRichard Purdie2021-09-231-0/+4
| | | | | | | | | | | | This sstate task is only needed when depended upon, it can be skipped if there are no tasks running that directly depend upon it. This reduced the number of sstate tasks in something like an image build. (From OE-Core rev: 601cee016da5c7505915e26641a085714de175ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure deploy tasks don't pull in toolchainsRichard Purdie2021-09-231-2/+2
| | | | | | | | | | | | | | If an image is built from sstate, the cross toolchain was being pulled in. This was due to the sstate dependencies for deploy tasks not being considered in the "depvalid" logic. do_deploy tasks do not need their populate_sysroot dependencies when installed from sstate so skip these. This reduces the sstate pulled in by an image build from sstate significantly. (From OE-Core rev: 8047c184f8266fc047c0e5a5524586d414528626) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure SDE is accounted for in package task timestampsRichard Purdie2021-09-161-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | When creating packages we build them with --clamp-mtime and use SOURCE_DATE_EPOCH as the maximum mtime. This makes the end packages reproducible. The data stored in sstate for do_package and the package task doesn't benefit from this though and have varying timestamps. This means their outhash varies and means hash equivalance isn't effective at all and doesn't work as intended/desired. We could create the sstate archives with the same clamping however that would lead to different results depending on whether a task was installed from sstate or not. Making that differ is a path to madness. It also wouldn't fix the outhash of the task to be determninistic without clamping of the date in the hash calculation code. Instead, iterate over the files in sstate output and clamp them at the code level. This isn't ideal but does make the file timestamps determnistic everywhere and means we don't have to change the hash calculation code. This issue can be clearly seen looking at the do_package outhash for a recipe which you then re-run the package task for after adding something like whitespace to the install task. The outhash shouldn't change but currently does. (From OE-Core rev: c3b3cc4745811b48b9193f83889946b2e1788932) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Avoid problems with recipes using SRCPV when fetching sstateRichard Purdie2021-09-111-0/+4
| | | | | | | | | | | | When fetching sstate, SRCPV can be expanded withuin PV which breaks the sstate fetcher as other variables like SRC_URI are changed by the fetcher code. Expand the variable before doing this to avoid those problems and hence avoid setscene task failures with remote sstate storage in git recipes. (From OE-Core rev: 0d9d7acebe0be42d4cf8a30ecbbbbcb2dfd16a4b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: get the number of threads from BB_NUMBER_THREADSJose Quaresma2021-08-201-3/+2
| | | | | | | | | | | | | | | | | | - bitbake BB_NUMBER_THREADS uses cpu_count from oe utils that uses the python os.sched_getaffinity and it is more acurrate. grep -nH ^BB_NUMBER_THREADS meta/conf/bitbake.conf meta/conf/bitbake.conf:806:BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" - multiprocessing.cpu_count() returns the number of CPUs on the host, not the number of usable CPUs on the host. If the user is using scheduler affinity then the number of usable CPUs may be less, so when determining how many cores we can use check the affinity instead. (From OE-Core rev: 45c52f08a289c6eb2329de50634a0406204d1d8e) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix error handling when sstate mirrors is roJose Quaresma2021-08-121-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The commit dd555537fc35c5f934af09d601d70772eb5955ae 'sstate.bbclass: fix errors about read-only sstate mirrors' adds an additional exception handler to silently mask read only rootfs errors thrown during the touch. The exception handler checks the error type with the python module errno but this module needs to be imported as it don't exist. Example of the error: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_task_postfunc(d) 0003: File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 778, function: sstate_task_postfunc 0774: 0775: omask = os.umask(0o002) 0776: if omask != 0o002: 0777: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask) *** 0778: sstate_package(shared_state, d) 0779: os.umask(omask) 0780: 0781: sstateinst = d.getVar("SSTATE_INSTDIR") 0782: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir']) File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 708, function: sstate_package 0704: except PermissionError: 0705: pass 0706: except OSError as e: 0707: # Handle read-only file systems gracefully *** 0708: if e.errno != errno.EROFS: 0709: raise e 0710: 0711: return 0712: Exception: NameError: name 'errno' is not defined (From OE-Core rev: 15f30ad144fbe25e9a5e71bc7e42e746d2039992) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Fix rebuilds when changing layer configRichard Purdie2021-08-021-0/+1
| | | | | | | | | | | | | When adding a layer which changed SSTATE_EXCLUDEDEPS_SYSROOT, the state hashes were changing when they should not. This was caused by wider use of setscene_depvalid which means the dependency on the variable was seen when it was previously not. Exclude the variable since this should be be included in the hashes. (From OE-Core rev: 09725a29365c69ccbd603fe3a1de72189f26d5ac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Drop pseudo exclusionRichard Purdie2021-07-111-2/+0
| | | | | | | | | | | | | Now that pseudo-native always includes statx support and we have sanity checks on pseudo-native to ensure it always contains a minimum feature set, we no longer need to mark pseudo-native as distro specific. This fixes eSDK build problems. [YOCTO #14428] (From OE-Core rev: 3279e43fcb469edb63c7c4eb60fdc565d5751f9d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/staging: Handle directory creation race issueRichard Purdie2021-07-021-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sstate code tries to be careful about racing around directory creation. In particular, the copyhardlinktree code creates the directory tree first allowing for "already exists" errors and ignoring them, then hardlinks the files in. Unfortunately the sstate removal code can race against this since it will try and remove empty directories. If there is some bad timing, a newly created directory can be removed before it was populated, leading to build failures. We could try and add locking but this would damage performance, we've been there before. It is also unclear where to actually place locks just based on the contents of a manifest file which may cover multiple sstate install locations for a given task. Instead, lets disable directory removal in the problematic "shared" core path. This could result in a few more empty directories being left on disk but those should be harmless and better than locking hurting performance or rare build races. [YOCTO #13999] [YOCTO #14379] (From OE-Core rev: 4f94d9296394bc7ce241439f00df86eb5912875f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix errors about read-only sstate mirrorsMichael Ho2021-06-291-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a read-only sstate mirror is used in conjunction with hash equiv, then OSError will be raised when an sstate-cache hit is achieved. This is because sstate_task_postfunc will try to "touch" the symlinks that point to the read-only sstate mirror when sstate_report_unihash has changed SSTATE_PKG. This commit adds an additional exception handler to silently mask read only rootfs errors thrown during the touch. The fix is also duplicated to sstate_eventhandler as the code is very similar but it may not be needed there. Example of the error: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_task_postfunc(d) 0003: File: '/poky/meta/classes/sstate.bbclass', lineno: 774, function: sstate_task_postfunc 0770: 0771: omask = os.umask(0o002) 0772: if omask != 0o002: 0773: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask) *** 0774: sstate_package(shared_state, d) 0775: os.umask(omask) 0776: 0777: sstateinst = d.getVar("SSTATE_INSTDIR") 0778: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir']) File: '/poky/meta/classes/sstate.bbclass', lineno: 703, function: sstate_package 0699: if not os.path.exists(siginfo): 0700: bb.siggen.dump_this_task(siginfo, d) 0701: else: 0702: try: *** 0703: os.utime(siginfo, None) 0704: except PermissionError: 0705: pass 0706: 0707: return Exception: OSError: [Errno 30] Read-only file system (From OE-Core rev: 244b3be0358a66e0cca4016fe26144e3d7323390) Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ignore sstate signing keyDaniel McGregor2021-05-211-0/+2
| | | | | | | | | | What key is used to sign sstate artefacts should not affect the hash of the object, otherwise everyone would need to use the same signing key. (From OE-Core rev: 57cc9429dba4f9bd23127633dbc1f57dc2d5dd16) Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Handle manifest 'corruption' issueRichard Purdie2021-05-161-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Under certain build patterns, warnings about missing manifests can appear. These are real issues where the manifest was removed and shouldn't have been. Martin Jansa was able to find a reproducer of: MACHINE=qemux86 bitbake zlib-native echo 'PR = "r1"' >> meta/recipes-core/zlib/zlib_1.2.11.bb MACHINE=qemux86-64 bitbake zlib-native MACHINE=qemux86 bitbake zlib-native <the zlib-native manifest is now removed along with the sysroot-components contents> The code maintains a per machine list of stamps but a per PACAGE_ARCH list of stamp/manifest/workdir mappings. The latter is only appended to for speed with the assumption that once stamps are gone, the code wouldn't trigger. The code only ever appends to the mapping list (for speed/efficency under lock) meaning that multiple entries can result where the stamp/workdir differs due to version changes but the manifest remains the same. By switching MACHINE part way through the build, the older stamp is referenced and the manifest is incorrectly removed as it matches an now obsolete entry in the mapping file. There are two possible fixes, one is to rewrite the mapping file every time which means adding regexs, iterating and generally complicating that code. The second option is to only use the last mapping entry in the file for a given manifest and ignore any earlier ones. This patch implments the latter. Also drop the stale entries if we are rewriting it. (From OE-Core rev: 63da9a4f889c5b0e41bc8ec08abe0acea1546479) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib/scripts: Use bb.utils.rename() instead of os.rename()Devendra Tewari2021-05-061-4/+4
| | | | | | | | | | | | | | | | | | Incremental build in Docker fails with: OSError: [Errno 18] Invalid cross-device link when source and destination are on different overlay filesystems. Rather than adding fallback code to every call site, use a new wrapper in bitbake which detects this case and falls back to shutil.move which is slower but will handtle the overlay docker filesystems correctly. [YOCTO #14301] (From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add documentation for eventhandlers and tweak namingRichard Purdie2021-03-231-6/+21
| | | | | | | | | | | It isn't immediately obvious what these do, add some better examples to document them. Also improve the naming for one of the handlers. This will help make it clear in any tracebacks which code is being used. (From OE-Core rev: 2b5c553744804f44a17b61c345e482027e45b71e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-09 07:13:01 +0000