summaryrefslogtreecommitdiffstats
path: root/meta/classes/sstate.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* sstate: Account for reserved characters when shortening sstate filenamesManuel Leonhardt2022-12-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | Previously, when shortening sstate filenames, the reserved characters for .siginfo were not considered, when siginfo=False, resulting in differently shortend filenames for the sstate and siginfo files. With this change, the filenames of the truncated sstate and siginfo files have the same basename, just as is already the case for untruncated filenames. Making sure that the .siginfo files always have the filename of the corresponding sstate file plus its .siginfo suffix, also when being truncated, makes it easier to manage the sstate cache and an sstate mirror outside of Bitbake/Yocto. (From OE-Core rev: 408bf1b4bb4f4ed126c17fb3676f9fa0513065ba) Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de> Cc: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c2e0e43b7123cf5149833e0072c8edaea3629112) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: inside the threadedpool don't write to the shared localdataJose Quaresma2022-03-111-1/+1
| | | | | | | | | | | | | | When inside the threadedpool we make a copy of the localdata to avoid some race condition, so we need to use this new localdata2 and stop write the shared localdata. (From OE-Core rev: 604146a242c3d5f5a9872bb756910f4bd1b58406) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 90fe6948a9df0b43c58120a9358adb3da1ceb5b9) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: A third fix for for touching files inside pseudoPeter Kjellerstedt2022-02-161-5/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | This continues where commit 676757f "sstate: fix touching files inside pseudo" and commit 29fc8599 "sstate: another fix for touching files inside pseudo" left off. The previous changes switched from trying to check if the sstate file is writable before touching it, to always touching the sstate file and ignoring any errors. However, if the sstate file is actually a symbolic link that links to nothing, this would actually result in an empty sstate file being created. And this in turn leads to that future setscene tasks will fail when they try to unpack the empty file. Change the code so that if an sstate file linking to nothing already exists, it is overwritten with the new sstate file. Also change it so that the temporary file that is used is always removed, even if ln fails to link the sstate file to it. Change-Id: I3800f98d0f2a0dd076352df85fad7c81460e733d (From OE-Core rev: f3cd092bf9f66d8d73075e5b777d89d8598691dd) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Avoid deploy_source_date_epoch sstate when unneededRichard Purdie2021-11-151-0/+4
| | | | | | | | | | | | | | | This sstate task is only needed when depended upon, it can be skipped if there are no tasks running that directly depend upon it. This reduced the number of sstate tasks in something like an image build. (From OE-Core rev: 884e44701ada57abe4d8ad9ece424435be25c6a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 601cee016da5c7505915e26641a085714de175ce) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure SDE is accounted for in package task timestampsRichard Purdie2021-11-151-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When creating packages we build them with --clamp-mtime and use SOURCE_DATE_EPOCH as the maximum mtime. This makes the end packages reproducible. The data stored in sstate for do_package and the package task doesn't benefit from this though and have varying timestamps. This means their outhash varies and means hash equivalance isn't effective at all and doesn't work as intended/desired. We could create the sstate archives with the same clamping however that would lead to different results depending on whether a task was installed from sstate or not. Making that differ is a path to madness. It also wouldn't fix the outhash of the task to be determninistic without clamping of the date in the hash calculation code. Instead, iterate over the files in sstate output and clamp them at the code level. This isn't ideal but does make the file timestamps determnistic everywhere and means we don't have to change the hash calculation code. This issue can be clearly seen looking at the do_package outhash for a recipe which you then re-run the package task for after adding something like whitespace to the install task. The outhash shouldn't change but currently does. (From OE-Core rev: 06b8f2a5a24be1a87f0eaf29fdba719ebe3bb06e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c3b3cc4745811b48b9193f83889946b2e1788932) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: another fix for touching files inside pseudoJose Quaresma2021-11-151-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch is a fixup for 676757f "sstate: fix touching files inside pseudo" running the 'id' command inside the sstate_unpack_package function shows that this funcion run inside the pseudo: uid=0(root) gid=0(root) groups=0(root) The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ] will always return true and the touch can fail when the real user don't have permission or in readonly filesystem. As the documentation refers: - the file test operator "-w" check if the file has write permission (for the user running the test). - the file test operator "-O" check if you are owner of file We can avoid this test running the touch and mask any return errors that we have. (From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876) (From OE-Core rev: e7d94a9cc5ab1b2c5d160fd06d643a4bc3409d26) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5b9210d66c78bb3f79056e5586cea7b0edd714a9) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: don't silently handle all exceptions in sstate_checkhashesRoss Burton2021-11-031-3/+4
| | | | | | | | | | | | | | | If checkstatus returns an exception we should silently handle FetchError, as this means the fetch failed for 'normal' reasons such as file not found. However, other exceptions may be raised, and these should be made visible. (From OE-Core rev: f16803718a19ace7b582c5b73924fe3112502927) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 50d99faf88a1d82cbd939b9bd6e33ebed2b1ffd8) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix touching files inside pseudoJose Quaresma2021-11-031-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | running the 'id' command inside the sstate_create_package function shows that this funcion run inside the pseudo: uid=0(root) gid=0(root) groups=0(root) The check for touch files [ ! -w ${SSTATE_PKG} ] will always return true and the touch can fail when the real user don't have permission or in readonly filesystem. As the documentation refers, the file test operator "-w" check if the file has write permission (for the user running the test). We can avoid this test running the touch and mask any return errors that we have. (From OE-Core rev: 590de1dd89cfd5f0ca7395880ba88b27ee35470d) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f6e7445c94443544e92fda97a017ce93393c5f84) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix error handling when sstate mirrors is roJose Quaresma2021-08-181-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The commit dd555537fc35c5f934af09d601d70772eb5955ae 'sstate.bbclass: fix errors about read-only sstate mirrors' adds an additional exception handler to silently mask read only rootfs errors thrown during the touch. The exception handler checks the error type with the python module errno but this module needs to be imported as it don't exist. Example of the error: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_task_postfunc(d) 0003: File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 778, function: sstate_task_postfunc 0774: 0775: omask = os.umask(0o002) 0776: if omask != 0o002: 0777: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask) *** 0778: sstate_package(shared_state, d) 0779: os.umask(omask) 0780: 0781: sstateinst = d.getVar("SSTATE_INSTDIR") 0782: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir']) File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 708, function: sstate_package 0704: except PermissionError: 0705: pass 0706: except OSError as e: 0707: # Handle read-only file systems gracefully *** 0708: if e.errno != errno.EROFS: 0709: raise e 0710: 0711: return 0712: Exception: NameError: name 'errno' is not defined (From OE-Core rev: ce78c16409363741d59a2f787aca66077bec93cd) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 15f30ad144fbe25e9a5e71bc7e42e746d2039992) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Fix rebuilds when changing layer configRichard Purdie2021-08-101-0/+1
| | | | | | | | | | | | | | | | When adding a layer which changed SSTATE_EXCLUDEDEPS_SYSROOT, the state hashes were changing when they should not. This was caused by wider use of setscene_depvalid which means the dependency on the variable was seen when it was previously not. Exclude the variable since this should be be included in the hashes. (From OE-Core rev: bae9c6482271d53dc28d3c801fba467e268003bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 09725a29365c69ccbd603fe3a1de72189f26d5ac) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Drop pseudo exclusionRichard Purdie2021-07-201-2/+0
| | | | | | | | | | | | | | | | Now that pseudo-native always includes statx support and we have sanity checks on pseudo-native to ensure it always contains a minimum feature set, we no longer need to mark pseudo-native as distro specific. This fixes eSDK build problems. [YOCTO #14428] (From OE-Core rev: d7e87a5851d717da047f552be394d5712efa0402) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3279e43fcb469edb63c7c4eb60fdc565d5751f9d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/staging: Handle directory creation race issueRichard Purdie2021-07-101-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sstate code tries to be careful about racing around directory creation. In particular, the copyhardlinktree code creates the directory tree first allowing for "already exists" errors and ignoring them, then hardlinks the files in. Unfortunately the sstate removal code can race against this since it will try and remove empty directories. If there is some bad timing, a newly created directory can be removed before it was populated, leading to build failures. We could try and add locking but this would damage performance, we've been there before. It is also unclear where to actually place locks just based on the contents of a manifest file which may cover multiple sstate install locations for a given task. Instead, lets disable directory removal in the problematic "shared" core path. This could result in a few more empty directories being left on disk but those should be harmless and better than locking hurting performance or rare build races. [YOCTO #13999] [YOCTO #14379] (From OE-Core rev: fa49622521b6386d8031b1e7519f087aa9d99b19) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4f94d9296394bc7ce241439f00df86eb5912875f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix errors about read-only sstate mirrorsMichael Ho2021-07-101-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a read-only sstate mirror is used in conjunction with hash equiv, then OSError will be raised when an sstate-cache hit is achieved. This is because sstate_task_postfunc will try to "touch" the symlinks that point to the read-only sstate mirror when sstate_report_unihash has changed SSTATE_PKG. This commit adds an additional exception handler to silently mask read only rootfs errors thrown during the touch. The fix is also duplicated to sstate_eventhandler as the code is very similar but it may not be needed there. Example of the error: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_task_postfunc(d) 0003: File: '/poky/meta/classes/sstate.bbclass', lineno: 774, function: sstate_task_postfunc 0770: 0771: omask = os.umask(0o002) 0772: if omask != 0o002: 0773: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask) *** 0774: sstate_package(shared_state, d) 0775: os.umask(omask) 0776: 0777: sstateinst = d.getVar("SSTATE_INSTDIR") 0778: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir']) File: '/poky/meta/classes/sstate.bbclass', lineno: 703, function: sstate_package 0699: if not os.path.exists(siginfo): 0700: bb.siggen.dump_this_task(siginfo, d) 0701: else: 0702: try: *** 0703: os.utime(siginfo, None) 0704: except PermissionError: 0705: pass 0706: 0707: return Exception: OSError: [Errno 30] Read-only file system (From OE-Core rev: f2360f796016cce93610a080ff07c8047bda6ee8) Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 244b3be0358a66e0cca4016fe26144e3d7323390) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ignore sstate signing keyDaniel McGregor2021-06-031-0/+2
| | | | | | | | | | | | | What key is used to sign sstate artefacts should not affect the hash of the object, otherwise everyone would need to use the same signing key. (From OE-Core rev: 0bfccdee11866d7f4e6c767127379329cd1dea2d) Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 57cc9429dba4f9bd23127633dbc1f57dc2d5dd16) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Handle manifest 'corruption' issueRichard Purdie2021-05-221-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Under certain build patterns, warnings about missing manifests can appear. These are real issues where the manifest was removed and shouldn't have been. Martin Jansa was able to find a reproducer of: MACHINE=qemux86 bitbake zlib-native echo 'PR = "r1"' >> meta/recipes-core/zlib/zlib_1.2.11.bb MACHINE=qemux86-64 bitbake zlib-native MACHINE=qemux86 bitbake zlib-native <the zlib-native manifest is now removed along with the sysroot-components contents> The code maintains a per machine list of stamps but a per PACAGE_ARCH list of stamp/manifest/workdir mappings. The latter is only appended to for speed with the assumption that once stamps are gone, the code wouldn't trigger. The code only ever appends to the mapping list (for speed/efficency under lock) meaning that multiple entries can result where the stamp/workdir differs due to version changes but the manifest remains the same. By switching MACHINE part way through the build, the older stamp is referenced and the manifest is incorrectly removed as it matches an now obsolete entry in the mapping file. There are two possible fixes, one is to rewrite the mapping file every time which means adding regexs, iterating and generally complicating that code. The second option is to only use the last mapping entry in the file for a given manifest and ignore any earlier ones. This patch implments the latter. Also drop the stale entries if we are rewriting it. (From OE-Core rev: 9039dd25e5d419dd1c60e1b27ff5f9d96c5b0fb5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 63da9a4f889c5b0e41bc8ec08abe0acea1546479) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* uninative: Don't use single sstate for pseudo-nativeRichard Purdie2021-02-031-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | pseudo-native is a bit special. It conditionally compiles in support for xattr, statx and statvfs amongst other options. If a pseudo-native binary is used on a system where these functions are present but it wasn't compiled in we see hard to debug permissions problems. An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target oe-selftest which shows a cryptic error: File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target self.assertEqual(filelist1, filelist2) File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual self.fail(msg) File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail raise self.failureException(msg) AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] First differing element 0: '-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor' '-rwxr-xr-x root root /etc/init.d/mdmonitor' This is due to a version of pseudo without statx being used on a system where ls uses statx, hence the files are displayed as 6000.6000 instead of root.root. Avoid this by always building pseudo-native for the specific distro in question rather than using a universal sstate feed. This hopefully fixes one of the mysterious AB-INT issues. (From OE-Core rev: 7c5881cf9baa7ea9fcfed382f24334cf8f96f9c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6e3785a3f1f3cf68f5fe101cd6bebe91db165973) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Check file ownership before doing 'touch -a'Norman Stetter2020-10-061-1/+1
| | | | | | | | | | | | | | | | | | In contrast to 'touching' a file without parameters 'touch -a' can only be performed by the file owner. In case of a shared sstate-cache served locally (e.g. over NFS), where the user running bitbake is not the owner of sstate-cache files, even if he has write access on group level, the sstate_unpack_package task will fail. Checking if the file is owned by the user before attempting to run 'touch -a' on it solves this. (From OE-Core rev: 8e7bb01564d5b06bcbda402cf4d29d7a0ff170c5) Signed-off-by: Norman Stetter <norman.stetter@garz-fricke.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f1554e8bd40c640fd58daac18ecf7b458e754109) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: When siginfo or sig files are missing, stop fetcher errorsMark Hatle2020-06-051-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prior to fetching, the system checks if the sstate file is present either locally or on the mirror. If it is, then it goes to the fetch stage. Up to three files can be fetched, sstate, sstate.siginfo and sstate.sig (if signature validation is enabled). The previous pstaging_fetch function would iterate over these, and if a download error occurred would spew forth a great amount of fetcher failure messages as well as stop fetching the next item in the set. This was resolved by adding a fetcher.checkstatus() call prior to the download. If the file isn't present, then the exception will be triggered, and no fetcher failure messages will reach the user. The exception handler is then modified to be a pass so that it will loop and pull the rest of the files that that are requested. Additionally, a check for the existance of the .sig file was added to the sstate_installpkg to avoid an error trying to load the .sig if it wasn't downloaded. (From OE-Core rev: ec58532ab6fc6343144da67789c928c751d36c06) Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a9085140434e2d26c0bb75bb53fcb7f7c19ef86d) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Do not fail if files cannot be touchedPeter Kjellerstedt2020-05-221-4/+11
| | | | | | | | | | | | It may be that a file is not allowed to be touched, e.g., if it is a symbolic link into a global sstate cache served over NFS. (From OE-Core rev: f528d6ffc9649536d21d625f65b415bfec7db258) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Drop obsolete check in hash validationRichard Purdie2020-03-061-4/+0
| | | | | | | | | | | | | | Now this function has a summary parameter we can drop this check. It could well be why the mysterious "locked sigs" selftest fails intermittently if this function were called with a single hash to check. [YOCTO #13605] (with luck) (From OE-Core rev: 02aabe0e59f73bf206d9bada1e7089832ceed254) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix issue while handling long sstate filenamesJaewon Lee2020-02-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | When moving to python3, divison using '/' now returns float instead of an integer. In upstream commit b8025e972081b70960ffcbcbe43a7118041556a1 sstate filenames longer than the limit are changed to just include necessary info + 3 fields just for information. The space left over after the necessary info is divided into 3 for each of the fields. Using '//' instead to do the division to solve the following error message: avail = (254 - len(hash + "_" + taskname + extension) - len(components[0]) - len(components[1]) - len(components[5]) - len(components[6]) - 7) / 3 > components[2] = components[2][:avail] components[3] = components[3][:avail] TypeError: slice indices must be integers or None or have an __index__ method (From OE-Core rev: 2acfee61a062c6520a413b2a797544d968bb0c76) Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Only show sstate mirror progress bar for >= 100 objectsPeter Kjellerstedt2020-01-271-4/+8
| | | | | | | | | | | | | | | With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly. Most of the times the number of objects scanned is very low and the progress bar completes almost immediately. To avoid all these unnecessary progress bars, set the minimum number of objects to 100 before the progress bar is shown. (From OE-Core rev: 114340e8d393cf60d0a596cb0800cf1b7fd198df) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve move into place op and touch fileRichard Purdie2020-01-131-1/+5
| | | | | | | | | | | | | | | | | Depending on the kernel and coreutils version, mv operations test for existence of files and can potentially race. It also leads to the file always changing which leads to potential problems if using and NFS share and there are other readers. Using ln instead means we don't overwrite the file if it already exists meaning other readers aren't disrupted and should work more reliably on NFS which is used for sstate on the autobuilder. Since we're not overwriting files, touch the file to show activity as would have been done it it were reused from sstate. (From OE-Core rev: d8e9a22a4e23616ad01627c1e472296b1e26f13c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Avoid races over rewritten siginfo filesRichard Purdie2020-01-061-3/+10
| | | | | | | | | | | | | | Try and avoid errors like: Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/x86_64-linux/libxslt-native/1.1.34-r0/temp/run.sstate_unpack_package.12343' failed with exit code 1: touch: setting times of 'build/sstate_devtool/80/4d/sstate:libxslt-native::1.1.34:r0::3:804d33f3f8d032b01ae92207669dd0a8f95a84917d563d122a77df19e786d73c_populate_lic.tgz.siginfo': Stale file handle WARNING: exit code 1 from a shell command. by not overwriting existing files. (From OE-Core rev: b2a389ad5111d587db3f95c6ce13fc2eafc22f27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Handle sstate filenames longer than 255 charactersRichard Purdie2020-01-061-1/+18
| | | | | | | | | | | | Many filesystems can't cope with filenames longer that 255 characters. Add code to detect this and truncate non-essential elements of the filename to stay within the limit. [YOCTO #13268] (From OE-Core rev: 90cc3d1ed1a12294a2d3ac97c1ba528ab315605d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Merge file name generation into single functionRichard Purdie2020-01-061-10/+12
| | | | | | | | | Move the task handling and extension handling into one common filename construction function. (From OE-Core rev: c0c158d38583648a801e959d91371f7b43a98da5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve SSTATE_PKG handlingRichard Purdie2020-01-061-10/+11
| | | | | | | | | | | Move the task handling code into the SSTATE_PKGNAME variable using a temporary variable. This makes the code more understandable as as well as allowing the length of the final sstate filename to be more easily accesses for following patches. (From OE-Core rev: be603dad6817948cfa09621d2311046631ec4ab2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add extra directory levelRichard Purdie2020-01-061-2/+2
| | | | | | | | | | | | | | | | | | We're having speed issues on the autobuilder due to the numbers of files in sstate directories. We previously split these by the first two characters of the hash. This change extends this to split by the next two characters as well, creating more layers of directories. This should signifiantly speed up eSDK builds on the autobuilder as the current sstate layout simply isn't scaling there but addresses a general complaint. gen-lockedsig-cache needed to be updated for the new split level sstate. Also update tests for new layout. (From OE-Core rev: d05bde16bdad761ed8f4c0a48de60c649aa33e85) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Fix for consistent whitespaceMartin Jansa2019-12-301-6/+6
| | | | | | | (From OE-Core rev: e874d067ace03774132ba3030a898678057c1304) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure mkdir happens before mktempRichard Purdie2019-12-111-3/+2
| | | | | | | | | | This avoids a directory not present error. Fix a comment typo whilst here. (From OE-Core rev: 1360d8d7d99b70a80c8cdbc1fc6d9e6752483139) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure target sstate directory exists if unihash changesRichard Purdie2019-12-111-2/+2
| | | | | | | | | The previous patches meant the mkdir might no longer match the final target directory. Fix this. (From OE-Core rev: 0af4dae84099e8632a9ea6a4afdbea2f232bb170) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure SSTATE_PKG is reloaded when handling siginfoRichard Purdie2019-12-101-1/+2
| | | | | | | | | STATE_PKG may have been changed by sstate_report_unihash so don't cache the variable's value. (From OE-Core rev: be29a25400c4ea285ab3f588c5831f00ba5d4f63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Stop overwriting SSTATE_PKGRichard Purdie2019-12-101-4/+4
| | | | | | | | | | | | Its rather antisocial to overwrite SSTATE_PKG with an expanded form for the variable and it stops the value of BB_UNIHASH being changed when the package is written out. Instead of expanding the variable, append to it instead to avoid this rather hard to figure out behaviour and allow the siggen code to behave as expected. (From OE-Core rev: 62eca02024b4c44d618ab9bcf87a3166c886dadb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add ability to hide summary output for sstateRichard Purdie2019-11-071-11/+12
| | | | | | | | | | Its confusing to keep seeing sstate summary messages when hash equivalency is active. This adds an option to control it. A default value is given which maintains compatibility with different bitbake versions. (From OE-Core rev: 038004866ff6650bcff7bb1bde36de6c0f451d29) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Remove SSTATE_HASHEQUIV_SERVERJoshua Watt2019-09-181-5/+0
| | | | | | | | | | Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This variable is redundant now that BB_HASHSERVE is present. (From OE-Core rev: 54b3adb6bc90a8e4b9e92952688772ee074d36e5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/sstatesig: Update to new form of BB_HASHCHECK_FUNCTIONRichard Purdie2019-08-061-42/+38
| | | | | | | | | Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION, adapt to this cleanup. This is an API breaking change. (From OE-Core rev: a4d413d8d809132b0e0a5dd673a36e2bd0e0be4c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Reduce race windowsRichard Purdie2019-08-031-1/+11
| | | | | | | | | | | | | | When we write to the sstate directory we try to do so atomically so consumers either see one version or another but never an imcomplete file. Unfortunately this is reliant on filesystem support and with some NFS configurations a replaced file would be lost from memory even if users held open descriptors. It makes sense to try and avoid replacing existing files where we can. (From OE-Core rev: 18cdc087fd5da30e2b31f3d4e81b153cd36ca844) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: regenerate sstate when signing enabledJoshua Lock via Openembedded-core2019-07-311-8/+12
| | | | | | | | | | | | | | | | This change ensures that the task signatures changes, and therefore sstate tasks are rerun, when signing is enabled. This has the positive outcome that if signing is enabled new signed shared state objects will be produced, rather than just signing shared state objects for tasks where no work has been performed yet. The downside of this change is that enabling/disabling sstate object signing alters the taskhash and results in rebuilding the world. (From OE-Core rev: ef60c65e3856682b90b17ae06a1b675756fd4496) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: don't use unsigned sstate when verification enabledJoshua Lock via Openembedded-core2019-07-271-1/+2
| | | | | | | | | | | | When signature verification of shared state objects is enabled (SSTATE_VERIFY_SIG) use of an unsigned object, even though it produces a warning, seems unexpected. Instead skip unsigned objects and force the non-accelerated task to be run. (From OE-Core rev: 571235978d98552e3734bf382454dd51272db782) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix log messageJoshua Lock via Openembedded-core2019-07-271-1/+1
| | | | | | | | | | | Referring to the sstate object as a staging package is an artefact of the code's origins. Switch to referring to an "Sstate package" in order to be more accurate and consistent with the rest of the file. (From OE-Core rev: 06559c1ed86dbce53505f9ed98111fe9d0b97ed7) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add tweak to avoid multiple sstate stats messagesRichard Purdie2019-07-151-0/+4
| | | | | | | | | | | | After the recent changes in bitbake to runqueue, we need to recheck sstate validity, particularly in multiconfig builds where tasks have the same checksum. Avoid printing summary messages in this case. Also avoid multiple events to toaster which may not be expecting that at later points in the code. (From OE-Core rev: 227125b96ad6fb0cf6e259e787d83415993db847) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Use bb.utils.to_boolean() for BB_NO_NETWORKRobert Yang2019-04-101-2/+4
| | | | | | | | | Make it consistent with bitbake (From OE-Core rev: 7fb540c3199bc2b82d60fff678b5e588ab4d1ad6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make sure changes to SSTATE_SCAN_FILES are not ignoredAlexander Kanavin2019-01-241-0/+1
| | | | | | | | | | | | | When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild, so if there's a sstate-cache available with "bad" sstate data in it that will still be used even though the recipe is updated to address this. [YOCTO #13144] (From OE-Core rev: ea3526961920a229e0bb5fb459952be89fce2255) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Update output hashJoshua Watt2019-01-221-62/+3
| | | | | | | | | | | | | | | | | Updates the output hash calculation for determining if tasks are equivalent. The new algorithm does the following based on feedback: 1) The output hash function was moved to the OE library. 2) All files are printed in a single line tabular format 3) Prints the file type and mode in a user-friendly ls-like format 4) Includes the file owner and group (by name, not ID). These are only included if the task is run under pseudo since that is the only time they can be consistently determined. 5) File size is included for regular files (From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: remove dependencies of SSTATE_EXTRAPATHWILDCARDRobert Yang2019-01-161-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | There might be do_cleansstate errors sometimes: ERROR: When reparsing /path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate, the basehash value changed from b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata is not deterministic and this needs to be fixed. The stable reproducer is: - Initial a fresh build, this is a must, otherwise we may can't reproduce it $ . oe-init-build-env build $ bitbake quilt-native -ccleansstate This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g., ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as SSTATE_EXTRAPATH did can fix the problem. (From OE-Core rev: 781117f9f02c0080dadc8797a8f8f9377a99b164) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar calls (again)André Draszik2019-01-141-2/+2
| | | | | | | | | | | | | | | | A couple have still been missed in the past despite multiple attempts at doing so (or simply have re-appeared?). Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Implement hash equivalence sstateJoshua Watt2019-01-081-8/+97
| | | | | | | | | | | | | | | | | | | | | Converts sstate so that it can use a hash equivalence server to determine if a task really needs to be rebuilt, or if it can be restored from a different (equivalent) sstate object. The unique hashes are cached persistently using persist_data. This has a number of advantages: 1) Unique hashes can be cached between invocations of bitbake to prevent needing to contact the server every time (which is slow) 2) The value of each tasks unique hash can easily be synchronized between different threads, which will be useful if bitbake is updated to do on the fly task re-hashing. [YOCTO #13030] (From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Handle unihash in hash checkJoshua Watt2019-01-081-1/+1
| | | | | | | | | | | | Handles the argument that passes task unique hash in the hash check function, as it is now required by bitbake [YOCTO #13030] (From OE-Core rev: 1c14b6969e58b51a325c1c1acf9c96e55675035c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: add support for caching shared workdir tasksMichael Ho2019-01-031-0/+6
| | | | | | | | | | | | | | | | | The sstate bbclass uses workdir as a hardcoded string in path manipulations. This means that the sstate caching mechanism does not work for the work-shared directory which the kernel uses to share its build configuration and source files for out of tree kernel modules. This commit modifies the path manipulation mechanism to use the work-shared directory if detected in the paths when handling the sstate cache packages. (From OE-Core rev: 27642449f95e38598f9c83948ce109c5891e5877) Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: catch ValueError from incorrectly written sstate manifestsMartin Jansa2019-01-031-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * in some rare cases the file format might be broken, e.g. when you run out of disk space while writing to it (hint: make sure to use BB_DISKMON_DIRS on _all_ the builds, to make sure that they can stop gracefully before this happens). * will show error like this: ERROR: Invalid line '/OE/build/luneos-thud/webos-ports' in sstate manifest '/OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch' when the file isn't complete, like in my case: $ tail -n 2 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch /OE/build/luneos-thud/webos-ports/tmp-glibc/stamps/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/manifest-allarch-org.webosports.app.calculator /OE/build/luneos-thud/webos-ports/tmp-glibc/work/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports * instead of much longer exception which doesn't really show what's wrong to selectively fix that (other than removing while TMPDIR): ERROR: Command execution failed: Traceback (most recent call last): File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/command.py", line 113, in runAsyncCommand self.cooker.updateCache() File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/cooker.py", line 1562, in updateCache bb.event.fire(event, self.databuilder.mcdata[mc]) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 225, in fire fire_class_handlers(event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 134, in fire_class_handlers execute_handler(name, handler, event, d) File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 106, in execute_handler ret = handler(event) File "/OE/build/luneos-thud/webos-ports/openembedded-core/meta/classes/sstate.bbclass", line 1083, in sstate_eventhandler2 (stamp, manifest, workdir) = l.split() ValueError: not enough values to unpack (expected 3, got 1) (From OE-Core rev: 870ba5d80e76e0f989971532fc2adc0ebe811ab6) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-24 10:05:46 +0000