summaryrefslogtreecommitdiffstats
path: root/meta/classes/sstate.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* sstate: inside the threadedpool don't write to the shared localdataJose Quaresma2022-03-221-1/+1
| | | | | | | | | | | | | | When inside the threadedpool we make a copy of the localdata to avoid some race condition, so we need to use this new localdata2 and stop write the shared localdata. (From OE-Core rev: 3d05b9b419b629001933fad7ee53b9f3bb366305) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1fa763b2022822a76fde541724e83e1977833d03) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: A third fix for for touching files inside pseudoPeter Kjellerstedt2022-02-101-5/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | This continues where commit676757f "sstate: fix touching files inside pseudo" and commit 29fc8599 "sstate: another fix for touching files inside pseudo" left off. The previous changes switched from trying to check if the sstate file is writable before touching it, to always touching the sstate file and ignoring any errors. However, if the sstate file is actually a symbolic link that links to nothing, this would actually result in an empty sstate file being created. And this in turn leads to that future setscene tasks will fail when they try to unpack the empty file. Change the code so that if an sstate file linking to nothing already exists, it is overwritten with the new sstate file. Also change it so that the temporary file that is used is always removed, even if ln fails to link the sstate file to it. (From OE-Core rev: 3eceda67a1098ab9641cb1b7fc789048b7daeae8) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> (cherry picked from commit b2a5d9bc61e0b2b7e0f187a262a514952ed30563) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: another fix for touching files inside pseudoJose Quaresma2021-11-211-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch is a fixup for 676757f "sstate: fix touching files inside pseudo" running the 'id' command inside the sstate_unpack_package function shows that this funcion run inside the pseudo: uid=0(root) gid=0(root) groups=0(root) The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ] will always return true and the touch can fail when the real user don't have permission or in readonly filesystem. As the documentation refers: - the file test operator "-w" check if the file has write permission (for the user running the test). - the file test operator "-O" check if you are owner of file We can avoid this test running the touch and mask any return errors that we have. (From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876) (From OE-Core rev: a5240d78f287ff5f3c5831af69e6e0ca3feaf769) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5b9210d66c78bb3f79056e5586cea7b0edd714a9) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Account for reserved characters when shortening sstate filenamesManuel Leonhardt2021-11-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | Previously, when shortening sstate filenames, the reserved characters for .siginfo were not considered, when siginfo=False, resulting in differently shortend filenames for the sstate and siginfo files. With this change, the filenames of the truncated sstate and siginfo files have the same basename, just as is already the case for untruncated filenames. Making sure that the .siginfo files always have the filename of the corresponding sstate file plus its .siginfo suffix, also when being truncated, makes it easier to manage the sstate cache and an sstate mirror outside of Bitbake/Yocto. (From OE-Core rev: 186062278e2d1e129815c4f8cf3d9003f3ac2809) Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de> Cc: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c2e0e43b7123cf5149833e0072c8edaea3629112) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix touching files inside pseudoJose Quaresma2021-11-031-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | running the 'id' command inside the sstate_create_package function shows that this funcion run inside the pseudo: uid=0(root) gid=0(root) groups=0(root) The check for touch files [ ! -w ${SSTATE_PKG} ] will always return true and the touch can fail when the real user don't have permission or in readonly filesystem. As the documentation refers, the file test operator "-w" check if the file has write permission (for the user running the test). We can avoid this test running the touch and mask any return errors that we have. (From OE-Core rev: 8acd5626b5b89fb48d5d7f0e4eee263793ba383f) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f6e7445c94443544e92fda97a017ce93393c5f84) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Avoid problems with recipes using SRCPV when fetching sstateRichard Purdie2021-09-241-0/+4
| | | | | | | | | | | | | | | When fetching sstate, SRCPV can be expanded withuin PV which breaks the sstate fetcher as other variables like SRC_URI are changed by the fetcher code. Expand the variable before doing this to avoid those problems and hence avoid setscene task failures with remote sstate storage in git recipes. (From OE-Core rev: 5232aa1c46380e7197098bed39fc6d5b8456caa7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0d9d7acebe0be42d4cf8a30ecbbbbcb2dfd16a4b) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix error handling when sstate mirrors is roJose Quaresma2021-08-261-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The commit dd555537fc35c5f934af09d601d70772eb5955ae 'sstate.bbclass: fix errors about read-only sstate mirrors' adds an additional exception handler to silently mask read only rootfs errors thrown during the touch. The exception handler checks the error type with the python module errno but this module needs to be imported as it don't exist. Example of the error: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_task_postfunc(d) 0003: File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 778, function: sstate_task_postfunc 0774: 0775: omask = os.umask(0o002) 0776: if omask != 0o002: 0777: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask) *** 0778: sstate_package(shared_state, d) 0779: os.umask(omask) 0780: 0781: sstateinst = d.getVar("SSTATE_INSTDIR") 0782: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir']) File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 708, function: sstate_package 0704: except PermissionError: 0705: pass 0706: except OSError as e: 0707: # Handle read-only file systems gracefully *** 0708: if e.errno != errno.EROFS: 0709: raise e 0710: 0711: return 0712: Exception: NameError: name 'errno' is not defined (From OE-Core rev: 2e5cf13aa46f25d8976ece55fcb4129bfd75659f) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 15f30ad144fbe25e9a5e71bc7e42e746d2039992) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Fix rebuilds when changing layer configRichard Purdie2021-08-141-0/+1
| | | | | | | | | | | | | | | | When adding a layer which changed SSTATE_EXCLUDEDEPS_SYSROOT, the state hashes were changing when they should not. This was caused by wider use of setscene_depvalid which means the dependency on the variable was seen when it was previously not. Exclude the variable since this should be be included in the hashes. (From OE-Core rev: 084d81058682e0c9074203c907f80d6eabe63534) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 09725a29365c69ccbd603fe3a1de72189f26d5ac) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Drop pseudo exclusionRichard Purdie2021-08-061-2/+0
| | | | | | | | | | | | | | | | Now that pseudo-native always includes statx support and we have sanity checks on pseudo-native to ensure it always contains a minimum feature set, we no longer need to mark pseudo-native as distro specific. This fixes eSDK build problems. [YOCTO #14428] (From OE-Core rev: 4c7c9326da92ea93fdef1ad804685c21a197f1d4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3279e43fcb469edb63c7c4eb60fdc565d5751f9d) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/staging: Handle directory creation race issueRichard Purdie2021-07-201-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sstate code tries to be careful about racing around directory creation. In particular, the copyhardlinktree code creates the directory tree first allowing for "already exists" errors and ignoring them, then hardlinks the files in. Unfortunately the sstate removal code can race against this since it will try and remove empty directories. If there is some bad timing, a newly created directory can be removed before it was populated, leading to build failures. We could try and add locking but this would damage performance, we've been there before. It is also unclear where to actually place locks just based on the contents of a manifest file which may cover multiple sstate install locations for a given task. Instead, lets disable directory removal in the problematic "shared" core path. This could result in a few more empty directories being left on disk but those should be harmless and better than locking hurting performance or rare build races. [YOCTO #13999] [YOCTO #14379] (From OE-Core rev: 3bea0da687d28c80f0b63e2ad81573a3601bb482) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4f94d9296394bc7ce241439f00df86eb5912875f) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix errors about read-only sstate mirrorsMichael Ho2021-07-061-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a read-only sstate mirror is used in conjunction with hash equiv, then OSError will be raised when an sstate-cache hit is achieved. This is because sstate_task_postfunc will try to "touch" the symlinks that point to the read-only sstate mirror when sstate_report_unihash has changed SSTATE_PKG. This commit adds an additional exception handler to silently mask read only rootfs errors thrown during the touch. The fix is also duplicated to sstate_eventhandler as the code is very similar but it may not be needed there. Example of the error: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_task_postfunc(d) 0003: File: '/poky/meta/classes/sstate.bbclass', lineno: 774, function: sstate_task_postfunc 0770: 0771: omask = os.umask(0o002) 0772: if omask != 0o002: 0773: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask) *** 0774: sstate_package(shared_state, d) 0775: os.umask(omask) 0776: 0777: sstateinst = d.getVar("SSTATE_INSTDIR") 0778: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir']) File: '/poky/meta/classes/sstate.bbclass', lineno: 703, function: sstate_package 0699: if not os.path.exists(siginfo): 0700: bb.siggen.dump_this_task(siginfo, d) 0701: else: 0702: try: *** 0703: os.utime(siginfo, None) 0704: except PermissionError: 0705: pass 0706: 0707: return Exception: OSError: [Errno 30] Read-only file system (From OE-Core rev: 1d86f3c8f5d47e1b3e3ca759076e24710ee7dc02) Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 244b3be0358a66e0cca4016fe26144e3d7323390) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ignore sstate signing keyDaniel McGregor2021-05-301-0/+2
| | | | | | | | | | What key is used to sign sstate artefacts should not affect the hash of the object, otherwise everyone would need to use the same signing key. (From OE-Core rev: 01a9358abe821c1da06c3243ccbcc93348042937) Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Handle manifest 'corruption' issueRichard Purdie2021-05-221-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Under certain build patterns, warnings about missing manifests can appear. These are real issues where the manifest was removed and shouldn't have been. Martin Jansa was able to find a reproducer of: MACHINE=qemux86 bitbake zlib-native echo 'PR = "r1"' >> meta/recipes-core/zlib/zlib_1.2.11.bb MACHINE=qemux86-64 bitbake zlib-native MACHINE=qemux86 bitbake zlib-native <the zlib-native manifest is now removed along with the sysroot-components contents> The code maintains a per machine list of stamps but a per PACAGE_ARCH list of stamp/manifest/workdir mappings. The latter is only appended to for speed with the assumption that once stamps are gone, the code wouldn't trigger. The code only ever appends to the mapping list (for speed/efficency under lock) meaning that multiple entries can result where the stamp/workdir differs due to version changes but the manifest remains the same. By switching MACHINE part way through the build, the older stamp is referenced and the manifest is incorrectly removed as it matches an now obsolete entry in the mapping file. There are two possible fixes, one is to rewrite the mapping file every time which means adding regexs, iterating and generally complicating that code. The second option is to only use the last mapping entry in the file for a given manifest and ignore any earlier ones. This patch implments the latter. Also drop the stale entries if we are rewriting it. (From OE-Core rev: fe468802f697d0be41cf3407df2460e1473e35f8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add documentation for eventhandlers and tweak namingRichard Purdie2021-03-231-6/+21
| | | | | | | | | | | It isn't immediately obvious what these do, add some better examples to document them. Also improve the naming for one of the handlers. This will help make it clear in any tracebacks which code is being used. (From OE-Core rev: 2b5c553744804f44a17b61c345e482027e45b71e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Remove stale objects before the main buildRichard Purdie2021-03-231-0/+45
| | | | | | | | | | | | | | | | | | | | | The split of util-linux-uuid out from util-linux caused some interesting sstate file overlap errors on existing build directories. This is a challenge to handle since util-linux depends on util-linux-uuid and has overlapping files in package data and deploy/packages directories. The util-linux build happens later and is what would clean up those files but it happens too late for uuid. Fixing this is hard as we don't know the taskhashes until the task graph is calculated. Once that is ready, we can compare the hashes with the existing hashes and know which sstate tasks are "stale". This patch adds a handler which iterates the sstate manifests looking for matching stamp paths and then removes the manifests along with the associated stamp files. (From OE-Core rev: 60e77b1777c6c304aa1d629c4cfdabe0daa22eb1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Split sstate summary into locally and network found artifactsAlejandro Hernandez Samaniego2021-02-191-1/+5
| | | | | | | | | | Modify sstate class to add extra output, differentiating between artifacts found locally and those found over the network. (From OE-Core rev: f017a2f677fa289c36b0e56ce480f69d7674ca11) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* uninative: Don't use single sstate for pseudo-nativeRichard Purdie2020-11-241-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | pseudo-native is a bit special. It conditionally compiles in support for xattr, statx and statvfs amongst other options. If a pseudo-native binary is used on a system where these functions are present but it wasn't compiled in we see hard to debug permissions problems. An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target oe-selftest which shows a cryptic error: File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target self.assertEqual(filelist1, filelist2) File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual self.fail(msg) File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail raise self.failureException(msg) AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] First differing element 0: '-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor' '-rwxr-xr-x root root /etc/init.d/mdmonitor' This is due to a version of pseudo without statx being used on a system where ls uses statx, hence the files are displayed as 6000.6000 instead of root.root. Avoid this by always building pseudo-native for the specific distro in question rather than using a universal sstate feed. This hopefully fixes one of the mysterious AB-INT issues. (From OE-Core rev: 6e3785a3f1f3cf68f5fe101cd6bebe91db165973) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: set mode explicitly when creating directories in sstate-cacheRoss Burton2020-09-301-1/+1
| | | | | | | | | | | When creating directories in the sstate-cache, explicitly set the mode passed to mkdir to 0775 so that the directories are group writable, as otherwise they cannot be shared with other users. (From OE-Core rev: 1b0624dffe1b8496533c86dfed873112c8c0a01b) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Check file ownership before doing 'touch -a'Norman Stetter2020-09-281-1/+1
| | | | | | | | | | | | | | | In contrast to 'touching' a file without parameters 'touch -a' can only be performed by the file owner. In case of a shared sstate-cache served locally (e.g. over NFS), where the user running bitbake is not the owner of sstate-cache files, even if he has write access on group level, the sstate_unpack_package task will fail. Checking if the file is owned by the user before attempting to run 'touch -a' on it solves this. (From OE-Core rev: f1554e8bd40c640fd58daac18ecf7b458e754109) Signed-off-by: Norman Stetter <norman.stetter@garz-fricke.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: When siginfo or sig files are missing, stop fetcher errorsMark Hatle2020-05-301-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Prior to fetching, the system checks if the sstate file is present either locally or on the mirror. If it is, then it goes to the fetch stage. Up to three files can be fetched, sstate, sstate.siginfo and sstate.sig (if signature validation is enabled). The previous pstaging_fetch function would iterate over these, and if a download error occurred would spew forth a great amount of fetcher failure messages as well as stop fetching the next item in the set. This was resolved by adding a fetcher.checkstatus() call prior to the download. If the file isn't present, then the exception will be triggered, and no fetcher failure messages will reach the user. The exception handler is then modified to be a pass so that it will loop and pull the rest of the files that that are requested. Additionally, a check for the existance of the .sig file was added to the sstate_installpkg to avoid an error trying to load the .sig if it wasn't downloaded. (From OE-Core rev: a9085140434e2d26c0bb75bb53fcb7f7c19ef86d) Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Do not fail if files cannot be touchedPeter Kjellerstedt2020-05-091-4/+11
| | | | | | | | | | It may be that a file is not allowed to be touched, e.g., if it is a symbolic link into a global sstate cache served over NFS. (From OE-Core rev: 462e3f505217c9b9e0d2c1cbdafb0c8910bb6a59) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Drop obsolete check in hash validationRichard Purdie2020-03-061-4/+0
| | | | | | | | | | | | | | Now this function has a summary parameter we can drop this check. It could well be why the mysterious "locked sigs" selftest fails intermittently if this function were called with a single hash to check. [YOCTO #13605] (with luck) (From OE-Core rev: 02aabe0e59f73bf206d9bada1e7089832ceed254) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: fix issue while handling long sstate filenamesJaewon Lee2020-02-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | When moving to python3, divison using '/' now returns float instead of an integer. In upstream commit b8025e972081b70960ffcbcbe43a7118041556a1 sstate filenames longer than the limit are changed to just include necessary info + 3 fields just for information. The space left over after the necessary info is divided into 3 for each of the fields. Using '//' instead to do the division to solve the following error message: avail = (254 - len(hash + "_" + taskname + extension) - len(components[0]) - len(components[1]) - len(components[5]) - len(components[6]) - 7) / 3 > components[2] = components[2][:avail] components[3] = components[3][:avail] TypeError: slice indices must be integers or None or have an __index__ method (From OE-Core rev: 2acfee61a062c6520a413b2a797544d968bb0c76) Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Only show sstate mirror progress bar for >= 100 objectsPeter Kjellerstedt2020-01-271-4/+8
| | | | | | | | | | | | | | | With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly. Most of the times the number of objects scanned is very low and the progress bar completes almost immediately. To avoid all these unnecessary progress bars, set the minimum number of objects to 100 before the progress bar is shown. (From OE-Core rev: 114340e8d393cf60d0a596cb0800cf1b7fd198df) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve move into place op and touch fileRichard Purdie2020-01-131-1/+5
| | | | | | | | | | | | | | | | | Depending on the kernel and coreutils version, mv operations test for existence of files and can potentially race. It also leads to the file always changing which leads to potential problems if using and NFS share and there are other readers. Using ln instead means we don't overwrite the file if it already exists meaning other readers aren't disrupted and should work more reliably on NFS which is used for sstate on the autobuilder. Since we're not overwriting files, touch the file to show activity as would have been done it it were reused from sstate. (From OE-Core rev: d8e9a22a4e23616ad01627c1e472296b1e26f13c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Avoid races over rewritten siginfo filesRichard Purdie2020-01-061-3/+10
| | | | | | | | | | | | | | Try and avoid errors like: Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/x86_64-linux/libxslt-native/1.1.34-r0/temp/run.sstate_unpack_package.12343' failed with exit code 1: touch: setting times of 'build/sstate_devtool/80/4d/sstate:libxslt-native::1.1.34:r0::3:804d33f3f8d032b01ae92207669dd0a8f95a84917d563d122a77df19e786d73c_populate_lic.tgz.siginfo': Stale file handle WARNING: exit code 1 from a shell command. by not overwriting existing files. (From OE-Core rev: b2a389ad5111d587db3f95c6ce13fc2eafc22f27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Handle sstate filenames longer than 255 charactersRichard Purdie2020-01-061-1/+18
| | | | | | | | | | | | Many filesystems can't cope with filenames longer that 255 characters. Add code to detect this and truncate non-essential elements of the filename to stay within the limit. [YOCTO #13268] (From OE-Core rev: 90cc3d1ed1a12294a2d3ac97c1ba528ab315605d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Merge file name generation into single functionRichard Purdie2020-01-061-10/+12
| | | | | | | | | Move the task handling and extension handling into one common filename construction function. (From OE-Core rev: c0c158d38583648a801e959d91371f7b43a98da5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Improve SSTATE_PKG handlingRichard Purdie2020-01-061-10/+11
| | | | | | | | | | | Move the task handling code into the SSTATE_PKGNAME variable using a temporary variable. This makes the code more understandable as as well as allowing the length of the final sstate filename to be more easily accesses for following patches. (From OE-Core rev: be603dad6817948cfa09621d2311046631ec4ab2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add extra directory levelRichard Purdie2020-01-061-2/+2
| | | | | | | | | | | | | | | | | | We're having speed issues on the autobuilder due to the numbers of files in sstate directories. We previously split these by the first two characters of the hash. This change extends this to split by the next two characters as well, creating more layers of directories. This should signifiantly speed up eSDK builds on the autobuilder as the current sstate layout simply isn't scaling there but addresses a general complaint. gen-lockedsig-cache needed to be updated for the new split level sstate. Also update tests for new layout. (From OE-Core rev: d05bde16bdad761ed8f4c0a48de60c649aa33e85) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Fix for consistent whitespaceMartin Jansa2019-12-301-6/+6
| | | | | | | (From OE-Core rev: e874d067ace03774132ba3030a898678057c1304) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure mkdir happens before mktempRichard Purdie2019-12-111-3/+2
| | | | | | | | | | This avoids a directory not present error. Fix a comment typo whilst here. (From OE-Core rev: 1360d8d7d99b70a80c8cdbc1fc6d9e6752483139) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure target sstate directory exists if unihash changesRichard Purdie2019-12-111-2/+2
| | | | | | | | | The previous patches meant the mkdir might no longer match the final target directory. Fix this. (From OE-Core rev: 0af4dae84099e8632a9ea6a4afdbea2f232bb170) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure SSTATE_PKG is reloaded when handling siginfoRichard Purdie2019-12-101-1/+2
| | | | | | | | | STATE_PKG may have been changed by sstate_report_unihash so don't cache the variable's value. (From OE-Core rev: be29a25400c4ea285ab3f588c5831f00ba5d4f63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Stop overwriting SSTATE_PKGRichard Purdie2019-12-101-4/+4
| | | | | | | | | | | | Its rather antisocial to overwrite SSTATE_PKG with an expanded form for the variable and it stops the value of BB_UNIHASH being changed when the package is written out. Instead of expanding the variable, append to it instead to avoid this rather hard to figure out behaviour and allow the siggen code to behave as expected. (From OE-Core rev: 62eca02024b4c44d618ab9bcf87a3166c886dadb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add ability to hide summary output for sstateRichard Purdie2019-11-071-11/+12
| | | | | | | | | | Its confusing to keep seeing sstate summary messages when hash equivalency is active. This adds an option to control it. A default value is given which maintains compatibility with different bitbake versions. (From OE-Core rev: 038004866ff6650bcff7bb1bde36de6c0f451d29) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Remove SSTATE_HASHEQUIV_SERVERJoshua Watt2019-09-181-5/+0
| | | | | | | | | | Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This variable is redundant now that BB_HASHSERVE is present. (From OE-Core rev: 54b3adb6bc90a8e4b9e92952688772ee074d36e5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/sstatesig: Update to new form of BB_HASHCHECK_FUNCTIONRichard Purdie2019-08-061-42/+38
| | | | | | | | | Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION, adapt to this cleanup. This is an API breaking change. (From OE-Core rev: a4d413d8d809132b0e0a5dd673a36e2bd0e0be4c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Reduce race windowsRichard Purdie2019-08-031-1/+11
| | | | | | | | | | | | | | When we write to the sstate directory we try to do so atomically so consumers either see one version or another but never an imcomplete file. Unfortunately this is reliant on filesystem support and with some NFS configurations a replaced file would be lost from memory even if users held open descriptors. It makes sense to try and avoid replacing existing files where we can. (From OE-Core rev: 18cdc087fd5da30e2b31f3d4e81b153cd36ca844) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: regenerate sstate when signing enabledJoshua Lock via Openembedded-core2019-07-311-8/+12
| | | | | | | | | | | | | | | | This change ensures that the task signatures changes, and therefore sstate tasks are rerun, when signing is enabled. This has the positive outcome that if signing is enabled new signed shared state objects will be produced, rather than just signing shared state objects for tasks where no work has been performed yet. The downside of this change is that enabling/disabling sstate object signing alters the taskhash and results in rebuilding the world. (From OE-Core rev: ef60c65e3856682b90b17ae06a1b675756fd4496) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: don't use unsigned sstate when verification enabledJoshua Lock via Openembedded-core2019-07-271-1/+2
| | | | | | | | | | | | When signature verification of shared state objects is enabled (SSTATE_VERIFY_SIG) use of an unsigned object, even though it produces a warning, seems unexpected. Instead skip unsigned objects and force the non-accelerated task to be run. (From OE-Core rev: 571235978d98552e3734bf382454dd51272db782) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: fix log messageJoshua Lock via Openembedded-core2019-07-271-1/+1
| | | | | | | | | | | Referring to the sstate object as a staging package is an artefact of the code's origins. Switch to referring to an "Sstate package" in order to be more accurate and consistent with the rest of the file. (From OE-Core rev: 06559c1ed86dbce53505f9ed98111fe9d0b97ed7) Signed-off-by: Joshua Lock <jlock@vmware.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Add tweak to avoid multiple sstate stats messagesRichard Purdie2019-07-151-0/+4
| | | | | | | | | | | | After the recent changes in bitbake to runqueue, we need to recheck sstate validity, particularly in multiconfig builds where tasks have the same checksum. Avoid printing summary messages in this case. Also avoid multiple events to toaster which may not be expecting that at later points in the code. (From OE-Core rev: 227125b96ad6fb0cf6e259e787d83415993db847) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: Use bb.utils.to_boolean() for BB_NO_NETWORKRobert Yang2019-04-101-2/+4
| | | | | | | | | Make it consistent with bitbake (From OE-Core rev: 7fb540c3199bc2b82d60fff678b5e588ab4d1ad6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make sure changes to SSTATE_SCAN_FILES are not ignoredAlexander Kanavin2019-01-241-0/+1
| | | | | | | | | | | | | When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild, so if there's a sstate-cache available with "bad" sstate data in it that will still be used even though the recipe is updated to address this. [YOCTO #13144] (From OE-Core rev: ea3526961920a229e0bb5fb459952be89fce2255) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Update output hashJoshua Watt2019-01-221-62/+3
| | | | | | | | | | | | | | | | | Updates the output hash calculation for determining if tasks are equivalent. The new algorithm does the following based on feedback: 1) The output hash function was moved to the OE library. 2) All files are printed in a single line tabular format 3) Prints the file type and mode in a user-friendly ls-like format 4) Includes the file owner and group (by name, not ID). These are only included if the task is run under pseudo since that is the only time they can be consistently determined. 5) File size is included for regular files (From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: remove dependencies of SSTATE_EXTRAPATHWILDCARDRobert Yang2019-01-161-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | There might be do_cleansstate errors sometimes: ERROR: When reparsing /path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate, the basehash value changed from b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata is not deterministic and this needs to be fixed. The stable reproducer is: - Initial a fresh build, this is a must, otherwise we may can't reproduce it $ . oe-init-build-env build $ bitbake quilt-native -ccleansstate This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g., ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as SSTATE_EXTRAPATH did can fix the problem. (From OE-Core rev: 781117f9f02c0080dadc8797a8f8f9377a99b164) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar calls (again)André Draszik2019-01-141-2/+2
| | | | | | | | | | | | | | | | A couple have still been missed in the past despite multiple attempts at doing so (or simply have re-appeared?). Search & replace made using the following command: sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \ -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \ | cut -d':' -f1 \ | sort -u) (From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Implement hash equivalence sstateJoshua Watt2019-01-081-8/+97
| | | | | | | | | | | | | | | | | | | | | Converts sstate so that it can use a hash equivalence server to determine if a task really needs to be rebuilt, or if it can be restored from a different (equivalent) sstate object. The unique hashes are cached persistently using persist_data. This has a number of advantages: 1) Unique hashes can be cached between invocations of bitbake to prevent needing to contact the server every time (which is slow) 2) The value of each tasks unique hash can easily be synchronized between different threads, which will be useful if bitbake is updated to do on the fly task re-hashing. [YOCTO #13030] (From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/sstate: Handle unihash in hash checkJoshua Watt2019-01-081-1/+1
| | | | | | | | | | | | Handles the argument that passes task unique hash in the hash check function, as it is now required by bitbake [YOCTO #13030] (From OE-Core rev: 1c14b6969e58b51a325c1c1acf9c96e55675035c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-15 16:38:31 +0000